Quality of Research
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Bennett, J. B., & Lehman, W. E. K. (2001). Workplace substance abuse prevention and help seeking: Comparing team-oriented and informational training. Journal of Occupational Health Psychology, 6(3), 243-254. 
Bennett, J. B., & Lehman, W. E. K. (2002). Supervisor tolerance-responsiveness to substance abuse and workplace prevention training: Use of a cognitive mapping tool. Health Education Research, 17(1), 27-42.
Bennett, J. B., Patterson, C. R., Reynolds, G. S., Wiitala, W. L., & Lehman, W. E. K. (2004). Team Awareness, problem drinking, and drinking climate: Workplace social health promotion in a policy context. American Journal of Health Promotion, 19(2), 103-113.
Lehman, W. E. K., Reynolds, G. S., & Bennett, J. B. (2002). Team and informational trainings for workplace substance abuse prevention. In J. B. Bennett & W. E. K. Lehman (Eds.), Preventing workplace substance abuse: Beyond drug testing to wellness (pp. 165-201). Washington, DC: American Psychological Association. Study 2Bennett, J. B., Patterson, C., Sledz, R., Klimback, E. M., Berish, J. E., & Cook, R. (n.d.). Small Business Wellness Initiative research report. Fort Worth, TX: Organizational Wellness & Learning Systems.
Bennett, J. B., & Patterson, C. R. (n.d.). Targeting working adult users of alcohol and drugs: Help-seeking and an experimental assessment in small businesses. Manuscript submitted for publication.
Patterson, C. R., Bennett, J. B., & Wiitala, W. L. (2005). Healthy and unhealthy stress unwinding: Promoting health in small businesses. Journal of Business and Psychology, 20(2), 221-247.
Supplementary Materials Bennett, J. B., & Lehman, W. E. K. (1999). Employee exposure to coworker substance use and negative consequences: The moderating effects of work group membership. Journal of Health and Social Behavior, 40(3), 307-322.
Bennett, J. B., Lehman, W. E. K., & Reynolds, G. S. (2000). Team awareness for workplace substance abuse prevention: The empirical and conceptual development of a training program. Prevention Science, 1(3), 157-172.
Bennett, J. B., Reynolds, G. S., & Lehman, W. E. K. (2003, March). The "black box" of health promotion: Training room behaviors predict outcomes. Poster session presented at Work, Stress & Health, 2003: Fifth Interdisciplinary Conference on Work Stress, Toronto, Canada.
Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237-256.
Intraclass correlations of ratings of participant responsiveness
Lehman, W. E. K., Bennett, J. B., & Reynolds, G. S. (2000). TCU workplace measures. Fort Worth: Texas Christian University, Institute of Behavioral Research.
Lehman, W. E. K., & Simpson, D. D. (1992). Employee substance use and on-the-job behaviors. Journal of Applied Psychology, 77(3), 309-321.
Reynolds, G. S., & Lehman, W. E. K. (2008). Workgroup temperance of alcohol and safety climate moderate the cognitive effects of workplace substance-abuse prevention. Journal of Applied Social Psychology, 38(7), 1827-1866.
Training Outline for Fidelity Checks
Training session rating form
Outcomes
| Outcome 1: Group climate |
|
Description of Measures
|
Supervisors from one large and one midsize suburban municipality and employees from the midsize municipality completed the Employee Health and Performance in the Workplace self-report questionnaire. One group-climate measure, group privacy regulation, consisted of 4 items addressing the extent to which employees (1) expected any personal information they shared to be "leaked into the grapevine" and (2) respected privacy and trusted each other with personal information. Items were rated on a scale from 1 (strongly disagree) to 5 (strongly agree) and then averaged. Higher scores indicated a greater ability to expect privacy and treat personal information as confidential.
In another study, employees from small businesses within three industries identified as at risk for substance abuse (construction, transportation, and hospitality/service) completed the Organizational Wellness Scale, a group climate measure comprising 8 self-report items (e.g., "Policies are flexible enough to help meet my personal and family needs").
|
|
Key Findings
|
City employees who attended the 8-hour Team Awareness training showed a mean increase in privacy regulation of 0.13 on a 5-point scale, while employees who did not receive any training showed a mean decrease of 0.07 (p = .04).
From pre- to posttest, supervisors who attended the 8-hour Team Awareness training demonstrated a significant increase in confidence in talking about problems with employees (p < .01), while those who attended a 4-hour informational training on the EAP and policy or received no training experienced no significant changes (p = .01). The effect size was small for the comparison between the intervention and informational training (Cohen's d = 0.44) and large for the comparison between the intervention and control condition (Cohen's d = 1.00).
In the study of small businesses in at-risk industries, organizational wellness significantly improved from pre- to posttest (p < .05) and from pretest to follow-up (p < .001) for employees who received Team Awareness-SB training (a 4-hour version of Team Awareness developed for small businesses), compared with employees who did not receive any training.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 2: Knowledge and attitudes related to substance use policies and the EAP |
|
Description of Measures
|
Employees from all city departments of a midsize suburban municipality and employees from three city departments designated as at risk for substance abuse (parks and recreation, transportation, and water) from a large municipality completed the Employee Health and Performance in the Workplace self-report questionnaire. (According to the developer, the three departments were identified as at risk based on previous surveys with the workforce and information from the EAP.) Measures assessed the following:
- Knowledge about how to get help: Employees responded "no" (scored 0) or "yes" (scored 1) to the question, "Do you know how to get help from the City for you or someone else who may have an alcohol/drug problem?"
- Knowledge about the EAP: Employees were asked, "How much information do you feel that you have about the EAP?" Response options ranged from 1 ("I don't know anything about it") to 5 ("I am very knowledgeable about the EAP").
- Knowledge about contacting the EAP: Employees were asked a multiple choice item, "If you needed to contact the EAP, how would you do it?" Incorrect responses were scored 0, and the correct response was scored 1.
- Knowledge of the EAP and substance use policies: True/false questions assessed knowledge of information presented in the trainings.
- Approval of substance use policies: Using a scale from 1 (strongly disagree) to 5 (strongly agree), employees rated 5 items (e.g., "I support the purpose and goals of the substance abuse policy").
|
|
Key Findings
|
City employees who attended either the 8-hour Team Awareness training or a 4-hour informational training showed significant improvements in knowledge about the EAP, substance use policies, and contacting and getting help from the EAP, compared with employees who received no training (p < .0001 to p < .05).
Over the 6 months following the training, attitudes toward substance use policies improved slightly among city employees from at-risk departments who received the 8-hour Team Awareness training. Attitudes worsened significantly among comparable employees who received no training.
City employees from at-risk departments who received either the 8-hour Team Awareness training or a 4-hour informational training showed relatively greater increases in knowledge about substance use policies and the EAP than did employees who received no training (p < .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.4
(0.0-4.0 scale)
|
| Outcome 3: Help-seeking attitudes, behavior, and encouragement |
|
Description of Measures
|
Supervisors from one large and one midsize suburban municipality and employees from the midsize municipality completed the Employee Health and Performance in the Workplace self-report questionnaire. Help seeking/encouragement was assessed by two constructs: "encouraged others" and "sought help or was encouraged." Three items asked if the employee or supervisor had called the EAP to help someone or had encouraged a coworker to call for work-related or non-work-related problems. Five items asked if the employee or supervisor had sought help for personal or drug/alcohol problems or had been encouraged to get help by other workers.
In another study, employees from small businesses selected from three at-risk industries (construction, transportation, and hospitality/service), provided ratings regarding help-seeking attitudes and behaviors. Three items asked how likely the employee would be to seek help for his or her own depression, stress, or a drug or alcohol problem (e.g., "If you were depressed, how likely would you go to a counselor, support group, or EAP for help?"). Response options ranged from 1 (very unlikely) to 5 (very likely). Four additional items asked respondents to indicate whether they had called or talked to a counselor, mental health professional, EAP representative, or spiritual or religious counselor for help with either a personal problem or a drug or alcohol problem.
|
|
Key Findings
|
City employees and supervisors who attended the 8-hour Team Awareness training showed a significant increase in seeking or being encouraged to seek help and in encouraging others to do so (p = .02 for supervisors; p < .01 for employees). Employees who attended a 4-hour informational training or received no training experienced no significant changes.
In the study of small businesses in at-risk industries, employees who received the 4-hour Team Awareness-SB training significantly improved in their willingness to seek help for themselves, compared with those who received customized health promotion training or no training (p < .03). In addition, employees with self-reported alcohol or other drug abuse received significantly more counseling from pretest to 6-month follow-up (p < .05). No significant changes were seen for employees who received customized health promotion training or no training.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 4: Alcohol and other drug use attitudes and drinking climate |
|
Description of Measures
|
Employees from three at-risk city departments (parks and recreation, transportation, and water) from a large municipality completed the Employee Health and Performance in the Workplace self-report questionnaire. The following measures assessed alcohol and other drug use attitudes and drinking climate:
- Drinking norms: Using a scale from 1 (never) to 5 (almost always), employees rated 5 items (e.g., "How often is the talk at work about drinking or activities involving drinking?").
- Informal sanctions: Using a scale from 1 (very unlikely) to 5 (very likely), employees rated 4 items (e.g., "If their own work was affected by a fellow employee who had a drinking or drug problem, how likely would your coworkers be to ignore the problem?").
- Stigma: Using a scale from 1 (strongly disagree) to 5 (strongly agree), employees rated 4 items (e.g., "My coworkers would think less of me if they thought that I had a drinking problem").
In another study, employees from small businesses selected from three at-risk industries (construction, transportation, and hospitality/service), answered the drinking norms and stigma questions described above. In addition, they responded to 4 items regarding disapproval of drug use (i.e., feeling about cigarettes, marijuana, alcohol, drinking and driving) and 4 items regarding perceived risk of drug use (i.e., risk of harm from cigarettes, marijuana, frequent heavy drinking).
|
|
Key Findings
|
Employees from at-risk city departments who received the 8-hour Team Awareness training reported significantly increased informal sanctions and decreased stigma, compared with employees who received either a 4-hour informational training or no training.
In the study of small businesses in at-risk industries, drinking climate significantly improved among employees who received the 4-hour Team Awareness-SB training, compared with those who received customized health promotion training or no training (p < .05). These reductions were most apparent among employees who were exposed to higher coworker risks (e.g., harassment by a coworker, coworker use of alcohol or illegal drugs).
Perceptions of the risks of substance use, particularly of smoking and heavy drinking, significantly increased among employees who received the 4-hour Team Awareness-SB training, compared with those who received customized health promotion training or no training (p < .001 to p < .05).
Disapproval of trying marijuana (p < .01) and of having one or two drinks per day (p < .05) significantly increased among employees who received the 4-hour Team Awareness-SB training, compared with those who received customized health promotion training or no training.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 5: Alcohol use and alcohol-related problems |
|
Description of Measures
|
Employees from three at-risk city departments (parks and recreation, transportation, and water) from a large municipality completed the Employee Health and Performance in the Workplace self-report questionnaire. Problem drinking was measured by 7 items that asked if alcohol caused problems at and away from work (e.g., getting into fights, driving while intoxicated, accidents, absences) and included the presence of any of the following five symptoms: drinking in the morning, having "shakes" and tremors because of a need to drink, drinking more than intended, staying drunk for a day or longer, and having blackouts. Response formats were recoded into a dichotomous measure representing no or never (0) versus any problem or symptom (1). Three additional measures asked about drinking behaviors in the past 6 months: frequent drinking, drunkenness, and job-related hangovers.
In another study, employees from small businesses selected from three at-risk industries (construction, transportation, and hospitality/service) provided responses to questions about their own substance use. A single item ("During the past 30 days, how many days have you used any alcohol?") assessed alcohol frequency. Responses were recoded on a scale from 0 (none) to 4 (22 to 30 days). Respondents also answered the 4 questions in the CAGE, a measure of alcohol problems:
- Have you ever felt you should cut down on your drinking?
- Have people annoyed you by criticizing your drinking?
- Have you ever felt bad or guilty about your drinking?
- Have you ever had a drink first thing in the morning to steady your nerves or to get rid of a hangover?
Item responses on the CAGE were scored 0 or 1, with a higher score being a greater indication of alcohol problems.
|
|
Key Findings
|
From pretest to 6-month follow-up, city employees from at-risk departments who received the 8-hour Team Awareness training demonstrated a significant decrease in the frequency of job-related hangovers (16% to 6%) and problem drinking (20% to 11%), compared with employees who received the 4-hour informational training (p = .007 for hangovers; p = .01 for problem drinking).
In the study of small businesses in at-risk industries, alcohol problems significantly decreased among employees who received either the 4-hour Team Awareness-SB training or customized health promotion training, compared with employees who received no training (p = .04). In addition, employees with self-reported alcohol or other drug abuse showed significant decreases in alcohol use frequency (p < .0001).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.4
(0.0-4.0 scale)
|
| Outcome 6: Personal health and well-being |
|
Description of Measures
|
Employees from small businesses selected from three at-risk industries (construction, transportation, and hospitality/service), completed the 6-item Spiritual Health subscale of the Perceived Wellness Survey (e.g., "It seems that my life has always had a purpose").
In addition, respondents indicated how often they performed each of four positive "unwinding" behaviors to relax, forget worries, and cope with stress, both in general and after work: calling or spending time with friends; meditating or praying; watching television, reading, or using other forms of entertainment; or exercising. Response options ranged from 1 (not at all) to 5 (very often). Ratings of general and after-work unwinding were averaged for all four positive unwinding behaviors.
|
|
Key Findings
|
Spiritual health significantly improved from pretest to 6-month follow-up for employees who received the 4-hour Team Awareness-SB training (p < .05). Employees who received either customized health promotion training or no training realized no significant changes in spiritual health.
Positive unwinding significantly increased among men who received either the 4-hour Team Awareness-SB training or customized health promotion training (p < .001 and p = .029, respectively) and significantly decreased among men who received no training (p = .027). Positive unwinding effects were not significant for women.
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
|
75.7% Male
24.3% Female
|
57.6% White
16.5% Black or African American
14.8% Race/ethnicity unspecified
11.1% Hispanic or Latino
|
|
Study 2
|
18-25 (Young adult)
26-55 (Adult)
55+ (Older adult)
|
56% Male
44% Female
|
53% White
27% Hispanic or Latino
16% Black or African American
4% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Group climate
|
2.2
|
2.6
|
2.7
|
2.0
|
2.2
|
3.5
|
2.5
|
|
2: Knowledge and attitudes related to substance use policies and the EAP
|
2.1
|
2.7
|
2.5
|
2.0
|
2.0
|
2.9
|
2.4
|
|
3: Help-seeking attitudes, behavior, and encouragement
|
2.2
|
2.9
|
2.7
|
2.0
|
2.2
|
3.5
|
2.5
|
|
4: Alcohol and other drug use attitudes and drinking climate
|
2.3
|
2.9
|
2.7
|
3.0
|
3.2
|
3.9
|
3.0
|
|
5: Alcohol use and alcohol-related problems
|
3.5
|
4.0
|
2.7
|
3.0
|
3.2
|
4.0
|
3.4
|
|
6: Personal health and well-being
|
2.4
|
2.2
|
2.9
|
2.1
|
2.4
|
4.0
|
2.7
|
Study Strengths The studies conducted were longitudinal and included multiple sites and random assignment to study conditions. Many of the scales used were straightforward and specific to the research protocol and also had evidence of convergent validity. In one study, there was more use of archival data, such as EAP activity. The researchers did an acceptable job of ensuring the fidelity of the intervention. Participants in the training groups were generally responsive and productive, showing self-disclosure and other group-process behaviors that Team Awareness was meant to instill, allowing the inference that the intervention was delivered as planned. A fidelity checklist was provided to Team Awareness facilitators in both studies, with a much better attempt to ensure and document fidelity in the second study. The researchers paid very close attention to issues pertaining to differential attrition by group and by key variables. In general, analyses were very sophisticated and used state-of-the-art methods; for example, hierarchical linear modeling was used to take into account group (i.e., business) membership, when appropriate. Covariates were included appropriately.
Study Weaknesses The statistical methods used to measure reliability of the 2-, 3-, and 4-item scales were not appropriate. Some measures were subject to a test/retest reliability procedure by correlating, for the control group for one study, pretest scores with 6-month follow-up scores, resulting in low correlations (few were above .60). Participant responsiveness is a poor measure of fidelity. Attrition was substantial and much greater in the Team Awareness condition. Dropouts tended to be the most "difficult to treat" (e.g., reported having five or more drinks in one sitting in the past month, hangovers that affect work), which may have prevented the training from reaching the employees who could have benefited the most from it. This differential subject self-selection may have inflated the program's effectiveness. One study found substantial differences following random assignment on several key variables. Several of the analyses involving subgroups (e.g., employees abusing substances, supervisors) relied on small sample sizes.
|
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
* Bennett, J. B., & Lehman, W. E. K. (2001). Workplace substance abuse prevention and help seeking: Comparing team-oriented and informational training. Journal of Occupational Health Psychology, 6(3), 243-254.
* Bennett, J. B., & Lehman, W. E. K. (2002). Supervisor tolerance-responsiveness to substance abuse and workplace prevention training: Use of a cognitive mapping tool. Health Education Research, 17(1), 27-42.
Bennett, J. B., Lehman, W. E. K., & Reynolds, G. S. (2000). Team awareness for workplace substance abuse prevention: The empirical and conceptual development of a training program. Prevention Science, 1(3), 157-172.
* Bennett, J. B., Patterson, C., Sledz, R., Klimback, E. M., Berish, J. E., & Cook, R. (n.d.). Small Business Wellness Initiative research report. Fort Worth, TX: Organizational Wellness & Learning Systems.
* Bennett, J. B., Patterson, C. R., Reynolds, G. S., Wiitala, W. L., & Lehman, W. E. K. (2004). Team Awareness, problem drinking, and drinking climate: Workplace social health promotion in a policy context. American Journal of Health Promotion, 19(2), 103-113.
Bennett, J. B., Reynolds, G. S., & Lehman, W. E. K. (2003, March). The "black box" of health promotion: Training room behaviors predict outcomes. Poster session presented at Work, Stress & Health, 2003: Fifth Interdisciplinary Conference on Work Stress, Toronto, Canada.
Lehman, W. E. K., Bennett, J. B., & Reynolds, G. S. (2000). TCU workplace measures. Fort Worth: Texas Christian University, Institute of Behavioral Research.
* Lehman, W. E. K., Reynolds, G. S., & Bennett, J. B. (2002). Team and informational trainings for workplace substance abuse prevention. In J. B. Bennett & W. E. K. Lehman (Eds.), Preventing workplace substance abuse: Beyond drug testing to wellness (pp. 165-201). Washington, DC: American Psychological Association.
* Patterson, C. R., Bennett, J. B., & Wiitala, W. L. (2005). Healthy and unhealthy stress unwinding: Promoting health in small businesses. Journal of Business and Psychology, 20(2), 221-247.
Reynolds, G. S., & Lehman, W. E. K. (2008). Workgroup temperance of alcohol and safety climate moderate the cognitive effects of workplace substance-abuse prevention. Journal of Applied Social Psychology, 38(7), 1827-1866.
|
