Quality of Research
Review Date: December 2006
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Hecht, M. L., Graham, J. W., & Elek, E. (2006). The Drug Resistance Strategies intervention: Program effects on substance use. Health Communication, 20(3), 267-276. 
Hecht, M. L., Marsiglia, F. F., Elek, E., Wagstaff, D. A., Kulis, S., Dustman, P., et al. (2003). Culturally grounded substance use prevention: An evaluation of the keepin' it REAL curriculum. Prevention Science, 4, 233-248.
Kulis, S., Marsiglia, F. F., Elek-Fisk, E., Dustman, P., Wagstaff, D., & Hecht, M. L. (2005). Mexican/Mexican American adolescents and keepin' it REAL: An evidence-based, substance abuse prevention program. Children and Schools, 27, 133-145.
Kulis, S., Nieri, T., Yabiku, S., Stromwall, L. K., & Marsiglia, F. F. (2007). Promoting reduced and discontinued substance use among adolescent substance users: Effectiveness of a universal prevention program. Prevention Science, 8(1), 35-49.
Marsiglia, F. F., Kulis, S., Wagstaff, D. A., Elek, E., & Dran, D. (2005). Acculturation status and substance use prevention with Mexican and Mexican American youth. Journal of Social Work Practice in the Addictions, 5, 85-111.
Warren, J. R., Hecht, M. L., Wagstaff, D. A., Elek, E., Ndiaye, K., Dustman, P., et al. (2006). Communicating prevention: The effects of the keepin' it REAL classroom videotapes and televised PSAs on middle-school students' substance use. Journal of Applied Communication Research, 34(2), 209-227.
Supplementary Materials Description of the measures used in the outcome study
Graham, J. W., Flay, B. R., Johnson, C. A., Hansen, W. B., Grossman, L. G., & Sobel, J. L. (1984). Reliability of self-report measures of drug use in prevention research: Evaluation of the Project SMART Questionnaire via the Test-Retest Reliability Matrix. Journal of Drug Education, 14(2), 175-193.
Hansen, W. B., & Graham, J. W. (1991). Preventing alcohol, marijuana, and cigarette use among adolescents: Peer pressure resistance training versus establishing conservative norms. Preventive Medicine, 20, 414-430.
Hansen, W. B., Johnson, C. A., Flay, B. R., Graham, J. W., & Sobel, J. (1988). Affective and social influences approaches to prevention of multiple substance abuses among seventh grade students: Results from Project SMART. Preventive Medicine, 19, 135-154.
Hecht, M. (n.d.). Keepin' it REAL curriculum guide: Teachers manual. University Park, PA: Author.
Hecht, M. E., Marsiglia, F. F., Elek-Fisk, E., Graham, J. W., Kulis, S., & Dustman, P. (2001, May/June). Evaluation of the Drug Resistance Strategies intervention: A test of cultural appropriateness in program content. Symposium presented at the 9th Annual Meeting of the Society for Prevention Research, Washington, DC.
Outcomes
| Outcome 1: Alcohol, cigarette, and marijuana use |
|
Description of Measures
|
Substance use was compared using self-reported quantity and frequency of use before the program and at 2, 8, and 14 months after students completed the curriculum. Questionnaires asked how often and how much students used alcohol, cigarettes, and marijuana (e.g., "On how many of the past 30 days did you use alcohol?" and "How many drinks have you had in the past 30 days?").
|
|
Key Findings
|
Curriculum participants reported lower alcohol, marijuana, and cigarette use than students who did not receive the program. Effects lasted up to 14 months for alcohol use and marijuana use and up to 8 months for cigarette use.
Matching students' language preference to a particular version of the curriculum (i.e., Spanish or English curriculum) did not improve outcomes for reported substance use.
Intervention students who received the Mexican American or the multicultural versions of the curriculum reported lower alcohol use than control students (p = .0018 and p = .0001, respectively). Students who received the multicultural version of the curriculum also reported a slower increase in marijuana use over time compared with control students (p = .0061).
Forty percent of participants who used alcohol at baseline reported reductions in alcohol use after receiving the curriculum, compared with 30% of control students who were baseline users (p < .001). Thirty-two percent of intervention students who used alcohol at baseline reported discontinuation of use, compared with 24% of control students who were baseline users (p < .01).
Positive outcomes occurred primarily among students who saw four or five of the curriculum videos. For example, compared with control students, intervention students who saw four or more intervention videos reported fewer days of alcohol use (p < .001), fewer drinks consumed (p = .029), fewer days of marijuana use (p = .007), and fewer "hits" of marijuana (p = .007). Curriculum participants who saw fewer than four videos did not report lower rates of substance use.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
| Outcome 2: Anti-substance use attitudes |
|
Description of Measures
|
Anti-substance use attitudes were measured with a questionnaire that asked students about their intention to accept offers to use alcohol, cigarettes, or marijuana in the future. It also addressed their beliefs about the positive consequences of substance use and their confidence that they could refuse offers to use substances from an acquaintance, a friend, or a family member.
|
|
Key Findings
|
At the 8- and 14-month follow-ups, students who received the curriculum reported lower expectations of positive consequences of substance use compared with students who did not receive the intervention.
Students who received the Mexican American version of the curriculum reported increases in their perceived ability to refuse offers to use substances and smaller increases in intentions to use substances in the future compared with students who did not receive the intervention.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
| Outcome 3: Normative beliefs about substance use |
|
Description of Measures
|
Antidrug normative beliefs were measured with a questionnaire that asked students what they believed and what their parents and peers believed was right or wrong about substance use. For example, students were asked if they believed it was "OK" for someone their age to use alcohol, cigarettes, or marijuana; how angry their parents would be if they used substances; and how their best friends would act toward them if they used substances. Students were also asked to estimate how many friends or peers in their school used drugs regularly or experimented with them occasionally.
|
|
Key Findings
|
Compared with control students, students receiving the curriculum reported lower personal acceptance of drug use 2 and 8 months after the intervention (but not 12 months afterward). The intervention group also reported smaller increases in estimates of the number of peers who experimented with drugs occasionally and used drugs regularly at 2, 8, and 12 months after the intervention compared with the control group. The intervention had no effect on perceptions of parental or peer norms.
Researchers found that students who received the Mexican American version of the curriculum, compared with control students, reported smaller increases in estimates of the number of their friends and peers who used drugs.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 4: Substance use resistance |
|
Description of Measures
|
Substance use resistance was measured using a questionnaire that asked students if they had used any of the intervention strategies to turn down an offer to use substances: Refuse ("just say no"), Explain ("giving an explanation or some other excuse"), or Leave ("just leaving the situation").
|
|
Key Findings
|
Students in the intervention group reported greater use of these strategies to resist marijuana use 2 months after the intervention and to resist cigarette use 2 and 8 months after the intervention. The effect was not found 12 months after the intervention.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
1.7
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
|
52.5% Male
47.5% Female
|
73.9% Hispanic or Latino
17.4% White
8.7% Black or African American
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Alcohol, cigarette, and marijuana use
|
3.0
|
3.0
|
2.0
|
2.8
|
2.5
|
3.0
|
2.7
|
|
2: Anti-substance use attitudes
|
2.5
|
3.0
|
2.0
|
2.8
|
2.5
|
3.0
|
2.6
|
|
3: Normative beliefs about substance use
|
2.5
|
2.5
|
2.0
|
2.8
|
2.5
|
3.0
|
2.5
|
|
4: Substance use resistance
|
0.0
|
0.5
|
2.0
|
2.5
|
2.3
|
3.0
|
1.7
|
Study Strengths The measures were developed and pilot tested with members of targeted ethnic groups to ensure cultural sensitivity. The psychometric properties were adequate and demonstrated stability over four waves of data collection. Students responded anonymously, which has been shown to enhance the validity of self-report.
Providing teachers with training and a standardized curriculum manual enhanced implementation fidelity, and the observation of 76% of teachers implementing the curriculum demonstrated high fidelity ratings (average fidelity ratings were 5.8 on a scale of 1.0-7.0).
Although attrition was high (only one in four students completed all four waves of the survey), it did not vary significantly by ethnic group, so attrition bias does not appear to be a significant concern. Moreover, missing data were handled appropriately, with multiple imputation using methods developed by Rubin and Little, along with generalized estimating equations (GEE).
Most analyses were sophisticated and appropriate, with large sample sizes and appropriate covariates.
Study Weaknesses Tests of cultural matching may have suffered from low statistical power caused by including so few African American and White students. There was no examination of the cultural content of programming received by control students.
Despite the use of sensitivity analyses and plans to address attrition, there was a high level of attrition, with two schools not participating in data collection 8 and 14 months after the intervention.
Confounding variables present some concerns. In addition, some of the participants that were at higher risk had stronger outcomes. Consequently, it is unclear if their risk status explains the results better than the success of the intervention.
Approximately 30% of participants saw all five videos; those who did not see at least four videos may have a problem with school attendance, a condition associated with numerous other negative outcomes. In addition, they probably did not receive other aspects of the curriculum at the same level as those who viewed four or five videos.
The resistance strategy measure was developed specifically for this study, and no evidence of reliability was presented. The items are face-valid, yet the scales are limited to yes/no reports of using three strategies. (The program acronym "REAL" reflects four resistance strategies; only Refuse, Explain, and Leave were evaluated, and it is unclear why Avoid was not included.) The emphasis on resistance strategies, which were a key component of the curriculum, varied between the different culturally grounded curricula.
Baseline self-reported use of strategies was controlled in the GEE analyses. However, it is impossible to discern whether the outcomes reflect an increase in the ability to use the strategies (i.e., an increase in skill level) or simply reflect a greater motivation to use the strategies.
|
|
