Quality of Research
Review Date: January 2007
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Laird, M., & Roden, J. (1991). Supplemental final report: Additional technical analysis of data on Quest's Skills for Adolescence program in Detroit schools. Newark, OH: Quest International. Study 2Laird, M. (1992). Evaluation of Lions-Quest "Skills for Adolescence" program: An analysis of students' attitudes, use patterns, and knowledge about harmful drugs. Newark, OH: Quest International. Study 3Laird, M., & Syropoulos, M. (1995). An evaluation of Lions-Quest's Skills for Adolescence. Unpublished report. Study 4Eisen, M., Zellman, G. L., & Murray, D. M. (2003). Evaluating the Lions-Quest "Skills for Adolescence" drug education program: Second-year behavior outcomes. Addictive Behaviors, 28, 883-897. 
Supplementary Materials Binda, K. P., & Nicol, D. G. (2003). Lions-Quest Manitoba program evaluation: Skills for Growing/Skills for Adolescence. Brandon, Canada: Brandon University.
Ray, N. G. (1990). The effects of participation in the Lions Quest Skills for Adolescence Program on student self-concept at the middle school level. Unpublished doctoral dissertation, University of La Verne.
Quest International. (n.d.). Report for U.S. Department of Education Expert Panel on Safe, Disciplined, and Drug-Free Schools. Lions-Quest Skills for Adolescence. Newark, OH: Author.
Tinzmann, M. B. (2000). Evaluation of Quest curriculum materials.
Outcomes
| Outcome 1: Social functioning |
|
Description of Measures
|
Social functioning was measured by an anger management test developed by the authors. The test measures students' knowledge about correct ways to handle peer conflicts and self-perceptions of internalizing statements that are associated with anger escalation.
|
|
Key Findings
|
Participants who received lessons from three of the seven units of the SFA curriculum increased their average scores on a 25-item knowledge test about correct ways to handle peer conflicts from 12.6 items (50%) to 14.7 items (59%). No change in scores was observed in students who did not receive the lessons.
|
|
Studies Measuring Outcome
|
Study 3
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.3
(0.0-4.0 scale)
|
| Outcome 2: Success in school |
|
Description of Measures
|
Success in school was measured by (1) the School Attitude Measure (SAM) (e.g., academic self-concept and the student's sense of control over performance, or the degree to which a student feels that his or her performance is a serendipitous result of luck, fate, or other factors beyond one's control); (2) school records (e.g., the number of absences, overall grade point average (GPA), and grades in specific school subjects); and (3) the California Achievement Test scaled scores for the reading and math dimensions.
|
|
Key Findings
|
SFA participants increased their average GPAs after the intervention from 2.07 to 2.31; no similar increase was reported in the comparison group.
In another evaluation, SFA participants averaged slight declines in self-concept about their performance in school and their sense of personal control over performance, while students in other classrooms experienced more dramatic declines (p = .041). Seventh-grade SFA participants averaged higher scores on the California Achievement Test in reading (p < .001) and math (p = .004). Additionally, compared with students in other classrooms, SFA participants averaged fewer days of absence in the full marking period following the intervention (p = .046), higher grades in language arts (p = .009) and math (p = .004), and higher overall GPAs (p = .025).
|
|
Studies Measuring Outcome
|
Study 1, Study 3
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
| Outcome 3: Misconduct |
|
Description of Measures
|
Misconduct was measured through teachers' daily logs of individual students' low-level aggressive acts (e.g., insubordination, verbal abuse, loitering or trespassing, refusal to identify self, smoking in school or on school property, gambling, demonstration, truancy, disruptive conduct, and unauthorized use of materials or equipment).
|
|
Key Findings
|
SFA participants had half the number of negative behaviors in the intervention and follow-up periods compared with other students (p < .022). SFA participants also received comparatively lower misconduct ratings from their teachers during the intervention.
|
|
Studies Measuring Outcome
|
Study 3
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.1
(0.0-4.0 scale)
|
| Outcome 4: Attitudes and knowledge related to alcohol and other drugs (AOD) |
|
Description of Measures
|
In one study, AOD attitudes and knowledge were measured using The Knowledge Test, an instrument consisting of 45 true/false, multiple choice, and open-ended short essay questions. AOD attitudes and knowledge were also measured through survey items asking the extent to which respondents thought various substances (beer, wine coolers, chewing tobacco, cigarettes, marijuana, and cocaine/crack) were harmful to their health; response options included "not at all," "somewhat," and "very much." In another study, students' sense of self-efficacy around refusing the use of alcohol and other drugs in various situations was measured using separate three-item scales.
|
|
Key Findings
|
Participation in the Living Healthy and Drug-Free unit of SFA increased student perceptions of risk associated with the use of beer, liquor, chewing tobacco, cigarettes, marijuana, and cocaine/crack (p = .05). Among 8th-grade students, participation also increased knowledge about substance abuse.
In a separate evaluation, SFA participants increased their average scores for self-efficacy in refusing offers of marijuana and alcohol in a variety of situations, compared with other peers, after controlling for baseline self-efficacy and drug use (p = .021 and p = .044, respectively).
|
|
Studies Measuring Outcome
|
Study 2, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 5: Tobacco use |
|
Description of Measures
|
Tobacco use was measured by asking students, in a survey called "Drug Use," about how often in the past month they chewed tobacco or smoked cigarettes. Response options ranged from "not at all" to "every day."
|
|
Key Findings
|
Participants in the Living Healthy and Drug-Free unit of SFA had lower levels of current chewing tobacco use than other students, after controlling for preintervention use (p < .001). No statistically significant effects were found for cigarette smoking.
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.3
(0.0-4.0 scale)
|
| Outcome 6: Alcohol use |
|
Description of Measures
|
In one study, current alcohol use was measured by asking students, in a survey called "Drug Use," how often in the past month they drank beer or wine coolers, or liquor. Response options ranged from "not at all" to "every day." Students were also asked how much they thought they would drink beer or wine coolers, or liquor in the next month. Response options were "not at all," "more than this month," "about the same as this month," and "less than this month." In a second study, alcohol use prevalence rates were assessed through a set of standard items that was modified, when necessary, following pretesting with the target population.
|
|
Key Findings
|
Participants in the Living Healthy and Drug-Free unit of SFA had lower levels of current beer use, lower predicted future use of beer, and lower predicted use of liquor than comparison groups, after controlling for preintervention use (p < .02). In schools using SFA, 27% of students with a history of binge drinking reported binge drinking at the end of the 8th grade, compared with 37% of binge drinking students in other schools (p < .01).
|
|
Studies Measuring Outcome
|
Study 2, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 7: Marijuana use |
|
Description of Measures
|
Recent and lifetime marijuana use was assessed using standard questions adapted from the Monitoring the Future survey.
|
|
Key Findings
|
About 27% of SFA participants and 30% of a comparison group reported lifetime use of marijuana following the intervention (p = .05). About 11% of SFA participants and 14% of the comparison group reported use during the past 30 days, after controlling for preintervention use and demographic and psychosocial variables (p = .03).
|
|
Studies Measuring Outcome
|
Study 4
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.5
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
|
Data not reported/available
|
Data not reported/available
|
|
Study 2
|
6-12 (Childhood)
13-17 (Adolescent)
|
53% Female
47% Male
|
Data not reported/available
|
|
Study 3
|
6-12 (Childhood)
13-17 (Adolescent)
|
Data not reported/available
|
Data not reported/available
|
|
Study 4
|
6-12 (Childhood)
13-17 (Adolescent)
|
51.7% Female
48.3% Male
|
33.9% Hispanic or Latino
25.7% White
17.6% Black or African American
14.3% Race/ethnicity unspecified
7.1% Asian
1.4% American Indian or Alaska Native
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Social functioning
|
2.0
|
1.0
|
2.0
|
2.0
|
3.0
|
4.0
|
2.3
|
|
2: Success in school
|
3.9
|
3.9
|
1.5
|
2.4
|
1.9
|
2.5
|
2.7
|
|
3: Misconduct
|
0.5
|
2.0
|
2.0
|
2.0
|
2.0
|
4.0
|
2.1
|
|
4: Attitudes and knowledge related to alcohol and other drugs (AOD)
|
3.5
|
3.5
|
2.0
|
3.5
|
3.0
|
3.0
|
3.1
|
|
5: Tobacco use
|
4.0
|
4.0
|
0.0
|
2.0
|
1.0
|
3.0
|
2.3
|
|
6: Alcohol use
|
3.8
|
3.8
|
2.0
|
2.3
|
2.5
|
4.0
|
3.0
|
|
7: Marijuana use
|
3.5
|
3.5
|
2.5
|
3.5
|
4.0
|
4.0
|
3.5
|
Study Strengths In general, the studies were conducted in a large number of schools with a large number of students participating. Several of the instruments have good or acceptable levels of reliability and validity. The most recent study included improvements in: (1) the use of appropriate sample selection procedures (i.e., random assignment of units to treatment conditions) that decreased potential bias in groups; (2) the use of appropriate and more extensive statistical analyses to examine program effects and test rival hypotheses regarding the role of mediating factors in affecting student outcomes and control for pretest differences; (3) strengthened intervention fidelity with more teacher training and monitoring of curriculum implementation; and (4) increased statistical power because of larger sample sizes and less attrition.
Study Weaknesses Flaws, especially in some of the earlier studies, include: (1) low intervention fidelity, since training of teachers who delivered the program was initially too short, and teachers were observed only once or not at all; (2) weak or inappropriate research design and analyses (e.g., in one study, too many statistical tests capitalized on chance); (3) problematic sampling (e.g., convenience samples were used and schools were not randomly selected nor randomly assigned to treatment groups); even the well-designed 2003 study revealed group nonequivalence that rivals program effectiveness; and (4) high attrition in most of the studies that was not satisfactorily addressed, especially the potential bias that can be introduced by differential group attrition.
|
|
