Quality of Research
Review Date: January 2009
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Jerabek, A., & Laney, R. (2006). Minnesota Department of Health and Human Services, CHD grant/final report--June 2006.
Jerabek, A. M., & Laney, R. (2007). Application for the National Registry of Evidence-based Programs and Practices: Building Assets--Reducing Risks. Unpublished manuscript.
Johnston, B. (personal communication, January 10, 2008). Statistical testing of groups in charts found in the Application for the National Registry of Evidence-based Programs and Practices: Building Assets--Reducing Risks.
Supplementary Materials Jerabek, A. M. (2004). Program assessment. Report to the Minnesota Department of Health and Human Services, Chemical Health Division (CHD).
Jerabek, A. M. (2005). Program assessment. Report to the Minnesota Department of Health and Human Services, Chemical Health Division (CHD).
Jerabek, A. M. (2008). Use of quasi-experimental design in evaluating Building Assets Reducing Risks. Unpublished manuscript.
Minnesota Departments of Education, Health, Human Services, and Public Safety. Minnesota Student Survey, 1992-2007 trends: Behaviors, attitudes, and perceptions of Minnesota's 6th, 9th, and 12th graders.
Sharma, A., & Griffin, T. (2003). Saint Louis Park 9th grade program: Saint Louis Park High School, Saint Louis Park School District (Independent School District 283). Summative evaluation report.
Outcomes
| Outcome 1: Class failure |
|
Description of Measures
|
Academic class failure was measured as the percentage of students who failed one or more classes and who failed two or more classes per school semester.
|
|
Key Findings
|
The percentage of 9th-grade students who failed one or more classes at the intervention school decreased from 44% before BARR implementation to 28% during the 1st year of implementation (p < .05) and 21% following 5 years of implementation (21% vs. 28% or 44%; p < .05). A smaller percentage of 9th-grade students in the intervention school failed two or more classes after 5 years of BARR implementation (10%) than in the year before or year after implementation (18% and 21%, respectively; p < .05). No data on a comparison group of students were provided.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
1.3
(0.0-4.0 scale)
|
| Outcome 2: Bullying at school |
|
Description of Measures
|
Bullying at school was measured using the Minnesota Student Survey (MSS), a voluntary, written, self-report instrument administered statewide every 3 years to 6th-, 9th-, and 12th-grade students in regular public schools and charter and tribal schools. The MSS covers a wide variety of youth-related behaviors, attitudes, and perceptions and includes questions on substance abuse, school climate, academics and school connectedness, school safety and violence, mental health, family structure, family connectedness, sexual behavior, gambling, and out-of-school activities. This study used one item in the perceptions of school safety domain that asks if the student has been threatened or made fun of because of race or cultural background.
|
|
Key Findings
|
Bullying data from the MSS for 9th-graders in the intervention school were compared with data for all other 9th-graders statewide 1 year before BARR implementation, after 2 years of implementation, and after 5 years of implementation. Among the findings from this study:
- From pre- to postintervention, the percentage reporting bullying declined more for boys in the intervention school than for statewide comparison boys (14% to 9% vs. 18% to 17%, respectively; p < .05).
- No statistically significant difference in bullying was found between girls in the intervention school and statewide comparison girls from pre- to postintervention.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
1.0
(0.0-4.0 scale)
|
| Outcome 3: School connectedness |
|
Description of Measures
|
School connectedness was measured using the MSS, a voluntary, written, self-report instrument administered statewide every 3 years to 6th-, 9th-, and 12th-grade students in regular public schools and charter and tribal schools. The MSS covers a wide variety of youth-related behaviors, attitudes, and perceptions and includes questions on substance abuse, school climate, academics and school connectedness, school safety and violence, mental health, family structure, family connectedness, sexual behavior, gambling, and out-of-school activities. This study used three items in the academics and school connectedness domain that ask the student about feeling positive about going to school, the perception that teachers are interested in the student as an individual person, and the perception that teachers at school show respect for students.
|
|
Key Findings
|
School connectedness data from the MSS for 9th-graders in the intervention school were compared with data for all other 9th-graders statewide 1 year before BARR implementation, after 2 years of implementation, and after 5 years of implementation. Among the findings from this study:
- From pre- to postintervention, the percentage reporting feeling positive about going to school increased for boys in the intervention school and decreased for statewide comparison boys (37% to 42% vs. 33% to 32%, respectively; p < .05). Similar results were found for the percentage of boys who reported their teachers were sincerely interested in them (36% to 46% vs. 38% to 36%, respectively; p < .05).
- From pre- to postintervention, the percentage reporting their teachers showed respect for students increased more for boys in the intervention school than for statewide comparison boys (66% to 77% vs. 63% to 68%, respectively; p < .05).
- From pre- to postintervention, the percentage reporting their teachers showed respect for students increased slightly for girls in the intervention school as well as for statewide comparison girls (77% to 79% vs. 68% to 71%, respectively), with no significant difference between the two groups.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
1.0
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
13-17 (Adolescent)
|
51% Male
49% Female
|
79.3% White
11% Black or African American
6.1% Asian
3.3% Hispanic or Latino
0.2% American Indian or Alaska Native
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Class failure
|
2.5
|
3.0
|
1.5
|
0.0
|
0.0
|
0.5
|
1.3
|
|
2: Bullying at school
|
1.0
|
2.0
|
1.5
|
0.0
|
0.0
|
1.5
|
1.0
|
|
3: School connectedness
|
1.0
|
2.0
|
1.5
|
0.0
|
0.0
|
1.5
|
1.0
|
Study Strengths School failure is an important scholastic benchmark for intervention at the 9th grade level. The Minnesota Student Survey appears to have face and content validity. The intervention is manualized and is based on a theory of adolescent development that emphasizes strengthening assets and reducing risk behaviors.
Study Weaknesses The absence of independent verification of school failure records is a concern, as the accuracy of school recordkeeping can vary by school. No reliability data were reported for the Minnesota Student Survey.
Intervention fidelity is difficult to determine; for example, it is unclear how often classes were observed, whether interviews were carried out with teachers and staff, and what was done with the information once collected.
Attrition and missing data are a major concern. The study's design did not allow the same students to be followed over time. In addition, limited information was provided on missing data.
No documentation was provided describing efforts to control for potential confounding variables; for example, although major changes were acknowledged for the student sample in the intervention school (e.g., an increase in the proportion of students with high-risk backgrounds), there was no corresponding information on whether similar changes occurred in the comparison student population, those in the statewide database, over the same period. These changes in the intervention student sample, as well as changes in program delivery staff, over the course of the evaluation make it difficult to determine how much of the findings were actually due to the intervention.
The analysis used a z-score test, which, unlike a t-test or ANOVA, is not appropriate for making inferences regarding outcome success because too much of the variance remains unexplained. The use of a z-score test for pre- and postintervention class failure rates is of particular concern because there was no comparison group, the same students were not followed over time in the intervention school, and the student samples in the intervention school changed demographically over time.
|
|
