Quality of Research
Review Date: November 2008
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Henggeler, S. W., Rowland, M. D., Randall, J., Ward, D. M., Pickrel, S. G., Cunningham, P. B., et al. (1999). Home-based multisystemic therapy as an alternative to the hospitalization of youths in psychiatric crisis: Clinical outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 38(11), 1331-1339. 
Huey, S. J., Henggeler, S. W., Rowland, M. D., Halliday-Boykins, C. A., Cunningham, P. B., Pickrel, S. G., & Edwards, J. (2004). Multisystemic therapy effects on attempted suicide by youths presenting psychiatric emergencies. Journal of the American Academy of Child and Adolescent Psychiatry, 43(2), 183-190.
Schoenwald, S. K., Ward, D. M., Henggeler, S. W., & Rowland, M. D. (2000). Multisystemic therapy versus hospitalization for crisis stabilization of youth: Placement outcomes 4 months postreferral. Mental Health Services Research, 2(1), 3-12. Study 2Rowland, M. D., Halliday-Boykins, C. A., Henggeler, S. W., Cunningham, P. B., Lee, T. G., Kruesi, M. J., & Shapiro, S. B. (2005). A randomized trial of multisystemic therapy with Hawaii's Felix class youth. Journal of Emotional and Behavioral Disorders, 13(1), 13-23.
Outcomes
| Outcome 1: Mental health symptoms |
|
Description of Measures
|
Mental health symptoms were assessed using:
- The Global Severity Index of the Brief Symptom Inventory (GSI-BSI), completed by adolescent to measure emotional distress (e.g., obsession-compulsion, depression, anxiety, hostility); responses were given on a 5-point scale ranging from 0 (not at all) to 4 (extremely).
- Child Behavior Checklist (CBCL), completed by the adolescent's caregiver and teacher to measure the adolescent's internalizing and externalizing symptoms; respondents rated how true each item was using a 3-point scale (0 = not true, 1 = somewhat or sometimes true, and 2 = very true or often true).
|
|
Key Findings
|
In one study, caregiver and teacher reports showed a significant reduction in MST-Psychiatric participants' externalizing symptoms from pretest to posttreatment (4 months after recruitment) relative to a comparison group of hospitalized youth who were released 1-2 weeks after recruitment (p < .01 and p < .04 respectively). No statistically significant differences were found between groups on internalizing symptoms and measures of emotional distress from pretest to posttreatment.
In another study, MST-Psychiatric participants reported a significant reduction in externalizing symptoms (p = .04) and internalizing symptoms (p = .02) 6 months after study entry relative to a comparison group of youth who received usual services from the State mental health agency.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.5
(0.0-4.0 scale)
|
| Outcome 2: Family relations |
|
Description of Measures
|
The Family Adaptability and Cohesion Evaluation Scale-III (FACES-III) was used to assess family relations (i.e., family adaptability and cohesion) using youth and caregiver reports.
|
|
Key Findings
|
One study found a significant improvement in family relations among MST-Psychiatric participants from pretest to posttreatment (4 months after recruitment) relative to a comparison group of hospitalized youth who were released 1-2 weeks after recruitment. Youth reports showed that families in the MST-Psychiatric condition had become more structured, while families in the hospitalization condition became less structured (p < .009). Caregiver reports also showed a significant difference between groups on family cohesion, which increased among MST-Psychiatric families and decreased among families in the hospitalized group from pretest to posttreatment (p < .004).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 3: School attendance |
|
Description of Measures
|
School attendance was measured using:
- The Service Utilization Survey, which tracked school attendance through monthly caregiver reports with corroboration by school contacts.
- Archival data on school placements. Each day the youth was in the study was coded with respect to school attendance and the restrictiveness of school placements.
For both measures, participants were considered to be in school (general education setting) if enrolled in an academic program with a standardized curriculum approved by the public school system.
|
|
Key Findings
|
In one study, MST-Psychiatric participants spent significantly fewer days out of school from pretest to posttreatment (4 months after recruitment; p < .018) relative to a comparison group of hospitalized youths who were released 1-2 weeks after recruitment.
In another study, 6 months after study entry, MST-Psychiatric participants spent 42% more days per month in general education settings than a comparison group of youths who received usual services from the State mental health agency. This difference was not statistically significant.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 4: Suicide attempts |
|
Description of Measures
|
Suicide attempts were assessed using:
- A single item from the CBCL representing the frequency of self-harm behavior from the caregiver's perspective ("deliberately harms self or attempts suicide"). Responses ranged from 0 (not true) to 2 (very true or often true).
- A single item taken from the Youth Risk Behavior Survey (YRBS) representing the number of times the youth attempted suicide in the past 12 months ("How many times did you actually attempt suicide?"). Responses ranged from 0 (0 times) to 4 (6 or more times).
|
|
Key Findings
|
One study found that MST-Psychiatric was significantly more effective at reducing self-reported attempted suicide over 16 months following recruitment (p < .001) than a comparison condition in which hospitalized youths were released 1-2 weeks after recruitment. No significant differences were found between groups on suicide attempts as rated by caregivers.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 5: Days in out-of-home placement |
|
Description of Measures
|
The number of days youth spent in out-of-home placement was measured using:
- The Service Utilization Survey. This telephone survey was administered monthly to caregivers by program staff at each data collection point. Out-of-home placement included foster care, therapeutic foster care, shelters, orphanages, group homes, residential treatment centers, psychiatric or substance abuse hospitals, detention centers, boot camps, reception and evaluation centers, jails, and prisons.
- Data collected by State agencies, including the Child and Adolescent Mental Health Division (CAMHD) and juvenile justice authorities. CAMHD provided information on inpatient, residential, foster care, and group home placements, and the juvenile justice authorities provided detention and incarceration data. The number of days spent in these placements was summed to indicate the total days in out-of-home placements per month.
|
|
Key Findings
|
In one study, MST-Psychiatric prevented hospitalization for 57% of participants (p < .001) and reduced the number of days hospitalized by 72% (p < .001) compared with a hospitalization condition in which youths were released from the hospital 1-2 weeks after recruitment. MST-Psychiatric reduced days in out-of-home placement by 49%.
In another study, MST-Psychiatric was found to be significantly more effective at maintaining youths in the community than a comparison condition in which youth received usual services from the State agency (p = .02). Specifically, youths in the MST-Psychiatric condition averaged 3.75 days of out-of-home placement services per month, versus an average of 11.83 days among comparison youth.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
13-17 (Adolescent)
|
65% Male
35% Female
|
64% Black or African American
34% White
1% Asian
1% Hispanic or Latino
|
|
Study 2
|
13-17 (Adolescent)
|
58% Male
42% Female
|
84% Native Hawaiian or other Pacific Islander
10% White
6% Asian
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Mental health symptoms
|
3.8
|
4.0
|
3.5
|
3.5
|
2.8
|
3.5
|
3.5
|
|
2: Family relations
|
2.5
|
3.5
|
3.8
|
3.5
|
2.8
|
3.5
|
3.3
|
|
3: School attendance
|
2.8
|
2.8
|
3.5
|
3.5
|
2.8
|
3.5
|
3.1
|
|
4: Suicide attempts
|
3.3
|
3.0
|
3.8
|
3.5
|
2.8
|
3.5
|
3.3
|
|
5: Days in out-of-home placement
|
2.0
|
2.5
|
3.5
|
3.5
|
2.8
|
3.5
|
3.0
|
Study Strengths Most of the outcomes used measures with good, well-documented psychometric properties (e.g., CBCL, GSI-BSI, FACES-III). Multiple methods with acceptable fidelity were used to address intervention fidelity, including structured supervision, use of standardized treatment adherence measures, and coded audiotapes of family treatment sessions. Attrition was low for both studies. Both studies used a randomized control group design. Analyses were appropriate for the data collected.
Study Weaknesses The measures used for school attendance and out-of-home placement did not have documented psychometric properties. One study had a small sample size. Statistical regression poses a threat to the internal validity of both studies reviewed because youths in the MST-Psychiatric group differed from those in the comparison condition at pretest; specifically, youths from the MST-Psychiatric group were more antisocial than the comparison group at pretest and had significantly higher rates of attempted suicide before treatment.
|
|
Readiness for Dissemination
Review Date: November 2008
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Cunningham, P. B., Rowland, M. D., Schoenwald, S. K., Swenson, C. C., Henggeler, S. W., Randall, J., & Donohue, B. (n.d.). Community reinforcement approach to support caregivers. Mount Pleasant, SC: MST Services.
Henggeler, S. W., Schoenwald, S. K., Rowland, M. D., & Cunningham, P. B. (2002). Serious emotional disturbance in children and adolescents: Multisystemic therapy. New York: Guilford Press.
Medical University of South Carolina, Department of Psychiatry and Behavioral Sciences, Family Services Research Center. (2008). Implementing contingency management for adolescent substance abuse in outpatient settings (2nd ed.). Charleston, SC: Author.
Multisystemic Therapy RFD Overview Packet
Multisystemic Therapy Web site, http://www.mstservices.com
Rowland, M. D., Cunningham, P. B., & Westlake, L. A. (2008, September). Basic steps to ensure in-home & community safety during MST with psychiatric support: Handouts. Mount Pleasant, SC: MST Services.
Rowland, M. D., Cunningham, P. B., & Westlake, L. A. (2008, September). Basic steps to ensure in-home & community safety during MST with psychiatric support: Trainer's notes. Mount Pleasant, SC: MST Services.
Rowland, M. D., & Goetz, P. (2008, September). The role of the psychiatrist in MST with psychiatric support: Outline/handouts. Mount Pleasant, SC: MST Services.
Rowland, M. D., & Westlake, L. A. (2006, Fall). Mental health multisystemic therapy psychiatry resource book. Mount Pleasant, SC: MST Services.
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
4.0
|
4.0
|
4.0
|
4.0
|
Dissemination Strengths An array of high-quality materials is provided to support program implementation. The specific skills and education required of personnel implementing the program or providing supervision are clearly described. The specialized role of the treating psychiatrist in the MST-Psychiatric team is emphasized in the intervention materials and addressed in training. The developer provides orientation training for clinicians, weekly telephone consultations, quarterly on-site booster trainings, and special guidance for supervisors to support initial installation of the program within an agency. The training materials use multiple well-placed case studies. Standardized measures for monitoring protocol adherence, data collection tools, and clear instructions for the use of these items are available to support a comprehensive quality assurance process. The developer provides additional quality assurance support services.
Dissemination Weaknesses No weaknesses were identified by reviewers.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Program development start-up fees (includes site readiness visit and 7-day, on-site orientation training)
|
$15,000 plus travel expenses
|
Yes
|
|
Annual program support and service fees (includes annual agency/team license fees, start-up kit, manuals, training materials, weekly case consultation, supervisor support calls as needed, quarterly on-site booster trainings, and use of Web-based adherence monitoring and outcome tracking system)
|
$96,500 per site plus travel expenses
|
Yes
|
|
2-day supervisor orientation training
|
$350 per participant plus travel expenses
|
Yes
|
|
Tape coding
|
About $7,920
|
Yes
|
|
Adherence data collection
|
About $3,600 per year
|
Yes
|
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
Stambaugh, L. F., Mustillo, S. A., Burns, B. J., Stephens, R. L., Baxter, B., Edwards, D., & DeKraai, M. (2007). Outcomes from wraparound and multisystemic therapy in a Center for Mental Health Services system-of-care demonstration site. Journal of Emotional and Behavioral Disorders, 15(3), 143-155.
|
|
|
