Quality of Research
Review Date: November 2009
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Sherbourne, C. D., Wells, K. B., Duan, N., Miranda, J., Unützer, J., Jaycox, L., et al. (2001). Long-term effectiveness of disseminating quality improvement for depression in primary care. Archives of General Psychiatry, 58(7), 696-703. 
Wells, K., Sherbourne, C., Schoenbaum, M., Duan, N., Meredith, L., Unützer, J., et al. (2000). Impact of disseminating quality improvement programs for depression in managed care primary care: A randomized controlled trial. Journal of the American Medical Association, 283(2), 212-220. 
Supplementary Materials Rubenstein, L., Jackson-Triche, M., Unützer, J., Miranda, J., Minnium, K., Pearson, M. L., & Wells, K. B. (1999). Evidence-based care for depression in managed primary care practices. Health Affairs, 18(5), 89-105. 
Schoenbaum, M., Unützer, J., Sherbourne, C., Duan, N., Rubenstein, L. V., Miranda, J., et al. (2001). Cost-effectiveness of practice-initiated quality improvement for depression: Results of a randomized controlled trial. Journal of the American Medical Association, 286(11), 1325-1330. 
Outcomes
| Outcome 1: Depressive symptoms |
|
Description of Measures
|
Depressive symptoms were assessed using two instruments:
- A revised, 23-item version of the Center for Epidemiologic Studies Depression scale (CES-D)
- The Composite International Diagnostic Interview (CIDI), which assesses probable depressive disorder in the past 6 months
The revised CES-D was used at baseline and 6-, 12-, and 24-month follow-up. The CIDI was used at 6-, 12-, and 24-month follow-up.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that received PIC materials or to a comparison group that received the Agency for Healthcare Research and Quality (AHRQ) depression practice guidelines by mail. All eligible patients at these clinics were screened for depression, and those with current depressive symptoms and 12-month, lifetime, or no depressive disorder were enrolled in the study.
For depressive disorder as assessed by the revised CES-D, the study found:
- There were no significant group differences at baseline in the rate of depressive disorder.
- At 6-month follow-up, 55.4% of intervention group patients and 64.4% of comparison group patients were reported as being depressed (p = .005).
- At 12-month follow-up, 54.5% of intervention group patients and 61.4% of comparison group patients were still reported to be depressed (p = .04).
- The difference between groups was still significant at 24-month follow-up (p = .04).
For probable depressive disorder as assessed by the CIDI, the study found:
- At 6-month follow-up, 39.9% of intervention group patients and 49.9% of comparison group patients were reported as having probable depression (p = .001).
- At 12-month follow-up, 41.6% of intervention group patients and 51.2% of comparison group patients were reported as having probable depression (p = .005).
- The difference between groups was still significant at 24-month follow-up (p = .04).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.6
(0.0-4.0 scale)
|
| Outcome 2: Mental health-related quality of life |
|
Description of Measures
|
Mental health-related quality of life was assessed using the Mental Component Summary (MCS) score from the self-administered 12-item Short-Form Health Survey (SF-12). The MCS provides an overall measurement of mental health based on factors such as vitality, social functioning, and emotional well-being. MCS scores were normalized to a general population mean of 50 (SD = 10), with higher scores indicating better health. This measure was administered at baseline and at 6-, 12-, and 24-month follow-up.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that received the PIC materials or to a comparison group that received the AHRQ depression practice guidelines by mail. All eligible patients at these clinics were screened for depression, and those with current depressive symptoms and 12-month, lifetime, or no depressive disorder were enrolled in the study. The study found:
- There was no significant difference at baseline between the intervention and comparison groups in mental health-related quality of life (mean MCS scores: 35.6% vs. 36.1%).
- At 6-month follow-up, significantly more intervention group patients than comparison group patients showed improved mental health-related quality of life (mean MCS scores: 41.6% vs. 39.8%; p = .009).
- The difference between groups was still significant at 12-month follow-up (mean MCS scores: 40.9% vs. 39.3%; p = .04) and at 24-month follow-up (p < .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.6
(0.0-4.0 scale)
|
| Outcome 3: Health service utilization |
|
Description of Measures
|
Health service utilization was assessed at baseline and 6- and 12-month follow-up using a self-administered patient survey that assessed whether the patient had any of the following in the past 6 months: medical visits, medical visits for emotional problems (i.e., visits at which emotional problems were discussed), and mental health specialty visits. An indicator was developed for each type of visit and the number of visits. Increased health service utilization was assumed to reflect improved quality of care for depression.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that received the PIC materials or to a comparison group that received the AHRQ depression practice guidelines by mail. All eligible patients at these clinics were screened for depression, and those with current depressive symptoms and 12-month, lifetime, or no depressive disorder were enrolled in the study. The study found:
- There were no significant group differences at baseline for any of the three types of visits.
- Significantly more intervention group patients than comparison group patients reported mental health specialty visits in the past 6 months at both 6- and 12-month follow-ups. The percentages of intervention and comparison group patients who reported such visits were 29.9% and 27.2% at baseline, 39.8% and 27.2% at 6-month follow-up (p = .001), and 29.1% and 22.7% at 12-month follow-up (p = .03).
- Significantly more intervention group patients than comparison group patients also reported having medical visits for emotional problems at 6-month follow-up, but the difference was no longer significant at 12-month follow-up. The percentages of intervention and comparison group patients reporting such visits were 48.4% and 46.5% at baseline, 47.5% and 36.6% at 6-month follow-up (p = .001), and 37.4% and 35.1% at 12-month follow-up.
- Rates of any medical visits did not differ significantly between groups in any period.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.6
(0.0-4.0 scale)
|
| Outcome 4: Quality of care for depression |
|
Description of Measures
|
Quality of care for depression was measured using a self-administered patient survey administered at baseline and 6- and 12-month follow-up. Patients were asked to report any prescription medications used in the past 30 days (for the baseline administration) and in the past 6 months (for the follow-up administration). Criteria from national guidelines (Depression Guideline Panel, 1993) were applied to each medication to assess whether the dosage was appropriate. In addition, patients reported whether they had received individual, group, or family counseling by a specialist in the past 6 months. Other survey questions were used to determine whether the patient had depressive symptoms or was free of probable disorder (see Outcome 1).
Overall care was considered appropriate if patients were free of probable disorder or had either specialty counseling or use of appropriate antidepressant medication during a given 6-month period.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that received the PIC materials or to a comparison group that received the AHRQ depression practice guidelines by mail. All eligible patients at these clinics were screened for depression, and those with current depressive symptoms and 12-month, lifetime, or no depressive disorder were enrolled in the study. The study found:
- There were no significant group differences at baseline in the rates of appropriate antidepressant use, specialty counseling, or overall appropriate care.
- The intervention group had significantly higher rates of appropriate antidepressant use relative to the comparison group at both 6- and 12-month follow-ups. The percentages of intervention and comparison group patients reporting appropriate antidepressant use were 27.6% and 27.0% at baseline, 34.7% and 25.1% at 6-month follow-up (p = .001), and 31.0% and 24.0% at 12-month follow-up (p = .01).
- Rates of specialty counseling use were also higher in the intervention group at both follow-ups. The percentages of intervention and comparison group patients who received specialty counseling were 29.5% and 26.9% at baseline, 38.2% and 25.6% at 6-month follow-up (p < .001), and 27.3% and 20.9% at 12-month follow-up (p = .03).
- Significantly more intervention group patients than comparison group patients received overall appropriate care for depression. The percentages of intervention and comparison group patients who received appropriate care were 44.2% and 42.5% at baseline, 50.9% and 39.7% at 6-month follow-up (p < .001), and 59.2% and 50.1% at 12-month follow-up (p = .006).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.4
(0.0-4.0 scale)
|
| Outcome 5: Employment |
|
Description of Measures
|
A self-administered patient survey was used to assess whether respondents were employed at baseline. Follow-up telephone surveys assessed current employment at 6- and 12-month follow-up.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that received the PIC materials or to a comparison group that received the AHRQ depression practice guidelines by mail. All eligible patients at these clinics were screened for depression, and those with current depressive symptoms and 12-month, lifetime, or no depressive disorder were enrolled in the study. Results showed that among patients who were initially employed, 89.7% of those in the intervention group and 84.7% of those in the comparison group were employed at 12-month follow-up (p = .05).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.6
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
26-55 (Adult) 55+ (Older adult)
|
72.3% Female 27.7% Male
|
57.4% White 29.2% Hispanic or Latino 6.9% Black or African American 6.5% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Depressive symptoms
|
3.3
|
3.3
|
3.5
|
4.0
|
3.5
|
4.0
|
3.6
|
|
2: Mental health-related quality of life
|
3.3
|
3.3
|
3.5
|
4.0
|
3.5
|
4.0
|
3.6
|
|
3: Health service utilization
|
3.3
|
3.5
|
3.5
|
4.0
|
3.5
|
4.0
|
3.6
|
|
4: Quality of care for depression
|
2.8
|
2.8
|
3.5
|
4.0
|
3.5
|
4.0
|
3.4
|
|
5: Employment
|
3.3
|
3.3
|
3.5
|
4.0
|
3.5
|
4.0
|
3.6
|
Study Strengths The majority of the measures used in the study have good psychometric information and acceptable levels of reliability and validity. Fidelity was assured by steps taken prior to and during the study, including extensive training, ongoing monitoring and feedback at the clinic and clinician level, and the distribution of supporting materials. Adherence rates were reported for each part of the intervention. Attrition was relatively low, and the methods used to address attrition were clearly described for every project phase, from patient screening through survey completion at each assessment point. Baseline factors that were associated with remaining enrolled were addressed in the analyses. The study was a group-level randomized controlled trial in which clinic clusters were grouped into matched blocks based on patient demographics, clinician specialty, and distance to mental health providers. The data analyses were very appropriate for detecting effects reflecting the primary hypothesis.
Study Weaknesses Psychometric information was not presented for one of the scales used to measure depression symptoms, the abbreviated adaptation of the CES-D. Similarly, psychometric information was lacking for the measures and methods used to gauge quality of care for depression (i.e., receipt of specialty counseling or use of appropriate antidepressant medication).
|
|
Readiness for Dissemination
Review Date: November 2009
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Munoz, R. F., & Miranda, J. (2000). Group therapy manual for cognitive-behavioral treatment of depression. Santa Monica, CA: RAND.
Munoz, R. F., & Miranda, J. (2000). Individual therapy manual for cognitive-behavioral treatment of depression. Santa Monica, CA: RAND.
Partners in Care program materials, http://www.rand.org/pubs/monograph_reports/MR1198/index.html
Partners in Care Web site, http://www.rand.org/health/surveys_tools/pic.html
RAND. (2000). Partners in Care quick reference cards. Santa Monica, CA: Author.
Rubenstein, L. V., Unützer, J., Miranda, J., Katon, W. J., Wieland, M., Jackson-Triche, M., et al. (2000). Clinician guide to depression assessment and management in primary care. Santa Monica, CA: RAND.
Rubenstein, L. V., Unützer, J., Miranda, J., Simon, B., Katon, W. J., Jackson-Triche, M., et al. (2000). Guidelines and resources for the depression nurse specialist. Santa Monica, CA: RAND.
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
3.3
|
2.8
|
3.0
|
3.0
|
Dissemination Strengths The program materials provide valuable guidance for evaluating patient needs, developing care plans, and monitoring patient progress. Both patient education materials and materials for use by practices are included. Thoughtful, written guidance is provided on how to train the clinicians and others involved in implementation. Each training session is coherently described in general terms before details are presented, facilitating understanding of each phase of the program. Quality assurance is supported through the use of assessment tools, patient self-reports, and team meetings. Forms, worksheets, and procedures for collecting data on patients and assessing their progress are provided. Many of the measures in these materials have been validated by research and offer a reliable way to monitor program quality.
Dissemination Weaknesses The program materials, most of which were developed in the 1990s, mention that implementers may want to revise the manuals to take into account recent advances (e.g., in psychotropic medications and informatics support for documenting case management), but no specific guidance on how to update the materials and procedures is offered. Certain inconsistencies in the materials may be confusing, and the language used, while appropriate for a research study, may not be practical in a primary care setting. The developer no longer provides training or support to implementers, although training on the basic PIC model is available through related projects. CBT training materials are presented as training for therapists but are focused only on the content of CBT sessions, not on the skills required to administer them, and the level of competence in CBT expected of therapists is unclear. There is little discussion of the role of team meetings and supervision and how this aspect of the program should be conducted. Since the recommended quality assurance procedures for the program were developed for the original research study, they may require more data collection than is feasible for some implementers.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
All program materials (electronic)
|
Free
|
Yes, one option is required
|
|
All program materials (hard copy)
|
$275 per set
|
Yes, one option is required
|
|
Replications
No replications were identified by the developer.
|
|
|