Quality of Research
Review Date: January 2010
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Fortney, J. C., Pyne, J. M., Edlund, M. J., Williams, D. K., Robinson, D. E., Mittal, D., & Henderson, K. L. (2007). A randomized trial of Telemedicine-Based Collaborative Care for Depression. Journal of General Internal Medicine, 22(8), 1086-1093. 
Pyne, J. M., Fortney, J. C., Tripathi, S., Maciejewski, M. L., Edlund, M. J., & Williams, D. K. (2009). Cost effectiveness analysis of a rural telemedicine collaborative care intervention for depression. Manuscript submitted for publication.
Supplementary Materials Fortney, J., & Pyne, J. (2009, June). Telemedicine-Based Collaborative Care models for rural primary care practices [PowerPoint slides]. Presented at the Future of Telehealth Workshop, National Institutes of Health, Bethesda, MD.
Fortney, J. C., Pyne, J. M., Edlund, M. J., Robinson, D. E., Mittal, D., & Henderson, K. L. (2006). Design and implementation of the Telemedicine-Enhanced Antidepressant Management Study. General Hospital Psychiatry, 28(1), 18-26. 
Fortney, J., Pyne, J., Mittal, D., & Hudson, T. (2009, July). Comparative effectiveness study of rural collaborative care models: The Outreach Study [PowerPoint slides]. Presented at the National Institute of Mental Health (NIMH) Mental Health Research Conference on Increasing the Efficacy of Research and Mental Health Services Delivery, Washington, DC.
Williams, J. W., Jr., Stellato, C. P., Cornell, J., & Barrett, J. E. (2004). The 13- and 20-item Hopkins Symptom Checklist Depression Scale: Psychometric properties in primary care patients with minor depression or dysthymia. International Journal of Psychiatry in Medicine, 34(1), 37-50. 
Outcomes
| Outcome 1: Medication adherence |
|
Description of Measures
|
Medication adherence was measured by self-report and pharmacy/electronic medical records. Prior to the 6- and 12-month follow-up surveys, a study psychiatrist examined each patient's electronic medical record to determine what antidepressants had been prescribed or refilled during the past 6 months. In the surveys, blinded telephone interviewers asked patients who had an active prescription about their most recently prescribed/refilled antidepressant medication, referring to the antidepressant explicitly by name. Patients were asked if they were currently taking the medication, and if so, how frequently they took the medication in the past month and if they took the dosage prescribed. Patients were categorized as adherent if their responses indicated they took the full dosage on 80% or more of days in the past month. Patients without an active prescription or who reported stopping antidepressants following their primary care provider's instruction were excluded from the adherence analysis.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that implemented Telemedicine-Based Collaborative Care or to a comparison group that provided usual care. Usual care involved the use of a two-stage clinical reminder system to facilitate depression screening and follow-up assessment, an in-clinic VA pharmacy, and interactive video equipment for telepsychiatry consultations at those sites lacking on-site psychiatrists.
Patients in the intervention group had significantly greater odds of being adherent to their medication protocol than those receiving usual care at both 6-month (p = .04) and 12-month follow-up (p = .01). The effect size for these findings was small (odds ratios = 2.1 and 2.7, respectively).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.2
(0.0-4.0 scale)
|
| Outcome 2: Treatment response |
|
Description of Measures
|
Treatment response was defined as a 50% or greater improvement in depression severity between baseline and follow-up. Depression severity was measured using the Hopkins Symptom Checklist 20-item scale (SCL-20), a widely used instrument for measuring symptoms of depression. The SCL-20 was administered via blinded telephone interview at baseline and at 6- and 12-month follow-up.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that implemented Telemedicine-Based Collaborative Care or to a comparison group that provided usual care. Usual care involved the use of a two-stage clinical reminder system to facilitate depression screening and follow-up assessment, an in-clinic VA pharmacy, and interactive video equipment for telepsychiatry consultations at those sites lacking on-site psychiatrists.
At 6-month follow-up, patients in the intervention group were significantly more likely to respond than those receiving usual care (p = .02), with a large effect size (odds ratio = 1.94). There were no significant findings for this outcome at 12 months.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.5
(0.0-4.0 scale)
|
| Outcome 3: Remission |
|
Description of Measures
|
Remission was defined as a depression severity score of less than 0.5 on the SCL-20. The SCL-20 was administered via blinded telephone interview at baseline and at 6- and 12-month follow-ups.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that implemented Telemedicine-Based Collaborative Care or to a comparison group that provided usual care. Usual care involved the use of a two-stage clinical reminder system to facilitate depression screening and follow-up assessment, an in-clinic VA pharmacy, and interactive video equipment for telepsychiatry consultations at those sites lacking on-site psychiatrists.
At 12-month follow-up, patients in the intervention group had significantly greater odds of achieving remission from depression compared with those receiving usual care (p = .02). This finding was associated with a small effect size (odds ratio = 2.4). There were no significant findings for this outcome at 6 months.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.6
(0.0-4.0 scale)
|
| Outcome 4: Health-related quality of life |
|
Description of Measures
|
Improvement in health-related quality of life was measured by the change in patient's score on the Quality of Well-Being scale (QWB) between baseline and follow-up. The QWB is an interviewer-administered tool that assesses four domains: mobility, physical activity, social activity, and symptomatic complaints that might inhibit function. Data were collected via blinded telephone interview at baseline and at 6- and 12-month follow-ups.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that implemented Telemedicine-Based Collaborative Care or to a comparison group that provided usual care. Usual care involved the use of a two-stage clinical reminder system to facilitate depression screening and follow-up assessment, an in-clinic VA pharmacy, and interactive video equipment for telepsychiatry consultations at those sites lacking on-site psychiatrists.
Patients' QWB scores improved significantly more in the intervention group than in the usual care group by 6-month follow-up (p < .01). There were no significant findings for this outcome at 12 months.
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 5: Cost effectiveness |
|
Description of Measures
|
Cost effectiveness was measured by calculating the ratio of total expenditures to quality-adjusted life years (QALYs). For each patient, two predicted expenditures were calculated: one prediction as if the patient had been randomized to the intervention, and the second as if the patient had been randomized to usual care. The difference between these two expenditures predictions represented the incremental effect of the intervention on expenditures for the particular patient. The difference between the two predicted values for each participant was then averaged across all patients to determine the incremental effect for the entire sample.
Expenditures and QALYs were measured for the entire 12-month study period. Expenditure data were derived from administrative records. QALYs were calculated using formulas developed by Brazier that convert data from the Short-Form Health Survey 12 (SF-12) using preference-weighted methods. The formulas are based on standard gamble methods and produce an overall preference-weighted index score that ranges from 0 (death) to 1.0 (perfect health). QALY data were collected via blinded telephone interview at baseline and at 6- and 12-month follow-ups.
|
|
Key Findings
|
Matched clinics participating in the study were randomized to an intervention group that implemented Telemedicine-Based Collaborative Care or to a comparison group that provided usual care. Usual care involved the use of a two-stage clinical reminder system to facilitate depression screening and follow-up assessment, an in-clinic VA pharmacy, and interactive video equipment for telepsychiatry consultations at those sites lacking on-site psychiatrists.
In a sensitivity analysis that included all inpatient expenditures and base case analysis expenditures, the incremental effect of the intervention on expenditures was significant (p = .002).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
26-55 (Adult) 55+ (Older adult)
|
91.7% Male 8.3% Female
|
74.7% White 18.2% Black or African American 4.1% Race/ethnicity unspecified 3% American Indian or Alaska Native
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Medication adherence
|
3.0
|
3.0
|
3.3
|
3.5
|
2.8
|
3.5
|
3.2
|
|
2: Treatment response
|
4.0
|
4.0
|
3.3
|
3.5
|
2.8
|
3.5
|
3.5
|
|
3: Remission
|
4.0
|
4.0
|
3.3
|
3.5
|
3.3
|
3.5
|
3.6
|
|
4: Health-related quality of life
|
3.5
|
3.0
|
3.3
|
3.5
|
3.3
|
3.5
|
3.3
|
|
5: Cost effectiveness
|
3.0
|
3.0
|
3.3
|
3.0
|
3.3
|
4.0
|
3.3
|
Study Strengths The study's methodological strengths include clearly specified eligibility criteria, a good acceptance rate among eligible participants, the use of outcomes appropriate to the intervention and measures appropriate to quantifying those outcomes, and the use of blinded telephone interviews to collect patient data. Most of the measures used in the study have strong psychometric properties and have been used with a variety of study populations, and the analysis was appropriate to the intervention. Attrition and missing data were addressed using statistical procedures. Evidence for intervention fidelity is strong based on the use of intervention scripts, branching of scripts, and techniques to measure fidelity.
Study Weaknesses It is unclear whether the measures have been tested with people of color. Some discussion of this segment of the population should have been included considering that more than one fifth of the participants were people of color and given the well-documented differential experiences of this population within mental health systems. The fact that many of the participants were already receiving treatment for depression may have affected study findings. Randomization was used with matched clinics but not at the patient or provider level.
|
|
Readiness for Dissemination
Review Date: January 2010
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Fortney, J. C., Pyne, J. M., Smith, J. L., Curran, G. M., Otero, J. M., Enderle, M. A., & McDougall, S. (2009). Steps for implementing collaborative care programs for depression. Population Health Management, 12(2), 69-79. 
NetDSS Web site, Version 2.5, https://www.netdss.net
Telemedicine-Based Collaborative Care implementation tools:
- Implementation Tool (Steps, Decisions, and Actions Items)
- NetDSS brochure and summary
- NetDSS Users Manual (Version 2.5). (2009). Available online at https://www.netdss.net/UManual.pdf
- Telephone Care Manager job posting
Telemedicine-Based Collaborative Care quality assurance tools:
- Fidelity Checklist and Score Sheet
- NetDSS Telephone Care Manager Date Range Report Summary
- NetDSS Telephone Care Manager Date Range Report Template
Telemedicine-Based Collaborative Care training and support tools:
- Depression Care Management Part I: Diagnosis and Treatment [PowerPoint slides]
- Depression Care Management Part II: Trial Management & Assessments [PowerPoint slides]
- Fortney, J., Davis, A., Pyne, J., Raney, W., & Ackerman, S. (2007). Depression Care Manager training manual (Version 1.1). Available online at https://www.netdss.net/CMManual.pdf
- Major Depressive Disorder: Recognition and Management in Primary Care [PowerPoint slides]
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
2.8
|
2.8
|
3.3
|
2.9
|
Dissemination Strengths The Web-based NetDSS decision support system supports implementation by providing a centralized repository of information related to patient assessment and treatment. The system includes a wealth of evidence-based instruments and scripts to guide treatment decisions, is intuitive to use, and comes with supplemental materials that include a user's guide and the Depression Care Manager training manual. The materials provided for training clinical and care management staff are high quality, detailed, and comprehensive. The quality assurance materials, resources, and procedures that are available to support implementation are well developed and easy to use. The NetDSS's automation of work flow and processes maximizes the use of evidence-based tools and resources, in turn enhancing fidelity.
Dissemination Weaknesses While the Web-based system and other implementation tools provide an abundance of information, potential implementers may find them difficult to access or sort through at first. No concise description of the program components, implementation steps, and contact information is offered to orient newcomers to the program. The program does not provide any hard-copy materials to give to patients to explain the care they will receive, including the role of follow-up telephone calls. There does not appear to be any systematic orientation, training, consultation, or ongoing technical assistance offered aside from the Web-based tools. No resources or guidelines for evaluating program outcomes are provided.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
All implementation materials (electronic format)
|
Free
|
Yes
|
|
NetDSS Web-based decision support tool (for primary care practice)
|
Free
|
Yes
|
|
1-day, off-site training
|
$3,000 per site
|
No
|
|
1-day, on-site training
|
$5,000 per site plus travel expenses
|
No
|
|
Consultation
|
$125 per hour
|
No
|
|
Fidelity checklist
|
Free
|
No
|
|
Replications
No replications were identified by the developer.
|
|
|