Quality of Research
Review Date: January 2012
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Hogarty, G. E., Flesher, S., Ulrich, R., Carter, M., Greenwald, D., Pogue-Geile, M., et al. (2004). Cognitive Enhancement Therapy for schizophrenia: Effects of a 2-year randomized trial on cognition and behavior. Archives of General Psychiatry, 61(9), 866-876. 
Hogarty, G. E., Greenwald, D. P., & Eack, S. M. (2006). Durability and mechanism of effects of Cognitive Enhancement Therapy. Psychiatric Services, 57(12), 1751-1757.
Hogarty, G. E., Greenwald, D. P., & Eack, S. M. (2006). Durability and mechanism of effects of Cognitive Enhancement Therapy [Supplemental material: Methods]. Psychiatric Services, 57(12), 1751-1757.
Hogarty, G. E., Greenwald, D. P., & Eack, S. M. (2006). Durability and mechanism of effects of Cognitive Enhancement Therapy [Supplemental material: Tables]. Psychiatric Services, 57(12), 1751-1757. Study 2Eack, S. M., Greenwald, D. P., Hogarty, S. S., Cooley, S. J., DiBarry, A. L., Montrose, D. M., et al. (2009). Cognitive Enhancement Therapy for early-course schizophrenia: Effects of a two-year randomized controlled trial. Psychiatric Services, 60(11), 1468-1476.
Eack, S. M., Greenwald, D. P., Hogarty, S. S., Cooley, S. J., DiBarry, A. L., Montrose, D. M., et al. (2009). Cognitive Enhancement Therapy for early-course schizophrenia: Effects of a two-year randomized controlled trial [Supplemental material: Table]. Psychiatric Services, 60(11), 1468-1476.
Supplementary Materials Eack, S. M. (2011). Cognitive Enhancement Therapy measurement supplement. Unpublished manuscript.
Eack, S. M., Hogarty, G. E., Greenwald, D. P., Hogarty, S. S., & Keshavan, M. S. (2007). Cognitive Enhancement Therapy improves emotional intelligence in early course schizophrenia: Preliminary effects. Schizophrenia Research, 89(1-3), 308-311.
Hogarty, G. E., & Flesher, S. (1999). Practice principles of Cognitive Enhancement Therapy for schizophrenia. Schizophrenia Bulletin, 25(4), 693-708.
Outcomes
| Outcome 1: Neurocognition |
|
Description of Measures
|
Neurocognition was assessed in both studies using a selected set of neuropsychological test instruments measuring memory, language, and executive mental functioning:
- Immediate and delayed recall items from stories A and B of the Wechsler Memory Scale-Revised, which measures memory functions
- List A total recall, short-term free recall, and long-term free recall from the California Verbal Learning Test, which measures different classes of verbal memory
- Digit span, digit symbol, picture arrangement, and vocabulary items from the Wechsler Adult Intelligence Scale-Revised, a general test of intelligence (defined as the global capacity of an individual to act purposefully, think rationally, and deal effectively with his or her environment) that consists of six verbal and five performance subtests
- Trail B time to completion and perseverative and nonperseverative errors, categories completed, and percentage of conceptual responses from the Wisconsin Card Sorting Test, which assesses executive frontal lobe brain functioning (e.g., strategic planning, organized searching, utilizing environmental feedback to shift cognitive sets, directing one's behavior toward achieving a goal, modulating impulsive responding)
- Trail-Making Test B, an assessment of task switching that measures visual search speed, scanning, speed of processing, mental flexibility, and executive functioning of respondents, who connect a set of 25 dots labeled with numbers, letters, or alternating numbers and letters
One study also used two additional instruments:
- The Tower of London Test, which assesses executive functioning to identify deficits in planning. The respondent is presented with various problem-solving tasks that involve the use of two boards with pegs and several beads of different colors.
- The Neurological Evaluation Scale, which consists of 26 items that assess three functioning domains--sensory integration, motor coordination, and sequencing of complex motor acts--and an "others" domain that includes short-term memory, frontal release signs (the reappearance of primitive reflexes, e.g., sucking and rooting, that are present at birth but absent in a normal mature brain), and eye movement abnormalities. The scale is also useful for assessing stereognosis (inability of the respondent to recognize commonplace items placed in his or her hand, such as keys, when his or her eyes are closed) or graphesthesia (inability of the respondent to recognize a letter drawn on the palm of his or her hand when his or her eyes are closed).
In both studies, scores from the individual neuropsychological test instruments were scaled to a common metric and averaged to form composite scores. The composite scores were then standardized to a baseline mean, and change scores were calculated from the baseline mean to the assessments at 12 and 24 months into the study.
|
|
Key Findings
|
In one study, a randomized clinical trial (RCT) was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-III-R or DSM-IV diagnosis of chronic schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of Enriched Supportive Therapy (EST). EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the neurocognition change score at 12 months into the study (p = .003) and at 24 months into the study (p = .02), after controlling for gender, length of illness, level of psychosis, and IQ level. These group differences were associated with a medium effect size at 12 months into the study (Cohen's d = 0.64) and a small effect size at 24 months into the study (Cohen's d = 0.46).
In another study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-IV diagnosis of early-course (defined as present for 8 years or less) schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of EST. EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the neurocognition change score at 24 months into the study (p = .023), after controlling for age, gender, length of illness, IQ level, and dose of medication. This group difference was associated with a small effect size (Cohen's d = 0.46). No significant between-group difference was found at 12 months into the study.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 2: Cognitive style |
|
Description of Measures
|
Cognitive style was assessed using two clinician-rated instruments: the Cognitive Styles and Social Cognition Eligibility Criteria and the associated Cognitive Styles Inventory.
The Cognitive Styles and Social Cognition Eligibility Criteria is a structured interview designed to elicit behaviors and verbal responses that provide information on the presence and severity of impairments in cognitive style and social cognition. The instrument includes direct questioning about interpersonal interactions and social problem solving, as well as vignettes that require the respondent to take the perspective of another person and appraise social context. For the cognitive styles component of the instrument, the clinician rates the severity of cognitive style impairment across three generalized domains, each of which includes disability and social handicap subgroups:
- Impoverished cognitive style (e.g., poverty of speech production, amotivation, reduced or flat affect)
- Disability examples: effortful planning or problem solving, difficulty initiating behavior, and effortful recall of information
- Social handicap examples: language not expressing needs, preferences, or opinions; lack of credible account of behavior; low stamina; and social withdrawal
- Disorganized cognitive style (e.g., spontaneous, labile affect; loose ideational content; poor attention; poverty of speech content)
- Disability examples: chaotic or imprecise planning, difficulty terminating behavior, and difficulty staying on track
- Social handicap examples: inappropriate responses not self-edited, difficulty using language coherently, hard-to-follow train of thought, impulsive, and readily changing plans or goals
- Rigid cognitive style (e.g., fixed, inflexible, ideational content; restricted cognitive schema; constrained affect; obsessive, repetitive thinking)
- Disability examples: plans, goals, and problem solving that are limited by inflexible thinking
- Social handicap examples: behavior that is restricted by preoccupation with details, tendency toward stereotyped views, and single-minded pursuit of inappropriate goals or career objectives
The Cognitive Styles Inventory is a 46-item clinician-rated micromeasure of the three generalized cognitive style domains used in the Cognitive Styles and Social Cognition Eligibility Criteria (i.e., impoverished, disorganized, and rigid).
In both studies, scores from the instruments were scaled to a common metric and averaged to form composite scores. The composite scores were then standardized to a baseline mean, and change scores were calculated from the baseline mean to the assessments at 12 and 24 months into the study.
|
|
Key Findings
|
In one study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-III-R or DSM-IV diagnosis of chronic schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of Enriched Supportive Therapy (EST). EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the cognitive style change score at 24 months into the study (p = .001), after controlling for gender, length of illness, level of psychosis, and IQ level. This group difference was associated with a medium effect size (Cohen's d = 0.69). No significant between-group difference was found at 12 months into the study.
In another study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-IV diagnosis of early-course (defined as present for 8 years or less) schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of EST. EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the cognitive style change score at 12 months into the study (p = .023) and at 24 months into the study (p = .003), after controlling for age, gender, length of illness, IQ level, and dose of medication. These group differences were associated with a medium effect size at 12 months into the study (Cohen's d = 0.68) and a large effect size at 24 months into the study (Cohen's d = 1.02).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
| Outcome 3: Social cognition |
|
Description of Measures
|
Social cognition was assessed in both studies using two clinician-rated instruments: the Cognitive Styles and Social Cognition Eligibility Criteria and the Social Cognition Profile.
The Cognitive Styles and Social Cognition Eligibility Criteria is a structured interview designed to elicit behaviors and verbal responses that provide information on the presence and severity of impairments in cognitive style and social cognition. The instrument includes direct questioning about interpersonal interactions and social problem solving, as well as vignettes that require the respondent to take the perspective of another person and appraise social context. For the social cognition component of the instrument, clinicians rate the severity of social cognition impairment across five generalized domains:
- Vocational ineffectiveness (e.g., unemployed or working below potential; reduced work stamina; unable to establish or maintain a routine; unable to use feedback from coworkers, supervisor, or authority figures)
- Interpersonal ineffectiveness (e.g., lack of empathy, flexibility, or reciprocity; unable to negotiate conflicts, express needs, act wisely, or take perspectives)
- Lack of foresight (e.g., unable to assess long-term consequences of behavior, good or bad; difficulty forming long-range plans)
- Gist extraction deficits (e.g., difficulty understanding formal or informal rules of conduct as social contexts change; unable to assess central point or norm of social situation)
- Adjustment to disability (e.g., unable to temporarily revise expectations; failure to understand or accept residual limitations imposed by illness)
The Social Cognition Profile is a 50-item clinician-rated instrument that assesses social cognition across four domain factors: tolerant (e.g., accepting, respectful, cooperative), supportive (e.g., empathic, reciprocal, friendly), perceptive (e.g., foresightful, self-aware, gistful), and self-confident (e.g., comfortable, assertive, involved). Clinicians rate each item on a 5-point scale, with higher scores reflecting better social cognition.
One study also used the Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT), a 140-item performance-based test of the respondent's emotional intelligence. The MSCEIT contains eight subtests that require respondents to identify emotions expressed by a face or in designs; generate a mood and solve problems with that mood; define the causes of different emotions and demonstrate an understanding of the progression of emotions; and determine how to best include emotion in one's thoughts during situations involving oneself or other people.
In both studies, scores from the instruments were scaled to a common metric and averaged to form composite scores. The composite scores were then standardized to a baseline mean, and change scores were calculated from the baseline mean to the assessments at 12 and 24 months into the study.
|
|
Key Findings
|
In one study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-III-R or DSM-IV diagnosis of chronic schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of Enriched Supportive Therapy (EST). EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the social cognition change score at 24 months into the study (p = .001), after controlling for gender, length of illness, level of psychosis, and IQ level. This group difference was associated with a medium effect size (Cohen's d = 0.72). No significant between-group difference was found at 12 months into the study.
In another study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-IV diagnosis of early-course (defined as present for 8 years or less) schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of EST. EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the social cognition change score at 12 months into the study (p < .001) and at 24 months into the study (p < .001), after controlling for age, gender, length of illness, IQ level, and dose of medication. These group differences were associated with large effect sizes (Cohen's d = 1.08 and 1.55 at 12 and 24 months into the study, respectively).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.7
(0.0-4.0 scale)
|
| Outcome 4: Social adjustment |
|
Description of Measures
|
Social adjustment was assessed in both studies with three clinician-rated instruments:
- The Major Role Adjustment Inventory, a 32-item measure of major role adjustment that assesses vocational, social, and household role functioning.
- The Global Assessment Scale, a global measure of patient functioning that is rated on a scale from 0 to 100, with higher ratings indicating better social adjustment.
- The Performance Potential Inventory, a 55-item measure of functional disability based on employment disability criteria used by the Social Security Administration. Items are rated on a 5-point scale, with higher scores indicating better social adjustment.
One study also used the clinician-rated Social Adjustment Scale-II, a 44-item structured interview-based measure that covers seven domains: work affinity (e.g., work performance, time lost at work because of illness), primary/family relations in the household (e.g., friction with parents or spouse), social functioning outside the home (e.g., friction with extended relatives or parents not living with the patient), interpersonal anguish (e.g., distress at work, loneliness, satisfaction with social adjustment), sexual relations (e.g., frequency, interest in sexual relations), social leisure (e.g., frequency of social contacts, participation in social leisure activities, interpersonal effectiveness), and self-care (e.g., personal hygiene). Each item is rated on a 5- or 6-point scale that ranges from 0 (good adjustment) to 4 or 5 (poor adjustment).
In both studies, scores from the instruments were scaled to a common metric and averaged to form composite scores. The composite scores were then standardized to a baseline mean, and change scores were calculated from the baseline mean to the assessments at 12 and 24 months into the study.
|
|
Key Findings
|
In one study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-III-R or DSM-IV diagnosis of chronic schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of Enriched Supportive Therapy (EST). EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the social adjustment change score at 12 months into the study (p = .046) and at 24 months into the study (p = .01), after controlling for gender, length of illness, level of psychosis, and IQ level. These group differences were associated with a small effect size at 12 months into the study (Cohen's d = 0.40) and a medium effect size at 24 months into the study (Cohen's d = 0.56).
In another study, an RCT was conducted at a university-based psychiatric research clinic with outpatients who had a DSM-IV diagnosis of early-course (defined as present for 8 years or less) schizophrenia or schizoaffective disorder at study entry. Participants were symptomatically stable but clinician rated as being cognitively and socially disabled (with the Cognitive Style and Social Cognition Eligibility Interview, specifically designed for this study) and were randomly assigned to the intervention group, which received 2 years of CET, or the comparison group, which received 2 years of EST. EST consisted of illness management through applied coping strategies and education. Assessments occurred at baseline and at 12 and 24 months into the study. Compared with participants who received EST, those who received CET had a greater improvement in the social adjustment change score at 12 months into the study (p = .001) and at 24 months into the study (p < .001), after controlling for age, gender, length of illness, IQ level, and dose of medication. These group differences were associated with large effect sizes (Cohen's d = 0.82 and 1.53 at 12 and 24 months into the study, respectively).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.8
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
26-55 (Adult)
|
59% Male
41% Female
|
89% White
11% Black or African American
|
|
Study 2
|
18-25 (Young adult)
26-55 (Adult)
|
69% Male
31% Female
|
69% White
19% Black or African American
10.3% Asian
1.7% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Neurocognition
|
3.3
|
3.1
|
2.9
|
2.6
|
2.9
|
3.2
|
3.0
|
|
2: Cognitive style
|
2.3
|
2.5
|
2.9
|
2.6
|
2.9
|
3.2
|
2.7
|
|
3: Social cognition
|
2.3
|
2.3
|
2.9
|
2.6
|
2.9
|
3.2
|
2.7
|
|
4: Social adjustment
|
2.5
|
2.8
|
2.9
|
2.6
|
2.9
|
3.2
|
2.8
|
Study Strengths Sample internal consistency and test-retest reliability were established for the clinician-rated instruments assessing the cognitive style, social cognition, and social adjustment outcomes. In both studies, the neurocognition outcome measures were from standard neuropsychological test batteries, the investigators included sample internal consistency and test-retest reliability for the neurocognition composite scores, and an independent psychometrician administered the neuropsychological test batteries, which further increased reliability. In both studies, efforts to ensure intervention fidelity included tracking of session attendance, indication of the intervention dosage received by study participants, adherence to a session agenda, and supervision by the program developers. Investigators used a fidelity scale to rate clinician drift in random audiotaped intervention sessions in one study and in random audio- or videotaped sessions in another study. In both studies, the intervention was manual driven, and random assignment was used to control many potential confounding variables. One study used an intent-to-treat approach that included data from all randomized participants in the analyses. Appropriate statistical modeling was used in both studies, with verification of multivariate main effects by effect sizes in addition to statistical significance and post hoc between-group testing of the individual measures composing each outcome's composite score.
Study Weaknesses Interrater and clinician reliability for rating the instruments assessing the cognitive style, social cognition, and social adjustment outcomes was not established during either study. In both studies, the construct validity of the four outcomes' composite scores was not tested and confirmed by independent investigators. The validity of using neuropsychological test instruments as longitudinal outcome measures, a function for which these tests were not designed, is questionable. There is no evidence that a tested fidelity scale with demonstrated psychometric properties was used in either study, and the ratings of intervention sessions for clinician drift were discontinued during the final year of treatment in one study because of funding restrictions. Clinician assessors who rated participants with the outcome measures were not blind to study condition in either study, which could have introduced a bias in the composite outcome findings. An intent-to-treat approach to the data analysis was not used in one study, and missing data were not handled by sophisticated statistical modeling in either study. In one study, there was no discussion of the differences between participants who dropped out of the study and those who remained in the study across the full 2-year period.
|
|
