Quality of Research
Review Date: September 2008
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Chronis, A. M., Fabiano, G. A., Gnagy, E. M., Onyango, A. N., Pelham, W. E., Lopez-Williams, A., et al. (2004). An evaluation of the summer treatment program for children with attention-deficit/hyperactivity disorder using a treatment withdrawal design. Behavior Therapy, 35, 561-585. Study 2Pelham, W. E., Burrows-MacLean, L., Gnagy, E. M., Fabiano, G. A., Coles, E. K., Tresco, K. E., et al. (2005). Transdermal methylphenidate, behavioral, and combined treatment for children with ADHD. Experimental and Clinical Psychopharmacology, 13(2), 111-126.  Study 3Fabiano, G. A., Pelham, W. E., Gnagy, E. M., Burrows-MacLean, L., Coles, E. K., Chacko, A., et al. (2007). The single and combined effects of multiple intensities of behavior modification and methylphenidate for children with attention deficit hyperactivity disorder in a classroom setting. School Psychology Review, 36(2), 195-216.
Supplementary Materials August, G. J., Realmuto, G. M., Hektner, J. M., & Bloomquist, M. L. (2001). An integrated components preventive intervention for aggressive elementary school children: The Early Risers program. Journal of Consulting and Clinical Psychology, 69(4), 614-626.
MTA Cooperative Group. (1999). A 14-month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder. Archives of General Psychiatry, 56(12), 1073-1086.
Pelham, W. E., Fabiano, G. A., Gnagy, E. M., Greiner, A. R., & Hoza, B. (2004). The role of summer treatment programs in the context of comprehensive treatment for attention deficit/hyperactivity disorder. In E. D. Hibbs & P. S. Jensen (Eds.), Psychosocial treatments for child and adolescent disorders: Empirically based strategies for clinical practice (2nd ed., pp. 377-409). Washington, DC: American Psychological Association.
Pelham, W. E., Gnagy, E. M., Greiner, A. R., Hoza, B., Hinshaw, S. P., Simpson, S., et al. (2000). Behavioral versus behavioral and pharmacological treatment in ADHD children attending a summer treatment program. Journal of Abnormal Child Psychology, 28(6), 507-525.
Pelham, W. E., & Hoza, B. (1996). Intensive treatment: A summer treatment program for children with ADHD. In E. D. Hibbs & P. S. Jensen (Eds.), Psychosocial treatments for child and adolescent disorders: Empirically based strategies for clinical practices (pp. 311-340). Washington, DC: American Psychological Association.
Outcomes
| Outcome 1: Rule-following and interpersonal behavior in recreational activities |
|
Description of Measures
|
Rule-following and interpersonal behavior in recreational activities were measured using a point system (i.e., token economy) in which children received immediate behavioral feedback and points that were exchanged for daily and weekly reinforcers (e.g., recess, field trips, social honors). The behaviors that were measured are commonly identified as targets of treatment for children with ADHD: (1) following activity rules, (2) noncompliance, (3) interruption, (4) complaining, (5) conduct problems, (6) negative verbalizations, and (7) rule violations. For each category, the daily total of behaviors exhibited by each child was averaged across days within the behavioral treatment condition.
|
|
Key Findings
|
In one study, a treatment withdrawal design was used in which the STP behavioral treatment components were withdrawn for 2 days during the 6th week of the 8-week program. On the days when the behavioral treatment was not in place, participants followed rules less and exhibited greater frequencies of negative behaviors (p < .001). The effect was significant for all measures (all p values < .001). Large effect sizes were found for noncompliance (Cohen's d = 7.38), interruption (Cohen's d = 1.71), complaining (Cohen's d = 2.42), conduct problems (Cohen's d = 3.50), negative verbalizations (Cohen's d = 6.63), and rule violations (Cohen's d = 1.98). The effect size for following rules was very small (Cohen's d = 0.06). Upon reinstatement of the STP behavioral components, behaviors returned to the levels observed before treatment was withdrawn.
In another study, a within-subjects factors design was used to compare the effects of behavior modification and medication, alone and in combination. Methylphenidate was administered in varying doses (placebo, 12.5 cm², 25.0 cm², and 37.5 cm²) with and without behavior modification. Each participant had 2 days in each medication condition without behavioral treatment and 4 days in each medication condition with behavioral treatment. All three dosage levels of medication had positive effects on all measures of rule-following and interpersonal behavior compared with placebo (p < .01). Behavior modification alone had positive effects on rule-following and interpersonal behaviors (p < .01) but did not have a significant effect on conduct problems. The combination of medication with behavior modification showed positive effects for complaining and negative verbalizations (p < .05).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 2: Academic productivity and rule-following in the classroom |
|
Description of Measures
|
Academic productivity and rule-following in the classroom were assessed by teachers using classroom measures. Children were assigned individual classroom work (e.g., reading, language arts, arithmetic) at the appropriate academic level and worked independently on these activities for 30 minutes. Academic productivity (percentage of assigned work that was completed) and accuracy (percentage of completed work that was accurate) were recorded daily. Teachers also observed and recorded children's rates of rule-following behavior in the classroom.
|
|
Key Findings
|
In one study, a treatment withdrawal design was used in which the STP behavioral treatment components were withdrawn for 2 days during the 6th week of the 8-week program. On the days when the behavioral treatment was not in place, academic productivity and rule-following decreased (all p values < .001). Large effect sizes were found for following classroom rules for seatwork (Cohen's d = 1.45), peer tutoring (Cohen's d = 1.20), and computer work (Cohen's d = 0.96). Effect sizes were also large for academic productivity (Cohen's d = 1.25) and accuracy (Cohen's d = 1.62). Upon reinstatement of the STP behavioral treatment components, behaviors returned to the levels observed before treatment was withdrawn.
In another study, a within-subjects factors design was used to compare the effects of behavior modification and medication, alone and in combination. Methylphenidate was administered in varying doses (placebo, 12.5 cm², 25.0 cm², and 37.5 cm²) with and without behavior modification. Each participant had 2 days in each medication condition without behavioral treatment and 4 days in each medication condition with behavioral treatment. All three dosage levels of medication decreased classroom rule violations and increased seatwork completion and accuracy compared with placebo (all p values < .01). Behavior modification increased classroom rule-following and academic productivity (p < .01) but did not significantly affect accuracy of seatwork. The combination of behavior modification with any dosage level of medication produced a decrease in classroom rule violations (p < .01).
A third study used a similar within-subjects design to investigate the effects of behavior modification (none, low intensity, and high intensity), methylphenidate in varying doses (placebo, 0.15 mg/kg, 0.30 mg/kg, and 0.60 mg/kg), and the combination of these treatments. Behavior modification conditions were implemented in 3-week blocks, with the order of the three conditions randomized by group of children. Medication dosage levels were randomly assigned for each child and varied daily during the 9-week course of the STP. Behavior modification alone had significant effects in increasing academic productivity and classroom rule-following (p < .001). All dosage levels of medication produced increases in academic productivity and rule-following compared with placebo (p < .001). The combination of medication and behavior modification also demonstrated positive effects on academic productivity and classroom rule-following (p < .001). Behavior modification had significant effects at all levels of medication, and medication had significant effects at all levels of behavior modification (p < .05). No differences were found between low- and high-intensity behavior modification as the dose of medication increased; however, both low- and high-intensity conditions were significantly better than no behavior modification, regardless of dosage level (p < .05). In general, medication dose-response curves flattened as the intensity of behavior modification increased.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
| Outcome 3: Child behaviors |
|
Description of Measures
|
Child behaviors were measured using:
- A counselor/teacher questionnaire developed for earlier studies of the STP. Counselors and teachers rated how pleasant their interactions with the child were, how much they liked the child, how much the child liked the camp, and how well the child got along with peers on that particular day. Scores ranged from 0 (less pleasant/liked less/got along worse with peers) to 6 (more pleasant/liked more/got along better with peers). The questionnaire was completed daily for each child.
- A child questionnaire developed for earlier studies of the STP. Children rated how much they liked the camp, how well they got along with other children, how well they thought they behaved themselves, and how well they thought other children in their group behaved. Scores were on a 7-point Likert scale ranging from 0 (not at all) to 6 (very much). The questionnaire was completed daily in one study and at three time points in another study.
- The Pittsburgh Modified Conners Rating Scale, which includes counselor-rated and teacher-rated measures on inattention/overactivity and oppositional/aggressive behavior. The sum score for each measure was used.
|
|
Key Findings
|
In one study, a treatment withdrawal design was used in which the STP behavioral treatment components were withdrawn for 2 days during the 6th week of the 8-week program. Counselors and teachers reported that children were more pleasant, liked the camp more, and got along better with peers when the behavioral treatment components were in place (p < .001). Large effect sizes were found for both counselor ratings (Cohen's d = 2.76) and teacher ratings (Cohen's d = 2.57). In addition, children reported getting along better with their peers when behavioral modification was in place (p < .01); the effect size for this finding was medium (Cohen's d = 0.78). Children's ratings of their own behavior during camp showed a near-significant effect of behavior modification (p < .10) with a small effect size (Cohen's d = 0.41), such that children reported being slightly better behaved when behavior modification was in place. Children's ratings of how much they liked camp were not significantly affected by behavioral treatment.
In another study, a within-subjects factors design was used to compare the effects of behavior modification and medication, alone and in combination. Methylphenidate was administered in varying doses (placebo, 12.5 cm², 25.0 cm², and 37.5 cm²) with and without behavior modification. Each participant had 2 days in each medication condition without behavioral treatment and 4 days in each medication condition with behavioral treatment. All three dosage levels of medication improved child behaviors compared with placebo (all p values < .01). Behavior modification also was associated with improvement in child behaviors compared with no behavior modification (p < .01). Medication combined with behavior modification produced an improvement in child behaviors as rated by counselors and teachers (p < .05), but the results for children's self-ratings were not significant. For children's ratings comparing behavior modification with no behavior modification, children rated others (but not themselves) as behaving better with behavior modification than without behavior modification (p < .01).
A third study used a similar within-subjects design to investigate the effects of behavior modification (none, low intensity, and high intensity), methylphenidate in varying doses (placebo, 0.15 mg/kg, 0.30 mg/kg, and 0.60 mg/kg), and the combination of these treatments. Behavior modification conditions were implemented in 3-week blocks, with the order of the three conditions randomized by group of children. Medication dosage levels were randomly assigned for each child and varied daily during the 9-week STP. Behavior modification did not significantly improve child behaviors compared with no behavior modification, but medication produced significant improvements in child behaviors compared with placebo (p < .01). Behavior modification had significant effects at all levels of medication, and medication had significant effects at all levels of behavior modification (p < .05). No differences were found between low- and high-intensity behavior modification as the dose of medication increased; however, both the low- and high-intensity conditions remained significantly better than no behavior modification (p < .05). In general, medication dose-response curves flattened as the intensity of behavior modification increased.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 4: Perceived effectiveness/stress among counselors and teachers |
|
Description of Measures
|
Perceived effectiveness/stress among counselors and teachers was measured using the Effectiveness and Stress Ratings Questionnaire. Each day, counselors and teachers rated their level of frustration and stress in interacting with the children, how effective they found the behavior management techniques they used that day, and their overall effectiveness in the treatment role (e.g., how successful they were in getting each child to complete his or her task). These ratings ranged from 0 (not at all) to 6 (very much). Responses to each domain were averaged together to produce an aggregate rating for counselors and an aggregate rating for teachers.
|
|
Key Findings
|
In one study, a treatment withdrawal design was used in which the STP behavioral treatment components were withdrawn for 2 days during the 6th week of the 8-week program. Counselors rated themselves as more effective and successful when behavioral modification procedures were in place and more frustrated and stressed during the behavioral modification withdrawal period (p < .001). Large effect sizes were found for counselor effectiveness ratings (Cohen's d = 3.48) and counselor frustration and stress ratings (Cohen's d = 3.79). Similarly, teachers rated themselves as more effective and successful (p < .001) and less frustrated and stressed (p < .001) when behavior treatments were in place than when they were withdrawn. Effect sizes were also large for teacher effectiveness ratings (Cohen's d = 2.75) and teacher ratings of frustration and stress (Cohen's d = 2.63). When the STP behavioral treatment components were reinstated, effectiveness and stress ratings returned to the levels reported before treatment was withdrawn.
In another study, a within-subjects factors design was used to compare the effects of behavior modification and medication, alone and in combination. Methylphenidate was administered in varying doses (placebo, 12.5 cm², 25.0 cm², and 37.5 cm²) with and without behavior modification. Each participant had 2 days in each medication condition without behavioral treatment and 4 days in each medication condition with behavioral treatment. Counselors and teachers rated themselves as more effective and less stressed when children received each of the three dosage levels of the medication compared with placebo (all p values < .01). No significant effects for behavior modification alone were demonstrated. However, counselors and teachers rated themselves as more effective and less stressed when children received both medication and behavior modification (p < .01).
A third study used a within-subjects design to investigate the effects of behavior modification (none, low intensity, and high intensity), methylphenidate in varying doses (placebo, 0.15 mg/kg, 0.30 mg/kg, and 0.60 mg/kg), and the combination of these treatments. Behavior modification conditions were implemented in 3-week blocks, with the order of the three conditions randomized by group of children. Medication dosage levels were randomly assigned for each child and varied daily during the 9-week STP. Teachers rated themselves as more effective when children received behavior modification compared with no behavior modification (p < .05), but there were no significant changes in teacher stress level. Teachers also rated themselves as more effective and less stressed when children received medication compared with placebo (p < .002). Behavior modification had significant effects at all levels of medication, and medication had significant effects at all levels of behavior modification (p < .05). No differences were found between low- and high-intensity behavior modification as the medication dosage level increased; however, both the low- and high-intensity conditions remained significantly better than no behavior modification (p < .05). In general, medication dose-response curves flattened as the intensity of behavior modification increased.
|
|
Studies Measuring Outcome
|
Study 1, Study 2, Study 3
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 5: Individualized target behavior |
|
Description of Measures
|
Individualized target behavior was measured by the Daily Report Card (DRC)/Individualized Target Behavior Evaluation (ITBE). All children received the DRC/ITBE as part of the STP treatment package. Behavioral goals and criteria were individualized for each child based on his or her areas of impairment. To measure how well each child behaved in these areas, the counselors tracked all target behaviors achieved in the DRC/ITBE, and the percentages of behavior goals met were calculated for each child for each day of assessment.
|
|
Key Findings
|
In one study, a treatment withdrawal design was used in which the STP behavioral treatment components were withdrawn for 2 days during the 6th week of the 8-week program. On the days when the behavioral treatment was not in place, children were significantly less likely to meet their individualized behavioral goals (p < .01). The effect size was large (Cohen's d = 3.10). Upon reinstatement of the STP behavioral treatment components, behaviors returned to levels observed before treatment was withdrawn.
In another study, a within-subjects factors design was used to compare the effects of behavior modification and medication, alone and in combination. Methylphenidate was administered in varying doses (placebo, 12.5 cm², 25.0 cm², and 37.5 cm²) with and without behavior modification. Each participant had 2 days in each medication condition without behavioral treatment and 4 days in each medication condition with behavioral treatment. All three dosage levels of medication increased the percentage of daily target criteria each child met compared with placebo (p < .01). Behavior modification increased the percentage of daily target criteria each child met compared with no behavior modification (p < .01). The combination of medication and behavior modification overall did not have statistically significant positive results for this outcome.
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
3.3
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
90% Male
10% Female
|
95% White
5% Race/ethnicity unspecified
|
|
Study 2
|
6-12 (Childhood)
|
92.6% Male
7.4% Female
|
92.6% White
3.7% American Indian or Alaska Native
3.7% Race/ethnicity unspecified
|
|
Study 3
|
6-12 (Childhood)
|
91.7% Male
8.3% Female
|
79% White
21% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Rule-following and interpersonal behavior in recreational activities
|
3.5
|
3.5
|
3.0
|
2.0
|
3.5
|
4.0
|
3.3
|
|
2: Academic productivity and rule-following in the classroom
|
3.5
|
3.5
|
3.0
|
2.5
|
3.5
|
4.0
|
3.3
|
|
3: Child behaviors
|
2.5
|
2.5
|
3.0
|
3.0
|
3.5
|
4.0
|
3.1
|
|
4: Perceived effectiveness/stress among counselors and teachers
|
2.0
|
2.5
|
3.5
|
2.5
|
3.5
|
4.0
|
3.0
|
|
5: Individualized target behavior
|
3.5
|
3.5
|
3.0
|
2.5
|
3.5
|
4.0
|
3.3
|
Study Strengths Psychometric properties were established by the investigators for most of the measures used in the studies. Treatment integrity and fidelity checks were conducted and showed that the intervention was implemented as intended. Staff members were intensively trained and supervised. The investigators identified and controlled for several confounding variables across studies, and the study designs were generally strong.
Study Weaknesses Attrition and small sample size were a concern in some of the studies, and there was inadequate attention to missing data (i.e., modeling of missing data was not used).
|
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
Carlson, C. L., Pelham, W. E., Milich, R., & Dixon, M. J. (1992). Single and combined effects of methylphenidate and behavior therapy on the classroom performance of children with attention-deficit hyperactivity disorder. Journal of Abnormal Child Psychology, 20(2), 213-232.
Chronis, A. M., Fabiano, G. A., Gnagy, E. M., Wymbs, B., Burrows-MacLean, L., & Pelham, W. E. (2001). Comprehensive, sustained behavioral and pharmacological treatment for ADHD: A case study. Cognitive and Behavioral Practice, 8, 346-359.
Coles, E. K., Pelham, W. E., Gnagy, E. M., Burrows-MacLean, L., Fabiano, G. A., Chacko, A., et al. (2005). A controlled evaluation of behavioral treatment with children with ADHD attending a summer treatment program. Journal of Emotional and Behavioral Disorders, 13(2), 99-112.
Evans, S. W., Pelham, W. E., & Grudberg, M. V. (1995). The efficacy of note taking to improve behavior and comprehension of adolescents' attention deficit hyperactivity disorder. Exceptionality, 5, 1-17.
* Fabiano, G. A., Pelham, W. E., Gnagy, E. M., Burrows-MacLean, L., Coles, E. K., Chacko, A., et al. (2007). The single and combined effects of multiple intensities of behavior modification and methylphenidate for children with attention deficit hyperactivity disorder in a classroom setting. School Psychology Review, 36(2), 195-216.
Fabiano, G. A., Pelham, W. E., Manos, M. J., Gnagy, E. M., Chronis, A. M., Onyango, A. N., et al. (2004). An evaluation of three time-out procedures for children with attention-deficit/hyperactivity disorder. Behavior Therapy, 35, 449-469.
Hoza, B., Pelham, W. E., Sams, S. E., & Carlson, C. L. (1992). An examination of the "dosage" effects of both behavior therapy and methylphenidate on the classroom performance of two ADHD children. Behavior Modification, 16, 164-192.
Pelham, W. E., Carlson, C., Sams, S. E., Vallano, G., Dixon, M. J., & Hoza, B. (1993). Separate and combined effects of methylphenidate and behavior modification on boys with attention deficit-hyperactivity disorder in the classrooms. Journal of Consulting and Clinical Psychology, 61, 506-515.
Waschbusch, D. A., Kipp, H. L., & Pelham, W. E. (1998). Generalization of behavioral and psychostimulant treatment of attention-deficit hyperactivity disorder (ADHD): Discussion and examples. Behaviour Research and Therapy, 36, 675-694.
Wymbs, B. T., Robb, J. A., Chronis, A. M., Massetti, G. M., Fabiano, G. A., Arnold, F. W., et al. (2005). Long-term, multi-modal treatment of a child with Asperger's syndrome and comorbid disruptive behavior problems: A case illustration. Cognitive and Behavioral Practice, 12, 338-350.
|
