Skip to Content

NREPP Review Criteria

In reviewing the research that has been submitted on each intervention, NREPP reviewers use predefined criteria to measure the Quality of Research and Readiness for Dissemination.

Criteria for Rating Quality of Research

Each reviewer independently evaluates the Quality of Research for an intervention's reported results using the following six criteria:

  1. Reliability
  2. Validity
  3. Intervention fidelity
  4. Missing data and attrition
  5. Potential confounding variables
  6. Appropriateness of analysis

1. Reliability

Outcome measures should have acceptable reliability to be interpretable. "Acceptable" here means reliability at a level that is conventionally accepted by experts in the field.

0 = Absence of evidence of reliability or evidence that some relevant types of reliability (e.g., test-retest, inter-rater, inter-item) did not reach acceptable levels.

2 = All relevant types of reliability have been documented to be at acceptable levels in studies by the applicant.

4 = All relevant types of reliability have been documented to be at acceptable levels in studies by independent investigators.

back to Criteria for Rating Quality of Research

2. Validity

Outcome measures should have acceptable validity to be interpretable. "Acceptable" here means validity at a level that is conventionally accepted by experts in the field.

0 = Absence of evidence of measure validity, or some evidence that the measure is not valid.

2 = Measure has face validity; absence of evidence that measure is not valid.

4 = Measure has one or more acceptable forms of criterion-related validity (correlation with appropriate, validated measures or objective criteria); OR, for objective measures of response, there are procedural checks to confirm data validity; absence of evidence that measure is not valid.

back to Criteria for Rating Quality of Research

3. Intervention Fidelity

The "experimental" intervention implemented in a study should have fidelity to the intervention proposed by the applicant. Instruments that have tested acceptable psychometric properties (e.g., inter-rater reliability, validity as shown by positive association with outcomes) provide the highest level of evidence.

0 = Absence of evidence or only narrative evidence that the applicant or provider believes the intervention was implemented with acceptable fidelity.

2 = There is evidence of acceptable fidelity in the form of judgment(s) by experts, systematic collection of data (e.g., dosage, time spent in training, adherence to guidelines or a manual), or a fidelity measure with unspecified or unknown psychometric properties.

4 = There is evidence of acceptable fidelity from a tested fidelity instrument shown to have reliability and validity.

back to Criteria for Rating Quality of Research

4. Missing Data and Attrition

Study results can be biased by participant attrition and other forms of missing data. Statistical methods as supported by theory and research can be employed to control for missing data and attrition that would bias results, but studies with no attrition or missing data needing adjustment provide the strongest evidence that results are not biased.

0 = Missing data and attrition were taken into account inadequately, OR there was too much to control for bias.

2 = Missing data and attrition were taken into account by simple estimates of data and observations, or by demonstrations of similarity between remaining participants and those lost to attrition.

4 = Missing data and attrition were taken into account by more sophisticated methods that model missing data, observations, or participants, OR there were no attrition or missing data needing adjustment.

back to Criteria for Rating Quality of Research

5. Potential Confounding Variables

Often variables other than the intervention may account for the reported outcomes. The degree to which confounds are accounted for affects the strength of causal inference.

0 = Confounding variables or factors were as likely to account for the outcome(s) reported as were the hypothesized causes.

2 = One or more potential confounding variables or factors were not completely addressed, but the intervention appears more likely than these confounding factors to account for the outcome(s) reported.

4 = All known potential confounding variables appear to have been completely addressed in order to allow causal inference between the intervention and outcome(s) reported.

back to Criteria for Rating Quality of Research

6. Appropriateness of Analysis

Appropriate analysis is necessary to make an inference that an intervention caused reported outcomes.

0 = Analyses were not appropriate for inferring relationships between intervention and outcome, OR sample size was inadequate.

2 = Some analyses may not have been appropriate for inferring relationships between intervention and outcome, OR sample size may have been inadequate.

4 = Analyses were appropriate for inferring relationships between intervention and outcome. Sample size and power were adequate.

back to Criteria for Rating Quality of Research

Criteria for Rating Readiness for Dissemination

  1. Availability of implementation materials
  2. Availability of training and support resources
  3. Availability of quality assurance procedures

1. Availability of Implementation Materials

(e.g., treatment manuals, brochures, information for administrators)

0 = No materials or resources are available that could assist with program implementation.

1 = Limited materials and resources are available that could facilitate but not directly assist with program implementation.

2 = Limited materials and resources are available that directly assist with program implementation.

3 = Adequate materials and resources are available that directly assist with program implementation.

4 = Adequate materials and resources are available that directly assist with program implementation, and they are of high quality and appropriate for the intended audience(s).

back to Criteria for Rating Readiness for Dissemination

2. Availability of Training and Support Resources

(e.g., tested training curricula, mechanisms for ongoing supervision and consultation)

0 = No materials, resources, and/or technical assistance are available that support training or coaching.

1 = Limited materials, resources, and/or technical assistance are available that could facilitate but not directly support training and coaching.

2 = Limited materials, resources, and/or technical assistance are available to directly support training and coaching.

3 = Adequate materials, resources, and/or technical assistance are available that directly support training and coaching.

4 = Adequate materials, resources, and technical assistance are available that directly support training and coaching, and they are of high quality and appropriate for the intended audience(s).

back to Criteria for Rating Readiness for Dissemination

3. Availability of Quality Assurance Procedures

(e.g., protocols for gathering process and/or outcome data, ongoing monitoring of intervention fidelity, supervision/training feedback)

0 = No materials, resources, and/or procedures are available that could help ensure quality assurance.

1 = Limited materials, resources, and/or procedures are available that could facilitate but not directly assist in ensuring quality assurance.

2 = Limited materials, resources, and/or procedures are available to directly support quality assurance.

3 = Adequate materials, resources, and/or procedures are available that directly support quality assurance.

4 = Adequate materials, resources and/or procedures are available that directly support quality assurance, and they are of high quality and appropriate for the intended audience(s).

back to Criteria for Rating Readiness for Dissemination