Skip to main content

You are here

NREPP Logo

Review Process

Pre-screening and Approval for Review

NREPP identifies programs for review in three ways:

  • Nominations from the field: SAMHSA announces an open submission period. The open submission period generally lasts several months and allows developers, researchers, practitioners, and other interested parties to submit programs for review.
  • Environmental scans: SAMHSA and NREPP contractor staff conduct environmental scans (including literature searches, focus groups, public input, and interviews) to identify potential interventions for review.
  • Agency nomination: SAMHSA identifies programs and practices addressing specific agency priorities.
Programs identified through the open submission process are prioritized for review.

Programs are pre-screened to ensure that at least one evaluation study meets the minimum criteria for review. Programs are submitted to SAMHSA for approval to move into the review process. Applicants are notified whether their intervention has been accepted for review or rejected.

Re-Review Process

  • A Federal Register notice issued July 7, 2015 announced the intent to re-review currently posted NREPP programs to comport with new review criteria and ratings.
  • The changes to the review process will apply to all re-reviews of programs currently posted on NREPP. Approximately 110 programs posted on NREPP prior to September 2015 will be reviewed each year over the next 3 years. Program contacts/developers will be notified at least 45 days prior to the re-review date that their program has been selected for re-review. The re-review will follow the procedures presented on this page.
Literature Search and Screening

NREPP contractor staff contact the developer to request any additional evaluation studies and information about resources for dissemination and implementation (RFDI). Applicants will be asked to complete the RFDI checklist. Although SAMHSA has determined that programs no longer need RFDI materials to be reviewed, programs with such materials will be prioritized for review.

To ensure a comprehensive report, a literature search is conducted to identify other relevant evaluation studies. All evaluation materials are screened and NREPP contractor staff determine which studies and outcomes are eligible for review. SAMHSA has determined that all eligible outcomes will be reviewed, but that programs with positive impacts on outcomes and populations of interest will be prioritized over programs without positive impacts.

All evaluation studies that meet minimum criteria, including being published within the past 20 years (1995 or later), and fall within a 10-year time frame—as defined by the most recent eligible article of a study—are eligible for review.

Expert Review

NREPP contractor staff identify two certified reviewers to conduct the review (Note: re-reviews of programs posted on NREPP before September 2015 may be completed by one reviewer). Reviewers must complete a Conflict of Interest form to confirm no conflict exists that would require recusal.

Review packets are sent to reviewers to assess the rigor of the study and the magnitude and direction of the program's impact on eligible outcomes. Studies identified during the submission process or by the literature search may be included in the review, but not all evaluation studies are necessarily included in the review packet. For instance, studies that will not be reviewed include those that do not meet the minimum criteria or assess only outcomes not included in the NREPP outcome taxonomy. The outcome taxonomy currently includes 16 mental health, 12 substance use, and 27 wellness outcomes, relevant to SAMHSA’s mission. An iterative process was applied to condense an extensive list of over 1000 outcome tags (extracted from studies reviewed on NREPP) into aggregate constructs. All outcomes were reviewed by scientists with expertise in the field of behavioral health and by SAMHSA staff.

Reviewers independently review the studies provided and calculate ratings using the NREPP Outcome Rating Instrument.

Outcomes are assessed on four dimensions. Study reviewers assign numerical values to each dimension in the NREPP Outcome Rating Instrument (with the exception of effect size). To support consistency across reviews, the dimensions include definitions, and the NREPP Outcome Rating Instrument provides other guidance that reviewers consider when rating findings. Reviewers also make note of any other information that should be highlighted as being of particular importance.

The study reviewer is responsible for making a reasonable determination as to the strength of the methodology, fidelity, and program effect, based on the provided documentation and his/her specialized knowledge with regard to program evaluation and subject matter. If the reviewers' ratings differ by a significant margin, a consensus conference to discuss and resolve the differences may be held.

In addition to this review by certified reviewers, NREPP staff also assess programs' conceptual frameworks.

NREPP Methodology

To assess the strength of evidence, each eligible reported effect or measure is rated on four dimensions; each dimension includes multiple elements (for instance, the strength of the methodology dimension includes design/assignment and attrition). Not all reported effects or measures will be assessed. For instance, those measured by a single item that is either subjective or not well-established are excluded. Those for which no effect size can be calculated are excluded. Also, findings not related to an NREPP outcome of interest are excluded. Once all eligible measures or effects are rated, all scores for those falling into one outcome are combined. This includes reported measures or effects across studies.

The effect size calculations are based on standard statistical methods. NREPP staff calculate Hedges g effect sizes for both continuous and dichotomous (yes/no) outcomes. Whenever possible, NREPP staff calculate an effect size that is adjusted for baseline differences.

Evidence Classes and Outcome Ratings The outcome ratings are based on four dimensions: rigor, effect size, program fidelity, and conceptual framework. Most weight is given to the ‘design/assignment’ element of the rigor dimension and the confidence interval of the effect size(s) contributing to the outcome.

The rigor and fidelity elements contribute to the evidence score; the confidence interval of the effect size determines the ‘effect class’; and both combined determine the ‘evidence class’ for each component measure of the outcome. Evidence classes for each measure are then assigned numeric weights and averaged to create an outcome score. The outcome rating is based on a combination of the outcome score and the conceptual framework rating.

The graphic below summarizes all components of the outcome rating.

Components of the Final Outcome Rating



The evidence class for each reported effect is based on a combination of evidence score and effect class.
  • The Evidence score is based on the rigor and fidelity dimensions and is rated as strong, sufficient, or inconclusive. For information about the elements of rigor and fidelity, go to: Program Review Criteria
  • The Effect class is based on the confidence interval of the effect size:
    1. Favorable: Confidence interval lies completely within the favorable range (greater than .10)
    2. Possibly favorable: Confidence interval spans both the favorable (greater than .10) and trivial range (from -.25 to .10)
    3. Trivial: Confidence interval lies completely within the trivial range (from -.25 to .10) or spans the harmful and favorable range (from -.25 to greater than .10)
    4. Possibly harmful: Confidence interval spans both the harmful (lower than -.25) and trivial range (from -.25 to .10).
    5. Harmful: Confidence interval lies completely within the harmful range (lower than -.25)
These two dimensions are then combined to categorize programs into one of seven evidence classes as depicted below.

DESCRIPTION OF EVIDENCE CLASSES
Evidence Class Evidence Description
Class A Highest quality evidence with confidence interval completely within the favorable range
Class B Sufficient evidence with confidence interval completely within the favorable range
Class C Sufficient or highest quality evidence with confidence interval spanning both the favorable and trivial ranges
Class D Sufficient or highest quality evidence with confidence interval completely within the trivial range
Class E Sufficient or highest quality evidence with confidence interval spanning both the harmful and trivial ranges
Class F Sufficient or highest quality evidence with confidence interval completely within the harmful range
Class G Limitations in the study design preclude from reporting further on the outcome


Outcome Score

The outcome rating (see table below) is based on a combination of the outcome score and the conceptual framework rating. The outcome score is based on the evidence class(es) of each component measure. The conceptual framework is based on whether a program has clear goals, activities, and a theory of change.

OUTCOME RATING
Outcome Evidence Rating Icon Definition
Effective effective The evaluation evidence has strong methodological rigor, and the short-term effect on this outcome is favorable. More specifically, the short-term effect favors the intervention group and the size of the effect is substantial.
Promising promising The evaluation evidence has sufficient methodological rigor, and the short-term effect on this outcome is likely to be favorable. More specifically, the short-term effect favors the intervention group and the size of the effect is likely to be substantial.
Ineffective ineffective The evaluation evidence has sufficient methodological rigor, but there is little to no short-term effect. More specifically, the short-term effect does not favor the intervention group and the size of the effect is negligible. Occasionally, the evidence indicates that there is a negative short-term effect. In these cases, the short-term effect harms the intervention group and the size of the effect is substantial.
Inconclusive inconclusive Programs may be classified as inconclusive for two reasons. First, the evaluation evidence has insufficient methodological rigor to determine the impact of the program. Second, the size of the short-term effect could not be calculated.


Reporting

The ratings and descriptive information are compiled into a program profile.

A courtesy copy of the program profile is shared with the developer or submitter of the program for review, who may suggest revisions to the profile.

The final program profile is submitted to SAMHSA for review, approval, and posting on the NREPP website.



Inquiring About or Appealing an Outcome Rating

Inquiries. SAMHSA is committed to the quality, fairness, and rigor of the NREPP review process. As part of this commitment, the program contact or others may inquire about an outcome evidence rating. The inquiry should be grounded in the following:
  • A fair reading of the evidence (please see the program's outcome ratings).
  • An understanding of NREPP rating procedures and processes
Please use the Contact Us form to submit an inquiry. If your inquiry concerns a specific outcome rating, we will provide additional information explaining why the rating was assigned. (Note that SAMHSA does not provide the identity of reviewers.)

Appeals Process. Following the inquiry and response, a program contact may request a formal appeal of one or more outcome ratings.

The NREPP Appeals Board will strive to provide a response that addresses all of an inquirer's concerns. Our goal is to complete the entire appeals process within 90 days.

The Appeals Board is composed of a Chair and six members, all of whom are certified NREPP reviewers. The Board can be supplemented with additional certified reviewers, if needed.

Please use the Contact Us form to submit an inquiry. If your inquiry concerns a specific outcome rating, we will provide additional information explaining why the rating was assigned. (Note that SAMHSA does not provide the identity of reviewers.)

Step 1.Use the Contact Us to request a formal appeal of one or more outcome rating(s). Upon receipt of the appeal request, NREPP staff will provide the inquirer with additional guidance that will require the inquirer to provide pertinent, specific information indicating the necessity of an appellate review.

Step 2.The Chair will select two members of the Appeals Board whose expertise is pertinent to the appeal and who have no conflicts of interest, to conduct the review. Each appellate reviewer will conduct an independent review of the appeal, the studies in the program's evidence base, and copies of the original reviewers' NREPP Outcome Rating Instruments. The identities of the original reviewers will remain anonymous until they are required to participate in the process.

Step 3.After the separate reviews are complete, the appellate reviewers will participate in a conference call led by the Chair to discuss the original review and outcome rating(s) and the concerns raised by the inquirer, and to reach consensus about the program's outcome rating(s). If the appellate reviewers agree the program's original outcome rating(s) should not be changed, they will provide a written explanation documenting the discussion and the reason for their final decision. If they believe new outcome rating(s) are warranted, they will provide a written explanation describing how and why their scores differ from the scores of the original reviewers. A final conference call will be held including the Chair, Appeals Board members, and original reviewers to discuss the disagreement on the scoring instruments and come to a consensus on final outcome rating(s).

Step 4.Once a final consensus rating is reached, NREPP staff will provide the inquirer with a written response describing the Appeals Board's final decision.

Step 5.If warranted as a result of the appeal, changes to a program's outcome rating(s) or additions to information in the program profile will be made on the NREPP website.

Revised 11/20/15