The following are a compilation of frequently asked questions (FAQs) that address information about established review procedures and criteria, updates
from the latest round of reviews, and a summary of findings. More detailed information on the specific process and criteria used to conduct the review can
be found on the
Review Process page.
If you have a question that is not answered here, please
contact us.
Last Updated December 2017.
GENERAL QUESTIONS ABOUT THE REVIEW
From 2009 to 2017, the U.S. Department of Health and Human Services (HHS) sponsored an ongoing systematic review of the teen pregnancy prevention research literature, to help identify programs with evidence of effectiveness in reducing teen pregnancy, sexually transmitted infections (STIs), and associated sexual risk behaviors. The main purpose of the Teen Pregnancy Prevention (TPP) Evidence Review was to review research to examine study quality and assess whether program models have demonstrated positive impacts on sexual risk behavior and sexual health outcomes. These programs reflect a range of approaches that exist in the field (e.g., youth development, comprehensive sex education, abstinence, HIV/STI education, etc.). In addition to being a resource to organizations that work to prevent teen pregnancy, the TPP Evidence review was used by the former OAH’s Teen Pregnancy Prevention program and ACF's Personal Responsibility Education Program (PREP) to identify individual program models that can be selected for replication by potential grantees.
The TPP Evidence Review was managed by the
Office of the Assistant Secretary for Planning and Evaluation (ASPE) in collaboration with the former
Office of Adolescent Health (OAH), and the Administration for Children and Families’
Family and Youth Services Bureau (FYSB) within the U.S. Department of Health and Human Services (HHS). The TPP Evidence Review was conducted through a contract with Mathematica Policy Research.
The TPP Evidence Review was a joint effort sponsored by three agencies within the U.S. Department of Health and Human Services: the Office of the Assistant Secretary for Planning and Evaluation, the Family and Youth Services Bureau within the Administration for Children and Families, and the former Office of Adolescent Health within the Office of the Assistant Secretary for Health.
There were 48 program models that have evaluation studies that met the TPP Evidence Review criteria for evidence of program effectiveness. The program models represent a range of different program approaches, including abstinence, comprehensive sex education, HIV/STI prevention, and youth development approaches.
The findings from the initial review were released in Spring 2010 as part of the former Office of Adolescent Health Teen Pregnancy Prevention grant announcement. The review findings were updated on a period basis as new research emerged. Findings from the most recent update were released in Winter 2017.
Within HHS, the former OAH's Teen Pregnancy Prevention (TPP) program and ACF's Personal Responsibility Education Program (PREP) relied on the TPP Evidence Review findings to identify individual program models that can be selected for replication by potential grantees. Questions about whether specific program models are eligible for federal funding should be directed to the appropriate program office.
UPDATES TO THE REVIEW FINDINGS
The review findings were updated most recently in Winter 2017 to cover research released from August 2015 to October 2016. As a part of this update, the review team identified four new programs meeting the review criteria for evidence of effectiveness, bringing the total number of evidence-based program models to 48.
There were no changes to the review criteria in the most recent update to the review findings.
The search strategy originally identified new studies for review in four ways: reviewing published research syntheses, reviewing the websites of relevant research and policy organizations, issuing public calls for studies to solicit new and unpublished research, and conducting keyword searches of electronic databases. When updating the review findings, the review team also scanned the tables of contents of relevant research journals and the schedules of professional conference proceedings.
REVIEW PROCEDURES AND CRITERIA
In developing the review criteria, HHS drew upon evidence standards used by several well established evidence assessment projects and research and policy groups, such as the What
Works Clearinghouse, Blueprints for Healthy Youth Development, and the National Registry of Evidence-Based Programs and Practices. Based on standards utilized in these other processes,
this review defined the criteria for the quality of an evaluation study and the strength of evidence for a particular intervention.
Using these criteria, HHS then defined a set of rigorous standards an evaluation must meet in order for a program to demonstrate evidence of effectiveness.
The high study quality rating was reserved for randomized controlled trials with low rates of sample attrition, no reassignment of sample members, no systematic differences
in data collection between the research groups, and more than one subject or group (school, classrooms, etc.) in both the treatment and control conditions. The moderate study
quality rating was considered for studies using quasi-experimental designs and for randomized controlled trials that did not meet all the review criteria for a high quality
rating. To meet the criteria for a moderate study quality rating, a study had to demonstrate equivalence of the intervention and comparison groups on race, age, and gender;
report no systematic differences in data collection between the research groups; and have more than one subject or group (school, classroom, etc.) in both the intervention and
comparison conditions. Studies based on samples of youth ages 14 or older also had to demonstrate equivalence of the intervention and comparison groups on at least one
behavioral outcome measure.
The reviews were conducted by trained researchers from Mathematica Policy Research. Each individual impact study was assessed by two team members; the first member conducted a
detailed review of the study following a protocol developed by Mathematica and approved by an HHS interagency work group; the second member assessed and verified the review
for accuracy and completeness.
No. In addition to studies that use a randomized design, the review also considered quasi-experimental (also known as matched comparison group design)
studies that do not employ random assignment.
Yes. The review was limited to studies of programs serving youth in the United States.
The age criterion of 19 or younger refers to age at the start of the intervention. Participants may have been older than 19 during the study period or when outcome measures were assessed.
Measures with serious limitations in terms of their validity or interpretation were excluded. For example,
the review did not consider reports from males of their female partners’ use of birth control pills, or scales of behavioral risk that combine multiple
measures into a single outcome.
Yes. In addition to findings for the full study sample, the review also considered findings for two subgroups based on: (1) gender and (2) sexual
experience at baseline. The review considered the same outcome measures for these subgroups as for the full study sample—namely, measures of sexual
risk behavior and its health consequences.
The subgroup assessment was limited to address concerns about “multiple comparisons” or “multiple hypothesis testing.” The issue was that as the number of subgroups examined in a particular study increases, the probability of finding a statistically significant impact also increases, just by chance. To address this issue, we chose to limit the number of subgroups considered for providing evidence of effectiveness. In selecting these subgroups, there were many relevant options to consider, such as race/ethnicity, gender, sexual experience, socioeconomic status, family structure, and many others. HHS ultimately chose to focus on gender and baseline sexual experience as the two subgroups to be considered by the review. Moreover, in many studies, the sample sizes are too small to assess impacts separately by race/ethnicity.
No. The review was not limited to peer-reviewed journal articles. The review also included studies reported as part of book chapters, government reports, unpublished manuscripts, or other documents.
By not restricting the pool of eligible studies to peer review publications, we were able to identify more recently evaluated studies. In addition, not all peer-reviewed publication venues are the same in terms of quality of the review. Instead, we focused on the quality of the evaluation study and assessed the impacts based upon the established review criteria. We required authors of unpublished reports to provide complete information needed to assess the quality of the evaluation study and its outcomes.
HHS determined that program models with evidence of effectiveness must demonstrate evidence of a positive, statistically significant impact on at least one of the
following outcomes: sexual activity (initiation; frequency; rates of vaginal, oral and/or anal sex); number of sexual partners; contraceptive use (consistency of
use or one-time use, for either condoms or another contraceptive method); STIs; pregnancy. It is possible that programs effective in influencing these behaviors also
affect other types of adolescent health-risk behaviors. However, to be included in the review, programs must examine program impacts on at least one measure of sexual
risk behavior or its health consequences.
In January 2015, implementation readiness scores were added for any existing programs meeting the review criteria for evidence of effectiveness. The score for each program was comprised of three component scores (curriculum and materials, staff training and supports, and fidelity monitoring tools and resources). The component scores were combined into a total implementation readiness score, which ranges from 0 to 8, with higher scores indicating programs most ready to implement. The implementation readiness scores are listed under the Find a Program page of the TPP Evidence Review website.
The Teen Pregnancy Prevention Evidence Review program is no longer active. The information found on this website is archival as of September 2019.
REVIEW FINDINGS
A full list of the studies reviewed is available under the
Reviewed Studies section of the website.
In 2014, the review team released a research brief summarizing findings from an ongoing effort to collect and report program effect size information from studies included the HHS Teen Pregnancy Prevention (TPP) Evidence Review. Findings indicate substantial variation in effect sizes across programs, but also a clear need for improved standards and reporting of effect size information in teen pregnancy prevention research. Visit the
Publications section of the website to download the research brief and accompanying effect size data.
Each evidence review used a slightly different set of procedures and criteria. Although there was usually overlap
across reviews in which program models meet evidence standards, differences in the criteria used to screen and assess
studies may have led to some difference in the program models identified as evidence-based. In addition, each evidence
review may have assessed a study’s evidence of effectiveness on a different outcome. For example, a youth development program
may have been reviewed as part of the TPP Evidence Review for evidence of effectiveness on contraceptive use behavior, and the
same program may be included in another evidence review to assess impacts on educational outcomes, violent behavior, or other outcomes.
The review does not aim to identify programs targeted to any specific population. Rather, the goal of the review is to identify programs with the strongest evidence of effectiveness.
If certain high-risk populations were underrepresented, it points to a gap in the underlying research literature and need for additional research to identify effective programs for
these groups.
The
About the Review section of the website provides a detailed description of the process used to determine the study ratings and an explanation of the criteria.
This information should help explain the rating given to any particular study.
There are five reasons a program may not have met the TPP Evidence Review criteria for evidence of effectiveness. First, a program may not be included because it was not evaluated.
Second, a program may not be included because even though it was evaluated, it did not meet screening criteria. Third, a program may not be included because its evaluation was not
sufficiently rigorous. Fourth, a program may not be included because it did not provide evidence of positive impacts on one of the key outcome measures
for either the full study sample or a priority subgroup. Fifth, a program may not be included because the evaluation provided evidence of negative impacts on one of the key outcome measures.
FUTURE PLANS FOR UPDATING THE REVIEW
The Teen Pregnancy Prevention Evidence Review program is no longer active. The information found on this website is archival as of September 2019.
It is up to the individual program offices to determine whether grantees will be able to replicate program models that show evidence of effectiveness in the future. Though the findings from the TPP Evidence Review may be used to inform any future funding announcements, program offices may consider additional factors beyond whether a program meets criteria for being evidence-based when deciding which programs are eligible for replication. For example, based on experience with the first round of teen pregnancy prevention initiative grant funding, program offices may focus on replicating those evidence-based program models that are implementation-ready and have training and support materials available.
The Teen Pregnancy Prevention Evidence Review program is no longer active. The information found on this website is archival as of September 2019.
No, we are not considering new studies for review . The Teen Pregnancy Prevention Evidence Review program is no longer active. The information found on this website is archival as of September 2019.