Serviceman and family looking at laptop

Understanding the Placement Process

Clearinghouse for Military Families at Penn State Stars

The Clearinghouse uses a rigorous process to review and categorize programs. Each program undergoes a systematic review. To determine a program’s placement on the Clearinghouse Continuum of Evidence, several factors are considered. These factors include ensuring that the program results in its desired outcome and has a lasting effect on its target population. To help ensure scientific rigor, only evaluations published in peer-reviewed journals are considered. View our continuum (PDF) to better understand the review criteria we use in our placement process.  To learn how the Clearinghouse differs from other organizations that review programs, please review our clearinghouse_comparisons_matrix_full (PDF), or clearinghouse_comparisons_matrix (PDF).

What is an effective program?

An effective program has been shown to have positive results in an evaluation using a randomized controlled trial (RCT) design or a well-matched quasi-experimental design. This evaluation must show a significant and sustained effect. Demonstrating a significant effect requires performing a rigorous statistical analysis of data that results in a statistically significant change in a highly desired outcome. For a program to demonstrate a sustained effect, the effect must last for at least one year beyond the end of the program or at least two years after beginning the program.

To be placed as an effective program, positive results (as described above) must have also been demonstrated in at least one other study conducted by researchers not involved with the original successful study. In other words, there must be independent confirmation of positive program outcomes. Effective programs also have evaluations that meet high scientific standards in research design, adequately addressing all of the following criteria:

  • A representative group of participants;
  • Adequate outcome measurement using reliable and valid assessments;
  • Indication of practical (vs. statistical) significance; and
  • Relatively few people leave the study before it is completed, so you can have more confidence in the conclusions.

The Clearinghouse assigns effective programs to one of the following two categories:

  • Effective (RCT) – Both the original and the replication study must be RCTs
  • Effective (Quasi) – One or both of the evaluations can be quasi-experimental
What is a promising program?

promising program has at least one study showing statistically significant effects, using a RCT or quasi-experimental design. A promising program shows sustained effects six months after the end of the program or one year from the beginning of the program. These programs do not have to show evidence of replication. Evaluations of promising programs must address at least two of the following criteria:

  • A representative group of participants;
  • Adequate outcome measurement using reliable and valid assessments;
  • Indication of practical (vs. statistical) significance; and
  • Relatively few people leave the study before it is completed, so you can have more confidence in the conclusions.
What is an unclear program?

An unclear program is one that has been evaluated with a quasi-experimental design that lacks sufficient methodological rigor, a pre/post-test design without a comparison group, or a purely descriptive evaluation (e.g. case study). Unclear programs could also have no evaluations of the program or have mixed results.
The Clearinghouse assigns unclear programs to one of the following three categories:

  • Unclear (Ø)—no evaluations performed or mixed results.
  • Unclear (+)—a program does not quite qualify for a Promising placement, but has potentially promising features.
  • Unclear (-)—a program does not quite qualify for an Ineffective placement but has potentially ineffective features.

Currently, many of the programs placed on the Continuum are categorized as unclear programs. This does not mean that they are ineffective but rather that the evaluations have not been sufficiently rigorous.

What is an ineffective program?

An ineffective program is one with an experimental or quasi-experimental research design that failed to demonstrate a significant or sustained effect in its initial study and at least one appropriate replication, or a program that has resulted in significant negative effects. Generally speaking, these programs should be avoided unless or until the Clearinghouse places them in the category of “unclear” or higher.