Learning From Evaluations of Youth-serving Programs

Following are highlights of reviews of many evaluations of youth programs. This material will be of interest to people in youth-serving organizations or who have research, funding, philanthropic, or community interests.

However, we also intend that this and other reports will be used to support a “wiki-discussion” paper on programs for youth and how they are organized and supported in Canada and the USA. The first draft of that paper will be the next major post in Politudes.

An Important General Conclusion From the Evaluation Literature

It appears that with some important exceptions, most programs are successful in helping youth to reduce risk factors, and in different degrees, to make improvements in their transitions to adulthood. They vary in cost-effectiveness, and some important factors of success are identified in the reviews. However, the ability or potential for society to improve the life trajectories for youth at risk does not seem to be in doubt. We will return to this point in our discussion paper.

 

Learning from evaluative research owes much to the Cochrane and Campbell Collaborations, which are network of academics who identify, review and assess hundreds of program evaluations. Only studies with good methodologies are used to contribute to their conclusions. In this post we rely heavily on “systematic reviews” carried out by the Campbell Collaboration, with a few additions from other literature.

 

1. Mentoring Interventions to Affect Juvenile Delinquency and Associated Problems

Patrick Tolan, David Henry, Michael Schoeny, Arin Bass, published 2008
http://www.campbellcollaboration.org.

This Campbell Review of youth programs reveals that most mentoring programs working with disadvantaged youth are successful in helping youth and are in essence, cost-effective.

“This analysis of 46 studies on four outcomes measuring delinquency or closely related outcomes of aggression, drug use, and academic functioning suggests mentoring for high-risk youth has a modest positive effect for delinquency and academic functioning, with trends suggesting similar benefits for aggression and drug use”

“Effects tended to be stronger when professional development was an explicit motive for participation of the mentors. Of four processes theorized as comprising the methods of effects in mentoring, we found evidence for significantly larger effects when emotional support and advocacy were emphasized. Although these findings support viewing mentoring as a useful approach for intervention to lessen delinquency risk or involvement, limited description of content of mentoring programs and substantial variation in what is included as part of mentoring efforts detracts from better understanding about what might account for the benefits. The valuable features and most promising approaches cannot be ascertained with any certainty. In fact, the body of studies is remarkably lacking in description of key features, program design organization, and theorized processes of impact that are typically provided in empirical reports of intervention effects.”

 

2. Scared Straight and Other Juvenile Awareness Programs for Preventing Juvenile Delinquency: A Systematic Review
Campbell Collaboration Systematic Review,
Anthony Petrosino, John Buehler, Carolyn Turpin-Petrosino
Published: May 2, 2013.
http://www.campbellcollaboration.org

The team reviewed literature on randomized criminological experiments from 1945-1993, and added searches of electronic databases, through to 2011. Experts were also consulted.

They conclude that “Scared Straight” and similar programs are likely to have a negative effect: that is, to increase rather than decrease, delinquency. Such programs are not recommended as crime prevention strategies, and agencies using them are urged to evaluate carefully what they are doing so as not to do more harm than good.

3. How Effective Are Mentoring Programs for Youth? A Systematic Assessment of the Evidence.

David L. DuBois1, Nelson Portillo1, Jean E. Rhodes2, Naida Silverthorn1, and Jeffrey C.Valentine3 (1University of Illinois at Chicago, USA; 2University of Massachusetts, Boston, USA; and 3University of Louisville, KY, USA)
Association for Psychological Science 2011
rhodeslab.org

This meta-analysis covered 73 independent evaluations of mentoring programs published between 1999 and 2010.

“Overall, findings support the effectiveness of mentoring for improving outcomes across behavioural, social, emotional, and academic domains of young people’s development. The most common pattern of benefits is for mentored youth to exhibit positive gains on outcome measures while non-mentored youth exhibit declines. It appears then that mentoring as an intervention strategy has the capacity to serve both promotion and prevention aims. Programs also show evidence of being able to affect multiple domains of youth functioning simultaneously and to improve selected outcomes of policy interest. …From a developmental stand- point, benefits of participation in mentoring programs are apparent from early childhood to adolescence and thus not confined to a particular stage of development. …Collectively, these findings point toward the flexibility and broad applicability of mentoring as an approach for supporting positive youth development.”

4. Interventions for Promoting Reintegration and Reducing Harmful Behaviour and Lifestyles in Street-connected Children and Young People: A Systematic Review

Campbell Collaboration Library of Systematic Reviews
http://www.campbellcollaboration.org
Authors:Esther Coren, Rosa Hossain, Jordi Pardo Pardo, Mirella Veras, Kabita Chakraborty, Holly Harris, Anne J. Martin
Published: 02.05.2013

This review covered evaluations of intensive, time-limited therapeutic interventions, compared with standard shelter or drop-in youth programs. Both kinds of programs achieved positive results, although the evaluations themselves and the data collected varied considerably, and significantly, no study reported on adverse results. Overall, there was not clear evidence that the therapeutic interventions were more effective than the standard services.

5. What happens to persistent and serious young offenders when they grow up?

This report of intensive interventions by the Youth Justice Board of the UK (previous post in Politudes) by Emily Gray of Keele University, showed that most programs achieved positive results but intensive interventions did not seem to have better results than more standard ones. A frequent factor of success seemed to be the quality of relationship established between the adult service providers and the youth.

6. Truancy Interventions: Effects on School Attendance Among Chronic Truant Students
Brandy R Maynard, Katherine Tyson McCrea, Michael S. Kelly
Campbell Systematic Review 05.07.2012

Another Campbell systematic review shows that programs aimed at chronic truant students tend to get positive results.

“The programs studied and included in the review improved attendance by an average of 4.69 days, almost a full school week. No differences were found between school-, court-, or community-based programs or between different modalities of programs. The duration of the intervention also did not demonstrate any association with effect size. The findings from this review also point to gaps in the evidence base, such as lack of inclusion of minority students and a lack of reporting and statistical analysis of demographic variables, particularly race/ethnicity and socioeconomic status (SES). Given that race and SES have been linked to absenteeism, the absence of this data was surprising.”

http://www.campbellcollaboration.org

7. Dropout prevention and intervention programs: Effects on school completion and dropout among school- aged children and youth
Sandra Jo Wilson, Emily E. Tanner-Smith, Mark W. Lipsey, Katarzyna Steinka-Fry, & Jan Morrison
Campbell Systematic Review 2011:8. http://www.campbellcollaboration.org

This Campbell review found that most dropout prevention and intervention programs are effective in decreasing school dropout when implemented well and adjusted to local context.

“The review suggests that no single prevention or intervention strategy stands out as better than any other. The results suggest that the particular program strategy chosen makes less of a difference in eventual outcome than selecting a strategy that can be implemented successfully by the school or agency. Therefore, the authors recommend that policy-makers and practitioners consider cost-effectiveness, adaptation to local needs and a good strategy for implementation when choosing a dropout prevention or intervention program.”

  

8. Self-control interventions for children under age 10 for improving self-control and delinquency and problem behaviours

Piquero A.R., Jennings W. G., Farrington D.P. Campbell Collaboration Systematic Review 2010
http://www.campbellcollaboration.org

When examining these questions, the authors identified 5,000 potentially relevant studies of which 34 met the quality and criteria for inclusion. The review shows that not only are self-control improvement programs effective at improving self-control and reducing delinquency and problem behaviors, but that the effect of these programs is quite strong.

“The authors conclude that the use of such programs should continue for children up to age 10, which is the age at which self-control becomes arguably relatively fixed. In particular, programs that are based on specific training efforts, that are focused and of short-duration. Based on the evidence, the recommended policy response is towards interventions aimed at improving socialization and child-rearing practices in the first decade of life-in lieu of the more cost-prohibitive incarceration policies of the recent past.”

9. Final Report 2010-2011 Evaluation of the Youth Gang Prevention Fund Program
http://www.publicsafety.gc.ca
(Full disclosure: The editor of Politudes, Terrance Hunsley, authored a synthesis of progress evaluation results in 2010 which was part of the material used in this report.)

This document reports on the results of a five year, $CAD 33.5 million Government of Canada fund to support experimental programs to deter youth from gangs. The report is equivocal on measures of outcomes, due mainly to difficulties in implementation and evaluation. Even so, with a projected potential result of dissuading up to 25% of youths engaged from joining or continuing with gangs, the program found that the costs involved would be more than recovered by reduced demand for other federal expenditures, particularly incarceration. No effort was apparent to estimate cost savings to other governments, private interests such as insurance companies, or to society as a whole. Nor was there any analysis of potential benefits to society of having more positively-contributing adults.

Some details of the report are excerpted below.

“Participant Outcomes: Overall, YGPF projects have successfully engaged approximately 1,100 youth. The projects are at capacity and project services are being delivered. A few projects have seen moderate success in raising the knowledge and awareness of project participants, and there is evidence of behaviour change across some of the risk factors. Although the precise extent of behaviour change cannot be determined, the maximum potential impact is up to one-quarter of program participants.”

“…according to mathematical modelling, the program can break even if a success rate of between 15% and 35% is realized (with a mid-range value of 20%); or if a total of between 150 and 400 participants either exit their gangs, or are prevented from entering a gang, by the end of the program. Based in literature review, the potential return at the provincial and societal levels is high if models are implemented with fidelity meaning that close attention is paid quality control and adherence to original program designs.”

“The assessment of economy focused on a single indicator (i.e., the cost savings to the federal government associated with a “gang exit”) to provide an assessment of the potential return on investment to the federal government. Using the data inputs and assumptions noted in Appendix G, a mathematical model was developed to calculate an estimated cost savings to the federal government by comparing the criminal behaviour of gang members and non-gang members and the associated costs (i.e., incarceration). For the analysis, a positive return on the federal investment was defined as cost savings lower than the project cost. The evaluation team undertook the analysis through a set of probabilities (e.g., determining the probability of committing a crime, the probability of jail time). The probability-weighted expense of federal jail time for both gang and non-gang members was compared to show potential cost savings resulting from the project intervention. Note that the average incarceration time for a youth gang member is not known with a great deal of precision. As such, a 95% confidence interval was calculated based on the likely incarceration range of between three and eight years for a gang member, with a median value of six years.”

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s