Systematic Review and Meta-analysis of the Effectiveness of Whole-school Interventions Promoting Mental Health and Preventing Risk Behaviours in Adolescence

Despite the potential of whole-school interventions in improving adolescent mental health and risk behaviour outcomes, the evidence of their effectiveness is inconclusive, in part due to the insufficient number of studies included in prior meta-analyses. This meta-analysis sought to update the evidence-base on the impact of whole-school interventions on outcomes including positive mental health, mental disorders, mental health literacy, substance use, bullying and aggression among adolescent populations.

This study identified that, of the fourteen outcomes meta-analyzed, whole-school interventions were effective in reducing cyber-bullying, cyber-aggression and regular smoking in adolescent populations. While Langford et al. (2015) demonstrated that whole-school interventions were effective in reducing the odds of being bullied among school-aged populations, the findings of this study discerned that this reduction was observed only for the odds of being bullied online, and not in-person, among adolescents. Similarly, while Langford et al. (2015) demonstrated that whole-school interventions were effective in reducing the odds of smoking among school-aged populations, the findings of this study pinpointed that this reduction was observed only for the odds of regular smoking, and not for any smoking, among adolescents. Interestingly, when restricted to low risk for selection bias, a sensitivity analysis supported the effectiveness of whole-school interventions in reducing the odds for regular alcohol use, whereas the non-significant impact on any alcohol use remained unchanged. This suggests that whole-school interventions are effective in relation to the secondary prevention of regular substance use, whereas the primary prevention of non-regular substance use (or experimental use) in adolescents presents as more reticent to change. This study further highlights an additional benefit that had not been identified by Langford et al. (2015), where whole-school interventions were found to be effective in reducing the odds of perpetrating cyber-aggression among adolescents. Given that the whole-school interventions evaluated were universal in nature, involving all adolescents rather than a select high-risk category, the population health benefits offered in relation to these outcomes is large. In an estimated adolescent population of 1.3 billion globally (UNICEF, 2024), the prevalence of cyber-bullying is estimated at up to 57.5% (Zhu et al., 2021), smoking at 19.3% (Nazir et al., 2019) and cyber-aggression at up to 46.3% (Zhu et al., 2021). The significant reduction identified by this review in the odds of cyber-bullying, regular smoking and cyber-aggression by 25%, 31 and 37% respectively, highlights that whole-school interventions can confer substantial benefit not only through the reduction of these highly-prevalent public health issues afflicting adolescents, but further, through averting their various adverse sequelae.

The analyses identified that whole-school interventions were ineffective in relation to the remaining eleven outcomes. This is a surprising finding, given that the HPSF offers a robust theoretical framework, which champions a holistic approach to promotion and prevention and involves the major stakeholders in an adolescent’s life. In unravelling this dichotomy, a key issue emerges in how the HPSF has been translated into practice. To qualify as a whole-school intervention in the literature, studies do not have to reference that the HPSF informed their intervention design. As a consequence, 75% of the studies included in this meta-analysis did not reference the HPSF (Balasooriya Lekamge et al., 2024). Instead, the literature has operationalised the definition of a whole-school intervention to be one that includes at least one programme component addressing each of the curriculum-, ethos and environment-, and community-levels outlined in Fig. 1 (Goldberg et al., 2019; Langford et al., 2015). This operationalisation of whole-school interventions in practice does not clearly align with what the World Health Organisation have advocated for through the HPSF, and as such, may not fully represent or realize the potential of the Framework. For example, a corresponding review mapped how frequently the whole-school interventions implemented by the included studies addressed the eight domains of the HPSF, identifying significant variability with which the domains were addressed (Balasooriya Lekamge et al., 2024). While 100% of interventions addressed the school curriculum domain, only 46% addressed the school physical environment domain, 29% the school health services domain, and 7% referenced the government policies and resources domain. Furthermore, studies occasionally demonstrated passive strategies to target particular domains, with posters serving as the most commonly-implemented strategy in the school physical environment domain. This begs a fundamental question: how whole-school are whole-school interventions (Balasooriya Lekamge et al., 2024)? Suboptimal application (or for many of the included studies, no application) of the HPSF may explain the dissonance between what is a theoretically robust construct and the non-significant findings observed. These findings urge reconsideration of whether interventions must explicitly be based on the HPSF to qualify as a whole-school intervention, and identify three foci for future research (Balasooriya Lekamge et al., 2024). Firstly, which of the eight domains of the HPSF are critical to the success of whole-school interventions? Secondly, does addressing more domains translate to greater impact? Thirdly, what is the relative effectiveness of intervention strategies within each domain, so that the most effective can be prioritized? Deepening our understanding of these research foci can facilitate optimal translation of the HPSF into practice to successfully promote mental health and prevent risk behaviours among adolescent populations.

In promoting the optimal application of the HPSF in practice, it is similarly critical to ensure that studies of whole-school interventions are responsive to the developmental needs of adolescents. This is particularly pertinent given that almost one-third of the included studies traversed multiple grade levels (Balasooriya Lekamge et al., 2024); for instance, a three-year study may target students as they progress from grade 7 through to grade 9. While this meta-analysis had intended to conduct a subgroup analysis to explore whether differential intervention effects were observed for adolescents in different grade levels, an insufficient number of studies for each meta-analysis precluded this analysis (Higgins et al., 2019). This developmental perspective, however, is equally crucial to consider from the lens of intervention design. In their two-year study, Cross et al. (2018) highlighted that participants found the intervention content less relevant in grade 9 than in grade 8, which corresponded with the intervention being evaluated as effective in grade 8, but not in grade 9. Sawyer et al. (2010a, 2010b) similarly identified that teacher ratings of the developmental-appropriateness of the intervention content was, on average, 59% over the three-year period, while adolescents rated their likelihood of recommending the programme to a friend as 32%. These authors subsequently cautioned against simply “aging up” intervention strategies from one grade to another, instead advocating for strategies to be tailored to each grade’s unique needs, as informed by the developmental literature (Cross et al., 2018). It is similarly imperative to highlight that the developmental needs of adolescents will be contingent on the local context. For example, reviews have identified that adolescents from socioeconomically disadvantaged settings are two to three times more likely to develop mental health problems (Reiss, 2013), with economic stress, chaos in the home environment and community violence serving as causal factors unique to this cohort (Devenish et al., 2017). Thus, in addition to grounding the design of whole-school interventions and their evaluations in the developmental literature, it is equally crucial to involve adolescents in the co-creation of intervention strategies to ensure that strategies truly capture their local development needs, preferences and interests.

This study further prompts reflection on the challenges faced in the evaluation of complex public health initiatives. While randomized controlled trials typically represent the most robust trial design for interventional studies, their use in the evaluation of whole-school interventions is contentious. While it is acknowledged that randomized controlled trials are not always best suited to capturing the complexity of systems-based approaches to public health issues (Barry et al., 2019), other scholars offer the counter-perspective that these are an appropriate trial design and can accommodate for the effects of local adaptation (Langford et al., 2015). Irrespective of the study design used, the findings of this study reinforce that the crux of evaluating whole-school interventions lies in the assessment of implementation, and the active use of implementation data in the interpretation of evaluation findings. While most studies (22 of 28) commented on implementation in some capacity, the majority focused on the aggregate-level of implementation among intervention schools. Only two studies (Andersen et al., 2019; Hamilton et al., 2005) provided a breakdown of implementation between intervention schools, finding that one-third or less of intervention schools classified as high-implementers, and up to one-half as low-implementers. These findings are consistent with previous literature, which demonstrates that multicomponent interventions, such as whole-school programmes, commonly encounter challenges with implementation (Durlak & DuPre, 2008). Both of these studies subsequently examined the impact of implementation on outcomes. For example, Andersen et al. (2019) identified that intervention schools classified as high-implementers demonstrated a significant 56% reduction in the odds of smoking when compared to the control; moderate-implementers a significant 30% reduction; and low-implementers no significant difference from the control condition. This association is mirrored in the literature, with a previous meta-analysis identifying that effect sizes were twice as high for intervention groups with high-levels of implementation as compared to those with low-levels of implementation (Durlak et al., 2011). Herein is emphasized the critical influence of implementation on outcomes. These findings evidence that an intervention evaluated as ineffective may be due to poor implementation of that intervention. This signifies that the intervention has deviated substantially from its intended design and this can lead to misleading conclusions about the intended intervention’s effectiveness (Barry et al., 2019). The findings of this study thus advocate for evaluations of whole-school interventions to not only assess implementation, as many of the included studies have done, but for the active use of this implementation data in the interpretation of evaluation findings. Given the ongoing challenges encountered in the implementation of whole-school interventions, the findings also highlight the need for training and support mechanisms to be incorporated into interventions to bolster implementation.

Consistent with previous literature, a preponderance of studies in this review have assessed implementation in relation to dosage and fidelity (Durlak, 2016). The remaining six components, however, remained largely neglected by studies. This meta-analysis illuminates one component in need of particular attention: the monitoring of the control condition. Only two studies monitored the implementation of preventive activities in the control condition (Dray et al., 2017; de Vries et al., 2006). These found that for several intervention strategies, there was no significant difference in the level of activities implemented by the intervention and control conditions, and that for certain intervention strategies, the control condition implemented significantly more activities than did the intervention condition. It is important to highlight that both of these studies described their control condition as “school-as-usual”. These findings emphasize the school as a complex and adaptive system in which a truly inactive control condition may be challenging to achieve. While a sensitivity analysis removing studies with an active control condition yielded findings that were unchanged from the overall meta-analysis results, the findings indicate that this study may have underestimated the number of studies that have described their control condition as “school-as-usual”, similar to the two studies referenced above, but have inadvertently involved active control conditions that would not have been accounted for in the sensitivity analysis. This means that evaluations of whole-school interventions may effectively be comparing two trial groups that are simply implementing different preventive activities, the level of which may remain unknown to trial authors and readers alike, unless explicitly measured. This study recommends that future evaluations monitor the preventive activities implemented by the control condition that may overlap with the whole-school intervention’s aim, as this would enable a more nuanced and accurate interpretation of evaluation findings. To facilitate this, this study advocates for greater investment in the development of standardized and comprehensive tools to measure implementation. One such tool that was recently developed by Vennegoor et al. (2023) to monitor the implementation of health-promoting schools’ activities is spotlighted. When used in both the intervention and control conditions, the tool can monitor all eight components of implementation and has preliminary evidence in support of its reliability and validity.

The findings of this study should be considered in light of several limitations. The first is the variability observed in the quality of included studies. Though most studies classified as either low or unclear risk for each of the risk-of-bias categories, the exception was Domain 4 (detection bias), where 25 of the 28 studies classified as high risk-of-bias. This was primarily due to the lack of information provided by studies on whether participants were blinded to the trial, in the broader context of the difficulty faced in blinding for this type of intervention. Secondly, while the protocol for this study outlined intentions to assess for publication bias using funnel plots, Egger’s test and trim-and-fill analyses, these could not be conducted due to an insufficient number of studies. Thirteen of the fourteen meta-analyses had less than ten studies, with best practice guidelines advising against these assessments in the case of less than 10 studies due to inadequate statistical power (Higgins et al., 2019). The third pertains to the high heterogeneity (>50%) observed for four of the outcomes meta-analyzed, creating challenges in meaningfully combining the effect estimates within these four outcome domains to draw definitive conclusions. The heterogeneity observed may be explained by various factors, including the inclusion of diverse samples, interventions that typify a whole-school approach in different ways, and the use of varying outcome measures by studies. It is worthwhile noting, however, that for the outcomes identified to be significantly different between the intervention and control conditions, two demonstrated low heterogeneity (cyber-bullying and cyber-aggression) and the other moderate heterogeneity (regular smoking). Traditional methods to further investigate heterogeneity, including subgroup analyses and meta-regression, could not be conducted due to the lack of studies for each outcome domain (Higgins et al., 2019). This similarly meant that this study was unable to address the variability in programme duration and explore its impact on intervention effectiveness. Finally, while previous reviews and the present study have focused on meta-analyzing individual-level outcomes, this study encourages future reviews to consider the inclusion of outcomes at broader levels of the socio-ecological model. For example, six studies reported on school-level outcomes, including school climate and safety (Bond et al., 2004a; Bonell et al., 2018; Cross et al., 2018; Shinde et al., 2018; Hunt et al., 2007; Sawyer et al., 2010a, 2010b). The inclusion of outcomes at the various levels of the socio-ecological model would be more aligned with the theoretical underpinnings of whole-school interventions, given that these interventions aspire for systems-level change.