The purpose of this study was to describe the extent to which EBPIs that have outcomes associated with child emotional/behavioral health and/or parenting include research methods, design, and interpretation strategies that are aligned with best practices for racial equity in research. Concerns associated with lack of relevance of EBPIs across populations, and particularly for minoritized and racialized populations, are well documented (Doyle et al.,
2023; Weisenmuller & Hilton,
2021) and substantial focus has been placed on supporting adaptations of EBPIs for these populations. We posit that, in service to this noble goal, it is imperative for research and evaluation around these programs to be conducted in accordance with CRE best practices. We further believe that adhering to these best practices does not compromise, and may likely enhance, rigor. This study demonstrated that while there is some use of these approaches within the programs and services that meet evidentiary standards for FFPSA, there is substantial room for continued growth. We acknowledge that this study was conducted with a specific focus on programs with policy relevance in the United States. However, this general approach may be relevant to any global social policy that uses evidence to justify implementation at a population level. Research and evaluation activities necessarily happen within the specific context in which they are being conducted. Therefore, it is important to consider the implications for people who may not be represented in the research and identify strategies to enhance the external validity of studies.
Utility of the Evaluation Framework in Research
The coding manual developed for the present study was an amalgamation of several different resources describing how to build racial equity into research, including best practices for conducting research and reporting findings. We hope that the coding manual (Supplemental Appendix I) will provide some guidance to researchers seeking to conduct culturally responsive and race equitable research through each stage, from design and execution through data analysis, interpretation, and dissemination. Considering strategies to address racial equity throughout the entirety of the research process can enhance the external validity of studies without compromising the internal validity of study designs.
Overall, we found the coding framework developed for this study to be feasible. The framework was used with a high degree of interrater reliability; discrepant codes were typically easily resolved through conversation. That said, our coding team was intentionally ‘generous’ in providing credit to studies for using racial equity domains and utilized a non-judgmental approach whereby we were simply exploring the degree to which various CRE practices were employed. Future studies could evaluate the extensiveness or completeness of studies in their incorporation of the various codes throughout the research process.
Most Frequently Observed Racial Equity Domains
The most used racial equity-related approaches observed in the papers and programs reviewed for this study were related to analysis and reporting. This includes assessing the validation of measures for the study sample and reporting participant race/ethnicity and socioeconomic status. This is unsurprising as there has been clear guidance over the past twenty years indicating that this is the best practice for all social science research, not just CRE (Boyd et al.,
2022; Cheng et al.,
2015). Ensuring that study participants are representative of the intended service population was another technique that was relatively more common in our reviewed studies. Historically, there have been critiques of the evidence-based practice literature that studies consisted predominantly of middle-class white participants (Langer et al.,
2021; Roberts et al.,
2020). Our review of studies demonstrates that, at least for the studies of EBPIs designated as having evidentiary support by the Clearinghouse, study participants were frequently from diverse backgrounds and of varied socioeconomic statuses. This finding is aligned with a recent study of the Blueprints for Healthy Youth Development program registry as well, in which most studies reported race (Buckley et al.,
2023). Across studies included in their review, 65% of the study sample consisted of participants identifying as non-White, though there were substantial inconsistencies in how race and ethnicity were reported (Buckley et al.,
2023). Finally, it was relatively common for considerations related to diversity to be included in the conclusion sections of reviewed papers. Paper authors frequently reflected on the overall diversity of the study sample and how that generalizes (or does not) to other populations or concluded additional research was warranted to further understand the generalizability of findings.
Most Infrequently Observed Racial Equity Domains
There were many best practices for racial equity that we attempted to find in the reviewed papers but were never (or only rarely) observed in our sample. Many of the codes that were less frequently observed were those that involved collaboration with community members. For example, use of CBPR/YBPR was nonexistent in our sample. This could also be a function of the type of research that is eligible for review by the Clearinghouse which limits eligibility to only randomized controlled trials (RCTs) and quasi-experimental designs (QEDs) (Wilson et al.,
2019) where CBPR/YBPR approaches remain less common. However, conducting CBPR/YBPR does not preclude use of RCT or QED designs. For example, a recent paper describes the utility of CBPR RCTs in American Indian populations (Rink et al.,
2020). Further, a recent systematic review indicates a substantial increase in the use of CBPR-based methods in recent clinical trials (McFarlane et al.,
2022). This review of ten years of clinical trials revealed a four-fold increase in the number of CBPR-based studies, improvement in population representation, and in general positive treatment effects for around 85% of reviewed studies (McFarlane et al.,
2022).
None of the studies indicated that they had trained evaluators on aspects of cultural competence or humility. The critical importance of ensuring that research staff, including evaluators, are trained and understand the unique cultural contexts of study participants has been extensively described (e.g., Skaff et al.,
2002). It is possible that, for some of the studies reviewed, this was a component of research training that was not described in the manuscript. As it has been described as an ethical imperative for researchers to engage in culturally competent and responsive practice (Woodland et al.,
2021), reporting of such training support can help the research field better understand strategies that improve research engagement and relevance for diverse populations.
None of the papers reviewed described using cognitive interviewing or pilot testing as a step to ensure participant understanding of measures. Study methods such as piloting test measures or identifying strategies to protect against differential attrition are CRE-informed strategies that enhance the internal validity of research studies. Nápoles-Springer et al. (
2006) conducted a study that illustrates the value of cognitive interviewing with a racially and ethnically diverse sample. This study started with 159 potential survey items and closely looked at respondent behavior and conducted interviews. They found 20% of study items were very problematic, exceeding a 15% threshold for concern. A further 79% of items that were below that threshold still had some concerns across one or more problem areas; concerns differed by racial/ethnic group (Nápoles-Springer et al.,
2006). In our own work, we have found that merely translating the language survey questionnaires may unintentionally introduce slight differences in interpretation (e.g., very much can be interpreted as too much) that impact responding.
We found only one paper and study that used specific retention efforts with the intention of reducing disproportionate attrition (Duggan et al.,
2005). This paper described comprehensive efforts including supporting home visiting staff to gain trust, use of incentives, gathering information about friends or other supports who could help find the family within the research window if needed, and collaboration with the post office to notify of address changes. Of course, as aligned with best practices, many studies reported overall attrition, and compared attrition across conditions. However, taking the step to report if attrition varies by different participant characteristics is valuable. It is possible that overall differential attrition is minimal, but a closer look at within-condition attrition could reveal systematic differences. These differences could be related to intervention acceptability, research methods acceptability, or both. One study of depression treatment looked closely at this issue (Murphy et al.,
2013). Study authors found engagement and treatment satisfaction were particularly important predictors of study retention for African Americans in the study sample.
Although still relatively infrequent across our paper sample, six papers representing three programs specifically included racial equity-related analyses as part of their primary research questions (Brief Strategic Family Therapy, Familias Unidas, and Family Check-Up), while two additional programs considered racial equity-related analyses post hoc (Multidimensional Family Therapy and Nurse Family Partnership). We were intentionally liberal in our application of this code, as our team wanted to capture any use of analyses that help differentiate what works for whom, including if programs work equally well across groups. All programs assessing racial equity as a primary research question and one that assessed racial equity post hoc met evidentiary requirements to achieve a “well-supported” designation in the Clearinghouse. Multidimensional Family Therapy, which explored such analyses post hoc, met criteria for “supported.” We did not further code the ways in which racial equity analyses were conducted, but this is an important area for future reviews. For example, to conduct racial equity analyses, care must be taken regarding how data are collected and how the population is understood within the context of the research (Edmonds et al.,
2021). Given we also infrequently observed use of strategies such as having participants self-describe their race/ethnicity and gender and rarely encountered consideration of place within analysis, it is not surprising that relatively few of the studies reviewed were positioned to be able to conduct racial equity analyses using best practices.
Only one study (of the Video Interaction Project) looked at intersectionality of race with other dimensions of identity. Intersectionality recognizes the ways that systemic inequalities related to different social identities often intersect to compound an individual’s experience of these inequalities (Crenshaw,
1991). This overall lack of consideration of intersectionality in evidence-based programming is of concern given that the dynamic intersection of determinants with a participant’s identity is touted as the main causes leading to health disparities. Previously mentioned factors such as poverty/lack of resources, limited access to high-quality health care, adverse life experiences, systems of discrimination, and oppression often interact with gender, race, LGBTQI + status, and ability status placing certain groups at higher risk of negative outcomes (Biglan et al.,
2023; Hawn et al., 2022). The lack of consideration of intersectionality can play out in a variety of ways for evidence-based programs including (1) excluding certain groups from intervention based on disability status, (2) underestimating the logistical challenges in accessing the intervention, (3) dismissing or ignoring major environmental constraints such as toxic physical environments; harmful media and marketing exposure and diminished neighborhood and school quality, and (4) minimizing the impact of trauma experienced by marginalized groups (women, people of color, LGBTQI +) as a result of system of discrimination (Biglan et al.,
2023).
To help ameliorate the issues associated with the impact of intersectionality on outcomes in evidence-based practice, Biglan et al. (
2023) highlight that not only do interventions need to be designed to be more comprehensive but public policy also needs to change. Currently, US-based policies tend to focus prevention efforts more on risk factors and evaluation of single programmatic interventions and not as much on health disparity and equity. Biglan et al. (
2023) propose shifting research focus to experimental evaluation of multiple interventions and the evaluation of comprehensive community interventions that simultaneously address risk factors and determinants of health disparities. With this approach, the detrimental impact of intersectionality on outcomes can be potentially reduced across domains.
Finally, we note that although many papers included considerations of diversity in the conclusions section, we did not observe any papers that specifically noted inclusion of community groups or program participants in the crafting of the conclusions and implications. Including community members at this stage is a best practice within CBPR and the CRE model, though it is one of the least utilized approaches (Cashman et al.,
2008). Involving community members in the dissemination of research findings is similarly important. McDavitt and colleagues describe several core strategies that can be used to ensure a transparent and supportive process, including having a flexible dissemination plan, tailoring presentations to the different audiences, having a community liaison, and following up after presentations. They further highlight the importance of intentionally building rapport and trust within communities toward implementing these strategies (McDavitt et al.,
2016).
Practical Implications of the Findings
In this section, Dr. Chaundrissa Oyeshiku Smith, one of Dr. Prinz’s doctoral students, who is currently a child/adolescent clinical psychologist and program director in a large Health Maintenance Organization (HMO) in the Southeast United States, reflects on the practical implications of this study.
In a clinical decision-maker role, it is critical that there is evidence that programs and services adequately address the needs of children and families. An important component of this is the extent to which these programs and services are relevant to a diverse patient population. Knowing that studies incorporate values aligned with racial equity helps increase confidence in the overall evidence base—it’s not enough to simply know that programs ‘work.’ We must understand what works for whom and under what circumstances. One way to help disentangle this is to embed racial equity practices within research. We cannot take for granted that because a body of evidence exists around a program’s effectiveness that it is necessarily applicable or translatable to another service setting, community, or population. As someone who supports practitioners to make everyday decisions about what services are best for particular clients, helping support their use of research knowledge and increasing their confidence in the validity and relevance of the studies is important. This study shows that there have been some modest improvements in the incorporation of racial equity approaches in research. If this trend continues, it will greatly help build transparency and confidence in the research literature.