The First Step is the Hardest: A Mixed Methods Single-Case Experimental Design Study of a VR-Enhanced Training Program in a Forensic Youth Care Setting

Street Temptations (ST) is an add-on intervention for adolescents with disruptive behavior problems. Adolescents engage in perspective-taking exercises using fictional and real-life scenarios. They create backstories for the characters involved and reflect on connections between observable behavior and internal mental states. This reflective process is the essence of mentalization, as previously explained (Choi-Kain & Gunderson, 2008; Fonagy et al., 1998).

An innovative strength is the use of virtual reality (VR). First, VR aids perspective-taking exercises by incorporating visual elements into therapy, reducing cognitive demands for creating mental representations of situations that are often used for such exercises (Cornet & van Gelder, 2020; van Gelder et al., 2019). This makes VR especially beneficial for ST. Second, this visual support is likely to benefit (part of) the target population, including adolescents with mild-to-borderline intellectual disabilities, who may require visual aids for effective treatment (de Wit et al., 2023; Moonen & Kaal, 2017; Vermaes et al., 2014). Third, VR may also appeal to a tech-savvy generation (Lochman et al., 2019; Weisz et al., 2018), while providing a safe yet realistic environment to challenge perspective-taking in social situations involving disruptive behavior (Bailenson, 2018). Fourth, adolescents may have difficulty engaging in therapeutic conversations (O’Keeffe et al., 2018; Stige et al., 2021). Using VR could help expedite the conversation between therapist and adolescent, having not only to rely on talking strategies. ST’s VR content, and the rest of the program, have been developed in close collaboration with adolescents and practitioners (Klein Schaarsberg et al., 2023, 2024).

By using mentalization as an important mechanism, ST’s primary aim is to challenge adolescents to reflect on both their own and others’ behavior and underlying mental states. In doing so, focus lies on the reinforcement of adolescents’ treatment responsivity, as not all adolescents with disruptive behavior problems seem to profit from existing evidence-based treatment options (Riise et al., 2021). More specifically, it is aimed to enhance two particular treatment responsivity indicators, being motivation for behavior change and self-serving cognitive distortions.

Motivation for behavior change is an important factor for treatment responsiveness (Miller & Rollnick, 2002; Olver et al., 2011). However, in the case of adolescent disruptive behavior, intrinsic motivation for behavior change can be particularly low (Brauers et al., 2016; Englebrecht et al., 2008; Harder et al., 2012; Hillege et al., 2017). Therefore, addressing motivation appears vital for sustainable behavioral changes and treatment success (Carl et al., 2020; Harder, 2018).

To foster motivation for behavior change, one must recognize problematic outcomes of certain behavior, desire different behavior, and be open to seeking help in this instance (Drieschner et al., 2004; Ward et al., 2004). To achieve this, a certain level of self-reflection and insight is required (te Velde et al., 2015). For that reason, as self-reflection and insight are part of mentalization, ST is assumed to be of added value for the enhancement of adolescents’ motivation. Through exercises that challenge adolescents to explicitly consider mental states while attributing meaning to certain behavior, it is attempted to create space for the evaluation of one’s own interpretations and behavior. In that way, ST intends to challenge adolescents to identify problematic outcomes of their own interpretations and behavior, for example, fostering the motivation to change this behavior.

Similarly, such exercises are assumed to be of added value in addressing cognitive distortions. Such self-serving distortions are described as inaccuracies or biases in relation to thoughts, for example, which are used to attribute meaning to behavior. An example is a tendency to interpret ambiguous behavior of someone as hostile, potentially leading to a hostile, inadequate reaction in return (Barriga & Gibbs, 1996; Barriga et al., 2000, 2001, 2008). A negative association between cognitive distortions and treatment response for treatment of aggressive behavior has been found (Smeijers et al., 2018), stating the importance of addressing cognitive distortions in addition to motivation, for instance. It is expected that ST, through taking on others’ perspectives and actively attributing meaning to behavior, will help to challenge possible inaccuracies and biases in this process of attributing meaning as well (Gibbs et al., 2013). In other words, during the ST exercises therapists can help adolescents to see a different perspective, for example, and to identify other interpretations of certain behavior.

Advancing on the expectation explained in the previous paragraph, it may be relevant to take emotion regulation strategies into account as a preliminary treatment outcome of participating in ST. Negative distortions, like the ones discussed, have been found to positively correlate with usage of non-adaptive regulation strategies, for example (Deperrois & Combalbert, 2022). Furthermore, emotion regulation is assumed to be highly relevant in relation to disruptive behavior (Njardvik et al., 2022; te Brinke et al., 2021). Although ST’s focus is not on teaching different emotion regulation skills per se, its primary focus on advancing reflection and perspective taking could translate into improvement of one specific emotion regulation strategy: “reappraisal.” This strategy applies to the meaning or self-relevance that is attributed to a certain situation (Gross, 1998, 2015). Therefore, it was deemed relevant to discover how ST’s focus on reflection and perspective taking, and the application thereof through the regulation strategy “reappraisal”, related to a participant’s perceived control over (impulsive) behavior in moments of high arousal.

The purpose of the present study was to provide a first and thorough exploration of ST’s added value as an innovative VR-based add-on intervention for youth with (serious) disruptive behavior problems. The main focus was on the effects of a mentalization-based approach on motivation for behavior change and cognitive distortions, as important treatment responsivity indicators, as well as on additional effects on emotion regulation as a potential treatment outcome. These effects were studied by applying a single-case experimental design (SCED) in two research settings, combined with additional quantitative measures as well as a qualitative evaluation of the intervention and the VR component more specifically.

A SCED study, involving repeated measurements on individual subjects before, during, and after participating in an intervention, allows individuals to serve as their own controls (Krasny-Pacini & Evans, 2018). Consequently, a SCED study ideally fits clinical research involving small and heterogeneous populations that are difficult to study using standard group designs like the randomized controlled trial. The population targeted by ST fits this description. Additionally, extensively studying a few participants enhances the understanding of individual cases and enables the discovery of individual intervention effects amidst performance variability (Normand, 2016). Altogether, these characteristics ensure that a SCED seems especially suited for this current study. By conducting our SCED study, we aimed to provide valuable insights into ST as an add-on intervention and contribute to the advancement of forensic youth care.

A non-randomized, non-concurrent, multiple-baseline design with replication across single participants was applied (see Fig. 1). Randomization across different baseline lengths was not performed, as baseline lengths would inherently vary due to natural circumstances in clinical practice. Moreover, mandatory waiting would most likely increase the already substantial risk of dropout within this population. Additionally, baselines would naturally be non-concurrent because it was impossible to assign all participants to a therapist at the same time, resulting in varying time lapses between phases across participants in any case. Blinding of participants, ST therapists, and researchers to any component of the study was not used, because this was not feasible due to the nature of the study.

Participants were asked to complete daily repeated measurements during a baseline phase (phase A), an intervention phase (phase B), and a post intervention follow-up phase (phase A’). The baseline phase functioned as a control and was therefore compared with the intervention and follow-up phases. The baseline phase lasted a minimum of 7 days, to keep the baseline length as short as possible to limit the burden on participants while still getting a representative estimate of the outcomes at baseline. The exact start date of the intervention depended on the participants’ presence and the ST therapists’ schedules. The intervention phase lasted at least four weeks, accommodating two ST sessions of 45–60 min per week. When this pace was not achieved, for example, because participants did not attend appointments, the intervention phase was extended. After completing all ST sessions, participants entered the follow-up phase, lasting a final 7 days.

Secondary outcomes were assessed by means of pre-, post-, and follow-up measurements. In addition, separate interviews with participants and ST therapists were performed at the end of the intervention phase. Lastly, 3 and 6 months after ending the study period, participants were approached again for an interview, to evaluate potential long-term effects.

Certain amendments to the original protocol have been made before and during the current study. These amendments followed both from practicalities as from our evolving understanding of the research design in combination with clinical practice. For a detailed description of the amendments, please see Supplementary Material 1.

Adolescents had to meet the following criteria to be eligible for inclusion, assessed by means of clinical judgment: (1) aged between 12 and 18 years, (2) antisocial or externalizing behavioral problems, (3) deficits regarding cognitive distortions or treatment motivation, (4) presence or risk of delinquent behavior, (5) assigned to ST after multidisciplinary consultation, (6) expected stay of at least 2 months, and (7) basic understanding of mobile apps. Clinical judgment, rather than standardized assessment, was chosen to reduce the burden on adolescents and clinicians. Additionally, creating a homogeneous sample—often achievable through standardized assessments—was not a goal of this study. Instead, we deliberately employed a SCED to accommodate the inherent heterogeneity of the population studied. A potential participant who met any of the following exclusion criteria was excluded from participation: (1) severe physical impairment such as deafness or blindness, (2) severe psychiatric problems such as psychosis or high risk of suicide requiring immediate intervention, (3) trauma from serious violence, (4) epilepsy or serious problems regarding motion sickness, and (5) insufficient understanding of the spoken and written Dutch language.

All ST therapists were extensively informed about the study procedures and the in- and exclusion criteria for participants. When an adolescent was referred to ST and was thought to be eligible to participate in the study based on pre-screening, ST therapists informed the adolescent about the study. In case the adolescent was interested and gave oral permission to be approached, an informed consent appointment was planned. Written informed consent was signed when the adolescent agreed to participate in the study. In the case of a minor (12–16 years), parents, or a legal guardian, had to sign a written informed consent as well. Adolescents who did not agree to participate in the study, did not start with ST and received care as usual.

The participants were recruited across two different settings. The first setting entailed a rural (secured) residential youth care facility in the province of South Holland, the Netherlands (residential treatment setting). This location also included a school providing secondary special education, partly connected to the residential care facilities. Participants were recruited from both the (secured) living groups as well as from the school. The other setting entailed a specific department of a rural youth care provider in the province of North Holland, the Netherlands (ambulatory treatment setting). This department provides resocialization trajectories for adolescents with severe behavioral problems who dropped out of school, for example. The populations of both settings are characterized by multiple and complex externalizing behavioral problems, sometimes in combination with internalizing problems. Psychiatric and addiction problems can exist as well. The adolescents often come from unstable and problematic family situations and delinquent behavior can be present.

Within the first setting (residential treatment), 14 adolescents have been screened between September 2021 and October 2023. Of these 14 adolescents, 7 were not approached for study participation due to a negative screening outcome, or because the organization decided that execution of the study was no longer feasible for their organization. Of the other 7 adolescents that were approached, 4 adolescents signed informed consent to participate in the study, of which 3 were included and 1 was not included because parents did not agree with participation of their child.

Participant 1 dropped out during the baseline phase. The intervention could not start because of covid quarantine, and in the meantime the problems of the participant worsened making other care more appropriate. Participant 2 dropped out between inclusion and pre-measurement. He lost his motivation to participate because it took too long to obtain permission from his legal guardian. Participant 3 dropped out during the baseline phase. Her problems worsened throughout the baseline phase, leading to a transfer to another care facility. None of the 3 participants completed measurements in at least two phases, therefore they will not be further discussed.

Within the second setting (ambulatory treatment), 25 adolescents have been screened between November 2022 and December 2023. Of these 25 adolescents, 12 were not approached for study participation due to a negative screening outcome, practical reasons, or because study inclusion had terminated in the meantime. Of the other 13 adolescents that were approached, 5 adolescents signed informed consent to participate in the study, and all 5 were included.

Participant 1 dropped out during the intervention phase. He disappeared for unknown reasons and upon return he was not motivated anymore to continue participation. Participant 2 completed the entire study period. Participant 3 dropped out during the intervention phase, because his motivation declined as it was not allowed that the time spent for ST and the study would shorten his penalty. Participant 4 dropped out during the intervention phase as well because another treatment focus became more important. Participant 5 completed the entire study period. Participants 3 and 4 did not complete measurements in two phases or more, therefore they will not be further discussed. Participants 1, 2, and 5 completed measurements in at least two phases, meaning that these participants will be further discussed.

This study was conducted in accordance with the principles of the World Medical Association Declaration of Helsinki (World Medical Association, 2013) and in accordance with the Medical Research Involving Human Subjects act. Ethics approval for this study was granted by the independent Medical Ethical Committee of Vrije Universiteit medical center (reference number: 2021.0114).

ST consisted of seven sessions that were divided over two modules. Each session lasted between 45 and 60 min and was executed individually. The first session served as an intake, the last session as a closing session. In module 1, several perspective-taking exercises were done, which were based on a 360° VR video (see Fig. 2 for a screen capture). By means of this video, youth were presented with a fictional situation depicting (violent) disruptive behavior. A detailed description of the VR video was presented in our recently published paper (Klein Schaarsberg et al., 2024). Several characters played a role in the story and each of the three sessions centered around one of the characters. The exercises were guided by the therapists, whilst providing room for adolescents to take charge by letting them choose different aspects such as which character is discussed when.

Individual backstories were created for each character, using several building block cards that focused on, for example, family, school, and living situation. These cards presented different questions to help adolescents to create a personal story behind the chosen character, such as “What kind of neighborhood does [character] live in?”, “What does [character] spend the most money on?”, “Is [character] a family person?”, or “With whom does [character] call or text the most?”. The therapists let the adolescents choose different cards and then they guided the adolescents in creating the backstory by asking the questions and providing further prompts. When the backstory had been created, adolescents took on the perspective of the respective character and reflected on their experience of the situation. They were explicitly encouraged to connect visible behavior to inner mental states such as thoughts and emotions, emerging from the created backstory. Following, adolescents were asked to reflect on similarities and differences between the character and themselves regarding the provided situation.

In module 2, adolescents chose a personal experience as a basis for the same perspective-taking exercises. They visualized their experience using a VR street view app (Wander, Parkline Interactive, 2019). This app visually transported adolescents to the exact location where their experience had occurred. Using this visual cue, the adolescents described the situation aloud. While they were immersed in the visualization, the VR glasses streamed to an additional screen, allowing the therapist to observe in real-time. The therapist could also ask questions to support the visualization, for example. For further details regarding ST, we refer to the protocol and the description of ST’s development (Klein Schaarsberg et al., 2022, 2023). For additional information about the equipment used and how procedural fidelity was monitored, please see Supplementary Material 1.

All data was summarized at individual participant level. For the daily repeated measurements, data was visualized graphically using the shiny app Single Case Design website (Bouwmeester, 2021) and inspected accordingly. For this inspection, we deliberately chose a descriptive approach rather than to follow available SCED guidelines for visual analysis such as reported by Lane and Gast (2014) including quantification of change in trend, level, and stability of data assessed within and between conditions. Both the nature and conduct of our study did not lend themselves adequately to the application of these guidelines, as the data contained many missing values, intrapersonal outliers, and fluctuations. Therefore, we decided to describe only the visible patterns and limit quantification of the data. Qualitative outcomes were approached alike and described per participant, to illustrate experiences and processes.

Data collected at pre-, post- and follow-up assessment were analyzed by means of a Reliable Change Index (RCI; Jacobson & Truax, 1991). When the RCI exceeded (-)1.96 it was assumed that a statistically reliable change had occurred between two phases, using a 0.05 significance level (Jacobson & Truax, 1991). Because the HIT provided a standardization that classifies scores as non-clinical, borderline clinical or clinical, we chose to use this standardization to indicate clinical significance rather than to focus on an RCI. No clear norms were available for the remaining questionnaires, so we calculated RCIs for those outcomes. A detailed description of how the RCIs were calculated can be found in Supplementary Material 1.

Jason was a 16-year-old boy, included in the study on January 10, 2023. Jason was proposed to participate in ST because of violent delinquent behavior, both as victim (for which he started trauma therapy) and as offender. There were suspicions of cognitive distortions that might have played a role in his delinquent behavior, particularly in his last offence. He appeared to be motivated for treatment.

Steve was a 16-year-old boy, included on February 13, 2023. Steve was proposed to participate in ST because of substantial problems with perception and interpretation, related to his violent behavior. He appeared to be motivated for participation.

Aron was a 17-year-old boy, included on October 12, 2023. He was proposed to participate in ST because of having trouble empathizing with others and talking mainly from his own point of view. He showed difficulty assessing how things are for someone else, making the perspective changing exercises particularly relevant.

Considering the outcomes of the daily measurements of the three participants collectively, there appeared to be positive changes in cognitive distortions and emotion regulation. However, these findings should be interpreted cautiously as they may be influenced by the study’s limitations. Nonetheless, looking at the items from both constructs, at least one item from each participant showed a positive pattern. Other items showed either no change, or a slightly negative change in pattern. Furthermore, cautiously positive patterns for 2 of the 3 participants in terms of their mood were noted (Jason and Steve). Considering motivation, we could not distill a positive effect of participating in ST. We observed either no change in scores, or a negative change, apart from 1 item from 1 of the participants (Aron).

The pre-, post-, and follow-up assessments of Steve and Aron showed different outcomes. Regarding cognitive distortions, almost all HIT-scores fell within the non-clinical range, indicating little problems. For Steve specifically, some scores had deteriorated at post-assessment, which had improved again at follow-up. On motivation, Steve did not show a statistically reliable change on the ATMQ, whereas Aron showed a statistically reliable increase. This increase was not consistent with his daily measurements. Looking at the RFQY, both participants did not show a statistically reliable change in reflective functioning. Regarding the SRIS-Y, Steve showed a statistically reliable increase in self-reflection. For insight, a statistically reliable increase could be noted from pre to post assessment, followed by a statistically reliable decrease in insight from post to follow-up assessment. Aron showed opposing results, with a statistically reliable decrease on the Self-Reflection subscale and a statistically reliable increase on the Insight subscale. The same was observed for perspective-taking; Steve’s scores on the PT-IRI reflected statistically reliable improvement, whereas Aron’s score indicated a statistically reliable decrease in perspective-taking skills.

Technical issues pertaining to the VR component emerged across the three participants. The main problem was failing attempts of streaming the VR footage to another screen, so a trainer could watch along with a participant. Another issue was stalling of the video when playing it, for example. These issues could not always be fixed on the spot, hindering execution as planned. Therefore, the technological constraints need further development to improve future execution.

Based on the current study and its outcomes, several implications for future directions can be noted. First, ST, as a VR-based add-on intervention, appears to offer relatively simple options to incorporate mentalization into treatment and in that way potentially help to enhance the treatment responsivity of adolescents. To further enhance outcomes, it may be worthwhile to consider other forms of delivering the program. For example, in line with the significant positive effects seen after brief intensive trauma-focused treatment (De Jongh et al., 2020; Voorendonk et al., 2020), it may be interesting to implement a brief intensive ST program that is delivered within 1 to 2 weeks. Moreover, we can note positive outcomes, even when the program is not completed in its entirety. Therefore, but possibly also in a brief intensive form, ST may be of specific value in the context of the often relatively short placements within the judicial, forensic setting (Buysse et al., 2021). Nevertheless, looking at the practical constraints we encountered, options like these would require considerable organizational efforts.

Second, the addition of VR components to treatment within the context of forensic youth care seems to hold promise and therefore deserves further attention. Using VR, especially the street view component, may hold value in other forms of usage as well. For example, it may be relevant for crime analysis, during which somebody gives a factual description of the events that led to a committed crime, as well as one’s thoughts, feelings, and behavior in relation to the crime. As illustrated by our study, the visual support of using VR street view may add to a reflective exercise like this. Also, VR components like the once discussed may be helpful in preparation of confrontations with possible victims, which is gaining interest by means of restorative justice practices (Rossner & Taylor, 2024; Suzuki, 2023). Considering VR, however, it is important to gain a better understanding of how to develop and use VR in a way that is both ethically responsible and suitable for the adolescents aimed at and the situations they can face. This would include a more extensive evaluation of the VR materials used, also consisting of quantitative assessment methods for example. Youth involved in forensic youth care can be used to quite some violence, unfortunately, which can make a relatively mild scenario like the current one not emotionally engaging enough. Therefore, it was suggested to incorporate weapons like guns or knives, for example. However, this is not something that can be done without question; it is not yet clear where the line lies between an emotionally engaging scenario and one that is irresponsible and possibly even harmful. Violent video games, for example, have been found to negatively impact aggression, other problem behaviors, and normative beliefs about aggression (Greitemeyer, 2022; Shao & Wang, 2019; Wei et al., 2022). There is also evidence against these negative effects available, in adults that is (Kühn et al., 2019), but caution is warranted at the very least. In addition, given the boundless opportunities presented by VR and the blurring of boundaries between virtual and real experiences, there exists a moral imperative in the development of VR experiences (Bailenson, 2018; Brey, 1999; Ford, 2001). Therefore, we must reflect on the question whether we should create everything.

Last, for future practice-oriented research it may be particularly beneficial to collaborate with adolescents and practitioners in every step of the research process. By collectively thinking about assessment methods and piloting these methods more extensively, for example, more room can be created to incorporate the language of adolescents themselves instead of primarily relying on therapeutic or academic language. This will likely allow for more idiosyncratically oriented research methods as well. In this way, we can try to get as close as possible to the experiences of the adolescents involved and the ways they express themselves. A good example may be found in the Youth Top Problems measure, for example, a psychometrically sound, client-guided approach to assessment (Weisz et al., 2011).

An important strength of this study was the ecological validity. The study was performed under daily, uncontrolled circumstances. While these conditions do create some limitations, as will be discussed below, they also make sure that the outcomes are of direct value to clinical practice. In addition, the results showed that progress and change can be difficult to translate into a set of numbers. Therefore, our study approach, including different data collection methods, moments, and informants, helped improve our understanding of the effects. Furthermore, this triangulation decreased possible deficiencies resulting from only using self-report questionnaires (Thurmond, 2001).

Despite these strengths, several limitations must be considered as well. One of the strengths, the ecological validity, simultaneously created one of the limitations. The nature of the study prohibits attribution of the observed outcomes to either ST or other specific circumstances or therapies, for example. Moreover, we observed some changes in the daily measurements already occurred during baseline (possibly due to an initial elevation bias, as explained in the beginning of the discussion). As a result, we cannot make convincing statements about whether the visible patterns could be attributed to participation in ST. Lastly, while clinical judgment was chosen to reduce the burden on adolescents and clinicians and to accommodate the population’s heterogeneity, this approach limits the generalizability of the findings to other contexts or populations. Future research should consider incorporating standardized assessments to strengthen the evidence base and to enhance comparability across studies.