Psychometric Properties of the Dutch Contextual Assessment of Social Skills (CASS): An Independent Observational Outcome Measure of Social Skills in Autistic Adolescents

This study involved 99 autistic adolescents who were recruited for the ACCEPT-study to investigate the effectiveness of the Dutch PEERS® intervention, performing a randomized controlled trial (RCT), while utilizing an active control condition i.e., Regulation, Organization, and Autonomy Didactics Training (ROAD) (Idris et al., 2022). The ROAD intervention provides psychoeducation on different adolescence-related themes such as identity/self-acceptance, autonomy in planning activities/schoolwork, physical appearance and changes, regulating emotions, developing friendships, and solo/partnered sexual activities/boundaries. The inclusion and exclusion criteria were (1) having a diagnosis of ASD conform with DSM-IV or DSM-V; (2) enrolment in secondary education; (3) age between 12 and 18 years old; (4) a total and verbal IQ > 70 (assessed with the WISC-IV or WASI); (5) fluent in the Dutch language (verbally and written); (6) motivation to participate in an intervention and in research; (7) no history of major mental illness (e.g., schizophrenia, bipolar disorder, or other types of psychotic disorders) or any visual, hearing or physical impairments that prohibited participation in the study. For further details, please see van Pelt et al (2020) and Idris et al (2022).

The original version of the CASS evaluates participant social skills under two social conditions: (1) an interested condition in which the confederate showed social interest and engagement; and (2) a bored condition in which the confederate demonstrated boredom and disengagement in the conversation. In both conditions, the adolescents had a conversation for three minutes with an unfamiliar confederate, first in the interested condition and then with another confederate in the bored condition (Ratto et al., 2011). We removed the bored condition in agreement with international colleagues (Corbett et al., 2020; Dolan et al., 2016; Rabin et al., 2018). The bored condition was removed from the CASS procedure, consistent with prior studies utilizing the CASS as an outcome measure for the PEERS® intervention (Rabin et al., 2018; White et al., 2015). The PEERS® intervention teaches adolescents to disengage in a situation where to conversational partner looks ‘bored’/uninterested, therefore it was considered an unsuitable and inappropriate outcome measure in the context of a study on the effectiveness of PEERS®. Moreover, the presentation of a “bored” confederate would potentially bias participant expectations and interaction styles for post-test assessment, thus leading to an invalid observation.

Thirteen typically developing adolescents aged between 13 and 23-years-old (male n = 6, female n = 7) were recruited from local schools as confederates in this study. They received a 1-day training from a certified CASS assessor and coder (KGL). All confederates underwent this formal training preceding their participation in the RCT, during which they received instructions and practiced their actual behaviours in role plays with the researchers and experienced confederates. In this study, there was no use of an extra assessment tool to measure confederate adherence. Rather, we used the CASS items to also observe the confederates’ behaviors and check if those corresponded to the instructions given. Investigating differences with the confederates’ behaviors was however not the primary objective of the current paper and was therefore not included in the current analyses. Yet, when using the CASS as the primary outcome measure in our RCT study, the variation in confederate behaviors was considered as a covariate, please see Idris et al., (2022). Importantly, to ensure that all confederates remained in line with the instructions in their interaction styles, we sent a reminder about of the instructions using an instant messaging application i.e., WhatsApp before each assessment time-point. In line with the CASS guidelines, all interactions were with a similarly aged and opposite biological sex peer. We maintained this policy, because, traditionally, one of the important social-emotional developmental tasks in adolescence is to learn to engage in interactions with the opposite biological sex. In our sample in this era (2017–2019), several participants did not identify with one particular biological sex, with 2 participants being in transition from one gender to the other (Idris et al, 2022). Therefore, we made sure to include confederates displaying a range of gender expressions (i.e., girly, tomboys, boyish, as well as transwoman). For more details, please see (Idris et al., 2022; van Pelt et al., 2020).

The CASS recording was used to code the verbal and non-verbal behaviors of both the autistic adolescents and the confederates. Behavioral coding was originally comprised of nine rating items: (1) Asking Questions, (2) Topic Changes, (3) Vocal Expressiveness, (4) Gestures, (5) Positive Affect, (6) Kinesic Arousal, (7) Social Anxiety, (8) Overall Involvement/Interest in the Conversation, and (9) Overall Quality of Rapport. The first two items are rated as frequency counts. The other seven items are rated on a Likert scale of 1 = low, 7 = high. Scores below 6 indicate some level of social skills deficit (Ratto et al., 2011). The original developers of the CASS (Ratto et al., 2011) calculated the total score consisting of four primary items (i.e., Asking Questions, Topic Changes, Overall Involvement, and Overall Quality of Rapport) indicative of the conversational skills to assess social skills of the adolescents. Coding was performed by fifteen undergraduate students. They were established reliability as the CASS coders by coding six training videos (i.e., three original US videos of typically developing young adults, and three new Dutch videos of autistic teenagers). We compared the scores of each coder against the scores of the original developer and the translators (i.e., the coding by A.B. Ratto and the coding by KGL and FtH). The 15 coders achieved at least 80% overall agreement with the original developer and at least 70% agreement on the new Dutch videos. The coders were masked to the time points of the RCT and to the assigned condition (PEERS versus ROAD) to minimize bias. They each viewed and independently coded the videos.

Diagnostic classification was determined using the Autism Diagnostic Observation Schedule Second Version (ADOS-2; De Bildt et al., 2009; Lord et al., 2012). Module 3 and 4 were used in the current study, based on the developmental age as well as on the adolescents’ language abilities. The total calibrated severity score (CSS) comprised of two subscales: Social Affect (SA) and Restricted, Repetitive Behavior (RRB) ranging from 1 to 10. The “Autism Spectrum Disorder” classification includes CSS scores in the range from 4 to 10, and the “Non-spectrum” classification includes CSS scores in the range from 1 to 3. If the ADOS-2 had been administered in the past 5 years, those scores were extracted from the patient file with permission from the parents. If a recent ADOS-2 was not available, a trained and licensed clinician administered the ADOS-2. Please note here that the ADOS-2 was assessed to obtain a more objective index of ASD severity, but it was not deemed necessary for ‘confirmation’ of clinical diagnoses.

Cognitive ability (IQ) If available, information of the IQ was extracted from the patient file with permission from the parents. If the test was administered more than 5 years earlier, the Wechsler Intelligence Scale for Children (WISC-IV; Wechsler, 2003) or the Wechsler Abbreviated Scales of Intelligence (WASI; Wechsler, 1999) was completed.

The RCT participants were recruited from three mental healthcare institutions in the Netherlands that provide specialist outpatient care for autistic individuals. Eligible participants were recruited via three methods; (1) directly referred by psychologists/psychiatrists/pedagogues, (2) other mental health institutions referred participants to either one of the three participating centres, and (3) adolescents/parents applied for participation in the study themselves after reading the information on websites, leaflets/posters or on social media or via referral by their general practitioner. Following referral, adolescents and their parents were contacted by phone to inform them about the study and detailed information was sent to them. The study design is described in more detail in van Pelt et al., 2020. Written informed consent of the adolescents and their parents was obtained for all assessments and for the videotaping of the CASS. The pre-treatment assessments were scheduled 1 week before the commencement of the intervention, while participants were not yet assigned to a condition. Before the CASS started, the adolescent was instructed by a research assistant outside of the assessment room. They were prompted to start and end a natural conversation with the unfamiliar conversational partner (i.e., a trained confederate) to “get to know each other”. The participant entered the room after the instruction, where the confederate was already seated. Then, after 3 min, a research assistant knocked on the door as a sign to end the conversation. After the conversation, the participant left the room. The video recorder was placed 5 feet away from the participant and the confederate. Subsequently, the participants as well as the confederates were asked to complete the brief questionnaire about how they experienced the conversation (i.e., the CRS). The CASS conversations were all videotaped for later behavioral coding.

This was a cross-sectional assessment validation study. It involved three phases (Hall et al., 2017; Tsang et al., 2017): (i) translation of the CASS and CRS from English to the Dutch language, including cultural adaptation (tailoring content); (ii) pre-testing of the Dutch CASS and CRS; and (iii) psychometric evaluation of the Dutch CASS and CRS which included reliability (inter-item correlations, Cronbach’s alphas and confirmatory factor analyses) and validity (i.e. convergent and divergent). Figure 1 describes the entire study process:

The study was approved by the medical ethical commission of the Erasmus Medical Centre, Rotterdam (MEC-NL.57472.078.16) following the Helsinki Declaration of 1975.

Statistical Program Social Sciences version 27.0 and AMOS version 28.0 were used to analyze data (IBM Corp, 2016). Before conducting the primary analyses, the data were examined for accuracy, missing values, outliers, and multivariate assumptions. Descriptive statistics were used to illustrate the demographic and diagnostic characteristics of the participants and responses to the items in the assessment. Inter-item correlations were calculated for all items. We calculated Cronbach’s alphas (α) to assess the reliability of a 4-item versus 7-item total scale, including the original 9 items. High internal consistency represents Cronbach’s alpha of .70–.80. Cronbach’s alpha between .60 and .70 indicated moderate internal consistency while .5–.60 is considered low internal consistency (Field, 2016; Tavakol & Dennick, 2011).

Subsequently, confirmatory factor analyses (CFA) were performed using maximum likelihood estimation to investigate the goodness-of-fit between the methods to calculate the CASS total score summed of 4-item (Rabin et al., 2018; Ratto et al., 2011) and summed of 7-item (Dolan et al., 2016). Several statistics were used: the Chi-square statistics (χ²), the CFI, the TLI, and the RMSEA. Then, the best model was used to calculate the convergent and divergent validity of the CASS. Before the correlations were examined using the Pearson correlation coefficient, the data were checked for any violations of the assumptions of the correlational analyses. No violations were found.

Originally, the CASS consists of nine rating items: (1) Asking Questions, (2) Topic Changes, (3) Vocal Expressiveness, (4) Gestures, (5) Positive Affect, (6) Kinesic Arousal, (7) Social Anxiety, (8) Overall Involvement/Interest in the Conversation, and (9) Overall Quality of Rapport. Besides the above mentioned original nine rating items, the Dutch developers have added four additional items to the D-CASS. The original item 1 was separated between (1a) Initiating and (1b) Follow-up Questions, as these behaviors are specifically instructed within PEERS®. The other three new items are binary coded items (i.e., yes/no), namely (0) Starting the Conversation, (10) Initiating the end of the conversation, (11) Giving a reason to end the conversation. The decision to add the new items was made based on the suggestions from earlier research and international experts to fit more closely to the PEERS® learning objectives. For example, initiating a conversation might depict some form of self-confidence. Floyd and Burgoon (1999) found that participants who initiate the conversation showed the most nonverbal liking behavior responses to the confederate. Obviously, the behaviors of participants at the beginning of a conversation may not tell the whole story. Therefore, we also include the items associated with ending the conversation, especially since these skills are also taught within PEERS.

First, an independent native Dutch language expert carried out the forward translation of the CASS rating manual. Then, the backward translation into the English language was carried out by another independent translator. Besides the translation of the CASS rating manual, the Dutch assessment procedure has also been adjusted. Originally, the procedure was introduced as a role-play. Based on the input of trained confederates from Phase 2, it was decided that introducing it as a natural, ‘getting-to-know-each other’ conversation is less stressful and is feeling more naturally. Second, the two original conditions of the CASS, the interested and the bored condition have been replaced by one interested condition in the Dutch version, consistent with prior trials using the CASS as an outcome for PEERS. Third, the behavioral coding forms of the Dutch CASS provide some space to write down whether a participant asked questions or made statements which were too personal or offensive (verbal content alert). Besides, the forms also provide space to write down some nonverbal inappropriateness (non-verbal behavioral alert) like getting too close, too amicable touches or inappropriate nonverbal behavior for the situation like staring. Fourth, the confederate initiated the conversation after 5-s instead of 10-s in the original CASS. Fifth, after a knock on the door, the adolescents with ASD got the opportunity to finalize the conversation. Finally, the behaviors of the confederates were also rated within the Dutch version. All these elements have been added to the Dutch rating manual by FtH.

The study also extended and adapted the CRS to create the D-CRS. Two items about self-confidence and three items about the perspective taking of the conversational partner on the conversation were added, to align more closely to the learning goals of the PEERS® program. The reason for adding two items on self-confidence is to (a) first measure their feeling/idea of trust in their own ability, probably based on their current self-esteem (potentially increased/decreased during intervention) with the item assessed before the actual conversation, and then (b) to assess their own (potentially more objective) judgement of their actual performance, with the item assessed after the conversation.

The percentage of missing values was minimal (< 5%). Because the number of missing values was small, pairwise exclusion of missing data was used to deal with the missing values. The normality of distributions was inspected using histograms and normal q–q plots. Because the distributions were normal, variable transformations were considered unnecessary. The Mahalanobis distance was used to identify multivariate outliers using p < .001 and no outliers were identified.

Out of 106 autistic adolescents who participated in the RCT, and who completed the initial D-CRS items before the start of the actual conversation, seven adolescents were excluded from the analyses on the D-CASS observational items, because there was no useful D-CASS video recording. Of these seven individuals, two adolescents—after filling out the D-CRS self-confidence item, subsequently both refused to perform the actual conversation, because they had realized they were too anxious/unconfident. For the other five adolescents, we did not have a suitable D-CASS video recording due to technical problems (n = 2) or because of the confederate not turning up in time (n = 3). Therefore, in total, reliable data on the D-CASS observational items were available for n = 99. Table 1 shows the descriptive data from the sample (n = 99) whose conversations were observed and coded.