Optimizing working memory assessment: development of shortened versions of complex spans, updating, and binding tasks

Working memory (WM) is a system with limited capacity that temporarily stores, manipulates, and retrieves information necessary for ongoing cognitive processes (Baddeley, 2012; Unsworth et al., 2009a). Individual differences in working memory capacity (WMC) are major predictors of high-order capacities, such as fluid intelligence (Gf) (Felez-Nobrega et al., 2018; Rey-Mermet et al., 2019) rationality (Burgoyne et al., 2023), and problem-solving (Friedman & Miyake, 2004). Additionally, WMC influences real-world behaviours, like multitasking (Colom et al., 2010) and emotion regulation (Barkus, 2020). Given the central role of WM in cognition, assessing this construct is ubiquitous in experimental and cognitive psychology (Conway et al., 2005). However, WM is also often measured in clinical, educational, and social investigations (Allen et al., 2020; Li et al., 2019; Mazerolle et al., 2012).

Multiple classes of cognitive tasks are used to measure WMC. Among them, the complex span is the most prevalent (Conway et al., 2005; Redick et al., 2012). However, there has been a recent increase in the use of binding and updating tasks (Bartsch et al., 2018; Waris et al., 2017; Wilhelm et al., 2013). Complex spans are dual tasks where participants must remember a series of stimuli presented in quick succession while performing a secondary task (Redick et al., 2012; Unsworth et al., 2009a, b). Updating tasks require participants to continuously refresh their mental representations (Ecker et al., 2010). Binding tasks involve combining different aspects of stimuli, such as their position and verbal content, to build new structures and associations (Oberauer et al., 2003). Although these task paradigms have different structures and features, they measure the same underlying construct — they loaded on the same latent factor in multiple studies (Lewandowsky et al., 2010; Schmiedek et al., 2009, 2014; Waris et al., 2017; Wilhelm et al., 2013). Moreover, they all serve as good predictors of Gf (Engle et al., 1999; Gray et al., 2017; Shelton et al., 2009).

However, most WM tasks take a long time to administer (Foster et al., 2015; Gonthier et al., 2016). For instance, completing a single complex span can take around 20 min (Unsworth et al., 2005). Many updating and binding tasks require approximately the same amount of time (Stollery & Christian, 2016; Waris et al., 2021), although some tests within these paradigms have shorter administration times (7 ~ 10 min) (Garcia et al., 2014; Kattner, 2021). These lengthy procedures can lead to long and tedious experimental sessions, potentially decreasing participant motivation and negatively impacting performance (Heitz et al., 2008). Furthermore, researchers often need to assess multiple constructs within limited timeframes, making it challenging to include several measures to evaluate all areas of interest (Stanton et al., 2002).

Balancing the limited amount of time to collect data and the extended administration times of most WM tasks has led many investigations to use a single test to measure WMC (Ma et al., 2017; Oswald et al., 2015). However, relying on a single task can result in biased interpretations, as no cognitive test is a perfect representation of the construct it is supposed to measure. Performance on these tasks is influenced not only by the construct of interest but also by specific features of the paradigm and task (e.g., the structure of the task or the content domain of the stimuli) and other forms of measurement error (Shipstead et al., 2012; Unsworth et al., 2009a). It is not possible to disentangle these sources of variance by applying a single task. Consequently, it becomes challenging to determine whether the individual differences detected by the test are mainly driven by the construct of interest or by unrelated sources of variation (Foster et al., 2015; Lewandowsky et al., 2010). Repeatedly using the same task in multiple studies can further confound these sources of variance, increasing the risk of interpreting task-specific variance as construct variance on a broader scale (Schmiedek et al., 2014).

To obtain a more accurate estimate of WMC, several authors recommended administering multiple WM tasks and either extracting the common variance at a latent level or computing a composite score by averaging the scores of all tests (Conway et al., 2005; Hicks et al., 2016). Estimates based on multiple tests from different paradigms and content domains (e.g., verbal, numeric, visuospatial) provide particularly accurate representations of WMC, as they allow the separation of construct-relevant variance from paradigm- and task-specific variance and measurement error (Gonthier et al., 2016; Waris et al., 2017). Additionally, using heterogeneous sets of valid measures that differ in construct-irrelevant features tends to lead to better estimates of the target construct than applying sets of similar tasks, as the former covers wider sources of relevant variance (Little et al., 1999). Schmiedek’s et al. (2014) suggested that this premise can be applied to WM tests. The outcomes of this investigation revealed that triplets of tasks from different classes correlated more strongly with a Gf factor than trios of tests from the same paradigm (e.g., updating tasks). However, given the prominence of the complex span in WM literature, a large number of studies used multiple versions of this paradigm to assess WMC (Engle et al., 1999; Kane et al., 2004; Unsworth et al., 2009b).

Applying multiple tests to assess WMC naturally increases the duration of data collection sessions (Felez-Nobrega et al., 2018). To address this issue, some researchers have developed and validated shortened versions of WM tasks. For example, Foster et al. (2015) divided three complex spans (reading, symmetry, and rotation spans) into blocks representing early, intermediate, and later trials. They then assessed the ability of different combinations of blocks and tasks to predict a set of Gf factor scores. Their findings indicated that the early trial blocks alone were sufficient to obtain a valid and reliable estimate of WMC — the first third of trials from the three complex spans was still able to account for 90% of the Gf explained by the full-length versions of these tasks. Moreover, these shortened versions of the three tasks produced a better WMC estimate than the complete version of any complex span alone. The shortened complex spans took approximately 43 min to complete, compared to 62 min for the full-length versions. In their turn, Oswald et al. (2015) created reduced versions of the reading, operation, and symmetry spans by removing all blocks of trials with the largest and smallest set sizes and the third block from all set sizes. These versions required 15–25 min less to administer than the original versions, which took approximately 39 min to complete. At last, Gonthier et al. (2016) developed a Composite Complex Span which also included shortened versions of the reading, operation, and symmetry spans. These reduced versions included one block of trials for the lowest and highest set sizes and two blocks of trials for all other set sizes. In general, these three sets of shortened WM tasks demonstrated strong internal and criterion validity, high internal consistency, and temporal stability, while reducing testing time by approximately 31–38%.

However, these studies have focused exclusively on shortening tasks within the complex span paradigm. Considering the advantages of deriving WMC estimates from different classes of tests and the limited amount of time researchers often have for data collection, creating shortened versions of WM tasks from different paradigms could have major benefits. These reduced versions would allow researchers to obtain reliable WMC estimates, free from paradigm- and task-variance, at a reduced time cost.

In this study, we employed a methodology similar to that of Foster et al. (2015) to evaluate whether WM tasks from different paradigms (complex span, updating tasks, binding tasks) could be shortened while maintaining good psychometric properties (internal consistency, and internal and criterion validity). To test this hypothesis, our participants completed the versions of the reading span, operation span, symmetry span, n-back task, working memory updating task, and binding and maintenance task included in the OpenWMB (Monteiro et al., 2024) as well as three reasoning tests. The OpenWMB is an open-source and automated battery containing multiple complex spans, updating, and binding tasks. The battery can be freely downloaded from the GitHub repository associated with the webpage https://​doi.​org/​10.​5281/​zenodo.​10600494 — to access this repository, locate and click on the GitHub URL presented on the Zenodo page. The WM tests were shortened through an iterative process, whose goal was to reduce each task to the minimum number of trials needed to achieve an α value ≥ 0.70 (Adadan & Savasci, 2012; McDonald, 1999). The remaining trials from each task were grouped into equal-sized blocks representing intermediate and later trials. To determine if shortening the WM tasks affected their internal and criterion validity, we compared the percentage of variance in a set of WMC and Gf factor scores explained by the reduced version of the WM tasks (which corresponded to the early blocks of trials from the complete WM tests) to that explained by the versions of the tasks that also included the intermediate and later blocks of trials.

Finally, we examined whether the findings of Schmiedek et al. (2014), which suggested that administering multiple WM tasks from different paradigms provided better WMC estimates than only applying tasks from the same class of tests, were replicated with our sample. We analyzed whether sets of tasks and blocks that only included tests from different paradigms had a better capacity to predict a set of Gf factor scores than combinations exclusively comprised of tasks from the same category.

Due to the wide range of issues addressed in this study, we have divided this section into several segments. Each segment details the analyses conducted to explore each research question and the respective results. Descriptive statistics for the shortened and full-length versions of the WM tasks and the Gf measures are presented in Table 2⁷. This table also includes the correlations between these tests. The statistics derived from the shortened versions of the WM tasks are displayed in parentheses.

The primary goal of this study was to determine whether WM tasks from the complex span (reading, operation, and symmetry spans), updating task (n-back and working memory updating tasks), and binding task (binding and maintenance task) paradigms could be effectively shortened and preserve robust psychometric proprieties. Our analysis demonstrated that the observed α values of the shortened WM tasks met or surpassed the conventional threshold for acceptable internal consistency (all αs ≥ 0.70) (Adadan & Savasci, 2012; McDonald, 1999). These values were comparable to, and in some cases exceeded, those reported in previous studies (Felez-Nobrega et al., 2018; Foster et al., 2015; Gonthier et al., 2016; Oswald et al., 2015). Additionally, the shortened tasks accounted for the vast majority of the variance in a set of WMC factor scores derived from the full-length versions of the WM tests (R² = 0.90). Furthermore, there was no significant difference between the shortened and full-length versions in their ability to predict Gf. These findings were consistent with the results reported by Foster et al. (2015). The shortened WM tasks reduced the total administration time by approximately 35%, requiring only 56 min to complete compared to 86.7 min for the full-length versions.

A notable finding from this study is that updating and binding tasks can be effectively reduced without compromising their psychometric integrity. To the best of our knowledge, this is the first study that evaluated this possibility. Previous studies with similar objectives primarily focused on complex spans (Felez-Nobrega et al., 2018; Foster et al., 2015; Gonthier et al., 2016; Oswald et al., 2015). Complex spans are highly reliable and valid tests, with numerous investigations employing either a single (Al-Rashidi et al., 2023; Unsworth & Engle, 2005) or multiple variants (Burgoyne et al., 2023; Engle et al., 1999; Kane et al., 2004) of this paradigm to measure WMC. However, estimating WMC based on a single task or paradigm risks conflating variation due to individual differences in WMC with task- and paradigm-specific variance (Lewandowsky et al., 2010; Schmiedek et al., 2014). Administering multiple tasks from different paradigms helps to separate these confounding influences from pure WMC variance, resulting in a more accurate measurement of this construct (Waris et al., 2017; Wilhelm et al., 2013). However, this approach can be time-consuming, as the participants must read different sets of instructions and adapt to various task structures. By creating shortened versions of WM tasks from different paradigms, we provide researchers with a method to obtain robust WMC estimates free of task- and paradigm-specific variance, at a reduced time cost.

It is important to note that using all shortened WM tasks is not necessary for an accurate WMC measurement. Depending on the specific aims and time available for task administration, different combinations of shortened and/or full-length WM tasks can be used to obtain reliable WMC estimates in a time-efficient manner. To assist researchers in selecting the configuration most suited to their needs, we computed a permutation analysis in which we assessed the efficacy of every possible combination of blocks and tasks. This analysis builds on the work of Foster et al. (2015), who examined how well different combinations of blocks of trials (early, early + intermediate, and early + intermediate + later) derived from three complex span tasks predicted a set of Gf factor scores and calculated the time required to complete each combination. However, they only tested tasks from a single paradigm and administered these tests in a fixed order, limiting the number of combinations they could assess (39). We included six tasks from three different paradigms in this analysis. Additionally, we did not have to account for possible order effects as we counterbalanced the order of presentation of the tasks in our study. This allowed us to test 2303 models.

The results of the permutation analysis suggested that employing the complete versions of the reading span, binding and maintenance task, and working memory updating task provided the best WMC estimate in the least amount of time based on the criterion that we adopted — only combinations that were able to explain 90% of the variance in both WMC and Gf factor scores were considered acceptable measures of WMC. This set of tasks took less than half the time to complete (40.7 min) compared to the entire set of full-length WM tasks (86.7 min) and was able to predict 90% of the WMC variance and 93% of the Gf variance accounted for by the full model. Thus, our findings suggested that researchers who intend to extract the most amount of WMC variance in the least amount of time should apply this set of tasks.

However, we would like to point out that this conclusion is based on the criterion adopted in this study. As previously stated, optimal task selection should be guided by the specific objectives and constraints of each investigation. Considering this, we provided two resources (the Excel file ‘Original_Results_Permutation_Analysis.xlsx,’ located at https://​doi.​org/​10.​5281/​zenodo.​13362470 and the interactive web application hosted in https://​shortwmtasks.​shinyapps.​io/​Results_​Permutation_​Analysis/​) that provide detailed information to help researchers identify the most suitable combination of tasks for their study. Researchers can use these resources to tailor their selection based on available testing time, the desired amount of Gf and WMC variance explained, the targeted WM components, and the inclusion or exclusion of specific tasks.

We have previously outlined examples detailing how task selection can be optimized based on the time available for task administration. For instance, if only 20 min are available, the permutation analysis indicates that the most effective approach would involve implementing the initial block of the binding and maintenance task alongside the complete version of the working memory updating task. Conversely, if the researcher has 40 min to administrate the WM tasks, the analysis suggests that the optimal combination would include the initial block of the reading span task, as well as the full-length versions of the n-back task, the binding and maintenance task, and the working memory updating task.

If the goal of the investigation is to obtain a multi-indicator estimate of the updating component of WM, the results of the permutation analysis indicated that administering the shortened versions of the n-back task and the working memory updating task is enough to account for 97% of the WMC and Gf variance explained by their full-length counterparts while reducing testing time by approximately 5 min. Alternatively, if the objective is to assess the ability to simultaneously store and process information, the researcher can account for 90% of the WMC and Gf variance explained by the full-length versions of the three complex span tasks by implementing only the early and intermediate blocks of the symmetry span and the full-length operation span. This approach saves approximately 24 min.

Thus, the results of the permutation analysis provide valuable insights for researchers, highlighting ways to achieve robust WMC estimates while minimizing testing time. Furthermore, they offer more efficient alternatives to the prevailing approach of estimating WMC with single or multiple versions of the complex span (Burgoyne et al., 2023; Engle et al., 1999; Al-Rashidi et al., 2023; Kane et al., 2004; Unsworth & Engle, 2005).

In addition to offering new methods to improve the time efficiency of WMC assessment, this study also contributes to other questions related to the optimal administration of WM tasks. For example, we explored whether heterogeneous combinations of tests from different paradigms and homogenous sets of tasks from the same class differed in their ability to predict Gf — a commonly used method to assess the criterion validity of WM tasks (Lewandowsky et al., 2010; Schmiedek et al., 2009). Our analysis revealed that heterogeneous combinations provided better representations of WMC than homogenous ones. These results align with the findings of Schmiedek et al. (2014). The superior predictive ability of heterogeneous combinations may be explained by the inclusion of indicators that covered wider sources of construct-relevant variation (Little et al., 1999): complex spans involve simultaneous storage and processing activities (Redick et al., 2012; Unsworth et al., 2009a, b), updating tasks require ongoing renewal of mental representations (Ecker et al., 2010), and binding tasks involve associating multiple stimulus features to create new relationships (Oberauer et al., 2003). Combinations that only included a single paradigm probably overlooked some of these dimensions. Moreover, as previously stated, using heterogeneous combinations helps to distinguish construct-relevant from task- and paradigm-specific variance, allowing for a more precise interpretation of WMC measurements (Gonthier et al., 2016; Waris et al., 2017).

One particularly surprising finding was that the working memory updating task predicted both WMC and Gf factor scores more accurately than any other measure. This was unexpected, given that the WMC factor scores were derived from a set of tests including three complex spans and a single version of the working memory updating task. This suggested that the updating processes assessed by this task may be particularly important for estimating both WMC and Gf (Ecker et al., 2010). Traditionally, the complex span has been the most widely used paradigm to measure WMC (Conway et al., 2005; Redick et al., 2012). However, our analyses indicate that the working memory updating task may provide more accurate estimates of WMC than any complex span.

While our research offers new perspectives for WMC assessment, it is important to acknowledge some limitations that may have influenced our findings. Future research might address these gaps to refine and build upon our results.

For instance, we only administered the WM tasks once in their full-length form. These tests were then segmented into blocks representing early, intermediate, and later trials. The shortened versions of the WM tasks were based on the blocks containing the early trials of each test. Thus, we did not administer the shortened versions by themselves. It would be important to assess whether the shortened WM tasks present similar levels of reliability and validity as those reported in this article when administered in isolation. One way to examine this would be to implement a split-sample design. In this approach, the sample would be divided into two groups: one group would complete both the full-length and shortened versions of the WM tasks, while the other would only complete the shortened versions. The first group would allow for direct comparisons of the psychometric properties between the two formats within the same individuals. On the other hand, the second group would provide insights into the reliability and validity of the shortened tasks in conditions more closely resembling their intended use in future studies. Moreover, it would be interesting to extend this approach to diverse populations, such as children, the elderly, and clinical groups, to determine if the shortened WM tasks provide reliable and valid WMC estimates across different populations.

The single administration of the shortened WM tasks introduces another potential limitation: while this version effectively explained the vast majority of the variance in the full-length tasks, supporting their utility in cross-sectional studies that seek to assess WMC, their ability to capture significant within-person variability over time remains uncertain. If the shortened tasks can reliably capture such variations, they could be particularly valuable for longitudinal research in cognitive psychology (e.g., behavioral studies investigating diurnal fluctuations in WM performance (Schmidt et al., 2015; Van Eekelen & Kerkhof, 2003) and neuroscience (e.g., neuroimaging investigations seeking to build personalized brain and behavior models to identify psychiatric biomarkers (Kraus et al., 2023; Makowski et al., 2024)). The shortened WM tasks could serve as a time-efficient alternative for repeated assessments of WM performance, facilitating the detection of intra-individual changes over time while maintaining the quality of the measurements and minimizing participant fatigue. To address this limitation, future research should explore whether the short WM tasks are able to effectively capture within-person variability in WM performance if they are administered multiple times within the same day and across multiple days.

Another aspect that needs to be addressed in future studies is the uneven distribution of paradigms in this study: our participants completed three complex spans, two updating tasks, and a single binding task. As a result, the factor scores derived from these tasks were probably not the most balanced representation of WMC. Considering the features of these paradigms, it is possible that these scores were overly influenced by the ability to simultaneously process and store information while underrepresenting the capacity to bind different stimulus features. This limitation may explain why the binding and maintenance task had the lowest ability to predict the WMC factor scores.

Another drawback of applying only a single binding task was the inability to confirm whether the robust metrics of the reduced version of the binding and maintenance task could be generalized to other tests within this paradigm. Future studies should explore this issue further.

Beyond the tasks themselves, another aspect that needs to be considered is the optimization of task instructions. The time required to read these instructions often consumes a significant portion of the administration time (Foster et al., 2015). In our study, 95% of participants took approximately 15 min to read the instructions, constituting 17% of the total administration time for the full-length tests and 27% for the shortened versions. Future research should examine whether streamlining the instructions can reduce reading time without sacrificing clarity.

Traditionally, assessing WMC relies on extensive cognitive tests comprising numerous trials (Stollery & Christian, 2016; Unsworth et al., 2005; Waris et al., 2021). The reason for including such a high number of trials is that most WM tasks were initially developed as independent measures capable of providing reliable and valid estimates of WMC on their own (Gonthier et al., 2016). However, our findings suggested that it is possible to substantially shorten tests from multiple paradigms and preserve their good psychometric proprieties. Implementing the shortened versions resulted in a 35% reduction in administration time, equivalent to a time savings of approximately 30 min. By creating reduced versions of WM tasks from different paradigms, we offer researchers an efficient solution to obtain accurate WMC estimates free of task- and paradigm-specific variance, at a reduced time cost. Additionally, the results of the permutation analysis identified less time-consuming combinations of shortened and/or full-length tasks that yield more robust WMC estimates than the conventional approach of using either a single or multiple versions of the complex span. We believe that these shortened WM tasks will be highly valuable for researchers, enabling more time to assess additional constructs and enhancing participant recruitment through quicker data collection.