Embedded Files
How consistent are behavioural flexibility competences over time? How reliable are behavioural measures of task performance, or latent performance factors estimated with mathematical modelling?
Components of this work were funded by the University of Reading
Purpose of the study
Purpose of the study
The study of learning and decision-making processes in psychology and neuroscience typically relies on the use of conditioning tasks to assay behaviour. These include instrumental conditioning tasks where subjects learn through experience the relevant associations between actions and outcomes, such as pulling a slot machine lever to win money. These associations either increase or decrease the likelihood of a given action in the future, depending on whether the outcome was rewarding or punishing. However, these associations also need to be updated flexibly for an individual to respond adaptively to dynamic and changing environments.
This complex competence can be assessed in the lab using behavioural tasks and measuring aspects of performance, such as how quickly people shift away from a previously succesful action when it stops being rewarded. We can also use computational models that try to disentangle the different processes that underpin successful performance, but it is not well understood whether such models are reliable across different participants, or within the same participant over time.
In SERE, we use repeated testing and reinforcement learning models to assess the extent of this reliability and the usefulness of these paradigms for studies of behavioural or physiological change over the timescale of weeks.
Williams, B., FitzGibbon, L., Brady, D., & Christakou, A. (2025). Sample size matters when estimating test–retest reliability of behaviour. Behavior Research Methods, 57(4), 123. https://doi.org/10.3758/s13428-025-02599-1
Intraclass correlation coefficients (ICCs) are a commonly used metric in test–retest reliability research to assess a measure’s ability to quantify systematic between-subject differences. However, estimates of between-subject differences are also influenced by factors including within-subject variability, random errors, and measurement bias. Here, we use data collected from a large online sample (N = 150) to (1) quantify test–retest reliability of behavioural and computational measures of reversal learning using ICCs, and (2) use our dataset as the basis for a simulation study investigating the effects of sample size on variance component estimation and the association between estimates of variance components and ICC measures. In line with previously published work, we find reliable behavioural and computational measures of reversal learning, a commonly used assay of behavioural flexibility. Reliable estimates of between-subject, within-subject (across-session), and error variance components for behavioural and computational measures (with ± .05 precision and 80% confidence) required sample sizes ranging from 10 to over 300 (behavioural median N: between-subject = 167, within-subject = 34, error = 103; computational median N: between-subject = 68, within-subject = 20, error = 45). These sample sizes exceed those often used in reliability studies, suggesting that sample sizes larger than are commonly used for reliability studies (circa 30) are required to robustly estimate reliability of task performance measures. Additionally, we found that ICC estimates showed highly positive and highly negative correlations with between-subject and error variance components, respectively, as might be expected, which remained relatively stable across sample sizes. However, ICC estimates were weakly or not correlated with within-subject variance, providing evidence for the importance of variance decomposition for reliability studies.
Williams, B., Rodriguez-Sobstel, C., FitzGibbon, L., Morriss, J., & Christakou, A. (2024). The influence of trait intolerance of uncertainty on behavioural flexibility (p. 2024.10.30.621039). bioRxiv. https://doi.org/10.1101/2024.10.30.621039
Identifying and responding adaptively to a change in our environment is an essential skill. However, differences in our ability to detect and our disposition to react to these changes mean that some individuals are better equipped to deal with change than others. Here, we investigate whether intolerance of uncertainty, a transdiagnostic dimension of psychopathology, is associated with performance during a reversal learning task. We assessed task performance of 145 subjects using behavioural measures and computational modelling across two time points (approximately 12 days apart). Intolerance of uncertainty and its prospective and inhibitory subscales were associated with task performance, irrespective of self-reported levels of trait anxiety. Intolerance of uncertainty and its inhibitory subscale were positively associated with increased sensitivity to reinforcement from positive, but not negative, feedback. Furthermore, the inhibitory subscale was positively associated with better performance, both overall (as indexed by accuracy) and immediately following a change in reward contingencies (as indexed by perseveration). Lastly, the prospective subscale interacted with the extent to which choice was driven by expected value across time points. These findings provide novel evidence for how trait intolerance of uncertainty may modulate behavioural flexibility in changeable environments. The study points towards exciting avenues for further research into the development of IU-related behaviours across the lifespan and their implications for mental health.
Google Sites
Report abuse