Machine learning prediction of cognition from functional connectivity: Are feature weights reliable?
No Preview Available
Citations
Scopus:
Altmetric:
Altmetric::
Associated Records
Authors
File Licence
Editors
Date
2021-12-15
Source Title
NeuroImage
Journal Title
Journal ISSN
Volume Title
Publisher
ACADEMIC PRESS INC ELSEVIER SCIENCE
Melbourne Authors
Affiliation
Psychiatry
Document Type
Journal Article
Subject Code
Citation
Tian, Y. & Zalesky, A. (2021). Machine learning prediction of cognition from functional connectivity: Are feature weights reliable?. Neuroimage, 245, https://doi.org/10.1016/j.neuroimage.2021.118648.
Access Status
Access this item via the Open Access location
Access URL
Open Access URL
Open Access at PMC
Linked Resource URL
NHMRC Grant Code
ARC Grant Code
Description
Abstract
Cognitive performance can be predicted from an individual's functional brain connectivity with modest accuracy using machine learning approaches. As yet, however, predictive models have arguably yielded limited insight into the neurobiological processes supporting cognition. To do so, feature selection and feature weight estimation need to be reliable to ensure that important connections and circuits with high predictive utility can be reliably identified. We comprehensively investigate feature weight test-retest reliability for various predictive models of cognitive performance built from resting-state functional connectivity networks in healthy young adults (n=400). Despite achieving modest prediction accuracies (r=0.2-0.4), we find that feature weight reliability is generally poor for all predictive models (ICC< 0.3), and significantly poorer than predictive models for overt biological attributes such as sex (ICC≈0.5). Larger sample sizes (n=800), the Haufe transformation, non-sparse feature selection/regularization and smaller feature spaces marginally improve reliability (ICC< 0.4). We elucidate a tradeoff between feature weight reliability and prediction accuracy and find that univariate statistics are marginally more reliable than feature weights from predictive models. Finally, we show that measuring agreement in feature weights between cross-validation folds provides inflated estimates of feature weight reliability. We thus recommend for reliability to be estimated out-of-sample, if possible. We argue that rebalancing focus from prediction accuracy to model reliability may facilitate mechanistic understanding of cognition with machine learning approaches.