On the stability of canonical correlation analysis and partial least squares with application to brain-behavior associations

Journal article

Associations between datasets can be discovered through multivariate methods like Canonical Correlation Analysis (CCA) or Partial Least Squares (PLS). A requisite property for interpretability and generalizability of CCA/PLS associations is stability of their feature patterns. However, stability of CCA/PLS in high-dimensional datasets is questionable, as found in empirical characterizations. To study these issues systematically, we developed a generative modeling framework to simulate synthetic datasets. We found that when sample size is relatively small, but comparable to typical studies, CCA/PLS associations are highly unstable and inaccurate; both in their magnitude and importantly in the feature pattern underlying the association. We confirmed these trends across two neuroimaging modalities and in independent datasets with n ≈ 1000 and n = 20,000, and found that only the latter comprised sufficient observations for stable mappings between imaging-derived and behavioral features. We further developed a power calculator to provide sample sizes required for stability and reliability of multivariate analyses. Collectively, we characterize how to limit detrimental effects of overfitting on CCA/PLS stability, and provide recommendations for future studies

Authors: Helmer, M; Warrington, S; Mohammadi-Nejad, A-R; Ji, JL; Howell, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Nature Research
• Journal: Communications Biology
• Volume: 7
• Issue: 1
• Pages: 217-217
• Article number: 217
• Publication date: 2024-02-21
• DOI: 10.1038/s42003-024-05869-4
• EISSN: 2399-3642
• ISSN: 2399-3642
• Language: English
• Keywords: Partial least squares regression; Artificial intelligence; Overfitting; Data mining; Correlation; Stability (learning theory); Computer science; Statistics; Pattern recognition (psychology); Interpretability; Multivariate statistics; Machine learning; Generalizability theory; Mathematics; Discriminative model; Canonical correlation; Artificial neural network; Partial correlation