Skip to main content

Advertisement

Springer Nature Link
Account
Menu
Find a journal Publish with us Track your research
Search
Cart
  1. Home
  2. Machine Intelligence Research
  3. Article

Multimodal Fusion of Brain Imaging Data: Methods and Applications

  • Review
  • Open access
  • Published: 15 January 2024
  • Volume 21, pages 136–152, (2024)
  • Cite this article

You have full access to this open access article

Download PDF
Machine Intelligence Research Aims and scope Submit manuscript
Multimodal Fusion of Brain Imaging Data: Methods and Applications
Download PDF
  • Na Luo  ORCID: orcid.org/0000-0002-2500-20831 na1,
  • Weiyang Shi  ORCID: orcid.org/0000-0002-1805-68841 na1,
  • Zhengyi Yang1,
  • Ming Song1 &
  • …
  • Tianzi Jiang  ORCID: orcid.org/0000-0001-9531-291X1,2,3,4 

  • 5190 Accesses

  • 26 Citations

  • 17 Altmetric

  • 2 Mentions

  • Explore all metrics

Abstract

Neuroimaging data typically include multiple modalities, such as structural or functional magnetic resonance imaging, diffusion tensor imaging, and positron emission tomography, which provide multiple views for observing and analyzing the brain. To leverage the complementary representations of different modalities, multimodal fusion is consequently needed to dig out both inter-modality and intra-modality information. With the exploited rich information, it is becoming popular to combine multiple modality data to explore the structural and functional characteristics of the brain in both health and disease status. In this paper, we first review a wide spectrum of advanced machine learning methodologies for fusing multimodal brain imaging data, broadly categorized into unsupervised and supervised learning strategies. Followed by this, some representative applications are discussed, including how they help to understand the brain arealization, how they improve the prediction of behavioral phenotypes and brain aging, and how they accelerate the biomarker exploration of brain diseases. Finally, we discuss some exciting emerging trends and important future directions. Collectively, we intend to offer a comprehensive overview of brain imaging fusion methods and their successful applications, along with the challenges imposed by multi-scale and big data, which arises an urgent demand on developing new models and platforms.

Article PDF

Download to read the full article text

Similar content being viewed by others

Multimodal Fusion of Structural and Functional Brain Imaging Data

Chapter © 2016

Multivariate Approaches for Multimodal Fusion of Structural and Functional Brain Imaging Data

Chapter © 2025

Persistent Feature Analysis of Multimodal Brain Networks Using Generalized Fused Lasso for EMCI Identification

Chapter © 2020

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.
  • Brain Mapping
  • Brain-machine Interface
  • Diffusion Tensor Imaging
  • Functional magnetic resonance imaging
  • Imaging Techniques
  • Magnetic Resonance Imaging
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

References

  1. J. Sui, T. Adali, Q. B. Yu, J. Y. Chen, V. D. Calhoun. A review of multivariate methods for multimodal fusion of brain imaging data. Journal of Neuroscience Methods, vol.204, no. 1, pp.68–81, 2012. DOI: https://doi.org/10.1016/j.jneumeth.2011.10.031.

    Article  Google Scholar 

  2. T. Z. Jiang. Brainnetome: A new-ome to understand the brain and its disorders. Neuroimage, vol.80, pp. 263–272, 2013. DOI: https://doi.org/10.1016/j.neuroimage.2013.04.002.

    Article  Google Scholar 

  3. S. Parisot, B. Glocker, S. I. Ktena, S. Arslan, M. D. Schirmer, D. Rueckert. A flexible graphical model for multi-modal parcellation of the cortex. Neurolmage, vol.162, pp. 226–248, 2017. DOI: https://doi.org/10.1016/j.neuroimage.2017.09.005.

    Article  Google Scholar 

  4. L. Ma, Y. Zhang, H. T. Zhang, L. Q. Cheng, J. J. Zhuo, W. Y. Shi, Y. H. Lu, W. Li, Z. Y. Yang, J. J. Wang, L. Z. Fan, T. Z. Jiang. BAI-Net: Individualized anatomical cerebral cartography using graph neural network. IEEE Transactions on Neural Networks and Learning Systems, to be published. DOI: https://doi.org/10.1109/TNNLS.2022.3213581.

  5. J. Sui, R. T. Jiang, J. Bustillo, V. Calhoun. Neuroimaging-based individualized prediction of cognition and behavior for mental disorders and health: Methods and promises. Biological Psychiatry, vol.88, no. 11, pp.818–828, 2020. DOI: https://doi.org/10.1016/j.biopsych.2020.02.016.

    Article  Google Scholar 

  6. R. T. Jiang, V. D. Calhoun, Y. Cui, S. L. Qi, C. J. Zhuo, J. Li, R. Jung, J. Yang, Y. H. Du, T. Z. Jiang, J. Sui. Multimodal data revealed different neurobiological correlates of intelligence between males and females. Brain Imaging and Behavior, vol.14, no. 5, pp. 1979–1993, 2020. DOI: https://doi.org/10.1007/s11682-019-00146-z.

    Article  Google Scholar 

  7. A. C. Feng, N. Luo, W. T. Zhao, V. D. Calhoun, R. T. Jiang, D. M. Zhi, W. Y. Shi, T. Z. Jiang, S. Yu, Y. Xu, S. Liu, J. Sui. Multimodal brain deficits shared in early-onset and adult-onset schizophrenia predict positive symptoms regardless of illness stage. Human Brain Mapping, vol.43, no. 11, pp. 3486–3497, 2022. DOI: https://doi.org/10.1002/hbm.25862.

    Article  Google Scholar 

  8. N. Luo, J. Sui, J. Y. Chen, F. Q. Zhang, L. Tian, D. D. Lin, M. Song, V. D. Calhoun, Y. Cui, V. M. Vergara, F. F. Zheng, J. Y. Liu, Z. Y. Yang, N. M. Zuo, L. Z. Fan, K. B. Xu, S. F. Liu, J. Li, Y. Xu, S. Liu, L. X. Lv, J. Chen, Y. C. Chen, H. Guo, P. Li, L. Lu, P. Wan, H. N. Wang, H. L. Wang, H. Yan, J. Yan, Y. F. Yang, H. X. Zhang, D. Zhang, T. Z. Jiang. A schizophrenia-related genetic-brain-cognition pathway revealed in a large Chinese population. Ebiomedicine, vol.37, pp.471–482, 2018. DOI: https://doi.org/10.1016/j.ebiom.2018.10.009.

    Article  Google Scholar 

  9. W. Y. Shi, L. Z. Fan, H. Y. Wang, B. Liu, W. Li, J. Li, L. Q. Cheng, C. Y. Chu, M. Song, J. Sui, N. Luo, Y. Cui, Z. W. Dong, Y. H. Lu, Y. W. Ma, L. Ma, K. X. Li, J. Chen, Y. C. Chen, H. Guo, P. Li, L. Lu, L. X. Lv, P. Wan, H. N. Wang, H. L. Wang, H. Yan, J. Yan, Y. F. Yang, H. X. Zhang, D. Zhang, T. Z. Jiang. Two subtypes of schizophrenia identified by an individual-level atypical pattern of tensor-based morphometric measurement. Cerebral Cortex, vol.33, no. 7, pp.3683–3700, 2023. DOI: https://doi.org/10.1093/cercor/bhac301.

    Article  Google Scholar 

  10. N. Luo, X. S. Luo, S. L. Zheng, D. R. Yao, M. Zhao, Y. Cui, Y. Zhu, V. D. Calhoun, L. Sun, J. Sui. Aberrant brain dynamics and spectral power in children with ADHD and its subtypes. European Child & Adolescent Psychiatry, to be published. DOI: https://doi.org/10.1007/s00787-022-02068-6.

  11. B. Wang, A. M. Mezlini, F. Demir, M. Fiume, Z. W. Tu, M. Brudno, B. Haibe-Kains, A. Goldenberg. Similarity network fusion for aggregating data types on a genomic scale. Nature Methods, vol.11, no. 3, pp. 333–337, 2014. DOI: https://doi.org/10.1038/nmeth.2810.

    Article  Google Scholar 

  12. F. Bao, Y. Deng, S. Wan, S. Q. Shen, B. Wang, Q. H. Dai, S. J. Altschuler, L. F. Wu. Integrative spatial analysis of cell morphologies and transcriptional states with MUSE. Nature Biotechnology, vol.40, no.8, pp. 1200–1209, 2022. DOI: https://doi.org/10.1038/s41587-022-01251-z.

    Article  Google Scholar 

  13. H. Hotelling. Relations between two sets of variates. Biometrika, vol.28, no.3–4, pp.321–377, 1936. DOI: https://doi.org/10.1093/biomet/28.3-4.321.

    Article  Google Scholar 

  14. Y. O. Li, T. Adali, W. Wang, V. D. Calhoun. Joint blind source separation by multiset canonical correlation analysis. IEEE Transactions on Signal Processing, vol. 57, no. 10, pp. 3918–3929, 2009. DOI: https://doi.org/10.1109/TSP.2009.2021636.

    Article  MathSciNet  Google Scholar 

  15. D. D. Lin, V. D. Calhoun, Y. P. Wang. Correspondence between fMRI and SNP data by group sparse canonical correlation analysis. Medical Image Analysis, vol. 18, no. 6, pp.891–902, 2014. DOI: https://doi.org/10.1016/j.media.2013.10.010.

    Article  Google Scholar 

  16. X. W. Zhang, J. Pan, J. Shen, Z. U. Din, J. L. Li, D. W. Lu, M. X. Wu, B. Hu. Fusing of electroencephalogram and eye movement with group sparse canonical correlation analysis for anxiety detection. IEEE Transactions on Affective Computing, vol.13, no. 2, pp. 958–971, 2022. DOI: https://doi.org/10.1109/TAFFC.2020.2981440.

    Article  Google Scholar 

  17. Z. M. Zhang, Z. D. Deng. A kernel canonical correlation analysis based idle-state detection method for SSVEP-based brain-computer interfaces. Advanced Materials Research, vol. 341–342, pp. 634–640, 2011. DOI: https://doi.org/10.4028/www.scientific.net/AMR.341-342.634.

    Google Scholar 

  18. J. Sui, H. He, G. D. Pearlson, T. Adali, K. A. Kiehl, Q. B. Yu, V. P. Clark, E. Castro, T. White, B. A. Mueller, B. C. Ho, N. C. Andreasen, V. D. Calhoun. Three-way (N-way) fusion of brain imaging data based on mCCA+jICA and its application to discriminating schizophrenia. Neurolmage, vol.66, pp. 119–132, 2013. DOI: https://doi.org/10.1016/j.neuroimage.2012.10.051.

    Article  Google Scholar 

  19. S. L. Qi, V. D. Calhoun, T. G. M. van Erp, J. Bustillo, E. Damaraju, J. A. Turner, Y. H. Du, J. Yang, J. Y. Chen, Q. B. Yu, D. H. Mathalon, J. M. Ford, J. Voyvodic, B. A. Mueller, A. Belger, S. McEwen, S. G. Potkin, A. Preda, T. Z. Jiang, J. Sui. Multimodal fusion with reference: Searching for joint neuromarkers of working memory deficits in schizophrenia. IEEE Transactions on Medical Imaging, vol.37, no. 1, pp.93–105, 2018. DOI: https://doi.org/10.1109/TMI.2017.2725306.

    Article  Google Scholar 

  20. V. D. Calhoun, T. Adali, N. R. Giuliani, J. J. Pekar, K. A. Kiehl, G. D. Pearlson. Method for multimodal analysis of independent source differences in schizophrenia: Combining gray matter structural and auditory oddball functional data. Human Brain Mapping, vol.27, no. 1, pp.47–62, 2006. DOI: https://doi.org/10.1002/hbm.20166.

    Article  Google Scholar 

  21. M. A. B. S. Akhonda, B. Gabrielson, S. Bhinge, V. D. Calhoun, T. Adali. Disjoint subspaces for common and distinct component analysis: Application to the fusion of multi-task FMRI data. Journal of Neuroscience Methods, vol.358, Article number 109214, 2021. DOI: https://doi.org/10.1016/j.jneumeth.2021.109214.

  22. J. Y. Liu, O. Demirci, V. D. Calhoun. A parallel independent component analysis approach to investigate genomic influence on brain function. IEEE Signal Processing Letters, vol.15, pp. 413–416, 2008. DOI: https://doi.org/10.1109/LSP.2008.922513.

    Article  Google Scholar 

  23. V. M. Vergara, A. Ulloa, V. D. Calhoun, D. Boutte, J. Y. Chen, J. J. N. Liu. A three-way parallel ICA approach to analyze links among genetics, brain structure and brain function. Neurolmage, vol.98, pp.386–394, 2014. DOI: https://doi.org/10.1016/j.neuroimage.2014.04.060.

    Article  Google Scholar 

  24. S. L. Qi, R. F. Silva, D. Q. Zhang, S. M. Plis, R. Miller, V. M. Vergara, R. T. Jiang, D. M. Zhi, J. Sui, V. D. Calhoun. Three-way parallel group independent component analysis: Fusion of spatial and spatiotemporal magnetic resonance imaging data. Human Brain Mapping, vol.43, no.4, pp. 1280–1294, 2022. DOI: https://doi.org/10.1002/hbm.25720.

    Article  Google Scholar 

  25. A. R. Groves, C. F. Beckmann, S. M. Smith, M. W. Woolrich. Linked independent component analysis for multimodal data fusion. Neurolmage, vol.54, no. 3, pp. 2198–2217, 2011. DOI: https://doi.org/10.1016/j.neuroimage.2010.09.073.

    Article  Google Scholar 

  26. W. K. Gong, C. F. Beckmann, S. M. Smith. Phenotype discovery from population brain imaging. Medical Image Analysis, vol.71, Article number 102050, 2021. DOI: https://doi.org/10.1016/j.media.2021.102050.

  27. W. K. Gong, S. Bai, Y. Q. Zheng, S. M. Smith, C. F. Beckmann. Supervised phenotype discovery from multimodal brain imaging. IEEE Transactions on Medical Imaging, vol.42, no.3, pp.834–849, 2023. DOI: https://doi.org/10.1109/TMI.2022.3218720.

    Article  Google Scholar 

  28. T. Adali, M. A. B. S. Akhonda, V. D. Calhoun. ICA and IVA for data fusion: An overview and a new approach based on disjoint subspaces. IEEE Sensors Letters, vol.3, no. 1, Article number 7100404, 2019. DOI: https://doi.org/10.1109/LSENS.2018.2884775.

  29. R. F. Silva, S. M. Plis, T. Adalı, M. S. Pattichis, V. D. Calhoun. Multidataset independent subspace analysis with application to multimodal fusion. IEEE Transactions on Image Processing, vol. 30, pp. 588–602, 2021. DOI: https://doi.org/10.1109/TIP.2020.3028452.

    Article  MathSciNet  Google Scholar 

  30. B. B. Avants, N. J. Tustison, J. R. Stone. Similarity-driven multi-view embeddings from high-dimensional biomedical data. Nature Computational Science, vol.1, no. 2, pp. 143–152, 2021. DOI: https://doi.org/10.1038/s43588-021-00029-8.

    Article  Google Scholar 

  31. A. Kumar, P. Rai, H. Daumé. Co-regularized multi-view spectral clustering. In Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, pp. 1413–1421, 2011.

  32. D. Y. Xie, X. D. Zhang, Q. X. Gao, J. L. Han, S. Xiao, X. B. Gao. Multiview clustering by joint latent representation and similarity learning. IEEE Transactions on Cybernetics, vol.50, no. 11, pp.4848–4854, 2020. DOI: https://doi.org/10.1109/tcyb.2019.2922042.

    Article  Google Scholar 

  33. G. E. Hinton, R. S. Zemel. Autoencoders, minimum description length and Helmholtz free energy. In Proceedings of the 6th International Conference on Neural Information Processing Systems, Denver, USA, pp. 3–10, 1993.

  34. R. Miotto, F. Wang, S. Wang, X. Q. Jiang, J. T. Dudley. Deep learning for healthcare: Review, opportunities and challenges. Briefings in Bioinformatics, vol.19, no. 6, pp. 1236–1246, 2018. DOI: https://doi.org/10.1093/bib/bbx044.

    Article  Google Scholar 

  35. T. L. Ma, A. D. Zhang. Integrate multi-omics data with biological interaction networks using multi-view factorization autoencoder (MAE). BMC Genomics, vol.20, no. S11, Article number 944, 2019. DOI: https://doi.org/10.1186/s12864-019-6285-x.

  36. D. Q. Zhang, D. G. Shen. The Alzheimer’s Disease Neuroimaging Initiative. Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease. Neurolmage, vol.59, no. 2, pp. 895–907, 2012. DOI: https://doi.org/10.1016/j.neuroimage.2011.09.069.

    Article  MathSciNet  Google Scholar 

  37. B. Jie, D. Q. Zhang, B. Cheng, D. G. Shen. The Alzheimer’s Disease Neuroimaging Initiative. Manifold regularized multitask feature learning for multimodality disease classification. Human Brain Mapping, vol. 36, no. 2, pp. 489–507, 2015. DOI: https://doi.org/10.1002/hbm.22642.

    Article  Google Scholar 

  38. L. Xiao, J. M. Stephen, T. W. Wilson, V. D. Calhoun, Y. P. Wang. A manifold regularized multi-task learning model for IQ prediction from two fMRI paradigms. IEEE Transactions on Biomedical Engineering, vol.67, no.3, pp. 796–806, 2020. DOI: https://doi.org/10.1109/TBME.2019.2921207.

    Article  Google Scholar 

  39. M. X. Liu, D. Q. Zhang, D. G. Shen. Relationship induced multi-template learning for diagnosis of Alzheimer’s disease and mild cognitive impairment. IEEE Transactions on Medical Imaging, vol.35, no.6, pp. 1463–1474, 2016. DOI: https://doi.org/10.1109/TMI.2016.2515021.

    Article  Google Scholar 

  40. S. M. Plis, D. R. Hjelm, R. Salakhutdinov, E. A. Allen, H. J. Bockholt, J. D. Long, H. J. Johnson, J. S. Paulsen, J. A. Turner, V. D. Calhoun. Deep learning for neuroimaging: A validation study. Frontiers in Neuroscience, vol. 8, Article number 229, 2014. DOI: https://doi.org/10.3389/fnins.2014.00229.

  41. W. Z. Yan, G. Qu, W. X. Hu, A. Abrol, B. Cai, C. Qiao, S. M. Plis, Y. P. Wang, J. Sui, V. D. Calhoun. Deep learning in neuroimaging: Promises and challenges. IEEE Signal Processing Magazine, vol.39, no. 2, pp.87–98, 2022. DOI: https://doi.org/10.1109/MSP.2021.3128348.

    Article  Google Scholar 

  42. Z. Zhang, Q. Zhu, G. S. Xie, Y. Chen, Z. M. Li, S. H. Wang. Discriminative margin-sensitive autoencoder for collective multi-view disease analysis. Neural Networks, vol.123, pp. 94–107, 2020. DOI: https://doi.org/10.1016/j.neunet.2019.11.013.

    Article  Google Scholar 

  43. G. Andrew, R. Arora, J. Bilmes, K. Livescu. Deep canonical correlation analysis. In Proceedings of the 30th International Conference on Machine Learning, Atlanta, USA, vol.3, pp. 1247–1255, 2013.

    Google Scholar 

  44. S. M. Gross, R. Tibshirani. Collaborative regression. Biostatistics, vol.16, no.2, pp.326–338, 2015. DOI: https://doi.org/10.1093/biostatistics/kxu047.

    Article  MathSciNet  Google Scholar 

  45. W. X. Hu, B. Cai, A. Y. Zhang, V. D. Calhoun, Y. P. Wang. Deep collaborative learning with application to the study of multimodal brain development. IEEE Transactions on Biomedical Engineering, vol. 66, no. 12, pp. 3346–3359, 2019. DOI: https://doi.org/10.1109/TBME.2019.2904301.

    Article  Google Scholar 

  46. W. X. Hu, X. H. Meng, Y. T. Bai, A. Y. Zhang, G. Qu, B. Cai, G. M. Zhang, T. W. Wilson, J. M. Stephen, V. D. Calhoun, Y. P. Wang. Interpretable multimodal fusion networks reveal mechanisms of brain cognition. IEEE Transactions on Medical Imaging, vol.40, no.5, pp. 1474–1483, 2021. DOI: https://doi.org/10.1109/TMI.2021.3057635.

    Article  Google Scholar 

  47. Y. Z. Chen, J. D. Yan, M. X. Jiang, T. Zhang, Z. B. Zhao, W. H. Zhao, J. Zheng, D. Z. Yao, R. Zhang, K. M. Kendrick, X. Jiang. Adversarial learning based node-edge graph attention networks for autism spectrum disorder identification. IEEE Transactions on Neural Networks and Learning Systems, to be published. DOI: https://doi.org/10.1109/TNNLS.2022.3154755.

  48. D. R. Yao, J. Sui, M. L. Wang, E. K. Yang, Y. Jiaerken, N. Luo, P. T. Yap, M. X. Liu, D. G. Shen. A mutual multi-scale triplet graph convolutional network for classification of brain disorders using functional or structural connectivity. IEEE Transactions on Medical Imaging, vol.40, no.4, pp. 1279–1289, 2021. DOI: https://doi.org/10.1109/tmi.2021.3051604.

    Article  Google Scholar 

  49. A. C. Evans, A. L. Janke, D. L. Collins, S. Baillet. Brain templates and atlases. Neurolmage, vol.62, no. 2, pp. 911–922, 2012. DOI: https://doi.org/10.1016/j.neuroimage.2012.01.024.

    Article  Google Scholar 

  50. B. Thirion, G. Varoquaux, E. Dohmatob, J. B. Poline. Which fMRI clustering gives good brain parcellations? Frontiers in Neuroscience, vol.8, Article number 167, 2014. DOI: https://doi.org/10.3389/fnins.2014.00167.

  51. K. Brodmann, Vergleichende Lokalisationslehre der Grosshirnrinde in Ihren Prinzipien Dargestellt auf Grund des Zellenbaues, Leipzig, Germany: Barth, 1909.

  52. Q. X. Wang, S. L. Ding, Y. Li, J. Royall, D. Feng, P. Lesnar, N. Graddis, M. Naeemi, B. Facer, A. Ho, T. Dolbeare, B. Blanchard, N. Dee, W. Wakeman, K. E. Hirokawa, A. Szafer, S. M. Sunkin, S. W. Oh, A. Bernard, J. W. Phillips, M. Hawrylycz, C. Koch, H. K. Zeng, J. A. Harris, L. Ng. The Allen mouse brain common coordinate framework: A 3D reference atlas. Cell, vol.181, no. 4, pp.936–953.e20, 2020. DOI: https://doi.org/10.1016/j.cell.2020.04.007.

    Article  Google Scholar 

  53. K. Amunts, H. Mohlberg, S. Bludau, K. Zilles. Julich-Brain: A 3D probabilistic atlas of the human brain’s cytoarchitecture. Science, vol.369, no.6506, pp.988–992, 2020. DOI: https://doi.org/10.1126/science.abb4588.

    Article  Google Scholar 

  54. B. T. T. Yeo, F. M. Krienen, J. Sepulcre, M. R. Sabuncu, D. Lashkari, M. Hollinshead, J. L. Roffman, J. W. Smoller, L. Zöllei, J. R. Polimeni, B. Fischl, H. S. Liu, R. L. Buckner. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of Neurophysiology, vol. 106, no. 3, pp. 1125–1165, 2011. DOI: https://doi.org/10.1152/jn.00338.2011.

    Article  Google Scholar 

  55. L. Z. Fan, H. Li, J. J. Zhuo, Y. Zhang, J. J. Wang, L. F. Chen, Z. Y. Yang, C. Y. Chu, S. M. Xie, A. R. Laird, P. T. Fox, S. B. Eickhoff, C. S. Yu, T. Z. Jiang. The human brainnetome atlas: A new brain atlas based on connectional architecture. Cerebral Cortex, vol.26, no. 8, pp. 3508–3526, 2016. DOI: https://doi.org/10.1093/cercor/bhw157.

    Article  Google Scholar 

  56. W. Li, L. Z. Fan, W. Y. Shi, Y. H. Lu, J. Li, N. Luo, H. Y. Wang, C. Y. Chu, L. Ma, M. Song, K. X. Li, L. Q. Cheng, L. Cao, T. Z. Jiang. Brainnetome atlas of preadolescent children based on anatomical connectivity profiles. Cerebral Cortex, vol.33, no.9, pp.5264–5275, 2023. DOI: https://doi.org/10.1093/cercor/bhac415.

    Article  Google Scholar 

  57. Y. Xiao, Y. Lin, J. J. Ma, J. H. Qian, Z. J. Ke, L. F. Li, Y. Y. Yi, J. B. Zhang, Z. J. Dai. Predicting visual working memory with multimodal magnetic resonance imaging. Human Brain Mapping, vol.42, no.5, pp. 1446–1462, 2021. DOI: https://doi.org/10.1002/hbm.25305.

    Article  Google Scholar 

  58. G. Qu, L. Xiao, W. X. Hu, J. Q. Wang, K. Zhang, V. D. Calhoun, Y. P. Wang. Ensemble manifold regularized multi-modal graph convolutional network for cognitive ability prediction. IEEE Transactions on Biomedical Engineering, vol.68, no. 12, pp.3564–3573, 2021. DOI: https://doi.org/10.1109/TBME.2021.3077875.

    Article  Google Scholar 

  59. N. U. F. Dosenbach, B. Nardos, A. L. Cohen, D. A. Fair, J. D. Power, J. A. Church, S. M. Nelson, G. S. Wig, A. C. Vogel, C. N. Lessov-Schlaggar, K. A. Barnes, J. W. Dubis, E. Feczko, R. S. Coalson, J. R. Jr.Pruett, D. M. Barch, S. E. Petersen, B. L. Schlaggar. Prediction of individual brain maturity using fMRI. Science, vol.329, no.5997, pp. 1358–1361, 2010. DOI: https://doi.org/10.1126/science.1194144.

    Article  Google Scholar 

  60. J. H. Cole, K. Franke. Predicting age using neuroimaging: Innovative brain ageing biomarkers. Trends in Neurosciences, vol.40, no. 12, pp.681–690, 2017. DOI: https://doi.org/10.1016/j.tins.2017.10.001.

    Article  Google Scholar 

  61. R. T. Jiang, D. Scheinost, N. M. Zuo, J. Wu, S. L. Qi, Q. H. Liang, D. M. Zhi, N. Luo, Y. C. Chung, S. Liu, Y. Xu, J. Sui, V. Calhoun. A neuroimaging signature of cognitive aging from whole-brain functional connectivity. Advanced Science, vol.9, no. 24, Article number 2201621, 2022. DOI: https://doi.org/10.1002/advs.202201621.

  62. R. A. I. Bethlehem, J. Seidlitz, S. R. White, J. W. Vogel, K. M. Anderson, C. Adamson, S. Adler, G. S. Alexopoulos, E. Anagnostou, A. Areces-Gonzalez, D. E. Astle, B. Auyeung, M. Ayub, J. Bae, G. Ball, S. Baron-Cohen, R. Beare, S. A. Bedford, V. Benegal, F. Beyer, J. Blangero, M. Blesa Cábez, J. P. Boardman, M. Borzage, J. F. Bosch-Bayard, N. Bourke, V. D. Calhoun, M. M. Chakravarty, C. Chen, C. Chertavian, G. Chetelat, Y. S. Chong, J. H. Cole, A. Corvin, M. Costantino, E. Courchesne, F. Crivello, V. L. Cropley, J. Crosbie, N. Crossley, M. Delarue, R. Delorme, S. Desrivieres, G. A. Devenyi, M. A. Di Biase, R. Dolan, K. A. Donald, G. Donohoe, K. Dunlop, A. D. Edwards, J. T. Elison, C. T. Ellis, J. A. Elman, L. Eyler, D. A. Fair, E. Feczko, P. C. Fletcher, P. Fonagy, C. E. Franz, L. Galan-Garcia, A. Gholipour, J. Giedd, J. H. Gilmore, D. C. Glahn, I. M. Goodyer, P. E. Grant, N. A. Groenewold, F. M. Gunning, R. E. Gur, R. C. Gur, C. F. Hammill, O. Hansson, T. Hedden, A. Heinz, R. N. Henson, K. Heuer, J. Hoare, B. Holla, A. J. Holmes, R. Holt, H. Huang, K. Im, J. Ipser, C. R. JackJr, A. P. Jackowski, T. Jia, K. A. Johnson, P. B. Jones, D. T. Jones, R. S. Kahn, H. Karlsson, L. Karlsson, R. Kawashima, E. A. Kelley, S. Kern, K. W. Kim, M. G. Kitzbichler, W. S. Kremen, F. Lalonde, B. Landeau, S. Lee, J. Lerch, J. D. Lewis, J. Li, W. Liao, C. Liston, M. V. Lombardo, J. Lv, C. Lynch, T. T. Mallard, M. Marcelis, R. D. Markello, S. R. Mathias, B. Mazoyer, P. McGuire, M. J. Meaney, A. Mechelli, N. Medic, B. Misic, S. E. Morgan, D. Mothersill, J. Nigg, M. Q. W. Ong, C. Ortinau, R. Ossenkoppele, M. Ouyang, L. Palaniyappan, L. Paly, P. M. Pan, C. Pantelis, M. M. Park, T. Paus, Z. Pausova, D. Paz-Linares, A. Pichet Binette, K. Pierce, X. Qian, J. Qiu, A. Qiu, A. Raznahan, T. Rittman, A. Rodrigue, C. K. Rollins, R. Romero-Garcia, L. Ronan, M. D. Rosenberg, D. H. Rowitch, G. A. Salum, T. D. Satterthwaite, H. L. Schaare, R. J. Schachar, A. P. Schultz, G. Schumann, M. Schöll, D. Sharp, R. T. Shinohara, I. Skoog, C. D. Smyser, R. A. Sperling, D. J. Stein, A. Stolicyn, J. Suckling, G. Sullivan, Y. Taki, B. Thyreau, R. Toro, N. Traut, K. A. Tsvetanov, N. B. Turk-Browne, J. J. Tuulari, C. Tzourio, É. Vachon-Presseau, M. J. Valdes-Sosa, P. A. Valdes-Sosa, S. L. Valk, T. van Amelsvoort, S. N. Vandekar, L. Vasung, L. W. Victoria, S. Villeneuve, A. Villringer, P. E. Vertes, K. Wagstyl, Y. S. Wang, S. K. Warfield, V. Warrier, E. Westman, M. L. Westwater, H. C. Whalley, A. V. Witte, N. Yang, B. Yeo, H. Yun, A. Zalesky, H. J. Zar, A. Zettergren, J. H. Zhou, H. Ziauddeen, A. Zugman, X. N. Zuo, 3R-BRAIN, AIBL, Alzheimer’s Disease Neuroimaging Initiative, Alzheimer’s Disease Repository Without Borders Investigators, CALM Team, Cam-CAN, CCNP, COBRE, cVEDA, ENIGMA Developmental Brain Age Working Group, Developing Human Connectome Project, FinnBrain, Harvard Aging Brain Study, IMAGEN, KNE96, The Mayo Clinic Study of Aging, NSPN, POND, The PREVENT-AD Research Group, VETSA, E. T. Bullmore, A. F. Alexander-Bloch. Brain charts for the human lifespan. Nature, vol. 604, no. 7906, pp. 525–533, 2022. DOI: https://doi.org/10.1038/s41586-022-04554-y.

    Article  Google Scholar 

  63. N. Luo, J. Sui, A. Abrol, D. D. Lin, J. Y. Chen, V. M. Vergara, Z. N. Fu, Y. H. Du, E. Damaraju, Y. Xu, J. A. Turner, V. D. Calhoun. Age-related structural and functional variations in 5 967 individuals across the adult lifespan. Human Brain Mapping, vol.41, no. 7, pp. 1725–1737, 2020. DOI: https://doi.org/10.1002/hbm.24905.

    Article  Google Scholar 

  64. F. Liem, G. Varoquaux, J. Kynast, F. Beyer, S. Kharabian Masouleh, J. M. Huntenburg, L. Lampe, M. Rahim, A. Abraham, R. C. Craddock, S. Riedel-Heller, T. Luck, M. Loeffler, M. L. Schroeter, A. V. Witte, A. Villringer, D. S. Margulies. Predicting brain-age from multimodal imaging data captures cognitive impairment. Neurolmage, vol. 148, pp. 179–188, 2017. DOI: https://doi.org/10.1016/j.neuroimage.2016.11.005.

    Article  Google Scholar 

  65. D. A. Engemann, O. Kozynets, D. Sabbagh, G. Lemaître, G. Varoquaux, F. Liem, A. Gramfort. Combining magnetoencephalography with magnetic resonance imaging enhances learning of surrogate-biomarkers. eLife, vol.9, Article number e54055, 2020. DOI: https://doi.org/10.7554/eLife.54055.

  66. X. Niu, F. Q. Zhang, J. Kounios, H. L. Liang. Improved prediction of brain age using multimodal neuroimaging data. Human Brain Mapping, vol.41, no.6, pp. 1626–1643, 2020. DOI: https://doi.org/10.1002/hbm.24899.

    Article  Google Scholar 

  67. D. Hu, H. Zhang, Z. W. Wu, F. Wang, L. Wang, J. K. Smith, W. L. Lin, G. Li, D. G. Shen. Disentangled-multimodal adversarial autoencoder: Application to infant age prediction with incomplete multimodal neuroimages. IEEE Transactions on Medical Imaging, vol.39, no. 12, pp. 4137–4149, 2020. DOI: https://doi.org/10.1109/TMI.2020.3013825.

    Article  Google Scholar 

  68. Y. C. Shi, Z. Wang, P. D. Chen, P. Y. Cheng, K. Zhao, H. X. Zhang, H. Shu, L. H. Gu, L. J. Gao, Q. Wang, H. S. Zhang, C. M. Xie, Y. Liu, Z. J. Zhang. Alzheimer’s Disease Neuroimaging Initiative. Episodic memory-related imaging features as valuable biomarkers for the diagnosis of Alzheimer’s Disease: A multicenter study based on machine learning. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, vol.8, no. 2, pp. 171–180, 2023. DOI: https://doi.org/10.1016/j.bpsc.2020.12.007.

    Google Scholar 

  69. S. G. Liang, Y. F. Li, Z. Zhang, X. Z. Kong, Q. Wang, W. Deng, X. J. Li, L. S. Zhao, M. L. Li, Y. J. Meng, F. Huang, X. H. Ma, X. M. Li, A. J. Greenshaw, J. M. Shao, T. Li. Classification of first-episode schizophrenia using multimodal brain features: A combined structural and diffusion imaging study. Schizophrenia Bulletin, vol.45, no.3, pp. 591–599, 2019. DOI: https://doi.org/10.1093/schbul/sby091.

    Article  Google Scholar 

  70. Z. Y. Ning, Q. Xiao, Q. J. Feng, W. F. Chen, Y. Zhang. Relation-induced multi-modal shared representation learning for Alzheimer’s disease diagnosis. IEEE Transactions on Medical Imaging, vol.40, no.6, pp. 1632–1645, 2021. DOI: https://doi.org/10.1109/TMI.2021.3063150.

    Article  Google Scholar 

  71. S. Zheng, Z. F. Zhu, Z. Z. Liu, Z. Y. Guo, Y. Liu, Y. C. Yang, Y. Zhao. Multi-modal graph learning for disease prediction. IEEE Transactions on Medical Imaging, vol.41, no. 9, pp. 2207–2216, 2022. DOI: https://doi.org/10.1109/TMI.2022.3159264.

    Article  Google Scholar 

  72. T. Tong, K. Gray, Q. Q. Gao, L. Chen, D. Rueckert. Multi-modal classification of Alzheimer’s disease using nonlinear graph fusion. Pattern Recognition, vol.63, pp. 171–181, 2017. DOI: https://doi.org/10.1016/j.patcog.2016.10.009.

    Article  Google Scholar 

  73. S. F. Liu, H. Y. Wang, M. Song, L. X. Lv, Y. Cui, Y. Liu, L. Z. Fan, N. M. Zuo, K. B. Xu, Y. H. Du, Q. B. Yu, N. Luo, S. L. Qi, J. Yang, S. M. Xie, J. Li, J. Chen, Y. C. Chen, H. N. Wang, H. Guo, P. Wan, Y. F. Yang, P. Li, L. Lu, H. Yan, J. Yan, H. L. Wang, H. X. Zhang, D. Zhang, V. D. Calhoun, T. Z. Jiang, J. Sui. Linked 4-way multimodal brain differences in schizophrenia in a large Chinese Han population. Schizophrenia Bulletin, vol.45, no. 2, pp. 436–449, 2019. DOI: https://doi.org/10.1093/schbul/sby045.

    Article  Google Scholar 

  74. Y. F. Zang, T. Z. Jiang, Y. L. Lu, Y. He, L. X. Tian. Regional homogeneity approach to fMRI data analysis. Neurolmage, vol.22, no. 1, pp.394–400, 2004. DOI: https://doi.org/10.1016/j.neuroimage.2003.12.030.

    Article  Google Scholar 

  75. N. Luo, L. Tian, V. D. Calhoun, J. Y. Chen, D. D. Lin, V. M. Vergara, S. Q. Rao, J. Yang, C. J. Zhuo, Y. Xu, J. A. Turner, F. Q. Zhang, J. Sui. Brain function, structure and genomic data are linked but show different sensitivity to duration of illness and disease stage in schizophrenia. Neurolmage: Clinical, vol.23, Article number 101887, 2019. DOI: https://doi.org/10.1016/j.nicl.2019.101887.

  76. S. Kinreich, V. V. McCutcheon, F. Aliev, J. L. Meyers, C. Kamarajan, A. K. Pandey, D. B. Chorlian, J. Zhang, W. P. Kuang, G. Pandey, S. S. S. D. Viteri, M. W. Francis, G. Chan, J. L. Bourdon, D. M. Dick, A. P. Anokhin, L. Bauer, V. Hesselbrock, M. A. Schuckit, J. I. Nurnberger, T. M. Foroud, J. E. Salvatore, K. K. Bucholz, B. Porjesz. Predicting alcohol use disorder remission: A longitudinal multimodal multi-featured machine learning approach. Translational Psychiatry, vol.11, no. 1, Article number 166, 2021. DOI: https://doi.org/10.1038/s41398-021-01281-2.

  77. Y. Y. Luo, T. L. Alvarez, J. M. Halperin, X. B. Li. Multimodal neuroimaging-based prediction of adult outcomes in childhood-onset ADHD using ensemble learning techniques. Neurolmage: Clinical, vol. 26, Article number 102238, 2020. DOI: https://doi.org/10.1016/j.nicl.2020.102238.

  78. M. Song, Y. Yang, J. H. He, Z. Y. Yang, S. Yu, Q. Y. Xie, X. Y. Xia, Y. Y. Dang, Q. Zhang, X. H. Wu, Y. Cui, B. Hou, R. H. Yu, R. X. Xu, T. Z. Jiang. Prognostication of chronic disorders of consciousness using brain functional networks and clinical characteristics. eLife, vol. 7, Article number e36173, 2018. DOI: https://doi.org/10.7554/eLife.36173.

  79. Y. B. Liu, L. Yue, S. F. Xiao, W. Yang, D. G. Shen, M. X. Liu. Assessing clinical progression from subjective cognitive decline to mild cognitive impairment with incomplete multi-modal neuroimages. Medical Image Analysis, vol. 75, Article number 102266, 2022. DOI: https://doi.org/10.1016/j.media.2021.102266.

  80. S. X. Luo, D. Martinez, K. M. Carpenter, M. Slifstein, E. V. Nunes. Multimodal predictive modeling of individual treatment outcome in cocaine dependence with combined neuroimaging and behavioral predictors. Drug and Alcohol Dependence, vol.143, pp. 29–35, 2014. DOI: https://doi.org/10.1016/j.drugalcdep.2014.04.030.

    Article  Google Scholar 

  81. A. Billot, S. Lai, M. Varkanitsa, E. J. Braun, B. Rapp, T. B. Parrish, J. Higgins, A. S. Kurani, D. Caplan, C. K. Thompson, P. Ishwar, M. Betke, S. Kiran. Multimodal neural and behavioral data predict response to rehabilitation in chronic poststroke aphasia. Stroke, vol.53, no.5, pp. 1606–1614, 2022. DOI: https://doi.org/10.1161/strokeaha.l21.036749.

    Article  Google Scholar 

  82. M. M. Schmitgen, I. Niedtfeld, R. Schmitt, F. Mancke, D. Winter, C. Schmahl, S. C. Herpertz. Individualized treatment response prediction of dialectical behavior therapy for borderline personality disorder using multimodal magnetic resonance imaging. Brain and Behavior, vol. 9, no. 9, Article number e01384, 2019. DOI: https://doi.org/10.1002/brb3.1384.

  83. S. Marek, B. Tervo-Clemmens, F. J. Calabro, D. F. Montez, B. P. Kay, A. S. Hatoum, M. R. Donohue, W. Foran, R. L. Miller, T. J. Hendrickson, S. M. Malone, S. Kandala, E. Feczko, O. Miranda-Dominguez, A. M. Graham, E. A. Earl, A. J. Perrone, M. Cordova, O. Doyle, L. A. Moore, G. M. Conan, J. Uriarte, K. Snider, B. J. Lynch, J. C. Wilgenbusch, T. Pengo, A. Tam, J. Z. Chen, D. J. Newbold, A. N. Zheng, N. A. Seider, A. N. Van, A. Metoki, R. J. Chauvin, T. O. Laumann, D. J. Greene, S. E. Petersen, H. Garavan, W. K. Thompson, T. E. Nichols, B. T. T. Yeo, D. M. Barch, B. Luna, D. A. Fair, N. U. F. Dosenbach. Reproducible brain-wide association studies require thousands of individuals. Nature, vol.603, no. 7902, pp. 654–660, 2022. DOI: https://doi.org/10.1038/s41586-022-04492-9.

    Article  Google Scholar 

  84. C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, M. Landray, B. Liu, P. Matthews, G. Ong, J. Pell, A. Silman, A. Young, T. Sprosen, T. Peakman, R. Collins. UK biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Medicine, vol.12, no. 3, Article number e1001779, 2015. DOI: https://doi.org/10.1371/journal.pmed.1001779.

  85. B. J. Casey, T. Cannonier, M. I. Conley, A. O. Cohen, D. M. Barch, M. M. Heitzeg, M. E. Soules, T. Teslovich, D. V. Dellarco, H. Garavan, C. A. Orr, T. D. Wager, M. T. Banich, N. K. Speer, M. T. Sutherland, M. C. Riedel, A. S. Dick, J. M. Bjork, K. M. Thomas, B. Chaarani, M. H. Mejia, D. J. Jr.Hagler, M. Daniela Cornejo, C. S. Sicat, M. P. Harms, N. U. F. Dosenbach, M. Rosenberg, E. Earl, H. Bartsch, R. Watts, J. R. Polimeni, J. M. Kuperman, D. A. Fair, A. M. Dale. The Adolescent Brain Cognitive Development (ABCD) study: Imaging acquisition across 21 sites. Developmental Cognitive Neuroscience, vol.32, pp.43–54, 2018. DOI: https://doi.org/10.1016/j.dcn.2018.03.001.

    Article  Google Scholar 

  86. D. C. Van Essen, K. Ugurbil, E. Auerbach, D. Barch, T. E. J. Behrens, R. Bucholz, A. Chang, L. Chen, M. Corbetta, S. W. Curtiss, S. Della Penna, D. Feinberg, M. F. Glasser, N. Harel, A. C. Heath, L. Larson-Prior, D. Marcus, G. Michalareas, S. Moeller, R. Oostenveld, S. E. Petersen, F. Prior, B. L. Schlaggar, S. M. Smith, A. Z. Snyder, J. Xu, E. Yacoub, WU-Minn HCP Consortium. The human connectome project: A data acquisition perspective. Neurolmage, vol. 62, no. 4, pp. 2222–2231, 2012. DOI: https://doi.org/10.1016/j.neuroimage.2012.02.018.

    Article  Google Scholar 

  87. Q. Y. Zhong, A. N. Li, R. Jin, D. J. Zhang, X. N. Li, X. Y. Jia, Z. H. Ding, P. Luo, C. Zhou, C. Y. Jiang, Z. Feng, Z. H. Zhang, H. Gong, J. Yuan, Q. M. Luo. High-definition imaging using line-illumination modulation microscopy. Nature Methods, vol.18, no. 3, pp. 309–315, 2021. DOI: https://doi.org/10.1038/s41592-021-01074-x.

    Article  Google Scholar 

  88. M. Goubran, C. Leuze, B. Hsueh, M. Aswendt, L. Ye, Q. Y. Tian, M. Y. Cheng, A. Crow, G. K. Steinberg, J. A. McNab, K. Deisseroth, M. Zeineh. Multimodal image registration and connectivity analysis for integration of connectomic data from microscopy to MRI. Nature Communications, vol. 10, no. 1, Article number 5504, 2019. DOI: https://doi.org/10.1038/s41467-019-13374-0.

  89. M. Zubair, S. R. Murris, K. Isa, H. Onoe, Y. Koshimizu, K. Kobayashi, W. Vanduffel, T. Isa. Divergent whole brain projections from the ventral midbrain in macaques. Cerebral Cortex, vol.31, no.6, pp. 2913–2931, 2021. DOI: https://doi.org/10.1093/cercor/bhaa399.

    Article  Google Scholar 

  90. M. Mancini, A. Casamitjana, L. Peter, E. Robinson, S. Crampsie, D. L. Thomas, J. L. Holton, Z. Jaunmuktane, J. E. Iglesias. A multimodal computational pipeline for 3D histology of the human brain. Scientific Reports, vol. 10, no. 1, Article number 13839, 2020. DOI: https://doi.org/10.1038/s41598-020-69163-z.

  91. K. Amunts, C. Lepage, L. Borgeat, H. Mohlberg, T. Dickscheid, M. É. Rousseau, S. Bludau, P. L. Bazin, L. B. Lewis, A. M. Oros-Peusquens, N. J. Shah, T. Lippert, K. Zilles, A. C. Evans. BigBrain: An ultrahigh-resolution 3D human brain model. Science, vol.340, no.6139, pp. 1472–1475, 2013. DOI: https://doi.org/10.1126/science.l235381.

    Article  Google Scholar 

  92. J. N. Acosta, G. J. Falcone, P. Rajpurkar, E. J. Topol. Multimodal biomedical AI. Nature Medicine, vol. 28, no. 9, pp. 1773–1784, 2022. DOI: https://doi.org/10.1038/s41591-022-01981-2.

    Article  Google Scholar 

  93. A. Shapson-Coe, M. Januszewski, D. R. Berger, A. Pope, Y. L. Wu, T. Blakely, R. L. Schalek, P. Li, S. H. Wang, J. Maitin-Shepard, N. Karlupia, S. Dorkenwald, E. Sjostedt, L. Leavitt, D. Lee, L. Bailey, A. Fitzmaurice, R. Kar, B. Field, H. Wu, J. Wagner-Carena, D. Aley, J. Lau, Z. D. Lin, D. Wei, H. Pfister, A. Peleg, V. Jain, J. W. Lichtman. A connectomic study of a petascale fragment of human cerebral cortex. BioRxiv, to be published. DOI: https://doi.org/10.1101/2021.05.29.446289.

  94. K. Amunts, T. Lippert. Brain research challenges supercomputing. Science, vol.374, no. 6571, pp. 1054–1055, 2021. DOI: https://doi.org/10.1126/science.abl8519.

    Article  Google Scholar 

  95. K. Amunts, C. Ebell, J. Muller, M. Telefont, A. Knoll, T. Lippert. The human brain project: Creating a European research infrastructure to decode the human brain. Neuron, vol.92, no.3, pp.574–581, 2016. DOI: https://doi.org/10.1016/j.neuron.2016.10.046.

    Article  Google Scholar 

  96. J. R. Ecker, D. H. Geschwind, A. R. Kriegstein, J. Ngai, P. Osten, D. Polioudakis, A. Regev, N. Sestan, I. R. Wickersham, H. K. Zeng. The BRAIN initiative cell census consortium: Lessons learned toward generating a comprehensive brain cell atlas. Neuron, vol.96, no.3, pp.542–557, 2017. DOI: https://doi.org/10.1016/j.neuron.2017.10.007.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. 82001450 and 82151307); Science and Technology Innovation 2030–Brain Science and Brain-Inspired Intelligence Project of China (No. 2021ZD0200201), the National Key Research and Development Program of China (No. 2022YFC3601200), the China Postdoctoral Science Foundation (No. BX2020 0364), the Chinese Academy of Sciences, Science and Technology Service Network Initiative (No. KFJ-STS-ZDTP-078), the Strategic Priority Research Program of the Chinese Academy of Sciences, China (No. XDB32030200), the Scientific Project of Zhejiang Laboratory, China (No. 2022ND0AN01).

Author information

Author notes

  1. These authors contribute equally to this work

Authors and Affiliations

  1. Brainnetome Center and National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Na Luo, Weiyang Shi, Zhengyi Yang, Ming Song & Tianzi Jiang

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China

    Tianzi Jiang

  3. Center for Excellence in Brain Science and Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Tianzi Jiang

  4. Research Center for Augmented Intelligence, Zhejiang Laboratory, Hangzhou, 311100, China

    Tianzi Jiang

Authors
  1. Na Luo
    View author publications

    Search author on:PubMed Google Scholar

  2. Weiyang Shi
    View author publications

    Search author on:PubMed Google Scholar

  3. Zhengyi Yang
    View author publications

    Search author on:PubMed Google Scholar

  4. Ming Song
    View author publications

    Search author on:PubMed Google Scholar

  5. Tianzi Jiang
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Tianzi Jiang.

Ethics declarations

The authors declared that they have no conflicts of interest to this work.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Na Luo received the Ph.D. degree in pattern recognition and intelligent system from University of Chinese Academy of Sciences, China in 2020. She is currently an associate professor with Institute of Automation, Chinese Academy of Sciences, China.

Her research interests include fusing multi-modal brain imaging and multi-omics data to help extend the understanding of brain in both health and disease status.

Weiyang Shi received the Ph.D. degree in pattern recognition and intelligent system from Institute of Automation, Chinese Academy of Sciences, China in 2023. He is currently an assistant professor with Institute of Automation, Chinese Academy of Sciences, China.

His research interests include machine learning in medical image analysis, computational psychiatry and brain-inspired artificial intelligence.

Zhengyi Yang received the Ph.D. degree in mechanical engineering from University of Hong Kong, China in 2003. He is currently an associate professor with Institute of Automation, Chinese Academy of Sciences, China.

His research interests include medical image analysis and applications on robot.

Ming Song received the Ph.D. degree in pattern recognition and intelligent system from University of Chinese Academy of Sciences, China in 2008. He is currently a professor with Institute of Automation, Chinese Academy of Sciences, China.

His research interests include brain imaging and analysis, brain network and brain-computer interface.

Tianzi Jiang (Fellow, IEEE) received the B.Sc. degree in computational mathematics from Lanzhou University, China in 1984, and the Ph.D. degree in computational mathematics from Zhejiang University, China in 1994. He is currently a member of the Academy of Europe, senior research professor, and director of Beijing Key Laboratory of Brainnetome and director of the Brainnetome Center at Institute of Automation, Chinese Academy of Sciences, China, and Adjunctive Senior Investigator of Zhejiang Laboratory in Hangzhou, China. He was the recipient of Hermann von Helmholtz Award (Lifetime Contribution Awards) of International Neural Network Society, Turan Itil Career Contribution Award of EEG & Clinical Neuroscience Society, Wu Wen-Jun AI Distinguishing Contribution Award of the Chinese Association of Artificial Intelligence, Natural Science Award of China, Beijing Natural Science Award (the first class), and Wu Wen-Jun AI Natural Science Award. He was elected as a fellow of IAPR and AIMBE. He is currently the Chair of Organization of Human Brain Mapping, an Associate Editor for IEEE Transactions on Cognitive and Developmental Systems, Neuroscience Bulletin, and Action Editor for Neural Networks.

His research interests include brainnetome atlas, neuroimaging, and their clinical applications in brain disorders.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, N., Shi, W., Yang, Z. et al. Multimodal Fusion of Brain Imaging Data: Methods and Applications. Mach. Intell. Res. 21, 136–152 (2024). https://doi.org/10.1007/s11633-023-1442-8

Download citation

  • Received: 16 December 2022

  • Accepted: 23 March 2023

  • Published: 15 January 2024

  • Version of record: 15 January 2024

  • Issue date: February 2024

  • DOI: https://doi.org/10.1007/s11633-023-1442-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Multimodal fusion
  • supervised learning
  • unsupervised learning
  • brain atlas
  • cognition
  • brain disorders

Profiles

  1. Ming Song View author profile
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

Advertisement

Search

Navigation

  • Find a journal
  • Publish with us
  • Track your research

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Journal finder
  • Publish your research
  • Language editing
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our brands

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Discover
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support
  • Legal notice
  • Cancel contracts here

Not affiliated

Springer Nature

© 2025 Springer Nature