Interpreting Age Effects of Human Fetal Brain from Spontaneous fMRI using Deep 3D Convolutional Neural Networks

In terpreting Age Effects of Human F etal Brain from Sp on taneous fMRI using Deep 3D Con v olutional Neural Net w orks Xiangrui Li 1 , Jasmine Hect 2 , Moriah Thomason 3 , and Dongxiao Zh u 1 1 Departmen t of Computer Science, W a yne State Universit y { fx7219,dzhu } @wayne.edu 2 Departmen t of Psyc hology , W a yne State Univ ersity ev4125@wayne.edu 3 Departmen t of Child and Adolescent Psyc hiatry , New Y ork Universit y moriah.thomason@nyulangone.org Abstract. Understanding h uman fetal neuro dev elopmen t is of great clinical imp ortance as abnormal developmen t is linked to adv erse neu- ropsyc hiatric outcomes after birth. Recen t adv ances in functional Mag- netic Resonance Imaging (fMRI) ha ve provided new insight into dev el- opmen t of the human brain before birth, but these studies hav e pre- dominately fo cused on brain functional connectivity (i.e. Fisher z-score), whic h requires man ual pro cessing steps for feature extraction from fMRI images. Deep learning approac hes (i.e., Con volutional Neural Netw orks (CNN)) hav e achiev ed remark able success on learning directly from im- age data, yet ha v ent b een applied on fetal fMRI for understanding fetal neuro dev elopmen t. Here, we bridge this gap b y applying a nov el appli- cation of deep 3D CNN to fetal bloo d o xygen-lev el dep endence (BOLD) resting-state fMRI data. Specifically , we test a supervised CNN frame- w ork as data-driven approach to isolate v ariation in fMRI signals that relate to younger v.s. older fetal age groups. Based on the learned CNN, w e further p erform sensitivity analysis to identify brain regions in which c hanges in BOLD signal are strongly asso ciated with fetal brain age. The findings demonstrate that deep CNNs are a promising approach for iden tifying sp ontaneous functional patterns in fetal brain activit y that discriminate age groups. F urther, we discov ered that regions that most strongly differentiate groups are largely bilateral, share similar distri- bution in older and y ounger age groups, and are areas of heightened metab olic activit y in early human develop ment. 1 In tro duction The human brain undergo es rapid developmen t in the prenatal p erio d. Previous studies show significan t links b etw een abnormal brain developmen t in uter o and adv erse p ostnatal outcomes [1, 4]. As a result, understanding fetal neuro dev elop- men t, particularly timescales and patterns for maturation of functional systems, is of great clinical imp ortance. 2 Xiangrui et al. Recen t metho dological adv ances in fetal functional Magnetic Resonance Imag- ing (fMRI) hav e attracted muc h atten tion [4, 7]. One promising adv ance is appli- cation of CNN-based metho ds to fetal brain MRI image segmen tation [11]. F or fetal neuro dev elopmental studies, existing works fo cus on analysis of the func- tional connectivity in fetal brain, such as discov eries of functional netw orks and iden tification of highly connected hubs [5, 18]. Those studies are based on func- tional connectivit y measures (i.e. adjacency matrix of Fisher z-scores) extracted from the spatio-temp oral 4D resting-state fMRI. How ever, from the p ersp ectiv e of computational metho dology , those approac hes rely hea vily on the data pro- cessing steps for extracting z-scores and only utilize temp oral information in the fMRI images, which miss the spatial information in the 2D or 3D fMRI images. There is preceden t for direct learning from MRI images in children and adults. F or example, CNN has b een used to classify neurological disorders suc h as Autism Sp ectrum Disorder (ASD) [10, 21] and Alzheimer’s disease [6, 8], and to c haracterize brain functional net works [20]. F urther, a small n umber of studies ha ve built CNN mo dels to predict age from fMRI data [9]. Probably due to very limited av ailability of fetal fMRI data, CNN mo dels hav e not b een applied for understanding fetal neuro dev elopmen t. T o bridge the metho dological gap, we propose a nov el application of deep CNN in conjunction with CNN interpretation tec hniques on the analysis of fe- tal brain fMRI data, by lev eraging CNN’s merits of learning high-lev el feature represen tations. T o the b est of our kno wledge, this work represen ts the first application of CNN models as a data-driven approac h in understanding fetal dev elopmental pro cesses. Sp ecifically , as age and brain dev elopment are closely link ed, we utilize the age group as the surrogate target (y ounger v.s. older) and train a sup ervised CNN directly on the residual fMRI images to capture as- so ciations b et w een age effect and bloo d oxygen-lev el dependence (BOLD). The trained mo del ac hiev es sup erior predictiv e p erformance o ver traditional machine learning algorithms, demonstrating its effectiveness in learning directly from fe- tal fMRI images. More importantly , since CNN is in terpretable, w e are able to explain the trained CNN to highlight brain regions that are strongly influen- tial in model predictions. The in terpretation results of important regions are clinically comp elling that matc h with existing studies. F urthermore, the trained CNN model also prop oses no vel clinically sensible regions that are p oten tially informativ e in c haracterizing brain developmen t in fetuses, demonstrating its strong promise as a data-driv en approach to enhance our understanding of brain dev elopment as well as facilitate fetal developmen tal studies. 2 Metho d 2.1 Data Acquisition and Prepro cessing Sub jects in our analysis were subset of a cohort in a study of longitudinal fetal brain developmen t. The cohort consists of 148 pregnant women that underwen t MRI with gestational age (GA) b etw een 24 and 40 weeks. W e manually chec k ed Title Suppressed Due to Excessive Length 3 Fig. 1. Data prepro cessing pipeline from raw MRI image to residual fMRI data. sub jects according to data quality (high motion, artifacts et al.) and inclusion- ary criteria (gestational age, b orn without nerv ous system abnormality et al.), lea ving 75 qualified sub jects: 30 sub jects with GA b et ween 26 and 29 weeks (y ounger group), and 45 b et ween 34 and 37 weeks (older group). MRI data were acquired on a Siemens V erio 70-cm open-b ore 3T MR sys- tem. Resting-state fMRI data were acquired using a gradien t echo planar imaging sequence: TR/TE 2000/30 ms, flip angle 80 ◦ , 360 frames, axial 4 mm slice thick- ness, v o xel size 3.4x3.4x4 mm 3 , repeated t wice. Bet ween 12 to 24 min utes of fetal resting-state fMRI data w ere collected p er sub ject. Extraction of fetal brain fMRI data requires m ultiple steps in prepro cessing ra w resting state fMRI images. In brief, p eriods of fetal quiescence w ere man ually iden tified using FSL image viewer, wherein individual segments must consist of at least 20 seconds (10 frames) of low motion ( < 2 mm translation and/or 3 degrees rotational mo vemen t) (A). After motion censoring, fetal brain masks w ere then created separately for each low-motion ep och from a single reference image using Brainsuite [16] (B). After masking, each temp oral segment was reorien ted man ually , realigned to the mean BOLD volume, resampled to 2 mm isotropic vo xels, and normalized to a 32-week fetal brain template [15] using SPM8 [13] (C). All normalized images from each segmen t w ere then concatenated in to one run, realigned to the mean BOLD volume, and smo othed with a 4 mm FWHM Gaussian k ernel (D-E). F urther prepro cessing was performed in CONN toolb o x (v14n) [19] including linear detrending, n uisance regression using aCompCor of five principal comp onen ts extracted from a 32-w eek fetal atlas white matter and CSF mask, six head motion parameters, and band-pass filtering at 0.008 to 0.09 Hz (F). Fig. 1 sho ws the prepro cessing pip eline from raw MRI to residual fMRI used in our mo del. 4 Xiangrui et al. In p u t D i m: ( 43, 51, 40) Cha n n el : 1 C o n v 1 Ker n el : ( 5, 5, 5) S tr i d e : 2 Cha n n el : 32 Pad d i n g : N o n e A c ti v at i o n : R eL U M ax Po o l i n g Ke r n e l : ( 2, 2, 2) S tri d e: 2 Co n v 2 Ke r n e l : ( 3, 3, 3) S tri d e: 1 C h an n e l : 32 Pad d i n g : N o n e A c ti v ati o n : R e L U M ax Po o l i n g Ker n el : ( 2, 2, 2) S tr i d e : 2 F C1 D i m: 256 O u tp u t: Yo u n g er o r O l d e r ? F la t t en Fig. 2. CNN architecture for age group prediction. 2.2 Mo deling Age Effect on Neuro dev elopmen t with 3D CNN F or each sub ject, the data are a time series of 3D fMRI volumes of dimension 43 × 51 × 40. T o exploit the spatio-temp oral information in fMRI, we take a sliding window approac h [10]. Sp ecifically , starting from the first 3D frame of 4D fMRI whose length is T , we tak e the mean 3D fMRI image in a window of size m ov er the time axis; as the windo w slides o ver the time axis with stride s , a total of  T − m s  + 1 3D fMRI images are generated, which significantly increases the sample size in training data. After initial exp erimen ts, m = 2 (or 3) and s = 1 achiev e go od p erformance and increasing windo w size leads to o ver- smo othing and degrade mo del p erformance. As mentioned in Section 1, younger (26 < GA < 29 weeks) and older (34 < GA < 37 weeks) age groups are used as the surrogate target. F or each 3D fMRI generated from sliding windo w, we lab el it with the sub ject’s age group. T o learn fetal neuro dev elopmental processes b y classifying 3D fMRI, w e build an effective 3D CNN to capture the asso ciation b et ween residual fMRI and age group. The prop osed CNN along with its architectural parameters (i.e. kernel size, stride, channel et al.) is sho wn in Fig. 2. It has tw o conv olutional lay ers with ReLU non-linear activ ation and one fully connected lay er, where eac h con- v olutional lay er is follow ed b y a max p o oling lay er. Mo del predictions are made with sigmoid activ ation on the output lay er after the fully connected lay er. The ob jective function is hence the binary cross en tropy plus L 2 regularization on the net work weigh ts: L ( θ ) = − X i y i log( f ( x i )) + (1 − y i ) log (1 − f ( x i )) + λ || θ || 2 2 , where y i = 1 (older group) or 0 (younger group) is the true lab el of x i , f ( x i ) is the probability p ( y = 1 | x i ) mo deled b y CNN, and λ is the tuning parameter of L 2 regularization. 2.3 Mo del In terpretation As our primary goal is to understand the age effect on fetal brain dev elopment, mo del interpretation is critical to our CNN approach. Once CNN is trained, we p erform sensitivity analysis (SA) [14, 17] on lea ve-out testing images to identify imp ortan t regions of interests (ROIs). The sensitivity score for each image pixel is the squared gradient with resp ect to the CNN output (i.e. probability of Title Suppressed Due to Excessive Length 5 b eing y ounger or older), which measures ho w sensitive the prediction is to the c hange of v oxel v alues. The calculation of sensitivit y score only needs one pass of bac kpropagation with resp ect to the input image. F or fetal fMRI images, a region (i.e. multiple v oxels in a neighborho od) in the brain with larger sensitivit y scores is more influen tial, hence more imp ortan t in predicting age group. 2.4 T raining Setup The CNN is trained with classic sto c hastic gradien t decen t algorithm (SGD) with momentum set to 0.8. The learning rate is set to 0.1 initially and decreased b y multiplying a factor 0.2 for every 7 ep o c hs. L 2 regularization with λ = 0 . 001 is used to preven t ov erfitting. The size of mini-batc h in each ep och is set to 128. W e apply early stopping as another regularization in mo del v alidation. In our practice, training usually is stopp ed at 15 epo chs. Mo del parameters are initialized with uniform distribution on ( − √ u, √ u ) where u is recipro cal of the n umber of weigh ts in each lay er. W e implemen t our CNN in Pytorch [12]. T o a void mo del seeing images of the same sub ject in testing and training (since each sub ject generates multiple 3D fMRI images), we split the dataset at sub ject level. In the exp eriments, data are divided with stratification in to train- ing, v alidation and testing sets by 80%/10%/10%. All images are normalized b y subtracting mean image and then divided b y the maximal absolute inten- sit y v alue, both calculated from training data. The splitting pro cedure results in ab out 9300 images in training, 1200 in v alidation and testing. W e rep eat this pro cedure 10 times in exp erimen ts. T o ev aluate mo del p erformance, w e use F1 score which is calculated as 2(precision × recall)/(precision+recall) and Area un- der R OC curve (AUC) as the ev aluation metric, and the av erage F1 and A UC o ver 10 runs are rep orted. 3 Exp erimen tal Results 3.1 Predictiv e Performance F or performance comparison, w e also test several baseline classification algo- rithms, including random forest (RF), gradient bo osting machine (GBM) and logistic regression (LR) with L2 regularization. These alternativ es w ere tested on t wo types of data: (1) fetal brain functional connectivity matrices extracted b y correlating resting-state fMRI time series data across 100 brain regions (Fisher z-score), (2) fetal BOLD fMRI images. F or fMRI, due to the high dimensionalit y (dim= 39984) 4 and strong correlations among neighboring vo xels in fMRI, w e use principal comp onent analysis (PCA) for dimension reduction before test- ing on baselines. W e select the optimal n umber of PCA comp onen ts as w ell as optimal parameters of classification models using the v alidation data. In our 4 W e remov e zero columns after flattening original 3D fMRI into a v ector of size 43 × 51 × 40 = 87720 6 Xiangrui et al. T able 1. Predictive p erformance (standard deviation) on testing sub jects. Note that Fisher z-score as functional connectivity measure is calculated based on brain parcel- lation and not applicable for CNN. Fisher z-score fMRI Image F1 A UC F1 AUC CNN - - 0.84 (0.05) 0.91 (0.06) RF 0.76 (0.03) 0.69 (0.18) 0.79 (0.06) 0.77 (0.15) GBM 0.73 (0.09) 0.69 (0.17) 0.81 (0.06) 0.77 (0.18) LR 0.77 (0.06) 0.75 (0.18) 0.77 (0.01) 0.67 (0.14) exp erimen ts, 100 comp onen ts ac hieves go od p erformance and including more comp onen ts do es not necessarily lead to p erformance gain. The predictive p erformance (F1 score and AUC) on testing data at sub ject lev el is sho wn in T able 1. Note that sub ject lev el prediction is made with soft v ot- ing: probability for each sub ject is av eraged across its corresp onding 3D images. The probability threshold for classification is chosen as 0.5 for calculating F1 score. W e see from T able 1 that CNN trained directly on fMRI images ac hieves the b est and robust p erformance (F1 0.84 with standard deviation 0.05, A UC 0.91 with 0.06), due to CNN’s merit of capturing spatial information of fMRI im- ages in classification. The impro vemen ts demonstrate that CNN can effectiv ely capture discriminative information that could reveal asso ciations b et w een age effect and brain dev elopment. 3.2 Inference Ab out F etal Neuro dev elopment from CNN Classification It was possible to discriminate older versus y ounger fetuses on the basis of sp on- taneous baseline resting-state BOLD measurements. BOLD reflects changing regional blo od concentrations of oxy- and deo xy-hemoglobin, which are influ- enced at least in part by regional metab olic activity [3]. Having observ ed that fetal age can b e differen tiated on the basis of these fluctuating signals suggests that patterns in baseline BOLD during resting-state reflect asp ects of brain mat- uration. Sensitivit y analysis was used to identify regions within each age group where c hange in v oxel v alues most altered strength of the prediction. Imp ortan t regions (iden tified with large sensitivity scores, see Section 2.3) for b oth y ounger and older fetus groups are shown in Fig. 3. W e found that sensitive regions (1) hav e a high degree of spatial o verlap across b oth age groups, (2) are largely bilater- ally distributed, (3) encompass brain regions that ha ve b een iden tified as having high baseline metab olic activit y in positron emission tomograph y (PET) stud- ies in h uman newb orns and infan ts, including sub cortex, thalamus, and medial temp oral lob e [2]. Title Suppressed Due to Excessive Length 7 Fig. 3. Sensitivity analysis for y ounger v.s. older age groups. Imp ortan t regions of high sensitivit y scores that are asso ciated with age effects are noted in green for younger fetuses and red for older fetuses. Sp ecifically , across groups we observe high sensitivity in bilateral o ccipital cortex, bilateral ven trolateral prefrontal cortex (vlPFC), sub-cortex, p osterior cingulate cortex (PCC), medial temporal lob e (MTL), thalamus, and cereb ellum. In addition, baseline BOLD in an terior cingulate cortex (A CC), hypothalamus, and insula regions are imp ortan t for group classification in older fetuses. 4 Conclusions In this pap er, we prop ose a nov el application of CNN for interpretation of fetal brain age effects directly from fMRI images. The predictiv e performance of CNN demonstrate that it can well capture asso ciations betw een age and v ariation in BOLD signal. T o b etter understand the relev ance of our predictive CNN to fetal developmen t, we use sensitivity analysis to isolate regions critical for CNN p erformance, and discov ered that our most sensitive regions were regions that are high in metab olic activit y in early human brain developmen t. The exp erimen tal results rev eal comp elling associations and demonstrate p oten tial promise of CNN applied to sp on taneous BOLD activit y as a data-driven approac h to understand fetal dev elopmental pro cesses. With those identified imp ortant ROIs, in our future studies, w e plan to p er- form seed-based analyses to examine ho w functional connectivities and net works of those R OIs are asso ciated with whole-brain age effects. 8 Xiangrui et al. References 1. Benk arim, O.M., et al: T ow ard the automatic quan tification of in utero brain de- v elopment in 3d structural mri: a review. Human brain mapping 38 (5), 2772–2787 (2017) 2. Ch ugani, H.T.: Imaging brain metab olism in the newb orn. Journal of child neu- rology 33 (13), 851–860 (2018) 3. F ukunaga, M., Horovitz, S.G., De Zw art, J.A., V an Gelderen, P ., Balkin, T.J., Braun, A.R., Duyn, J.H.: Metab olic origin of b old signal fluctuations in the absence of stimuli. Journal of Cerebral Blo od Flow & Metab olism 28 (7), 1377–1387 (2008) 4. v an den Heuvel, M.I., Thomason, M.E.: F unctional connectivity of the human brain in utero. T rends in cognitiv e sciences 20 (12), 931–939 (2016) 5. v an den Heuvel, M.I., T urk, E., Manning, J.H., Hect, J., Hernandez-Andrade, E., Hassan, S.S., Romero, R., v an den Heuvel, M.P ., Thomason, M.E.: Hubs in the h u- man fetal brain netw ork. Dev elopmental cognitive neuroscience 30 , 108–115 (2018) 6. Hosseini-Asl, E., Gimel’farb, G., El-Baz, A.: Alzheimer’s disease diagnostics b y a deeply sup ervised adaptable 3d conv olutional netw ork. arXiv preprint arXiv:1607.00556 (2016) 7. Konk el, L.: The brain b efore birth: Using fmri to explore the secrets of fetal neu- ro dev elopmen t (2018) 8. Korolev, S., et al: Residual and plain conv olutional neural netw orks for 3d brain mri classification. In: ISBI. pp. 835–838. IEEE (2017) 9. Li, H., Satterth waite, T.D., F an, Y.: Brain age prediction based on resting-state functional connectivity patterns using conv olutional neural netw orks. In: ISBI. pp. 101–104. IEEE (2018) 10. Li, X., Dvornek, N.C., Zhuang, J., V entola, P ., Duncan, J.S.: Brain biomark er in terpretation in asd using deep learning and fmri. In: MICCAI. pp. 206–214. Springer (2018) 11. Makrop oulos, A., Counsell, S.J., Ruec kert, D.: A review on automatic fetal and neonatal brain mri segmentation. NeuroImage 170 , 231–248 (2018) 12. P aszke, A., Gross, S., Chintala, S., Chanan, G., Y ang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in p ytorch (2017) 13. P enny , W.D., F riston, K.J., Ash burner, J.T., Kiebel, S.J., Nichols, T.E.: Statistical parametric mapping: the analysis of functional brain images. Elsevier (2011) 14. Riek e, J., et al: Visualizing conv olutional netw orks for mri-based diagnosis of alzheimers disease. In: Understanding and Interpreting Machine Learning in Med- ical Image Computing Applications, pp. 24–31. Springer (2018) 15. Serag, A., et al: Construction of a consistent high-definition spatio-temp oral atlas of the developing brain using adaptive kernel regression. NeuroImage 59 (3), 2255– 2265 (2012) 16. Shattuc k, D.W., Leah y , R.M.: Brainsuite: an automated cortical surface identifi- cation to ol. Medical Image Analysis 6 (2), 129–142 (2002) 17. Simon yan, K., V edaldi, A., Zisserman, A.: Deep inside con volutional net- w orks: Visualising image classification mo dels and saliency maps. arXiv preprin t arXiv:1312.6034 (2013) 18. Thomason, M.E., et al: Age-related increases in long-range connectivity in fetal functional neural connectivit y net works in utero. Developmen tal Cognitiv e Neuro- science 11 , 96 – 104 (2015) 19. Whitfield, S., Nieto, A.: Conn: a functional connectivity toolb ox for correlated and an ticorrelated brain net works. Brain connectivity 2 (3), 125–141 (2012) Title Suppressed Due to Excessive Length 9 20. Zhao, Y., et al: Automatic recognition of fmri-deriv ed functional netw orks using 3-d conv olutional neural net works. IEEE T ransactions on Biomedical Engineering 65 (9), 1975–1984 (2018) 21. Zhao, Y., Ge, F., Zhang, S., Liu, T.: 3d deep conv olutional neural netw ork rev ealed the v alue of brain net w ork ov erlap in differentiating autism sp ectrum disorder from health y controls. In: MICCAI. pp. 172–180. Springer (2018)