Demonstration of Vector Flow Imaging using Convolutional Neural Networks

Demonstration of V ector Flow Imaging using Con volutional Neural Networks 2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating ne w collectiv e works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. *IEEE policy provides that authors are free to follow public access mandates to post accepted articles in funding agency repositories. When posting in a funding agency repository , the IEEE embargo period is 24 months. Howe ver , IEEE recognizes that posting requirements and embargo periods v ary by funder , and IEEE authors may comply with requirements to deposit their accepted manuscripts in funding agency repositories where the embargo is less than 24 months. Demonstration of V ector Flo w Imaging using Con v olutional Neural Networks Thomas Robins, Antonio Stanziola, Kai Riemer , Peter D. W einber g, Meng-Xing T ang † Department of Bioengineering, Imperial College London, United Kingdom † Email: mengxing.tang@imperial.ac.uk Abstract —Synthetic Apertur e V ector Flow Imaging (SA-VFI) can visualize complex cardiac and vascular blood flow patterns at high temporal resolution with a large field of view . Convolutional neural networks (CNNs) are commonly used in image and video recognition and classification. Howev er , most recently presented CNNs also allow for making per -pixel predictions as needed in optical flow velocimetry . T o our knowledge we demonstrate here for the first time a CNN architecture to produce 2D full flow field predictions from high frame rate SA ultrasound images using supervised learning. The CNN was initially trained using CFD-generated and augmented noiseless SA ultrasound data of a realistic vessel geometry . Subsequently , a mix of noisy simulated and real in vivo acquisitions were added to increase the network’ s rob ustness. The r esulting flow field of the CNN resembled the ground truth accurately with an endpoint-error percentage between 6.5% to 14.5%. Furthermore, when confronted with an unknown geometry of an arterial bifurcation, the CNN was able to predict an accurate flow field indicating its ability for general- ization. Remarkably , the CNN also performed well for rotational flows, which usually r equires advanced, computationally intensive VFI methods. W e have demonstrated that conv olutional neural networks can be used to estimate complex multidirectional flow . Keyw ords — V ector Flow Imaging, Convolutional Neural Network, Deep Learning, Echo-PIV , Ultrasound T oolbox, FieldII I . I N T RO D U C T I O N Abnormalities in blood flow and complex flo w patterns hav e been linked to the de velopment of cerebro vascular and coronary heart disease [ 1 ]. Con ventional Doppler ultrasound based methods are angle-dependent and can only measure the axial-component of blood flow in straight vessels, making them unsuitable for quantifying complex flow patterns [ 2 ]. V ector Flo w Imaging (VFI) is largely angle-independent and capable of visualizing highly resolv ed spatio-temporal flow patterns in all directions [ 3 ]. Se veral methods of VFI hav e been proposed such as Directional Beamforming [ 4 ], T ransverse Oscillation [ 5 ], Multi-angle Doppler V ector Projectile Imaging [ 6 ] and Echo P article Image V elocimetry (Echo-PIV) [ 7 ]. Echo-PIV is based on tracking the speckle pattern of two consecutiv e B-Mode images through cross-correlation to form a vector displacement field. From the pixel displacement and the time between two frames, a full 2D velocity field can be created [ 7 ]. In Synthethic Aperture (SA) imaging these techniques are further enhanced through a wider field of view of the diver ging waves, which allo w imaging of regions with a limited acoustic window , such as in transthoracic cardiac Fig. 1. Modified FlowNetSimple architecture for predicting 2D velocity fields from consecutive synthetic aperture ultrasound frames. imaging [ 8 ]. Howe ver , VFI comes at the cost of high beam- forming loads, angle dependent errors and aliasing artifacts and can be time-consuming in post-processing [ 9 ]. The problem of estimating 2D velocity fields from sets of consecutiv e images is widely explored in the computer vision community under Optical Flo w estimation. Beside hand crafted models [ 10 ], machine learning approaches have also been used in this context, for example using principal component analysis of natural flo w fields [ 11 ] or neural net- works [ 12 ], [ 13 ]. Among various neural network architectures, con volutional neural networks (CNNs) are a class for which each neuron of a hidden layer is connected to a local subset of neurons of the previous layer , and the weights of such connections are shared across the neurons of the same hidden layer . This allo ws the network to learn sets of hierarchical filters tailored for dif ferent characteristics of the data [ 14 ]. Fig. 2. Simulated synthetic aperture flow datasets from a 3D arterial CFD simulation. (a) D high resolution rabbit artery model used and extracted cross section (marked in blue), (b) 2D velocity field sampled from the plane cross section of the 3D CFD simulation, (c) Simulated ultrasound datasets using randomly sampled CFD velocity field flow particles to be used for training the VFI-CNN CNNs hav e been used to estimate Optical Flow in a supervised manner using sev eral v ariants of a conv olutional architecture named Flo wNet. Remarkably , FlowNet has developed as far as reaching real time optical flow detection up to 140 fps, outperforming state of the art methods [ 15 ], [ 16 ]. In this study , we demonstrate the feasibility of using a CNN based on the Flo wNetSimple architecture for velocity estimation of ultrasound images. I I . M E T H O D S A. Neural network Figure 1 illustrates the architecture of the modified FlowNetSimple CNN, where each con volutional layer but the last is using ReLU activ ation functions. The input of the network are image patches of 128 x 128 pixels and the output is a two channel image of the same size, where each channel represents a component of the 2D velocity vector . The size of the conv olution filters is 5 pixels for the first layer and 4 pixels for all follo wing layers, while between con volutional layers we perform a max-pooling operation over a neighbourhood of 2 by 2 pixels. This is followed by upsampling and refinement of the network with 4 by 4 kernels. Here we perform ’unpooling’ and con volution while concatenating with feature maps from our con volutional layers, as seen in Fig. 1 (red to blue), to preserve high-level information through our CNN [ 15 ]. B. T raining Data T o learn how to perform displacement vector field estimation, the neural network requires a sufficiently large and di verse training dataset of vascular flo w images. For each training dataset, a corresponding ground truth velocity field is required so that supervised learning can be performed. While ground truth information is difficult to determine for real world data, one solution is to generate simulated ultrasound images of vascular flo w with kno wn predetermined vector fields. This generated data, howe ver , may not be sufficiently realistic to let the network generalize well for real world scenarios. Furthermore, it is computationally expensi ve and time consuming to build up large training sets of simulated data compared to sampling from external sources. An alternativ e approach is to use real world acquisitions and to attempt to approximate the ground truth using a gold standard VFI method. Consequently , to generate a versatile and lar ge set of training data, both methods were used. 1) V ascular Flow Simulation: An artificial dataset of vascular flo w was generated using StarCCM+ v11.02.01-R8 and a high-resolution luminal arterial surface of a 12- month-old male New Zealand White rabbit. Blood flow was prescribed time-dependent with rigid walls, Newtonian rheology and a no-slip boundary condition. T o dissipate initial transients, three cycles ov er a total of 0.9 second divided in 1200 timesteps were simulated. Blood density was ρ = 1044 . 0 kg m 3 and dynamic viscosity was µ = 4 . 043e − 3 P a s . A physiological cardiac wa veform and flow splits to the main branches were obtained from the literature [ 17 ]. V elocity fields were saved for a number of 800x400px axial planar sections distributed along the vessel (Fig. 2 a). 2) Flow F ield W indow Sampling: From these axial planar sections, velocity fields and flow particle positions were repeatedly extracted using a 128x128px region of interest with random position, orientation and point in time, for fi ve consecutiv e time samples (Fig. 2 b). 3) Synthetic Apertur e Image Generation: Using the Ul- traSound T oolBox [ 20 ] and Field II (v3.24) [ 18 ], [ 19 ] a time-resolved moving speckle phantom was created from the sampled flow field windows (Fig. 2 c). T o scan the phantom, a synthetic aperture setup with five virtual point sources was implemented. For the imaging sequence a 128-element linear array probe with a centre frequency of 8 MHz and 60% bandwidth was modelled. In transmission a Hanning apodization was implemented. The excitation signal consisted of a tapered 3 cycle sinusoidal with a 50% Tuk ey window and the pulse repetition frequency was 5 kHz. Using this approach, a training dataset of a total of 2400 unique B-Mode sequences was generated. 4) IUS 2018 SA-VFI Challenge Datasets: T o further increase the div ersity of our training data and to make the CNN more robust, we also sampled the datasets provided by the IUS 2018 SA-VFI Challenge using the flow field window sampling (T able I ). As these came without velocity field information we generated a ground truth v elocity data using T ABLE I. Data sets from IUS 2018 SA-VFI Challenge Name Simulation Real Carotid Bifurcation X Straight V essel at 105 X X Straight V essel at 90 X X Spinning Disk X T ABLE II. Overview of number of training, validation and testing datasets Name T raining V alidation T est Simulated CFD 1920 240 240 iUS SA-VFi Challenge 4980 300 300 iUS SA-VFi Challenge SVD 5200 500 500 our previously published gold standard Echo-PIV algorithm [ 7 ]. Cross-correlation was performed using 32 by 32 pixel interrogation windows, which size was halved at each of the three iterations to refine the results. T o demonstrate that our CNN is capable of generalizing, the Carotid Bifurcation model (T able I ) was excluded from the training data and was instead used exclusi vely for testing the performance of our CNN. An overvie w of the challenge training sets used can be seen in T able II . 5) In Line Data Augmentation: Due to the Hilbert trans- form applied during beamforming the training data is complex in nature. T o use a real valued CNN, we extracted and concate- nated the real and imaginary part of the input images, doubling the input data size. By in verting the complex intensity , we further augmented the training dataset, doubling its size again. Each image was normalized by its maximum absolute value. T o extract moving blood speckle from tissue a Singular V alue Decomposition filter (SVD) was applied. The datasets were ev aluated with and without a binary vessel mask. C. Error measur e T o e valuate our algorithm, we use a percentage endpoint error measure (EPE), which is defined as EPE = 100 · E h q ( v x − v 0 x ) 2 + ( v y − v 0 y ) 2 i V max (1) where v x and v y are the velocity components in axial and lat- eral dimension of the ground truth, v 0 x and v 0 y are the estimated components, V max is the maximum velocity magnitude in the image and E [ · ] is the sample av erage across all pixels. I I I . R E S U L T S A N D D I S C U S S I O N While our method works well on simulated dataset, with a median EPE of 6.5%, we acknowledge that the highest error can be found for the unfiltered challenge datasets, with a median EPE of 14.5%. W e also observed, that the whiskers of the T ukey boxplots in Fig. 3 extend sev eral tens of percent abov e the median v alue, suggesting the presence of some flow fields for which the velocity field was not correctly Fig. 3. Results obtained with different datasets. Note that the Bifurcation scan has not been used in training. T ABLE III. Median EPE for different datasets EPE Dataset T rain T est Simulated CFD 7.3 11.5 Simulated CFD + m 6.5 10.6 Challenge 14.4 14.5 Challenge + m 13.6 13.2 Challenge SVD 12.9 12.8 Challenge SVD + m 12.1 12.2 Bifurcation 1 n.a. 8.1 Bifurcation 1 + m n.a 7.0 estimated. Areas of stagnation or zero movement were distinguished well from the flow field (Fig. 4 ). Rotational and straight flow is represented faithfully . When the network was applied to region 1 of the bifurcation model, which samples the External Carotid Artery (ECA), as shown in Fig. 5 , the result yielded a median EPE error of only 7.0% (see T able III ), despite the fact that the network had never been trained on a bifurcation phantom. Howe ver , when applied to the more complex flow of the Common Carotid Artery (CCA), the method did not accurately predict the velocity . Future work should focus on the optimization of the neural network architecture, for example by adding a cross corre- lation stage in the architecture, as done for example in the more advanced FlowNet model [ 15 ], [ 16 ]. The training and ev aluation must be performed on a wider range of di verse ultrasound data: the latter could be done for example by designing an unsupervised training approach, that could lev er- age different ultrasound scans without the need of a known ground truth velocity field. Also, regularization of the cost function, for example by imposing a smooth velocity field, should be ev aluated to increase robustness. Lastly , the structure of the erroneously estimated flow fields should be carefully in vestigated to understand and circumvent the limitations of the approach. Fig. 4. Example of results using the proposed approach. Fig. 5. Example of results for samples of the Carotid Bifurcation for regions of the External Carotid Artery (top) and Common Carotid Artery (bottom). I V . C O N C L U S I O N S In this study , we hav e shown that CNN can be used to estimate vector displacement fields from 2D flo w ultrasound images. Furthermore, the preliminary results suggest that our approach is able to generalize the input data and to adapt to previously unknown geometries. The use of CNNs could lead to real time ev aluation of large 2D flow fields. In addition, it could be used to automatically extract clinically relev ant information and to pro vide care teams with useful interpretations. V . A C K N OW L E D G M E N T W e gratefully ackno wledge the support of the NVIDIA Corporation for the their donation of a NVIDIA TIT AN Xp GPU which was used to carry out this research. W e would also like to acknowledge the BHF Centre of Research Excellence and the EPRSC for their funding. Finally , we thank the ULIS group for their support and Antonia Creswell for helpful feedback. R E F E R E N C E S [1] A. Lusis. Atherosclerosis. Nature, 2000, V olume 407, Issue September , pp. 233-241 [2] A. Goddi, C. Bortolotto, I. Fiorina, M. Raciti, M. Fanizza, E. T urpnin, G. Boffelli, F . Calliada. High-frame rate vector flo w imaging of the carotid bifurcation. Insights into Imaging, 2017, V olume 8, Issue 3, pp. 319-328 [3] J. Jensen, Comparison of vector velocity imaging using directional beamforming and transverse oscillation for a conv ex array transducer . Proceedings of SPIE Medical Imaging conference, San Diego, USA, 2014, pp. 904012 [4] J. A. Jensen and S. I. Nikolov , ”Transv erse flow imaging using synthetic aperture directional beamforming, ” 2002 IEEE Ultrasonics Symposium, 2002. Proceedings., Munich, Germany , 2002, pp. 1523-1527 vol.2. [5] J. A. Jensen and P . Munk. A new method for estimation of velocity vectors. IEEE T ransactions on Ultrasonics, Ferroelectrics, and Frequency Control, pp. 837-851, 1998 [6] B.Y . Y iu, S. S. Lai, and C. H. Alfred. V ector projectile imaging: Time- resolved dynamic visualization of complex flo w patterns. Ultrasound in medicine & biology , 2014, 40(9), pp. 2295-2309. [7] C. H. Leow , E. Bazigou, R. J. Eckersley , A. C. H. Y u, P . D. W einberg, M.-X. T ang, Flow V elocity Mapping Using Contrast Enhanced High- Frame-Rate Plane W a ve Ultrasound and Image Tracking: Methods and Initial in-V itro and in-V i vo Evaluation, Ultrasound in Medicine & Biology , V olume 41, Issue 11, 2015, Pages 2913-2925, [8] C. V illagomez-Hoyos, M. Stuart, T . Bechsgaard, M. Nielsen, J. Jensen. High frame rate synthetic aperture vector flow imaging for transthoracic echocardiography . Proceedings of SPIE, 2016, pp. 979004 [9] J. Jensen, C. Hoyos, M. Stuart, C. Ewertsen, M. Nielsen, J. Jensen. Fast Plane W av e 2-D V ector Flow Imaging Using Transverse Oscilla- tion and Directional Beamforming. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 2017, V olume 64, Issue 7, pp. 1050-1062 [10] D. Sun, S. Roth, and M. J. Black. A quantitative analysis of current practices in optical flo w estimation and the principles behind them. International Journal of Computer V ision, 106(2), 115-137, 2014. [11] J. Wulf f, and M. J. Black. Efficient sparse-to-dense optical flow esti- mation using a learned basis and layers. In Proceedings of the IEEE Conference on Computer V ision and Pattern Recognition (pp. 120-130), 2015. [12] P . W einzaepfel, J. Rev aud, Z. Harchaoui, and C. Schmid. DeepFlow: Large displacement optical flo w with deep matching. In Proceedings of the IEEE International Conference on Computer V ision, pp. 1385-1392, 2013. [13] A. Ranjan, and M. J. Black . Optical flow estimation using a spatial pyramid network. In IEEE Conference on Computer V ision and Pattern Recognition (CVPR), July 2017. [14] N. Aloysius and M. Geetha, ”A revie w on deep conv olutional neural networks, ” 2017 International Conference on Communication and Signal Processing (ICCSP), Chennai, 2017, pp. 0588-0592. [15] A. Dosovitskiy , P . Fischer, E. Ilg, P . Hausser, C. Hazirbas, V . Golkov , D. Cremers and T . Brox. Flownet: Learning optical flow with conv olutional networks. In Proceedings of the IEEE International Conference on Computer V ision, pp. 2758-2766, 2015. [16] E. Ilg, N. Mayer, T . Saikia, M. Keuper , A. Dosovitskiy , and T . Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE conference on computer vision and pattern recognition (CVPR), July 2017. [17] A. Barakat, R. Marini, C. Colton. Measurement of flow rates through aortic branches in the anesthetized rabbit. Laboratory animal science, 1997, V olume 47, Issue 2, pp. 184-189 [18] J. A. Jensen. FIELD: A Program for Simulating Ultrasound. Medical and Biological Engineering and Computing, pp. 351-352, 1996 [19] J.A. Jensen and N. B. Svendsen: Calculation of pressure fields from arbitrarily shaped, apodized, and excited ultrasound transducers, IEEE T rans. Ultrason., Ferroelec., Freq. Contr., 39, pp. 262-267, 1992. [20] USTB toolbox, www .ultrasoundtoolbox.com