Left Ventricular Wall Motion Estimation by Active Polynomials for Acute Myocardial Infarction Detection

1 Abstract — Echocardiogram (echo) is the e arliest and the primary tool for identifying regional w all motion abnormalities (RWMA) in order to diagnose myocardial infarction (MI) or commonly known as heart attack. This paper proposes a novel approach , Active P olynomial , which can accurately and robustly estimate the globa l motion of the Left Ventricula r ( LV ) wall from any echo in a ro bust and accurate wa y. The proposed algorith m quantifies the true wall m otion occurring in LV wall segments so as to assist cardio logists diagnose early signs of an acute MI. It further enables medical experts to gain a n enhanced v isualization capability of e cho images through color-coded segments along with their “maximum motion displ acement ” plots helping them to better assess wa ll motion and LV Ejection -Fraction (LVEF). The outputs of the method can further help echo-technicians to assess and improve the quality of the echocardiogram re cording. A major contribution of this stud y is the first p ublic echo database collection composed by physic ians at the Hamad Medic al Corporation Hospital in Qatar. Th e so -called HMC-QU database will serve a s the benchm ark for the forthcoming relevant studies . The results over HMC-QU dataset show that the proposed approach can achieve high accuracy, sensitiv ity and precision i n MI detection even though the echo quality is quite poor, and the temporal resolution is low. I nd ex Te r ms — Echocardiogra m, Le ft Ventricular Wall Motion Estimation, Myocardial Infarction I. INTRODUCTION arly d etection of an acute myocardial infarction (MI) [1] , [2] or in general a coron ary artery disease (CAD) requ ires an accurate estimation of t he regiona l and global motion of th e le ft ventricle (LV) o f the heart. Early an d fundamental sign s of a CA D are b elieved to show in LV wall motion as ab normalities in one or several seg ments of the LV wall , where a segment may move “abnormally” or “non - uniformly”. This abno rmality can be defined as a “weak motion”, known as hypokinesia , “no motion”, known as akinesia or “out o f sy nc”, known as dyskinesia . The primary too l to detect, identify and quantify such regional wall motion abnormalities (RWMA ) is the p atient’s e chocardio gram (echo), which is notoriously difficult, and subjective and, therefore, Serkan Kiranyaz ( mkiran yaz@qu.edu.qa ), Rayyan Ahm ed , Rayaan Abouhasera, Junaid M alik ( hafiz.malik@qu.edu. qa ), and Ridha Hamila ( hamila@qu.e du.qa ) are w ith the Department of Engineering, Qatar University, Doha, Qatar. Tahir Ham id and Rashi d Mazhar are wi th HMC, Doha, Qa tar. highly operator dependent. Although RWMA is the first abnormality to set in with the o nset of myocardial ischemia, preceding metab olic and electrocardio graphic abnormalities, it is currently on ly us ed as a secondary diag nostic too l in p atients with non -diagnostic ECG or when diagnosis is not evidenced (or shown/proven) by “standard” means, despite the fact th at echo and particularly myocardial strain imaging p rovide an early diag nosis of an acute MI when RWMA is p resent. The reasons for n ot using echo as a first line d iagnostic tool, for suspected MI patients , are largely interpretation al. Echo cardiographic in terpretation, such as ultrasound scan, is highly operator-dep endent as it depends upon the visual estimation of the left ventricle (L V) m uscle-wall motion, its radial displacement and d eformations. As a result, the final diagnosis su ffers s everely from the h igh inter -observer an d in tra- observer variability, making it prone to human errors and misjudgments. To address these challenges, there is a n eed for an auto mated, robust and accu rate tool that can ass ist cardiologists and echo- technicians understand and interpret echo more accurately, which may lead to saving lives . Despite the need, th ere are only few studies in the literatu re wh ich p roposed an auto matic method for the LV wall motion estimation and abnormality detectio n from echo [3] . This is not surprising because first of all, there is no publicly av ailable benchmark echo database with ground -truth labels. Second, capturing the global motion of an arbitrary shaped LV segment is difficult especially when the quality and/o r spatial/temporal resolution of the echo is low. Finally , “ motion estimation ” in a video is known to be an ill -posed problem [3]-[5] even for natural vid eos with distinct objects/ textures. Th is is true both for den se (pix el-based) or local (few p ixels or blo ck-based) motion estimation. Therefore, for a typical echo, which m ay be too noisy , estimating the true motion of th e entire LV wall from such local (gro up of) pixels will be difficult and in some cases , impossible [6] . A recent study [7] has attempted to ad dress this problem by two Machine Learning approaches. Both approaches have obtain ed low accuracies varying 57% to 85.4% and specificities varying 47% to 77. 6% despite the fact that the echo videos were in relative high quality and pre-processed . Due to these limitations and drawbacks, echocardiographic strain and strain-rate imagin g (deformation imaging), instead, became th e main focus o f many studies as a non -invasive metho d Aysen Degerli ( aysen.de gerli@tuni.fi ), Morteza Zahibi ( morteza.zahib i@tuni.fi ), Moncef Gabbouj ( moncef.ga bbouj@tuni.fi ) are with the F aculty o f Informat ion Technolog y a nd Communica tion S ciences, Tam pere University, Tam pere, Finland. Left V entricular W all Motion Estimation by Active Polynomials for Acute Myocardial Infarction Detection Serkan Kiranyaz, A ysen Degerli, Tahi r Hamid, Rash id Mazhar, Rayyan Ah med , Rayaan Abouhasera, Morteza Zabihi, Juna id Malik, Ridha Ha mila, and Mon cef Gabbouj E 2 for th e assessment of myocardial fun ction. LV wall motion and wall deformation (strain) are different assess ments. First of all, while th e motion can be estimated and assessed by human experts in a subjective way, this is not po ssible for the strain becau se it represents th e amount (and rate) of deformation of th e LV wall such as longitudinal shortening (negative strain) and radial thickening (positive strain) during myocardial contraction. A human eye, no matter how trained and experien ced the practitioner can b e, cann ot sense or measure this from an echo . On th e o ther hand, strain and strain rate (SR) measurements are also d erived from the myocardial velo cities over th e LV wall. The most common technique is called “Speckle Tracking” [6]-[23], which attempts to capture the motion by trackin g “spec kles” ( natural acoustic markers) in the 2D ultrasonic image (echo). Speckles are the brightest patches and the y usually are about 20 to 40 pixels . In prior studies, they ar e assumed to be “sta ble” from fr ame to frame. So, under the assumption of an accurate frame- by -frame trackin g, the change o f a speckle position gives its velocity and thus the LV wall motio n is somewhat reflected b y the motion of the s peckles. They are chosen at the LV segment b orders to produ ce the motion curves from which (negative) strain (i.e., shortening) and SR can be estimated. Therefore, the accuracy of the strain and SR estimation, too, so lely depend on the accuracy of the motion estimation (and trackin g) of each join t speckle during one or more cardiac cycles. Although the motion estimation of a lo cal group of pixels is much easier than the estimation of the global motion of the entire LV wall, it still s uffers from this ill -posed nature of the problem, e.g ., due to small sensor movements, medium to very high noise levels, poo r temporal and s patial resolution, and other sources of distortions [6]. For example, the minimum fra me r ate required for a reasonable sp eckle trackin g is 6 0 fps. A higher frame rate is another challeng e sin ce it reduces the spatial resolution resulting in poo r tracking. Moreover, robu stness is a crucial issue since it is a well-known fac t that different speckle tracking algorithms produce d ifferent results. In this study, we shall demonstrate that even when robust key-points are u sed instead of those speck les, robustn ess still remains the main prob lem. Eventually, the curves resulting from strain imagin g are highly variable and their interpretation for diagnosis is subjectiv e and experience-depend ent [11] . Even under ideal c onditions, many studies [11] , [19]-[23] reported around 80 -85% sensitivity rate for the detection of the infarcted segments. For instance, in an earlier study on 30 patients Leitman n et al. [1 9] found that 8 0.3% of the infarcted segments and 97.8% of normal segments were adequately recognized by speckle tracking based 2D -strain im aging . Even when the Doppler echo i s used, the longitudinal Doppler s train data displayed 85% sensitivity and specificity for the detection of infarcted segments [20]. As a resu lt, despite some pro mising and recently published studies [21]-[23], s train imaging b ased on speckle tracking is not ready yet for routine assessment of MI viability [11]. In this study, we propose a novel Computer Vision me th od , called Active Polynomials (APs) th at can capture the global motion of the LV wall in a robust and accurate way. Our objective is to mimic an expert cardiologist who can analyze the echocardiogram records by vis ually searching for any RW MA for the early detection of an acute myocardial dysfun ction. While th e cardiologist can only perform this subjectively, APs can be used to capture and measure the true motion of the LV wall; th erefore, it can identify and quantify the region al motion abnormalities. In order to accomplish this, APs are formed on the endocardial boundary of the LV wall and chamber. This boundary is the most promising salient feature o f an echo where the maximum contrast usually occ urs. In order to capture this boundary , we u se the Activ e Contour (or snake) [24] with an artificial constrain t embedded on the LV wall with the Ridge Polynomials (RPs). RPs will ensure a converging snake initiated in t he LV ch amber; however, due to the high noise le vel, the snake may still partially fail to capture the true endocardial boundary of the LV wall. This is why we shall then f it a 4 th order polyno mial o ver the snake in order to obtain the APs that can cover the boundary o f the entire LV wall in a smooth and continuous manner. On ce th is is repeated for all fra mes in th e echo , then th e global motion of the segments o n the LV wall can be modelled b y “motio n activity” curves and their maximum displacement can be measured. W hile the APs can be used as an automatic tool to detect and identify objectiv ely a possible segment motion abnormality (and hence to identify the in farcted seg ments causing a possible MI), it can also be used as an en hanced visualization platform over the raw ech o to assist cardiologists o r echo operators for a more accurate diag nosis and echo quality assessment. Fin ally, th e proposed method is tested extensively over the first benchmark echo dataset, HMC-QU, solely created for this purpose by th e physician s in Hamad Medical Corp. (HMC) Hospital an d researchers in Qatar Univ ersity (QU) . HMC-QU encapsulates 160 echos, 89 of wh ich are from acute MI patients and the rest from normal (non-MI) patients . HMC- QU is th e first publicly available dataset and is th e larg est by far among the non - public ones u sed in prior wo rk s [6]-[23]. The 4-chamber v iew echos of the MI patients are labelled by a group of physicians in HMC Hospital, and the proposed method is evaluated bas ed on these ground -truth segment labels of each echo. The rest of the paper is organized as follows. Section II provides preliminary work on MI detection on echo and Active Contours . Section III presents the proposed LV wall motio n estimation for MI d etection and identification . The benchmark echo dataset, HMC-QU will first be intro duced in Section IV. Then b oth quantitative and qu alitative evaluations of the proposed app roach over th e HMC-QU d ataset will be detai led and the MI detection performance will be analyzed together with a computational complexity analysis. Finally, Section V concludes the paper and suggests topics for future research . II. P RELIMIN ARIES A. MI Detection by Loca l Motion Estimation MI is a major cause o f death worldwid e and gaining momentu m especially during the last decade. In pathology, MI can be defined as myocardial cell d eath du e to prolonged ischemia. When a coronary artery is blocked, the CAD sho ws th e first signs of perfusion ab normalities due to the lack of o xygenated blood flow to the LV tis sue within minutes from the occlusion of the coron ary artery, leading to severe isch emia, which p roduces regio nal wall motion ab normalities (RWMAs). These RWMAs can be visualized by echo . This is the onset of a MI that can be even before th e pat i ent feels a ch est pain or angina. That is why echo is an essential tool to detect the onset of myocardial ischemia and to identify the arteries with blockage. 3 Figure 1: LV 17-segment model and respective views [18]. In an echo , there are different standardized LV segmentation models, such as 16-segment, 17-segment, and 18-segment mod els. The American Hea rt Association Writing Group on Myocardial Segmentation and Registration for Cardiac Imaging recommendation is to use the 1 7-seg ment model [18 ] shown in Figure 1. In this study, the proposed tech nique has been developed and extensively tested on the 4-chamber view; howev er, it can directly be used on th e other views. As illustrated in Fig ure 2, i n the 4-chamber view, the LV has 7 segments where 6 of them except the apical cap (segment-4) exhibit a u niform motion activity. Pr ior studies that attempt to co mpute the longitudinal strain by speckle - tracking echocardiography fix a speckle at each segment boundar y and attempt to track it during on e or few cardiac cycles. Due t o the aforementioned lim itations and drawbacks, even in ideal cases (i.e., low n oise, high frame rate, and full contrast), the speckle tracking methods can ach ieve around 80 -85% sens itivity and specificity levels. End ocardi al Bord er Epi cardi al Bord er LV W al l Seg me n t bo unda rie s Seg me n t Refe ren ce bo rder Sta rt Poi nt Ape x Poi nt End Poi nt Figure 2: The LV wall and its bo rders (right) in th e 4 -chamber view. The segmentation of the LV wall and the s tart-end points (left). In this study, we first investigated whether m ore stable and robust key-poin ts can indeed cure the drawbacks of speckles. So, instead of a motion analysis based solely on a single speckle on each segment bou ndary along with its lo cal region , we extract a large number of hig hly robust key-po ints on the LV wall by using the met ho d called, “Speed ed up Robust Feature” (SURF) [25] , which belongs to the family of well-known key-point extractors in Computer Vision [2 5]-[28] including the first and perhaps the most popular one, “Scale Inv ariant Feature Tra nsform” (SIFT). Juan and Gwun in [25] evaluated the performances of SIFT, PCA -SIFT and SURF methods for scale, rotation, and affine transforms as well as for blur and illumination changes. This study has sho wn that SIFT performed s lightly superior in most experiments but with the slowest speed (highest computational complexity). Some other experiments have shown that SURF was the fastest and th e most stable [26]. Obviously, both SIFT and SURF points show a superior robustness over the n aïve speckles with the s ole feature of “high brightness” which ma y change abrup tly due to n oise, sensor disturbance or other possible factors. Figure 3: Accurate (top) v s. erroneous (bottom) tracking of the SURF points on the LV wall. Our aim is to investigate whether a large nu mber of key -points can indeed be us ed to capture the global motion of the LV wall. Accordingly, we can also find ou t wh ether t hey can be used to compute the strain in a ro bust and accurate manner. Fo r the fo rmer, the results have shown that e specially when the noise level is high, even the majority of the r obust SURF points may lead to er roneous tracking as shown in Figure 3. W hile SURF points are coherent and able to capture the global motion for those echos o n top of the figure, the zo omed s ections o f th e echo s on the bottom clearly show that the majority of t he SURF points, despite their robust and stable nature, could n ot be tracked due to th e high level of noise appearing on the next frame. Obviously for the latter aim, tracki ng of SURF points on the b oundaries of the segments may fail too and this will result in erroneous strain computations, which in turn yield misdiagnosis of the heart status. This shows that such “bottom - up” approaches to capture the true motion of the LV wall using the local ke y-points may neither be r obust nor reliable for MI detection. This clearly indicates that the global motio n should in stead be captured in a “top - d own” fashion. The two pos sible solutions for this approach are the (accurate) segmentation of the LV wall or extraction of the entire endocardial boundary at each frame of the echo. Th ere are few recent att empts for the former using recent Deep Learning paradigms [ 29]-[32]; however, they can not s till guarantee an accurate seg mentation especially wh en the echo quality is poor. In th is study, we shall focus on the latter, the extraction o f the endocardial bound ary, which can indeed be performed with a hig h accuracy usin g the proposed approach. The starting point for th is is the Active Contours [24], which will b e 4 reviewed next. B. Active Contours An active contour (or snake) is an elastic 2D spline whose contour is guided by internal (smoothness and curvature) and external (image gradien ts and edges) constraints. The problem is transferred to the minimization of a joint (total) energy,   , that can be expressed as follows:            󰇛   󰇛  󰇜       󰇛  󰇜    . (1) where   is internal and   is external energy terms that define the respective constraints ,  is the regu larization coefficient. In this study, we used a more recent and improved version of snake [33] the details of which are covered in Supplementary (A) . Figure 4 : Sn ake method for LV segmentation on 6 echos. Reasonable (top) vs . erroneous (bottom) results. Snake method has been directly used for LV segmentation in a recent study [34]; however, it has only been tested on the frames of a single echo. Although the res ult was satisfactory, obviously such a limited evaluation is not su fficient. Especially when the qu ality of echo degrades, the snake may fail to conv erge to the true boundary of the LV wall. Typical examples can be seen in Figure 4 (bottom) wh ere the snake not on ly failed to converge to the true boundary due to lack of co ntrast, it also presents severe noise sensitivity on the boundaries as sh own by the white arrows. Thi s basically demonstrates the fact that the “snake -o nly ” ap proach cannot exhibit the requ ired robustness and accuracy to cap ture the LV endocardial boundary along with its global motion. In the next section, we shall detail how the proposed method addres ses effectively this drawback by using the proposed appro ach with Active Polynomials (APs). III. M ETHODOLOGY The propos ed method consists of two co nsecutive phases, as illustrated in Figure 5. The first phase is the LV wall extraction. In the secon d phase, using the APs formed on the LV wall, 7 segments are extracted for M I detectio n, identification and for furth er enhanced visualization capabilities to assist cardiologists perform their diagnosis. In the following sections, we shall d etail each phase. A. LV Wall Extraction As illustrated in Figure 5, the formation o f Active Polyno mials (APs) is perf ormed in three stag es over each frame of the echo. Th e first stage is the formation o f an artificial wall, Ridge Po lynomials (RPs), to preven t the div ergence of the snake. The second stage is the formation of the snake withi n the LV. Finally, the third stag e is the composition of the APs over the snake. In the following sub - sections, we shall detail each stage. 1) Ridge Polynomia ls The proposed approach to capture th e endocardial bound ary is designed to address the two drawbacks of the active contours on echos. The first and the foremost problem is the partial divergence of the snake when the contrast is po or, e.g., s ee th e three ech os in Figure 4 (bottom) wh ere the sn ake fails to co nverge to the boundaries of s egments 4, 5 and 6. To preven t this, we artificially enhance th e contrast by b uilding a wh ite wall o n top of the brightest section (i.e., the ridge) o f th e LV wall. In tho se prob lematic echos in the figure, the ridge can even be invisib le to the nak ed eye due to the lack of con trast; h owever, it still exists with low brightness values wh ile still having the maximum intensity in a local neighborhood. So, the idea is to build a sufficiently thick wall ( e.g. 6 pix els) b y incrementally increasing the intensity value (e.g. 200 to 255 for 8-bit image representation) on top o f the conn ected series of brightest pixels (i.e. the rid ge). To obtain the anchor points on the ridge, we use the start and end p oints as illustrated in Figure 2 along with the topmost point of th e apical cap . As shown in Fi gure 6 considering that the LV boundary is divided into left an d right parts equ ally from the apex point, two lines are fitted from the top to the start and end points. The ridg e (maximum intensity ) points are detected as movin g th e 14 equally distanced anchor po ints towards the boundary. New s et of anchor points are defined b y stretching the a nchor points horizontally t o obtain a pair of left a nd right an chor points shown in Figure 6 . Finally, o ver the left and right anchor points , two 4 th order ridge polynomials (RPs) are initially fit which will con stitute the borders of th e search region, i.e., the “Region of Interest” (RoI) inside of which the actual ridge points will be searched . Once the ridge points are detected, the 4 th order ridg e polynomial is fit to these ridge points u sing the regularized Least-Square (LS) optimization. The details of fitting an n th order polynomial o ver m>n+1 p oints u sing regularized LS method is given in Supplementary (B). The RPs are then used to create the artificial wall (barrier ) so that the snake is gu aranteed to converge to th e LV wall. 2) The Constrained S nake As illustrated in Figu re 7 (left), th e sn ake is initialized as a mini - form of the RPs within the LV wall which is encapsulat ed by the actual RPs. After 300 iterations, the snake converges to the true endocardial boundary of the LV wall in Figu re 7 (right). In this particular echo, without the artificial wall made by the two RPs, the snake would have diverged on the right side of the LV wall du e to lack of co ntrast of t he ec ho shown in this figure. In t his echo a nd on several o thers wh ere the snake is divergent, the artificial wall solved this problem. However, the second d rawback o f the s nake approach, the h igh n oise sensitivity, is still evident. The sn ake may fail to co nverge to th e true boundary du e to the noisy speckles within the blood chamber. Furthermore, excess noise level usually makes the snake unnecessarily detailed at the b ound aries as shown in Fig ure 4 (wh ite arrows at th e bottom). In o rder to address these drawbacks, the formation of the proposed Active Polynomials (APs) will be detailed in the next sub-section. 5 Ridge Polynomials Constrained Snake Active Polynomials Segmentation (7-Segments) LVEF Computation LVEF 55 % Norma l Echo MI ( akines ia ) LVEF < 15 % 55 %>LVEF> 15 % Echo Yes MI No LVEF < 30 % MI ( akines ia ) Yes No Norma l Echo Segment Classification: normal =1 MI =2 Visual ization Pre-Proc essin g: - Filt ering - Star t-End point s Bi -Products: - Colo r- coded E cho visua lization - Segm . D isp . P lots - Max . Disp . Snaps hot Formation of APs Detection, Identification and Visualization Outputs            L V E F 31.06% Se g-1 (1) Se g-2 (2) Se g-3 (2) Se g-5 (2) Se g-6 (1) Se g-7 (1) M oti on 26.7 9% 17.76 % 15.8 7 % 14.4 6 % 24.74 % 30.2 1 % Figure 5: The overview o f the proposed method for MI detection and iden tification. The yellow shaded blo cks generate the outputs on the right-most sid e. An cho r poin t s R i dg e Pol y nomial s Figure 6 : (left) The anch or points (yellow) on LV wall and (right) final ridge polynomial (RP). sn ake(0) sn ake(N -1) L e ft RP R i gh t RP Figure 7: Initialization o f the snake inside the two RPs (t=0) and the final snake obtained after N = 3 00 iterations. 3) Ac tive Polynomia ls Although some sections of the snake suff er fr om occasional “over - fitting” p roblem possibly du e to th e excess nois e, eve n su ch a problematic snake can still serve as the initial “reference” to capture the endocardial bound ary with smooth p o lynomials. For th is pu rpose, a pair of 4 th order p olynomials, for the left and right side of the LV wall are fit to the poin ts of the snak e using the regularized LS . For th e left one, the 9 equally spaced snake p oints between th e start to apex , and for the right one, betwe en ap ex and the end are used . Sin ce each polynomial will assu me a smo oth shape of the active contour, we call them Active Polynomials (APs). As shown i n Figure 8, th e two parts of the snake (pu rple and yellow) are used to compose a pair o f APs and finally, they are used to create 7 segments (segments 1 to 7 counter -clockwise) of the 4-chamber view ech o. STAR T END APEX Segme nt-1 Segme nt-2 Segme nt-3 Segmen t-4 Segme nt-6 Segme nt-7 Segme nt-5 STAR T END APEX Figure 8: Snake poin ts (left) are used to create a pair of APs (right) that are used to create 7 segments of th e 4-chamber view. B. MI Detection, Identification and Visualization This is the second block in Figure 5 which u ses th e outp ut of the first block, APs, to perform the glob al motion analysis. APs are divided into 7 seg ments as shown in Figure 8 (right) and their movement (displacement) is monitored. Once the global moti on of each seg ment is captured by simply ev alu ating the “rate o f displacement”, we can mimic a typical cardiologist ’s diagnosis of a motion anomaly b y detecting which segment or seg ments are showing signs of abnormal ( non -uniformity o r lack of) motion activi ty . However, before going into motion analys is, th e LV Ejection-Fraction (LVEF) ratio is first computed as follows: 6              (2) where  and  are the e nd -diastolic and end-systolic volumes, resp ectively. In a 2D echo, o ne can estimate them by computing   and   , which are the minimum and maximum area of the LV chamber, respectively . They are proportional to the total number of p ixels encapsulated by the snake or by t he tw o APs. The recommendation for LVEF to indicate a “reference” (normal) and “sev erely abnormal” LV activ ities for both men and women, are LVEF ≥ 55% and LVEF < 30% , respectively [35] . Following this recommendation , the proposed motion an alysis will n o longer be performed when LVEF ≥ 55 % and the echo can directly be classified as normal . However, for the lower limit, we empirically use a m ore conservative threshold, L VEF ≤ 15, which is obviously a sensitive marker of myo cardial dy sfunctio n and a clear s ign of MI . For this severe case of myocardial dysfunction , the echo with all the segments can directly be classified as MI . Th erefore, motion analysis will only be performed when 55% > L VEF > 15% a nd the outcome of the motion analysis will determine whether the e cho is normal o r MI. In this case, if th ere is at leas t one myocardial segment with abnormal ( hypokin esia o r akin esia ) motion activity then MI is detected an d th e correspond ing arteries with blo ckage can be identified. Figure 9 : Computation o f the normalized maximu m displacemen t of the 6 segments of th e 4 -chamber v iew echo with the ground - truth labels (normal = 1, infarcted = 2) When a cardiologist visually evaluates the motion activity o f a 4-chamber view echo, th e infarcted segments that show a “reduced” motion (or almost no motion at all) compared to o ther segments are identified either as hypokinesia or akinesia. The motion assessment is obviously independent fr om the resolution of the echo . The stu dy over the segments labeled as abno rmal by the cardiologists in HMC-QU datab ase has indicated that those segments that move less than 20% o f th e minimum interval to the corresponding segment on the other side o f the chamber are diagnosed as MI. Since the minim um interval between corresponding segments is resolution dependent, the ratio of maximum disp lacement to this minimum interval will therefore allow us to mimic the cardio logist’ s evaluation in a qu antitative way. Let   be the maximum displacement of the segment,  where   󰇝 󰇞 . Let  󰇛    󰇜  be the minimum interval during a cardiac cycle o f an echo o f the corresponding segments,       where              respectively. Since the maximum motion ( of a segment),   , is proportional t o the ( maximum) displacement o ccurred from en d-diastole to end-systole , the proposed method computes   as the maximum dis placement of each segment and normalizes them by  󰇛    󰇜  . Both measurements are one -pixel accu rate and the ratio is then compared with an empirical threshold (e.g., 19%) that is selected just below 20%. Figure 9 illustrates a sample echo f rom a MI patient where cardiologists labeled segment 3 as akinetic, and the rest as normal . The two APs extracted for the frames corresponding to end- diastole and en d-systole are shown in t he figure. Knowing t he end- diastole as t he first frame, one can easily find the f rame of the end- systole by simply searching for the maximum overall s egment displacement. However, in this study instead o f co nsidering the segments in the end-systole frame, we search for the maximum displacement of each individual segment, which may not necessarily come from the end-systole frame. Experiments show that most of th e segment-wise maximum disp lacement indeed occurs at the end-systole frame; ho wever, occasionally it may also occur at the frames within a close v icinity (e.g., ±1 -2 frames). It is straightforward to notice the “reduced ” motio n from the gap between 󰇧    󰇛    󰇜  󰇨 , ratios of segment 2 and th e rest where th e latter group has ratios above 19 %. Therefore, the resu lts are in full agreement with the g round-truth labels made by the cardiologists. In order to compute the maximum d isplacement of a segment, at first, uniformly sampled   points are taken ov er the segment and then the segment displacement can be app roximated by averaging the poin t-wise distances, as expressed b elow for the 2 nd segment ( S =2) shown in the figure (e.g., for     ).          󰇛 󰇜     (3) where   󰇛 󰇜 is the i th point ’s maximum displacement of the segment S =2 as shown in the figure. There are several options to compute individual point-wise distances,   󰇛 󰇜 . When they visually assess echo, cardio logists consider th e motion in bo th x and y directions. Therefore , we can u se L 1 , L 2 o r L ∞ n orms to compute   which are expressed as below.                         󰇛      󰇜   󰇛      󰇜       󰇛                󰇜 (4) where 󰇛     󰇜 and 󰇛     󰇜 are th e x- and y-coordinates of the corresponding i th points on the segment at the end-diastole and end- systole frames, respectively . A clos er look will reveal the fact that   does not exactly mimic the aforementioned way th e 7 cardiologist assess motion since   only reflects the dis tance in either x- or y -direction. B etween the remaining n orms, we use the L 2 norm since it is the natural distan ce m etric for the human perception. An alternative way to assess the segment motion is the segment displacement plots as shown in Figu re 10 , which show the displacement of each segment during on e cardiac cycle. This plot is in fact more informative than the maximum motion ratios given in Figure 9 because the instantaneous and average mot ion o f each segment can also be computed besides their maximum displacements. However, we still perform our motion analysis based o n the m aximum displacement due to the simple fact that the derivative operator is noise sensitive and the displacement curves will inevitably bear certain level of measurement noise. Finally, th e proposed method presents several enhanced visualization op tions th at will si gnificantly ass ist medical experts perform their diag nosis. For instance, the color-coded segments formed over the two APs provide a cardiologist with a better motion estimation than the one from the raw (gray -scale) echo since the cardio logist can now see , distingu ish and assess each individual seg ment displacement and (instantaneous) motion in a visually e nhanced manner. Another bi-product of the p roposed method is the maximum displacement snapshot as sho wn in Figure 9, which allows cardiologists to v isualize the displacement anomaly. Figure 10 : Se gment displacement plots for the e cho shown in Figure 9 with respect to the first ( end-diastole ) frame. IV. E XPERIMENTAL R ESULTS A. HMC-QU Benchma rk Dataset HMC-QU benchmark echo dataset h as been created by collaboration between Qatar Univ ersity (QU) and Hamad Medical Corporation (HMC) Hosp ital. HMC-QU contain s 1 60 4-chamber view echo recordings obtained at the HMC hospital between 2018 and 2019 . These cases are from over 10 000 echo s performed i n a year including more than 8 00 cases admitted with acute ST elevation Myocardial infarction. The echos included in our assessment belonging to the 89 MI patients (all first tim e and acute MI) an d the rest are no rmal. T here are 13 women and 76 men in the MI patient group. All MI echos were o btained from patients who were admitted with a diagnosis of acute MI with evidence obtained from ECG, cardiac enzymes and who underwent coronary angiogram/angioplasty to treat the MI . These patients had echos obtained within 24 hours of adm ission or in some cases b efore they underwent coronary angioplasty. All “Normal” echos were defined, as the echos of the patients no t admitted for MI (acute or previous) b ut acquired for other reasons in cluding h ealth check and investigation of murmurs. All 4-chamber view echos hav e been labelled segment -wise by cardiologists in HMC hospital. The 6 segments of 4 -chamber v iew o f each ech o is labelled as: n ormal =1, hypokinetic =2 and akinetic =3. Echos ar e acquired by devices from different vendors, such as Phillips Ultrasound machines an d GE Vivid (GE-Healthcare-USA) U ltrasound Machine. The temporal resolution (frame rate p er second ) o f the echos is 25 fps. The spatial resolution also varies from 422 × 636 to 768×1024 pixels. The duration of each echo taken for analysis is one card iac cycle. B. Results In this study, each echo is categorized as norma l or MI while each segment in a 4 -chamber view echo is catego rized as normal (1 ) or infarcted (either hypokinesia = 2 or akin esia = 3). As illus trated in Figure 5 the motion analysis is on ly performed if 1 5% < LVEF < 55 %. If LVEF ≥ 55 %, all segments are assumed to be normal (1) and if LVEF ≤ 15 % all segments ar e assumed to be akinetic . Otherwise, th e motion analysis will d etermine whether the echo is normal or MI . If there is at least one infarcted segment with abnormal motion activity , then the echo is assumed to be MI ; otherwise, normal . A segment is assu med to be infarcted if its motion ratio is belo w 19 % . The thresholds, 55% and 30 % for LVEF are recommended in [35] . Once all echos in the dataset together with their segments are categorized by the pro posed algorithm, then the confusion matrices (CM) are formed by evaluating the assigned categories with re spect to the ground-truth labels. Th is enables us to co mpute the following s tandard performance metrics for MI detection and infarcted segment identification performances: classification accuracy ( Acc ), sensitivity ( Sen ), specificity ( Spe ), and p ositive pr edictivity ( Ppr ). CM elements are the h it/miss coun ters such as true positive ( TP ), true negativ e ( TN ), false p ositive ( FP ), and false negative ( FN ). The following standard performance metrics can now be expressed using them : accura cy is the ratio of the nu mber of correctly detected echos (or s egments) to the total number of echos (segments) ; sensitivity (or Recall ) is the rate of correctly detected MI ec ho s amon g all MI echos in the dataset ; s pecificity is the s en sitivity of the normal echos; and , p os itive predictivity (or Precision ) is the rate o f correctly detected MI echos in all the echos detected as MI. Finally, the false alarm r ate ( FAR ) can b e defined as: FAR = 1 – Spe . Figure 11 shows the end-diastole , middle a nd end-systole frames of a normal and MI echos where the 7 s egments are color- coded over th e two APs along with th eir maximum displacement snapshots. Th e relative motion (displacement) of th e segments 1 , 2, 3 an d 5 makes it straightforward to detect the motion abnormality on the MI echo while ap parently all segments of th e normal echo move in a uniform manner. Despite the fact that the quality is quite p oor with a sig nificant noise in th e interior (blood) chamber and the temporal resolution is low (25 fps) in both echos, the proposed method successfully captures the glob al motion of the LV wall. The MI detection and identification (detect ion of the segments with abnormal motion activity) performances are presen ted in Table 1 (per segment), Table 2 (o ver all segments) and Table 3 (over all echo s), resp ectively. Several important observations can be made based on these results. First of all, all results are based on the selected motion ratio threshold, 1 9%. No optimizatio n or fine- tuning was performed on this thresh old for maximizing certain criteria. The high accuracies achieved for detecting inf arcted segments and echos approve the validity o f this threshold; however, th ere is still room for improvement. The most cruc ial 8 performance criterion is of cou rse sens itivity ( Recall ) for infarcted segments and especially MI ech os. Especially the latter , Sen(MI) > 91%, indicates an elegant performance level co nsidering the low temporal resolutio n and th e poor quality of many echo s in the dataset. T he secondary o bjective is to m inimize the f alse alarms ( or equivalently to maximize the specificity ). This is a misdiagnosis case for MI, which can be co rrected by the cardiologists when the proposed method causes a false alarm. The FA R was rather low ( i.e., < 9%) for detecting seg ment motion ab normality; however, the results in Table 3 show a rather high FAR . The main reason is that m is -diagnosing a normal segment as infarcted suffices to misclassify the echo as MI. On the other hand, th e fixed threshold used fo r detection, 19%, may yield such misdiagnosis because the proposed metho d has a certain sensitivity for capturing the global motion, i.e., in the vicinity of ±2 pixels and, therefore, a slight variation from the actual displacement may cause such a misclassification even if it is as low as 1-2 p ixels, e.g. assume tha t for a normal segment,  󰇛    󰇜     and   is measured as 1 8 pix els with +2 p ixels bias. This will caus e a false alarm since 󰇧    󰇛    󰇜  󰇨     . End-Diastole End-Systole Max . Displacement Snapshot Figure 11 : End-diastole, middle and end-sy stole frames of a normal (top) and MI (bottom) echos. Their maximum displacement snapshots are shown on the right. Table 1: T he performance of th e proposed m ethod (per- segment) on detecting the infarcted seg ments. Segment Sensitivity Specificity False alarm Precision F1 -score Accuracy 1 0.8966 0.9389 0.0 61 0.7647 0.8254 0.9313 2 0.7843 0.8807 0.1193 0.7547 0.7692 0.85 3 0.8676 0.8587 0.1413 0.8194 0.8429 0.8625 5 0.8545 0.9362 0.0638 0.8868 0.8704 0.906 6 0.6875 0.9187 0.0813 0.6875 0.6875 0.871 7 0.7 0.9348 0.0652 0.6087 0.6512 0.9051 Table 2: T he performance of the pr oposed method on detec ting the infarcted segment s. Sensitivity Specificity False alarm Precision F1 -score Accuracy 0.8157 0.9141 0.0859 0.7790 0.7969 0.8875 Table 3: MI detection performance o f the proposed method over the HMC -QU benchmark d ataset with 160 echos. Sensitivity Specificity False alarm Precision F1 -score Accuracy 0.9101 0.7606 0.2394 0.8265 0.8663 0.8438 9 Table 4: T he performance of the p roposed method (per - segment) on detecting the infarcted segments over ech os with a reasonable quality . Segment Sensitivity Specificity False alarm Precision F1 -score Accuracy 1 0.9286 0.9469 0.0531 0.8125 0.8667 0.9433 2 0.8125 0.8817 0.1183 0.7800 0.7959 0.8582 3 0.9048 0.8974 0.1026 0.8769 0.8906 0.9007 5 0.9020 0.9444 0.0556 0.9020 0.9020 0.9291 6 0.7586 0.9464 0.0536 0.7857 0.7719 0.9078 7 0.7647 0.9435 0.0565 0.6500 0.7027 0.9220 Table 5: T he performance of the pr oposed method on d etecting the infarcted segments over echo s with a reasonable qu ality. Sensitivity Specificity False alarm Precision F1 -score Accuracy 0.8602 0.9295 0.0705 0.8252 0.8423 0.9102 Table 6: M I detection performance of the proposed m ethod over echos with a reasonable quality in the HMC -QU benchmark dataset. Sensitivity Specificity False alarm Precision F1 -score Accuracy 0.9512 0.8136 0.1864 0.8764 0.9123 0.8936 The main reason for some o f the false alarms encountered was the extremely poor image quality of some o f the echos in the dataset (19 out of 160), which degrades sign ificantly the actual performance level o f th e proposed method. In s uch cases , even an expert Cardiologist may not perform an accurate diagnosis on these echos. The reason we h ave included t hem in the d ata set is to accomplish a realistic case and show the m ain source of misdiagnosis. When those 19 echos with su ch poor quality are excluded from the evalu ation, the actual performance of the proposed method is presented in Table 4 , Table 5 and Table 6 . Finally, the poor temporal reso lution is also the common d rawback among all the echos and as dis cussed earlier, this, alone, is sufficient to render out the usage of any method based on Speckle Tracking o r local motio n estimation. The results presen ted in the above tables indicate that the propo sed approach is quite robust against this drawback and certain level o f quality degradations. C. Computational Comp lexity Analysis Due to the unoptimized and sequential execution of the proposed method, its computational complexity is the sum of the individual computational complexities of the individual blo cks illustrated in Figure 5 . Please refer to Supplementary (C) for the computational complexity of each block along with the overall computational times in an unoptimized implementation . V. C ONCLUSI ONS AND F UTURE W ORK In this study, we propose a global method for finding the true motion of the LV wall by Active Polynomials which are formed at the endocardial boundary of the LV wall for each frame of an echo . Since th e proposed method does not depend o n local motion estimation and trackin g, it is largely immune to the well-known ill- posed behavior and noise sensitivity of the 2D motion esti mation. The propo sed method is designed to “mimic” an expert cardiologist to capture the global motion in a similar manner so as to assess the regional motio n with respect to the motion uniformity . Th e global extraction of the true motio n of the LV wall enables to detect and identify the regional wall motion abnormalities (RWMA) which in turn can diagnose a myocardial infraction (MI) in an o bjective way. Since echo is the primary tool that can indicate th e o nset of a Myocardial Ischemia long befo re the ECG, this will help the detection of a MI at the earliest possible stages -practically as soon as the echo is acquired. Moreover, th e proposed method voids the subjectivity and operator dep endability of th e echo interpretatio n and ass essment since it can quantify the true measures o f LV w all motion, (maximum) displacemen t an d LVEF. Besides a standalone diagnostic tool, the proposed method can also offer sev eral assistive bi-products such as enhanced visualization ca pabili ties by color-coded APs over the raw echo, a sn apshot o f the maximum segment displacements to localize the motion abnormalities, segment d isplacement plots that can provide a d eeper motion analysis and ev en a quality assessment to ol for the echo acquisition. The last feature is especially importan t since an echo technician can fine-tune th e echo prob e manually until the proposed method can successfully compose the APs o ver the LV wall bound ary. Especially, when the d iagnosis of the p roposed method and the (group of) cardiologist(s) dif fers, it can also b e used as a “verification” tool that can give a second chance to the cardiologists to re -assess the cases o r seg ments with mismatching diagnosis. Finally, when a range is used in stead of a fixed threshold for classifying the segments, th e proposed method can draw the attention for the Unsure cases where th e segment motion is not definitive. Cardiologists can handle such difficult cases and make the final decision perhaps by ass essing other echo views. A crucial objective of th is study i s to achieve an utmost robustness again st the high noise level in e chos with a poo r temporal resolution. Experiments over such echos show that this objective h as been well-acc omplished. This is particularly important because it is in feasible to analyze su c h echos with the current state- of -the- art MI detection methods su ch as “Speckle Tracking” or any other method based on local motion estimation. This has been v erified in this study by a case study using the SURF key-points for tracking. Finally, for a proper performance evaluation, the HMC-QU dataset con tains ground -truth labels. This is not only the first benchmark dataset that will b e publicly available for the research community, it is also the largest collection e ver compiled which co nsists of n ormal echos and echos of both male and female acu te MI p atients with different ag es. An extensive set o f exp eriments ov er this benchmark dataset demonstrates that the proposed method achiev es an elegant sensitivity and precision for detecting MI and diagnosing th e 10 RWMA. For the latter, sp ecificity is quite high yielding a low false- alarm rate. However, for the MI d etection, th e proposed method yields a significant false alarm rate; although not as severe as the false negativ es, we aim to reduce false alarms with t he joint analysis of the other views. Moreover, we aim to improve the speed of the proposed method by par allel computing paradigms and optimized implementation to achieve real-time an alysis of the acquired echo. We ex pect to achieve an ev en better detectio n and identification performance with a prop er analysis on th e motion curves usi ng a Machine Lear ning approach. T hese will be the topics of our future work. R EFERENCES [1] K. Thygese n, J. S. Alpert and H. D. White, “Universal Definition of Myocardial I nfarct ion,” Circulation, 200 7. [2] K. T hygesen, J. S. Alpert, A . S . J affe, M. L. Simoo ns, B . R. Chaitman and H. D. Whi te , “Third universal definiti on of myoc ardial infarctio n”, Re v. Esp. Cardiol. v ol. 66, pp.232, 201 3. DOI : 10.1016/j.rec.2012.11.00 6 [3] V. Sudarshan, U R Acharya, E. Yin-Kwee Ng, C. S. Meng, R. S. Tan, and D. N Ghista, "Automated identifica tion o f infarcted myocardium tissue characterizati on using ultrasound images: a revi ew." IEEE reviews in biom edical engineer ing, vol 8, pp. 86 -97, 2015. [4] J. Konrad, and E. Dubois, "Bayesian estimat ion of motion vector fields", IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 14, pp. 910-927, 1992. [5] R. Kordasiewicz, M. Gallant, a nd S. Shirani, "Affine motion prediction based o n translational motion vectors," IEEE Transactions on Circuits and Systems f or Video Tech nology, vol. 17, pp.1 388 - 1394 , 2007. [6] M. Dandel, H. Lehmkuhl, C. Knosall a, N. Suramelash vili and R. Hetzer, "S train and Strain R ate Imag ing by Ec hocardiograp hy – Basic Concepts and Clinical Applicability", Current Ca rdiology Reviews, vol. 5, pp. 133-148, 2009. [7] H. A. Omar, A. Patra, J. S. Domingos, P. Leeson, and A. J. Noblel. "Automated Myocardial Wall Motion Cla ssification Using Handcrafted Features vs a Deep CNN-Base d M apping." In 2018 I nt. Conf. of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 3140 -43, 2018. https://doi. org/10.1109/EMB C.2018.851306 3 [8] A. Heimdal, A . Støy len, T. S kjærpe a nd H. T orp, “Real -Time S train Rate Imaging of t he Left Ventricle by Ultrasound,” Journal of the American Society of Echo cardiography, vol. 1 1, no. 11, pp. 1 013-1019, 1998. [9] M. B ansal and P. P. Sengup ta, “Speckle Tracking Echocardio graphy,” in Perioperative Tr ansesophagea l Echocardiogr aphy , Elsevier Inc, pp. 298-300, 2014. [10] D. A. Bluemke, J. A. Noble an d M. Tee, “I maging techniques for cardiac strain and deformation : comparison of echocardiography, cardiac magnetic resonan ce and cardiac computed tomography,” Expert Review of Cardiovasc ular Therapy, vol. 11, n o. 2, pp. 221-231, 201 3. [11] O. A. Smiseth, H . Torp, A. Opdahl, K. H. Haugaa and S. Urheim, “Myocardial strain imaging: how usefu l is it in clinical decision making?,” European Heart Journal, vol. vol. 37, no. no. 15, pp. 1196- 1207, 2015. [12] A. S. a. S. Solomo n, “Myocardial Deformation Imaging: Current Status and F uture D irections,” Circulat ion, vol. 125, no. 2 , pp. 244-e248, 2 012. [13] J. H. Riffel, M. Keller, M. Aurich and S. J. Buss, “Assessment of global longitudinal strain using standard ized myocardial deformation imaging: a modalit y independent software approach,” Clinical Research in Cardiology, vol. 1 04, no. 7, pp. 5 91-602, 2015. [14] A. Stoylen, “Basic st rain ultraso und for clinicians. Princi ples and technology for strain and strain rate imaging b y echocardiograph y.,” Norwegian University of Scuen ce an d Tec hnology, June 2016. [O nline]. Available: http://f olk.ntnu.no/sto ylen/strainrate/meas urements.h tml . [15] Y. Nagata, “Intervendor Variability of Two-Dimension al St rain Using Vendor-Specific and Ven dor- Independent Sof tware,” Journal of the American Society of Echocar diography, vol. 28, no. 6, pp. 630 -641, 2015. [16] J. - U. Voigt, “ Defi nitions for a Common Standard for 2D Spec kle Tracking Echocard iography: Consens us Document of the EACVI/AS E/Industry Task Force to Standardize Deformati on Imaging,” Journal of the American Society of Echocardiograph y, vol. vol. 28, no. no. 2, pp. 183-193, 2015. [17] P. Biaggi, “ Compariso n of Two Different Speckl e Tracking Softwar e Systems: Does the M ethod Matter?,” Echocardiography, vol. 28, no. 5, pp. 539-547, 2011. [18] S. A. Coulter, “ Echocard iographic evaluation of coronary arte ry d isease,” Coronary Arter y Disease (978 1447128274), pp. 217-252, 2 015. [19] M. Leitman, P. Lysyansky, S. Sidenk o, et al.. “Two -dimensional strain - A novel software for real-time quantitative echocardiograp hic assessment of myocar di al function ”, Journa l of t he America n Socie ty of Echocardiograp hy vol. 17, pp.102 1-1029. 2004. 10.1016/j.echo. 2004.06.019. [20] Jamal F, Strotmann J, Weidemann F, Kukulski T, D’hooge J, Bijnens B, Van de Werf F, De Scheerder I, Sutherland GR. Noninvasive quantification of the contrac tile reserve of stunned myocar dium by ultrasonic strain rate and strain. Circulation. vol . 104, pp.1059 - 1065, 2001. [21] M. Ba nsal, L. Jef friess, R. Leano , J . Mundy, T. H . Marwic k "Assessment of m yocardial viabi lity a t dobutamine ec hocardi ography by deformation analysis u sing tissue v elocity and speckle-"JACC Cardiovasc Imaging, vol. 3, pp.121-131, 2010. [22] M. Becker, R. Hoffmann, H. P. Kuhl, H. Grawe, M. Katoh, R. Kramann, A. Bucker, P. Hanra th P, A. Heussen, "Analysis o f myocardial deformation based on u ltrasonic pixel tracking to determine transmurality in chronic m yocardial infarction", in European H eart Journal, vol.27, pp. 2560-2566, 2006. [23] V. Delgado, R. J. van Bo mmel , M. Be rtini, C. J. Borleffs, Marsan NA, Arnold CT, Nucifora G, van de Veire NR, Ypenburg C, Boersma E, Holman ER, Schalij MJ, Bax JJ. "Relative merits of left ventricular dyssynchrony, left ventric ular lead position, a nd myocardial scar t o predict long-term survival of ischem ic heart failure patients underg oing cardiac resynchronizati on therapy", Circulation, vol. 123, pp.70 -78, 2011. [24] M. Kass, A . Witk in and D . Terz opoulos, "Sna ke: ac tive cont our mode l",” Int.Journal of Com puter Vision, vol . 1, pp. 321-331, 198 7. [25] L. Juan, O . Gwu n, "A compariso n of SI FT, PCA-SIFT and SURF," Int. J. Image Process. v ol. 3 (4) pp. 14 3-152, 2009. [26] D. Mistry, A. Banerjee, "Comp arison of Feature Det ectio n and Matchin g Approaches : SIFT and SURF", GRD Jo urnals- Global Research and Development Journal for Engineeri ng, Vol. 2, Issue 4, March 2017. ISSN: 2455-57032 017 [27] T. Tuytelaars, K. Mikolaj czyk," Local invarian t feature detectors: a survey, " in Found. Trends Comp. Graph. Vis. vol. 3 (3) pp.177-280, 2007. http://dx.doi .org/10.1561/0 600000017 . [28] A. Canclini, M. C esana, A. Redondi, M. Tagliasacchi, J. Ascens o, R. Cilla, "Eva luation of low-complexity visual feature detectors and descriptors," in Int.Conf. on Digital Signal Processing, pp. 1 -7, 2013. http://dx.doi.or g/10.1109/ICD SP.2013.6622757 . [29] Carneiro, Gustavo, and Jacin to C. Nascim ento. "Combining m ultiple dynamic models and deep learning architectures for t racking the left ventricle endocardium in ultrasound data." IEEE transac tions o n pattern analysis and m achine intellige nce 35.11 (2013): 25 92-2607. [30] Dong, Suyu, et al. "A left ventricul a r segmen tation method on 3D echocardiography using deep lea rnin g and snake." 2016 Computing in Cardiology Confer ence (CinC) . IEEE , 2016. [31] Luo, Gongnin g, et al. "A deep learning netw ork fo r right ventricle segmentation in short-axis MRI." 2016 Computi ng in C ardiology Conference (CinC) . I EEE, 2016. [32] Carneiro, Gustavo, Jacinto C. Nascimento, and António Freitas. "The segmentation of the left ventricle of the heart from ult rasound data using deep learning architectures and derivative-based search metho ds." IEEE Transactions on I mage Processi ng 21. 3 (2011): 9 68-982. [33] T. F. Chan and L. A . V ese, “Active contours wit hout e dges,” I EEE Trans. image Process., v ol. 10, no. 2, p p. 266 – 277, 2001. [34] M. Landgren, N. Christian O. and A. Heyden, "Segmentat ion o f the Left Heart Ventricle in Ultrasound Images Using a Region Based Snake," Medical Imagin g: Image Processi ng, Proc. of SPI E Vol. 8669, 2 013. [35] Lang RM1 et al.; Chamber Qua ntification W riting Gr oup; Am erican Society of Echocardiogra phy's Guidelines and Standar ds Committee; European Association of Echocardiography, "Recommendations for chamber quantificatio n: a report from the America n Society of Echocardiograp hy's Guideli nes and Standards Committee and the Chamber Quantification Writing Group, developed in conjunction with the E uropean A ssociation of E chocardiogra phy, a branch of t he E uropean Society of Cardiology," Journal of American Society of Echocardiograp hy, vol.18(12), pp.1440-1463, 2005. 11 SUPPLEMENTARY MATERIAL A. Active Contours The snake , s , at time, t , co nsists of finit e nu mber of vertices ,  󰇛  󰇜 as shown in Figure 12 , a nd each vertex will move (slowly) to another state (location) in order to minimize the total energy,   . (x 0 ,y 0 ) (x 1 ,y 1 ) v i =( x i ,y i ) v n-1 =( x n-1 ,y n-1 ) s = sn a ke (t) Figure 12 : A snake with n v ertices at iteration, t . The next state of each v ertex is shown with a red arrow. The internal energy measures the b ending (elasticity and stiffness) co st that can be expressed in terms of vertices, v(s) over the snake , s , as follows:    󰇛  󰇜           󰈅       󰈅  (5) Where fo r simp licity  an d  can be assumed to be constant parameters and the 1 st and 2 nd derivativ e can be approximated as,                           󰇛      󰇜  󰇛      󰇜            (6) Note that, the term,   , forces f or a “smooth” spline whereas the term,       , forces for th e min imal cur vatur e (no corner s and discontinuo us edges). Setting,    will, therefore, allow snake to converge to corners/edges. The reason that this energy term is called “internal” is because both d erivatives depends only on th e relative position of the consecu tive vert ices; i.e., there is nothin g about the image intensity o r g radient v alues. The secon d energy term,   , is design ed to for ce th e snake to converge towards certain image attributes such as “l ight/dark” objects or “object boundar ies”. Since in this s tudy , we aim to capture the LV wall en docardial boundary , we shall set this term to the negative gr adient of the image to minimize the external energy (to maximize the likelihood of the edge pixels to constitute the snake). This yields,    󰇛  󰇜      󰇛  󰇜      󰇛  󰇜  (7) As a result, in 2D image grid of an echo frame, Eq. (1) can be expressed as,                                           󰇛  󰇜      󰇛  󰇜     . (8) The objective is to find the snake,  󰇛  󰇜 , which yields th e minimum to tal energy, i.e.,   󰇛  󰇜     󰇛  󰇜    󰇛  󰇜  (9) Using the C alculus of Variations, it can be shown that the states of the minimu m total energy will co rrespond to the zero crossings of the Euler-L agrange equation that can be expressed as,                     󰇛  󰇜    ( 10 ) In ord er to create an iterative solution starting from an initial (estimate) of t he snake,  󰇛    󰇜 , w e can par ametrize Euler- Lagrange equation with time  󰇛  󰇜   󰇛    󰇜 , the mo tion of convergence to a (local) optimum can be ex pressed,  󰇛   󰇜       󰇛    󰇜         󰇛    󰇜         󰇛    󰇜    󰇛    󰇜       󰇛    󰇜         󰇛    󰇜         󰇛    󰇜    ( 11 ) Eq. ( 11 ) can then be solved by the discrete- time approximation and Lin ear Algebra which is u sed in th is study. Alternatively, there ar e various m ethods to find the optimal state of the snake such as Gr adient Descen t, greedy search, evolu tionary search, Dynamic Pro gramming, etc. In this study , we used a rec ent variation o f snake method proposed in Chan-Vese, [3 3]. I n this variant, the prob lem defined by [33] is the minimizatio n of an en ergy-based segmentation. Con sider a bounded open subset  of   , th e image       , the ev olving curve  in   , as the boundary of an open subset  of  . Then, th e  󰇛  󰇜 represen ts the region  , and   󰇛 󰇜 is the regio n    . T he bo undary of the image   is deno ted as   . Then, the fitting term,   is defined as in Eq. ( 12 ) wh ere   the snake curve, and the variables is       are th e averages of   inside  and o utside  , respectively. 12   󰇛  󰇜    󰇛  󰇜      󰇛    󰇜          󰇛 󰇜      󰇛    󰇜          󰇛 󰇜 ( 12 ) In this case, the optimal curve is the bou ndary of the image   which is also the m inimizer term in the Eq. ( 13 ) as follows:    󰇝   󰇛  󰇜     󰇛  󰇜󰇞       󰇛   󰇜     󰇛  󰇜 ( 13 ) By adding some regularization terms to the minimization equation abov e, the final energy function al  󰇛        󰇜 is defined by,  󰇛        󰇜     󰇛  󰇜      󰇛  󰇜          󰇛    󰇜        󰇛 󰇜         󰇛    󰇜  󰇛 󰇜       ( 14 ) where    ,    and        are the fixed parameters. Therefore, the minimization problem proposed b y the Chan - Vese method is as f ollows,        󰇛        󰇜  ( 15 ) The problem ca n be redefined in a level set fo rm where    represented by     as ; 󰇱     󰇝 󰇛    󰇜     󰇛    󰇜  󰇞  󰇛  󰇜    󰇝󰇛   󰇜      󰇞  󰇛  󰇜      󰇝󰇛  󰇜      󰇞 ( 16 ) The ev olution of curve,   in the ab ove level of set fun ctions of Eq.( 16 ) is illustrated in Figur e 13 which also reflects the relation between Chan -Vese [33] and Kass et al. [24 ] methods. Using the Heaviside f unction,  in Eq.( 17 ) and Dirac measure,   in Eq. ( 18 ) the energy function can be r ewritten as Eq.( 19 ),  󰇛 󰇜  󰇥        ( 17 )   󰇛  󰇜     󰇛 󰇜 ( 18 ) Ou tside  > 0 I nside  < 0 N  = 0  N N Figure 13 : The p ropagation of curve C={(x ,y): ϕ (x,y )=} in normal direction .  󰇛        󰇜      󰇛    󰇜    󰇛    󰇜       󰇛 󰇛    󰇜 󰇜          󰇛    󰇜         󰇛    󰇜          󰇛    󰇜      󰇛     󰇛    󰇜 󰇜  ( 19 ) The variables    and    are fixed, and the energy function in Eq.( 19 ) is minimized with respect to   which conclu des the Euler-Lag range e quation for  . The descent direction is chosen as an artificial time   where th e par tial d ifferentiation eq uation is expressed as f ollows;      󰇛  󰇜              󰇛      󰇜     󰇛      󰇜        󰇛   󰇜     󰇛     󰇜    󰇛    󰇜      󰇛  󰇜       󰇍       ( 20 ) where  󰇍   represents the exterior n ormal to the boundary  , and   󰇍  d enotes th e normal derivative of   at th e bou ndary. Lastly , Eq.( 20 ) ca n converge to its solution by gr adien t descent method. 13 B. Regularized Least-Square for n th Order Polyno mial Fitting Assume that we have m points in 2D surf ace ,    󰇛      󰇜       , and we wan t to f it n th ord er polynomial wh ere m> >n . The n th order polynom ial,  󰇛 󰇜 can be expressed with n +1 coefficients as fo llows:    󰇛 󰇜          ( 21 ) One can turn this to a LS optimization p roblem b y min imizing the total erro r,             󰇛 󰇜        󰇛     󰇛  󰇜 󰇜      ( 22 ) By definin g the matrix,  coef ficient vec tor, x and outpu t vector, b, this pr oblem can be turned to a linea r system as,                                                                                                                                    ( 23 ) or equivalently in a linear system equ ation,  󰇛 󰇜      ( 24 ) where  is the mx(n+1) matr ix with th e n th power of the x coordinates of the 2D poin ts are the elem ents of the ( n +1)- dimensional column v ector , c, of the poly nomial coef ficients, ( c u ) and b is the m -d imensional colum n vector o f y coo rdinates of the 2D points . For m>n+1 , this is clearly an over-deter mined linear system, which means that there a re more con straints (linear e quations) than unknowns (parameter s). In such systems there is no (exact) solution, one may on ly get a u nique least- square (LS) solu tion as defined in Eq. ( 22 ) o r equivalen tly, th e LS solution of this equation,   , can be expressed as fo llows:                󰇛    󰇜     ( 25 ) However,  may not b e even of full ran k matr ix, i.e.,  󰇛 󰇜 = r < n +1 , in which case,    will b e singular and the inver se cannot be compu ted. T o address this pro blem, we make use o f the Singular Value Decomposition (SVD) o f  as follo ws,                  ( 26 ) where U and V ar e mxm and (n+1)x(n+1) o rthogonal matrices which ho lds the eigenvectors of th e square matrices,   and    , r espectively, as the column vectors. The mx(n+1) matrix,  , can be expressed as,                                                                                                                                           ( 27 ) where            are the singular values or equivalently the eigenvalues of m atrices,    and   . This can yield the LS solution,   , regardless wh ether or n ot A is singular,                        ( 28 ) However, the LS solution,   , ca n still yield large values, the so - called “exp losion” of the LS solution, due to noisy values in matrix  (the inpu t = powers of x coordin ates o f the m p oints ) or in vector b (the ou tput = y co ordinates of the m points ), or both. A crucial d isadvantage here is that the smaller non-zero singular values will resu lt in ev en lar ger exp losion o f   . In order to prevent this, w e shall regularize the LS solu tion by optimizing the LS error togeth er with the magnitude of th e LS solution as,        󰇛             󰇜 ( 29 ) where  is the reg ularization parameter. It is straightforward to show that this join t optimization can be expr essed as,         󰇼󰇡   󰇢  󰇡   󰇢  󰇼               ( 30 ) where   is now an 󰇛     󰇜  󰇛    󰇜 full-rank matrix ( r= n +1 ) an d therefore, the LS solution over   can be ob tained by using Eq. ( 25 ) as,   󰇛󰇜                              󰇛        󰇜     ( 31 ) The i th eigenvector of        can be obtained by solv ing,         󰇛        󰇜    󰇛       󰇜   ( 32 ) So it is clear that matrix      has t he same eigenvector,   , as matrix    but a larger eigenvalu e, 󰇛       󰇜 . Ther efore, using the o rthogonality of the eigenvectors, o ne can write the eigenvector decomposition of      , and its inverse as f ollows: 14                                                          ( 33 ) Finally, using Eqs. ( 26 ) and ( 33 ) yield s the regularized LS solution,   , expressed as,      󰇛󰇜             󰇭                 󰇮                󰇭                  󰇮 ( 34 ) A direc t com parison of Eq. ( 28 ) and ( 34 ) will reveal the fact that the regularized LS solution will no longer be effected from the noisy eigenvectors with v ery small eigenvalues,   since the when     , also          too with a rea sonable cho ice of the regular ization parameter,  (e. g.,    ). C. Compu tational Comp lexity Analysis In this ap pendix, we shall first detail the computational complexities o f the three m ain blocks: Active Conto urs (snakes) with N vertices, n th order po lynomial fitting and motion (displacemen t) computation of each segment. 4) Active Contou rs The activ e contour method used in the proposed method is Chan-Vese [3 3] which def ines a m inimization pr oblem of an energy-b ased segmentation. The en ergy f unctional  󰇛        󰇜 is defined by [33] is as follo ws;  󰇛        󰇜     󰇛  󰇜      󰇛  󰇜         󰇛    󰇜        󰇛  󰇜        󰇛    󰇜  󰇛  󰇜       ( 35 ) where   ≥ 0,  ≥ 0 and      > 0 are th e fixed par ameters. The algorithm has a co mplexity of  󰇛  󰇜 for each iter ation (N = 300) where  is the n umber of rows and,  is the n umber of columns in th e image. The sizes of th e echo records in th e dataset vary f rom 422×636 to 768×1024 pixels. 5) n th Order Polynomia l Fitting The regularized LS solution for the n th order polynomial fitting to the m points in 2D is expressed in Eq. ( 22 ). The regularized LS solution,    is expressed as,      󰇛󰇜            ( 36 ) which can b e expressed as a linear system as follows:            ( 37 ) where   is an 󰇛      󰇜  󰇛    󰇜 full-rank matrix and hence       is a 󰇛    󰇜  󰇛    󰇜 is a f ull-rank square matrix,    is a    co lumn v ector. The solutio n of a linear system, Ax=b where A is an    full-rank matrix is in 󰇛  󰇜 in the worst case. This is negligible in this applicatio n since 4 th order polynomials are used both for RPs and APs, an d thus n +1=5 is the ran k of the m atrix A . Since we have m po ints in 2D surfac e,    󰇛      󰇜       , an d m> >n , th e significant computation al co mplexity will result fr om the m atrix multiplication,      , which is in 󰇛   󰇛   󰇜󰇜 an d matrix - vector multiplication, , which is in 󰇛  󰇜 . Therefore, both computation al complexities are linearly pro portional with t he number of 2D p oints, m . 6) Segment Motio n Estimation and LVEF computation This is the least com putationally d emanding b l ock of all since the segm ent d isplacement can be approximated by averaging the point-wise distance s as expressed in Eq. (3) . With L 2 norm and     points, this req uires only 5 summ ations and 10 differences, a total of 15 summation operation, which is negligible. This will be repeated by 6 segmen ts for each frame in the echo , and f or all frames so as to compu te the maximum displacement of each segment. Similarly, the LVEF computation is even simpler than the segment -wise motion estimation since it r equires o nly th e coun ting (incrementin g) the number of pixels in side the evolved snake. 7) Computationa l Times In the current un -parallelized and un-o ptimized MATLAB (version 2018 a) implementation of the proposed method over a PC with 3.2 GHz CPU an d 16 GB memory , the to tal execution time to process a cardiac cycle is abou t 36.9 second s b y 2 seconds per fram e. The individual time share of each operation as illustrated in Figu re 5 is given in Table 7. It is o bvious that the majority of th e computational comp lexity arises from the formation of the snake and then the formation of the both types of polynom ials, RPs and APs. Table 7: E xecution time percentages o f each block of the proposed meth od illustrated in Fig ure 5 . Process Time (%) RPs 17.45 Snake 78.15 APs 0.25 LVEF 0.19 Disp. Ratio 0.057