Dictionary-based Method for Vascular Segmentation for OCTA Images

Dictionary-based Metho d for V ascular Segmen tation for OCT A Images Astrid M. E. Engb erg, V edrana A. Dahl, and Anders B. Dahl T echnical Univ ersity of Denmark, Kgs. Lyngby , Denmark { asteng,vand,abda } @dtu.dk Abstract. Optical coherence tomograph y angiography (OCT A) is an imaging tec hnique that allows for non-in v asive inv estigation of the mi- cro v asculature in the retina. OCT A uses laser light reflectance to mea- sure moving bloo d cells. Hereby , it visualizes the blo od flow in the retina and can b e used for determining regions with more or less blo o d flow. OCT A images contain the capillary netw ork together with larger bloo d v essels, and in this pap er we propose a metho d that segments larger v essels, capillaries and background. The segmentation is obtained using a dictionary-based mac hine learning metho d that requires training data to learn the parameters of the segmentation mo del. Here, w e give a de- tailed description of how the metho d is applied to OCT A images, and w e demonstrate ho w it robustly lab els capillaries and bloo d vessels and hereb y provides the basis for quantifying retinal bloo d flow. 1 In tro duction Optical coherence tomograph y angiograph y (OCT A) is a relatively no v el imaging metho d commercially a v ailable in 2014 [16]. Compared to alternative retinal imaging modalities, it is fast and pro vides high-resolution, depth-resolved images of the retinal microv asculature without any inv asive pro cedures such as contrast agen ts [12]. Despite the rapid acquisition, images can easily b e corrupted by motion artefacts and noise. Noise induced by a lo w signal-to-noise ratio can o ccur due to eye conditions such as cataract [16], where the laser light, that illuminates the retina as part of the OCT A imaging system, is scattered by the cataract-affected lens. F urthermore, the image intensit y in OCT A images can v ary o ver the image plane and giv e rise to bias in the image. These effects m ust b e accoun ted for when choosing a metho d for obtaining an automatic segmentation of the retinal micro v asculature. OCT A imaging is a 3D acquisition metho d [12], but the clinical scanners emplo yed in our studies perform a preprocessing of the data that segments the v olume into so-called en face angiograms of different retinal lay ers. W e fo cus on t wo lay ers: the sup erficial retinal lay er (SRL) and the deep retinal lay er (DRL). An example is shown in Fig. 1. Here, the larger blo o d vessels (arterioles and v enules) are seen as thick er bright structures and the capillary netw ork is a finer net work of bright structures b et w een the larger vessels. The dark area in the middle is the fo veal av ascular zone (F AZ). 2 A. Engb erg et al. (a) (b) Fig. 1. Example of an OCT A image including (a) the superficial retinal la yer (SRL) and (b) the deep retinal lay er (DRL). The problem w e address is to segmen t OCT A images in to three classes includ- ing larger vessels, capillaries, and background. By distinguishing b et ween larger v essels and capillaries, the t wo structures can be analyzed separately . This allo ws for removing the influence of the size of the larger vessels, when quantifying the retinal capillaries, and hence not ov erestimating their density . W e will solve this as a pixel lab eling problem, such that we assign each pixel to one of three lab els using a dictionary-based segmen tation metho d. The ma jorit y of clinical studies focus on solely obtaining quantitativ e metrics of the microv asculature, and segmentation of the microv asculature from OCT A images is a problem that has b een addressed in only a few studies. Most studies obtain a segmen tation through thresholding and filtering schemes [1,13]. A few studies utilize man ually annotated data to create segmentation mo dels, such as probabilistic mo dels [8], conv olutional neural netw orks [15], and Hessian- and deep learning-based metho ds [6] to segmen t all vessels. A single study [6] au- tomatically segments main vessels and capillaries separately in retinal images using deep learning. One anatomical difference betw een larger vessels (arterioles and ven ules) and capillaries is their thic kness. While the diameter of the capillaries is around 4-9 µ m [12] and is determined b y the size of the red bloo d cells, the larger vessels are thic ker than capillaries and they v ary more in size. Since the larger v essels and capillaries are connected, it is not trivial to design a model that separates the t wo anatomical structures. It cannot b e accomplished by a simple thresholdning metho d, whic h are commonly applied to OCT A data. Instead, we propose to use the dictionary-based segmen tation metho d from [2,3,4], where the segmentation mo del is learned from annotated training data. W e ha ve used this metho d for segmen ting retinal microv asculature from OCT A images in [9,10,11]. Dictionary-based Segmentation 3 Fig. 2. Illustration of the pipeline for the dictionary segmen tation method. It consists of a training and a segmentation part. The training is based on a training image with a corresp onding label image. Here, the training labels are blue for larger v essels, red for capillaries, and black for background. Based on patches sampled in the training image we p erform a clustering (T1), whic h makes up the dictionary . Dictionary lab el patc hes are computed from the clustering of the training image and the lab el image (T2). In the segmentation part, w e assign the dictionary to an input image (S1) and then compute pixel-wise probabilities of the lab els using the dictionary lab els (S2). Finally , we obtain the segmentation illustrated here. Larger vessels are marked with cy an, capillaries are magenta, and bac kground is black. W e in tro duced the fundamen tals of the dictionary-based segmentation metho d in [2]. It has later b een extended for efficient computation of lab el probabilities whic h allow ed iteratively up dating lab el probabilities, which we used for com- puting deformable b oundary mo dels in [3,4], where probabilities are computed in each iteration. F urther, we extended the mo del to allow for interactiv e seg- men tation in [5], which allows for computing lab el probabilities from partially annotated data. In this pap er we will fo cus on details related to segmenting microv asculature from OCT A images. Core elements of the metho d are describ ed in e.g. [5], but to give a complete description of the metho d we will also describ e them here. F urthermore, we use a feature-based representation to characterize local texture instead of using intensit y patches. This has not previously b een described, so we will pro vide the details here. 4 A. Engb erg et al. 2 Metho d The basic principle of the dictionary-based segmentation mo del was introduced in [2], which is inspired by sparse co ding metho ds [7]. Sparse co ding metho ds op erate on image patc hes, and were originally applied to problems like denois- ing and texture mo deling. Our dictionary-based segmen tation metho d is similar to sparse co ding metho ds b ecause it emplo ys a dictionary of image patc hes. Ho wev er, here we assign each patch to only one dictionary elemen t, which is differen t from sparse co ding, where an image patc h is t ypically represented by a small n umber of dictionary patches. The idea of our method is that image patches with similar appearance should ha ve the same lab el. W e exploit this idea by clustering image patc hes (unsup er- vised part of training), computing patc h label information from user input on the training image (sup ervised part of training), and then pasting this information in a testing image (using the mo del). An ov erview of our segmen tation pip eline is sho wn in Fig. 2. 2.1 T raining the mo del F or training the segmentation mo del we need training data, consisting of image data and user-provided lab eling. The image data used for training needs to b e represen tative of the segmentation problem. This is usually one image or a small set of images. User-provided lab eling should provide the desired segmentation for the images. It is not a requirement that all image data used for training is lab eled by the user, but lab eling should co ver the v ariability of the structures to b e segmented. Extracting patc h v ectors. W e aim to construct a feature descriptor c harac- terizing the lo cal app earance around ev ery image pixel. W e start b y extracting an N × N patc h around the pixel. W e choose N to be odd suc h that the patch can b e centered on a pixel, and we rearrange the patch into a vector of length N 2 . Reducing the dimensionality of the patc h v ectors. W e preform principal comp onen t analysis (PCA) to reduce the dimensionality of the patch vectors. Here, we randomly select K patc h vectors of length N 2 , denoted v k . F or this set of vectors we compute a mean ¯ v , such that we can compute patch vectors cen tered in origo v k − ¯ v , and arrange those vectors in the rows of a matrix V , whic h will hav e size K × N 2 . No w, we p erform eigendecomp osition of a matrix U = V T V , (1) and keep eigenv ectors corresp onding to q largest eigenv alues in a N 2 × q matrix S . F or each image pixel i and its patc h vector, we can now compute the pro jec- tion f i = ( v i − ¯ v ) T S , and use it as a feature vector of length q . Dictionary-based Segmentation 5 (a) I (b) I x (c) I y (d) I xy (e) I xx (f ) I yy Fig. 3. Subfigures sho wing the comp onen t with the largest v ariability of the lo cal features of the raw intensities, I , and the first ( I x , I y ) and second order deriv atives ( I xy , I xx , I yy ) with a feature patch size of 7 × 7 pixels. Incorp orating image deriv ativ es. W e strengthen the features emplo yed in our metho d b y incorp orating the v alues of the first and second deriv atives of the image I x = ∂ I ∂ x , I y = ∂ I ∂ y , I xx = ∂ 2 I ∂ x 2 , I xy = ∂ 2 I ∂ x∂ y , I y y = ∂ 2 I ∂ y 2 . (2) F or eac h of these five images we follow the pro cedure as described for the intensit y image I , i.e. patch vector extraction and dimensionality reduction using PCA. This results in five additional feature vectors of length q . When all these are concatenated, we are left with a 6 q feature vector p er every image pixel. The PCA feature vector describ es the local appearance of the image around the pixel. Fig. 3 illustrates the PCA features computed in an OCT A image. Clustering. The PCA feature vectors are clustered using a k -means hierar- c hical clustering to obtain the dictionary . As a distance measure, w e use the Euclidean distance b et w een PCA feature patches. Hierarc hical clustering is chosen instead of a conv en tional k -means because it sp eeds up the clustering dramatically . This allo ws for large dictionaries, and giv es a very efficient search structure for assigning new feature vectors to the dictionary . 6 A. Engb erg et al. The k -means hierarc hical clustering is go verned by tw o parameters, a branc h- ing factor b and a depth t . A set of feature vectors is clustered by first clustering all vectors to b clusters using conv entional k -means, and then clustering each of these groups into b sub-clusters. This is rep eated t times or until there are less than b feature vectors in a cluster. The result is a tree graph, the k -means tree, where the no des represent cluster centers. F or our purp oses, clustering is p erformed on PCA feature v ectors corresp ond- ing to patches extracted from the training image. W e therefore exp ect that fea- tures b elonging to the same cluster corresp ond to image patches whic h hav e a similar app earance. Incorp orating user lab elings. As mentioned previously , w e exp ect similar image features to hav e similar lab els. Having computed clusters of the image features, w e need to define a lab eling for each cluster. F or this we need user lab elings. This step constitutes the sup ervised part of building the dictionary . F or an image of the size n × m , user lab elings are stored as an n × m × C arra y L , where C is the num b er of lab els. In our case C is three since we are in terested in larger vessels, capillaries and background. Elemen ts of L are binary , suc h that each pixel p osition b elongs to only one lab el, indicated by a v alue 1 in one la yer of L . W e now compute lab eling information for eac h cluster in the dictionary by com bining lab eling information of all the members of the cluster. F or this we extract patches of size M × M × C from L , where M is chosen to b e o dd such that patc hes are centered on a pixel. F or each dictionary cluster, w e extract suc h patches at the same spatial lo cations as the image patches b elonging to the cluster. This set of patc hes is then a veraged, and we obtain lab eling information for each dictionary cluster. Due to the av eraging of binary v alues, lab eling information asso ciated with dictionary patc hes is not binary , but can rather b e interpreted as probabilities. 2.2 Using the mo del Once we ha ve a dictionary and the dictionary probabilities, w e can use our mo del to pro cess a new image (a testing image) and obtain pixel-wise probabilities of b elonging to each of C lab els. Dictionary assignmen t. The first step in pro cessing the testing image is dic- tionary assignmen t. F or this, w e first extract a PCA feature vector from the patc h around the pixel. Then we search the k -means tree to assign the pixel to the dictionary cluster. Our assignment is done b y a simple greedy searc h through the k -means tree, and then assigning a feature vector to the nearest no de in the tree. This do es not guarantee that the feature vector is assigned to the closest no de in the tree, but will generally ensure that similar patterns are group ed together. An illustration of the assignmen t is shown in Fig. 4a. Dictionary-based Segmentation 7 (a) Assignment image (b) Probabilit y image for larger vessels (c) Probability image for capillaries (d) Probabilit y image for background Fig. 4. Here, we show an assignment image in (a), where the color is the index of whic h dictionary element the patch around that pixel is assigned to. W e also sho w the resulting probability images for the three differen t classes (b-d). Computing probability image. F rom the dictionary assignment, we build a probabilit y image P of size n × m × C by visiting all image pixels for eac h lab el, obtaining the probability information asso ciated with their dictionary cluster, and adding it to P in the spacial p osition corresp onding to the pixel. Finally , w e normalize P suc h that the C lab el probabilities sum to one. T o capture the lo cal app earance of the image, it can sometimes b e adv an- tageous to c ho ose a sligh tly larger patch for computing the feature vector than the label patch, meaning that N > M . In that case, the ( N − M ) / 2 b oundary pixels will not b e lab eled. But in practice, this does not influence our analysis. Ho wev er, M and N can b e chosen indep enden tly such that it makes sense for the giv en segmentation problem. The final step b efore obtaining the segmentation is to choose the most prob- able lab el from the C lab els. 3 Exp erimen ts T o train the mo del, w e are using one OCT A image (a training image) with cor- resp onding lab eling, b oth shown in Fig. 5. The OCT A image has b een acquired 8 A. Engb erg et al. b y a sw ept source DRI OCT T riton, T op con Medical Systems, Inc. The image w as initially 320 × 320 pixels, but was upscaled by a factor of tw o to a final size of 640 × 640 pixels. Before using the image to train the dictionary , we applied an adaptive histogram equalization, which impro ved the contrast of the image [14]. F or adaptive histogram equalization the image is divided into patches of size 40 × 40 pixels with a contrast enhancement limit of 0.004 prev enting ov er- saturation in homogeneous regions. The three classes for describing the retinal micro v asculature include cap- illaries, larger v essels (arterioles and ven ules), and background. Larger vessels are defined as v essels with a radius of at least t wice the radius of the capillar- ies. Image patches for computing the PCA feature use 7 × 7 pixels ( N = 7). T o enhance the v ariabilit y of the patches, the features are extracted from b oth the original image, as well as a 90 degree rotation of the image. 50000 random patc hes ( K = 50000) are used to compute the PCA features where the q = 10 biggest comp onen ts are used. Next, lab el patc hes of size 13 × 13 pixels are used for the probabilit y dictionary ( M = 13). The patch size is on the same scale as the capillaries that we wish to iden tify , and it has b een determined through a parameter study . Figure 3 sho ws the largest comp onen t of each of the six feature groups. No w, each non- b oundary pixel in the image has six 10 × 1 vectors, which are concatenated into a 60-dimensional feature v ector. W e then p erform k -means clustering of 100000 feature vectors by building a search tree with five lay ers and a branching factor of five (with a maxim um of 3905 clusters). W e end up with 3905 dictionary elements. The assignment image A is created, see Fig. 4a, where the color corresp onds to the index of the dictionary element for the image patch around that pixel and the corresp onding probabilit y images are shown in Fig. 4b-d. T o optimize the segmentation, a 3 × 3 weigh t matrix W is computed such that the resulting class probabilities of the training image equal the annotated class lab els by W = min W k ˆ L − ˆ P W k 2 2 , (3) where ˆ L is the label image arranged into an nm × C matrix, where eac h row con- tains the pixel-wise lab el probabilities, and ˆ P is the probability image arranged in the same w ay . This is solved as linear least squares problem. The OCT A images, that we hav e work ed with, consist of b oth the sup erficial retinal lay er (SRL) and the deep retinal lay er (DRL). Only the SRL is used to create the dictionary , as we wish to ha ve training information containing larger v esse ls. Since there are mainly capillaries presen t in the DRL, the detected capillaries and the detected larger vessels are com bined in to one class in this la yer. Fig. 6 shows some examples of the resulting segmentation in b oth the sup erficial and deep retinal la yers. It should be noted that these segmentations are obtained from a mo del that has b een trained using one single annotated training image. It is very time consuming to annotate the detailed micro v ascular structures, and therefore it is adv an tageous that only one image is needed to obtain this result. Dictionary-based Segmentation 9 (a) T raining image (b) Manual lab eling Fig. 5. The dictionary is build from the training image seen in (a) and the corresp ond- ing man ually lab elled image sho wn in (b). Larger vessels are mark ed in blue, capillaries in red, and background in black. The segmentation mo del is relatively fast to train and run. Computing the PCA feature model tak es around 4 . 25 seconds, building the dictionary tak es around 7 . 01 seconds, and computing the segmentation from an unseen image using the trained mo dels takes around 5 . 95 seconds. The most time consuming part is the man ual annotation of the training image. 4 Conclusion W e hav e presented our dictionary-based segmentation metho d that allows seg- men tation of the retinal micro v asculature from OCT A images. F rom a single annotated image, w e obtain a clear separation into the three classes of capillar- ies, larger vessels, and background. The model is fast to compute, and gives an accurate separation of the three classes, allowing for quan titative assessment of the retinal micro v asculature. 10 A. Engb erg et al. (a) SRL (b) SRL output (c) DRL (d) DRL output (e) SRL (f ) SRL output (g) DRL (h) DRL output (i) SRL (j) SRL output (k) DRL (l) DRL output (m) SRL (n) SRL output (o) DRL (p) DRL output (q) SRL (r) SRL output (s) DRL (t) DRL output Fig. 6. Examples of segmentation output. Subfigures (a)-(d) show the segmentation of sub ject 1, (e)-(h) sho w sub ject 2, (i)-(l) sho w sub ject 3, (m)-(p) sho w sub ject 4, and (q)-(t) show sub ject 5. Dictionary-based Segmentation 11 References 1. Ch u, Z., et al.: Quan titative assessment of the retinal microv asculature using op- tical coherence tomography angiography . J. Biomed. Opt 21 (6), 066008 (2016) 2. Dahl, A., Larsen, R.: Learning dictionaries of discriminative image patches. In: Pro c. BMVC. pp. 77.1–77.11. BMV A Press (2011) 3. Dahl, A.B., Dahl, V.A.: Dictionary snakes. In: Pattern Recognition (ICPR), 2014 22nd International Conference on. pp. 142–147. IEEE (2014) 4. Dahl, A.B., Dahl, V.A.: Dictionary based image segmentation. In: Scandinavian Conference on Image Analysis. pp. 26–37. Springer (2015) 5. Dahl, V.A., T rinderup, C.H., Emerson, M.J., Dahl, A.B.: Conten t-based propa- gation of user markings for interactiv e segmentation of patterned images. arXiv preprin t arXiv:1809.02226 (2018) 6. Deng, W.: Measurements of retinal microv asculature in mice and humans with deep learning. PhD, Universit y of Iow a, Iow a City , Iow a, USA (Aug 2018). h ttps://doi.org/10.17077/etd.84ques5w 7. Elad, M.: Sparse and redundant representations: from theory to applications in signal and image pro cessing. Springer Science & Business Media (2010) 8. Elada wi, N., Elmogy , M., Helmy , O., Ab oelfetouh, A., Riad, A., Sandhu, H., Schaal, S., El-Baz, A.: Automatic blo od vessels segmentation based on different retinal maps from OCT A scans. Computers in Biology and Medicine 89 , 150–161 (Oct 2017). https://doi.org/10.1016/j.compbiomed.2017.08.008 9. Engb erg, A., Amini, A., Willerslev, A., Larsen, M., Sander, B., Kessel, L., Dahl, A.B., Dahl, V.A.: Automated Quan tification of Macular V asculature Changes from OCT A Images of Hematologic Patien ts. In: Pro o ceedings of International Symp o- sium of Biomedical Imaging (2020) 10. Engb erg, A.M.E., Erichsen, J.H., Christensen, A.N., Conradsen, K., Sander, B., Kessel, L., Dahl, A.B., Dahl, V.A.: Quan tifying c hanges in the macular v asculature after cataract surgery . Submitted to: Medical Image Analysis (2020) 11. Engb erg, A.M.E., Erichsen, J.H., Sander, B., Kessel, L., Dahl, A.B., Dahl, V.A.: Automated quantification of retinal microv asculature from OCT angiography using dictionary-based vessel segmentation. Communications in Computer and Informa- tion Science (2019) 12. Kashani, A.H., et al.: Optical coherence tomography angiograph y: A comprehen- siv e review of current metho ds and clinical applications. Prog Retin Eye Res. 60 , 66–100 (2017) 13. Kim, A.Y., et al.: Quantifying microv ascular densit y and morphology in diab etic retinopath y using sp ectral-domain optical coherence tomography angiography . In- v est Ophthalmol Vis Sci 57 (9), OCT362–OCT370 (2016) 14. Pizer, S.M., Amburn, E.P ., Austin, J.D., Cromartie, R., Geselowitz, A., Greer, T., ter Haar Romeny , B., Zimmerman, J.B., Zuiderveld, K.: Adaptive histogram equalization and its v ariations. Computer vision, graphics, and image processing 39 (3), 355–368 (1987) 15. Pren taic, P ., Heisler, M., Mammo, Z., Lee, S., Merkur, A., Nav a jas, E., Beg, M.F., arunic, M., Loncaric, S.: Segmentation of the fov eal micro v asculature using deep learning netw orks. Journal of Biomedical Optics 21 (7), 075008 (Jul 2016). h ttps://doi.org/10.1117/1.JBO.21.7.075008 16. Spaide, R.F., F ujimoto, J.G., W aheed, N.K., Sadda, S.R., Staurenghi, G.: Optical coherence tomography angiography . Progress in retinal and eye research 64 , 1–55 (2018)