U-SLADS: Unsupervised Learning Approach for Dynamic Dendrite Sampling

U-SLADS: UNSUPER VISED LEARNING APPR O A CH FOR D YNAMIC DENDRITE SAMPLING Y an Zhang 1 , Xiang Huang 2 , Nicola F errier 2 , Emine B. Gulsoy 3 , Charudatta Phatak 1 1 Materials Science Di vision, Argonne National Laboratory , Lemont, IL 2 Mathematics and Computer Science Di vision, Argonne National Laboratory , Lemont, IL 3 Department of Materials Science and Engineering, Northwestern Uni veristy , Evanston, IL ABSTRA CT Nov el data acquisition schemes hav e been an emerging need for scanning microscopy based imaging techniques to reduce the time in data acquisition and to minimize probing radi- ation in sample exposure. V aries sparse sampling schemes hav e been studied and are ideally suited for such applications where the images can be reconstructed from a sparse set of measurements. Dynamic sparse sampling methods, particu- larly supervised learning based iterativ e sampling algorithms, hav e sho wn promising results for sampling pixel locations on the edges or boundaries during imaging. Howe ver , dynamic sampling for imaging sk eleton-like objects such as metal den- drites remains dif ficult. Here, we address a new unsupervised learning approach using Hierarchical Gaussian Mixture Mod- els (HGMM) to dynamically sample metal dendrites. This technique is v ery useful if the users are interested in fast imag- ing the primary and secondary arms of metal dendrites in so- lidification process in materials science. Index T erms — Dynamic sampling, unsupervised learn- ing, computational imaging, Gaussian mixture model 1. INTR ODUCTION In most commonly used conv entional point-wise imaging modalities, each pix el measurement can take up to a few seconds to acquire, which can translate to hours or e ven days for middle to high resolution image (e.g. 1024 × 1024 or 2048 × 2048 pixels) measurements. The sample exposure to a highly focused electron or X-ray beam for extended periods of time may furthermore damage the underlying object. Thus, minimizing the image acquisition time and radiation damage is of critical importance. Static sampling methods, such as random sampling, uniform spaced sampling and lo w-discrepancy sampling methods have been widely studied and used [1][2][3]. Recently , sampling techniques where previous measurements are used to adaptively select new sampling locations hav e been presented. These methods, known as dynamic sampling methods, hav e been shown to significantly outperform traditional static sampling methods [4][5][6][7][8]. SLADS [6] and SLADS-Net [8] are two dynamic sam- pling methods based on supervised learning approaches. The goal of SLADS and SLADS-Net is to select the measure- ments that minimizes the reconstruction error during the sam- pling process. In order to train SLADS/SLADS-Net, one needs corresponding pairs of representati ve extracted features and pre-calculated reconstruction errors from historical im- ages. The mapping from features to reconstruction errors could be learned using various learning algorithms, such as linear regression, support vector regression or (deep) neural networks. In testing or experiment, SLADS/SLADS-Net will compute the reconstruction error as a score using extracted features for unmeasured pixel locations and select the one that has lowest score v alue for the next measurement. The key local descriptors for SLADS/SLADS-Net feature extrac- tion are the gradients and variances close to edges in images. [6] and [8] sho wed that these algorithms successfully sam- pled the pixels on the ”informativ e” boundaries of an object. The PSNR were high and the distortions were low between reconstruction and original images. Howe ver , in the imaging fields, some researchers are more enthusiastic about the skeleton of an object instead of the edges. One important area is the metal solidification research, where researchers care more about the formation of metal dendrites during solidification [9], as shown in Figure 1. An algorithm that can iteratively sample along the main direc- tion of the object formation is of critical importance. Since it is relati vely dif ficult to define the k ey features related to this phenomenon nor the metric for reconstruction, unsupervised learning approach is used to estimate the next measurement locations based on the calculated distribution using current measurements. Here, we use Hierarchical Gaussian Mixture Models (HGMM) for dynamic sampling and we name it U- SLADS because of its unsupervised learning fashion. 2. U-SLADS FRAMEWORK The core idea of U-SLADS is to use two-dimensional Gaus- sians to model the primary and secondary arms of dendrites. W e discretize the 2D sample and then vectorize it as X ∈ R N , Fig. 1 : Metal dendrite image [9]. The measured pixel loca- tions are used for Gaussian mixture model and the measured intensities are used for computing threshold. and define by X s the pixel value at location s . Assuming t pixels hav e been measured, we can construct the current mea- surement v ector as a combination of locations and intensities: Y ( t ) =    s (1) , X s (1) . . . s ( t ) , X s ( t )    . Using the measurement vector Y ( t ) , we then compute a threshold from the t pixel intensities, and perform cluster- ing of locations only for pixels with intensity larger than the threshold. In U-SLADS, the ke y step is to select the next measure- ment pixel to update the measurement v ector as Y ( t +1) =  Y ( t ) s ( t +1) , X s ( t +1)  This process is repeated until the stopping criterion is met. In SLADS and SLADS-Net, the dynamic sampling meth- ods are based on supervised learning strategies which require training images and feature extractions. In each iteration of SLADS or SLADS-Net, local descriptors, such as gradients, variance and density , are computed to form feature vectors. Unmeasured locations having high feature scores are usually close to edges of training images and experimental objects. Howe ver , in dynamic sampling process for dendrites imag- ing, those features have little contribution to the measure- ment of dendrites formation of an object. Our proposed U- SLADS algorithm is thus based on unsupervised learning ap- proach which iterativ ely update the distributions of measured locations and estimate the next measurement locations which might contribute to the e xisting clusters data distributions. In U-SLADS, we use HGMM to select the next measure- ment locations in each iteration. W e start with 5% random sampling and use Otsu’ method [10] to calculate a thresh- old for all measured intensities. W e then apply GMM [11] clustering on all measurement locations which ha ve intensi- ties larger than the threshold. For the unmeasured locations in each cluster , we calculate the weighted distances (Maha- lanobis distances) to their cluster centroids. W e later perform measurements on the unmeasured locations of top n distances (n is user’ s choice). W e apply the same procedure in a hierar- chical fashion for each cluster until only one single cluster is found. Fig. 2 : U-SLADS Frame work. The stopping criterion of the orange box is controlled by maximum iteration in each GMM layer . The iteration of the yellow box stops when only one single cluster is found. The outer loop is controlled by the sampling ratio for the total measurements. 3. U-SLADS ALGORITHMS W e describe U-SLADS as three parts: the main function, layer-wise HGMM function and Bayesian Information Cri- terion (GMMbic) [12] function. 3.1. U-SLADS main function The main function of U-SLADS is to store the layer-wise measurements of the clusters in each layer . In each itera- tion, it calls lay er GM M function which is used to calculate the locations to be measured and perform the measurement. Then the updated number of clusters of the current updated measurement vector Y ( t +1) is ev aluated. If there’ s only one cluster , then Y ( t +1) is pushed into a Stack which is a record of the measurements in each hierarchical layer of GMM models. If there exists multiple clusters, then Y ( t +1) k for each cluster k is pushed into a Queue which will be used for next layer GMM computation. If the Queue is not empty , then a sub- image is constructed using the measurement vector Y ( t +1) k pop out from the Queue and repeat the whole process of the main function. Algorithm 1: U-SLADS main Function Y ( t +1) ← U S LAD S ( X ( t ) ) Queue = [ ], stack = [ ]; while ( sampling r atio < = φ % ) do X ( t +1) = lay erGM M ( X ( t ) ) ; Y ( t +1) = Y ( t ) ∪  s ( t +1) , X s ( t +1)  ; if G.components == 1 then stack.push( Y ( t +1) ); else for k=1:G.components do Queue.push( Y ( t +1) k ); end end if Queue not empty then Y ( t +1) i = Queue.pop(); X ( t +1) i = C onstr uctI mage ( Y ( t +1) i ) ; else break; end end return Y ( t +1) 3.2. Layer -wise HGMM function The layer-wise HGMM function first computes a threshold τ using all measured intensities by Otsu’ s method [10]. The measurement locations in the measurement vector Y ( t ) .s hav- ing intensity values larger than the threshold are preserved and used by GMMbic function to calculate the mixture of Gaussian distributions [11]. The unmeasured locations u ( t ) are then used to calculate their predicted cluster labels using the GMM of current step. The unmeasured locations in each assigned cluster ˆ u ( t ) k are used to compute their weighted dis- tance from cluster centroids using the mean vectors µ k and cov ariance matrices Σ k . The weighted distances D are sorted and the unmeasured locations having the first  closest dis- tances will be measured. 3.3. GMMbic function The GMMbic function is to calculate the BIC scores [12] for different hypothesized number of clusters. The number of clusters ha ving the lo west BIC score will be the GMM model parameter to be used in the current step. A binary search method may be used to improv e the search efficienc y . 4. HYPER-P ARAMETERS A total of four hyper-parameters can be tuned. The sampling r atio in Algorithm 1 is the stopping criterion re- gards to the percentage of dynamic sampling. The maxiter Algorithm 2: Layer-wise HGMM Function X ( t +1) ← lay er GM M ( X ( t ) ) for i = 1 : maxiter do τ = otsu ( X ( t ) ) ; Y ( t ) = S eg ment ( X ( t ) > = τ ) ; G ( t +1) = GM M bic ( Y ( t ) .s ) ; label s = G ( t +1) .pr edict ( u ( t ) ) ; ˆ u ( t ) k = label sor t  u ( t )  ; for k = 1 : K do D = r  ˆ u ( t ) k − µ k  T Σ k − 1  ˆ u ( t ) k − µ k  ; end idx = arg sor t ( D ) ; if len ( D ) <  then X ( t +1) = P er f or mM easur ement ( ˆ u ( t ) , idx ) ; else X ( t +1) = P er f or mM easur ement ( ˆ u ( t ) , idx [1 :  ]) ; end end return X ( t +1) in Algorithm 2 indicates the number of iterations in each layer of GMM and  defines the maximum allo wance of the loca- tions to be measured in each GMM run. The n in Algorithm 3 is set to be maximum number of clusters in BIC score search. 5. RESUL TS AND CONCLUSION W e applied U-SLADS algorithm on a simulated dendrite image as sho wn in Figure 1. W e used maxiter = 10 and  = 10 , started with 5% random sampling initially , and se- lected the future measurements iterati vely using dynamic sampling up to sampl ing r atio = 40% . Figure 3 sho ws the resulted sampled masks (measured locations) and sampled images (measured intensities). W e can see that at 10% , the algorithm coarsely estimated the Gaussian distrib ution of the four primary arms and only a fe w secondary arms of the metal dendrites. Since then, U-SLADS started to find the sub-sets of Gaussian distrib utions that belongs to secondary arms in a hierarchical f ashion. At 40% , U-SLADS has almost found all the feature distributions of the dendrites. Our U-SLADS algorithm is a nov el strate gy for dynamic dendrite sampling. It outperforms the traditional random sam- pling method as sho wn in Figure 4 (U-SLADS has a PSNR of 11.31 dB over 7.25 dB by random sampling and struc- tural similarity of 0.65 o ver 0.46 at 40% measurements) and provides an alternate when boundary-focused dynamic sam- pling is not applicable. U-SLADS is an unsupervised learning based approach so training is not required. One limitation of U-SLADS is that the computation time increases exponen- tially with the sampling ratio, as shown in Figure 5. (a) 10% sampled mask. (b) 10% sampled image. (c) 20% sampled mask. (d) 20% sampled image. (e) 30% sampled mask. (f) 30% sampled image. (g) 40% sampled mask. (h) 40% sampled image. Fig. 3 : Sampled masks and images at dif ferent sampling ra- tios of U-SLADS. Left column is the sampled mask (mea- sured locations) and right column is the corresponding sam- pled image (measured intensities). 6. A CKNO WLEDGMENTS This material is based upon work supported by Laboratory Direct Research and De velopment (LDRD) funding from Ar - gonne National Laboratory , provided by the Director , Office of Science, of the U.S. Department of Ener gy under Contract No. DE-A C02-06CH11357. Algorithm 3: GMMbic Function G ( t ) ← GM M bic ( Y ( t ) ) bic = [ ]; for i = 1 : n do if Y ( t ) .siz e () > n then G ( t ) = GaussianM ixtur e ( Y ( t ) .s, i ) ; bic.push(G.score); end end if len ( bic ) > 0 then k = argmin(bic) + 1; else k = 1; end G ( t ) = GaussianM ixtur e ( Y ( t ) .s, k ) ; return G ( t ) Fig. 4 : Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) at dif ferent sampling ratios using sampled image without reconstruction. Fig. 5 : T ime used at dif ferent sampling ratios. 7. REFERENCES [1] Anderson, Hyrum S., Jov ana Ilic-Helms, Brandon Rohrer , Jason W . Wheeler, and K urt W . Larson. ”Sparse imaging for fast electron microscop y . ” In Computational Imaging, p. 86570C. 2013. [2] Ohbuchi, Ryutarou, and Masaki Aono. ”Quasi-Monte Carlo rendering with adaptiv e sampling. ” IBM T okyo Re- search Laboratory (1996). [3] Hujsak, K., Myers B. D., Roth, E., Li, Y , Dravid V . P .. ”Suppressing Electron Exposure Artifacts: An Electron Scanning Paradigm with Bayesian Machine Learning. ” Microscopy and Microanalysis v ol. 22 (2016): 778–788. [4] Merryman, T ., K ov ace vic. J.. ”An adapti ve multirate al- gorithm for acquisition of fluorescence microscopy data sets. ” IEEE T rans. on Image Processing vol. 14 (2005): 1246–1253. [5] Jackson, C., Murphy R. F ., K ov acevic. J.. ”Intelligent ac- quisition and learning of fluorescence microscope data models. ” IEEE Trans. on Image Processing v ol. 18 (2009): 2071–2084. [6] Godaliyadda, GM Dilshan, Dong Hye Y e, Michael D. Uchic, Michael Anthony Groeber , Gregery T . Buzzard, and Charles A. Bouman. ”A Framew ork for Dynamic Image Sampling Based on Supervised Learning. ” IEEE T ransactions on Computational Imaging (2017). [7] Zhang, Y an, GM Dilshan Godaliyadda, Nicola Fer- rier , Emine B. Gulsoy , Charles A. Bouman, and Charu- datta Phatak. ”Reduced electron exposure for energy- dispersiv e spectroscopy using dynamic sampling. ” Ultra- microscopy 184 (2018): 90-97. [8] Zhang, Y an, G. M. Godaliyadda, Nicola Ferrier , Emine B. Gulsoy , Charles A. Bouman, and Charudatta Phatak. ”SLADS-Net: Supervised learning approach for dynamic sampling using deep neural networks. ” arXiv preprint arXiv:1803.02972 (2018). [9] Ferreira, Ale xandre Furtado, Ivaldo Leo Ferreira, Janaan Pereira da Cunha, and Ingrid Meirelles Salvino. ”Sim- ulation of the Microstructural Evolution of Pure Mate- rial and Alloys in an Undercooled Melts via Phase-field Method and Adaptiv e Computational Domain. ” Materials Research 18, no. 3 (2015): 644-653. [10] Otsu, Nobuyuki. ”A threshold selection method from gray-lev el histograms. ” IEEE transactions on systems, man, and cybernetics 9, no. 1 (1979): 62-66. [11] Christopher , M. Bishop. P A TTERN RECOGNITION AND MA CHINE LEARNING. Springer-V erlag Ne w Y ork, 2016. [12] Claeskens, Gerda, and Nils Lid Hjort. ”Model selection and model av eraging. ” Cambridge Books (2008).