CNN-Based Deep Architecture for Reinforced Concrete Delamination Segmentation Through Thermography

1 CNN -Based Deep Architecture for Reinforced Concrete Delamination Segmentation Through Thermography Chongsheng CHENG 1 , Zhexiong SHANG 2 and Zhigang SHEN 3 1 The Durham School of Architectural Engineering and Construction, University of Nebraska-Lincoln, 113 Nebraska Hall, L incoln, NE 68588-0500; e-mail: cheng . chongsheng@ huskers .unl.edu 2 The Durham School of Architectural Engineering and Construction, University of Nebraska-Lincoln, 113 Nebraska Hall, Lincoln, NE 68588 -0500; e-mail: szx0112@huskers.unl.edu 3 The Durham School of Architectural Engineering and Construction, University of Nebraska-Lincoln, 113 Nebraska Hall, Lincoln, NE 68588 -0500; e-mail: shen@unl.edu ABSTRACT Delamination assessment of the bridge deck plays a vital role for bridge health monitoring. The rmograph y a s one of the nondestructive technologies for delamination detection has the advantage of efficient data acquisition . But there are challenges on the interpretation of data for accurate d elamination shape profiling. Due to the environmental variation and the irr egular presence of delamination size and depth, conventional processing methods based on temperature contrast fall short in accurate segmentation of delamination. Inspired b y the re cent development of d eep learning architecture for image segmentation, t he Convolutional Neur al N etwork (CNN) based framework was investigated for the applicability of delamination segmentation under variations in temperature contrast and shape diffus ion. The models were developed based on Dense Convolutional Network (DenseNet) and trained on thermal images collected for mimicked delamination in concrete slabs with different de pths under experimental setup. The results suggested satisfactor y performance of accurate profiling the delamination shapes. 1. INTRODUCTION Delamination as the horizontal crack embedded in the sub su rface of the bridge deck is often a re sultant of corrosion-induced deterioration of reinforcement. I ts continuous progress will e ventually affect the struc tural integrity of the bridge deck (Gucunski and Counc il 2013). Thus, profiling its e xtent is essential in the point of view of structural health monitoring (SHM). Prac tically, determination of its location, boundary (shape), and area (ratio over the entire deck) are typical ly desired. Compar ed to the conventional method such as hammer sounding and chain dragging, nondestructive detection (NDT) methods su ch as Infrared Thermography provided a fast and effective way to de tect shallow delamination of bridge de ck (Dabous et al. 2017 ; Kee et al. 2011 ; Maierhofer et al. 2006 ; Omar and Nehdi 2017 ; Washer e t a l. 2013). Studi es revealed the mechanism of detection through thermography was based on the principle of the developed tempe rature contrast between delaminated and intact areas during a day. Thus, the focus of previous researches was on investi gating the 2 detectability based on temperature contrast variations in terms of different environmental conditions and config urations of delamination geometry (Hiasa et al. 2017 ; Hiasa et al. 2017 ; Sultan and W asher 2017 ). So far, the image processing methods proposed in the literature were relied on the optimal threshold selection (Sultan and Washer 2017 ), k-mean clustering ( Omar and Nehdi 2017), and region regrowth based on contrast (Abdel-Qader et al. 2008 ; Ellenberg et a l. 2016). Given the context of profiling the delamination boundary (shape), th ere were limited studies to utilize the spatial features of delamination through thermography. In this paper, we introduce the Convolutional Neur al Network (CNN) based deep learning architecture to tackle the problem. This framework has been reported that overperform ed conventional methods in object detection and seg mentation in computer vision which the objects in an ima ge could be represented by features in the more abstract levels (LeCun et al. 2015 ). Inspired by densely connected CNN architecture (DenseNet) for image seman tic segmentation (Huang et al. 2017), the delamination profiling (segmentation) in the thermal image could be addressed through pix el-wised labeling under supervised learning scheme. The purpose of this study aims to investigate the performance of this architecture i n delamination segmentation under experimental settings. 2. RELATED WORK Many CNN-based applications were found in t he field of medical image processing for tissue segmentation (Peka la et al. 2018 ; Schleg l et al. 2017). I t is rare to find such applications in concrete delamination NDT through thermography. Th is paper focused on the architecture proposed b y Hu ang et al. (2017), which a dense block module was introduced. The block modul e consisted a sequence of densely connected convolutional layers that the information not only propagated through layer sequentially but also concatenated to the later la y ers. In this way, the features could be re -used, and the v ariations were increased so that the model were e asy to train and highly parameter efficient. Between dense blocks, the transition layer was used which consisted of b atch normaliz ation and 1x 1 convolut ional layer then a max pooli ng layer. This architecture has been evaluated on several benchmark datasets (e .g. C IFAR-10, CIFAR-100, SVHN and ImageNet) and si gnificant improvement was found over the state-of-the-art at the time for c lassification tasks. Although the original DenseNe t wa s designated for classification, it could be extended for semantic segmentation with ‘end - to - end’ scheme . Based on its key module of the dense block, the architec ture was then built for our delamination segmentation task. 3. METHOD Architecture Development Figure 1 shows the propose d architecture with three dense blocks followed by three upscale structures. Each dense block had four convolutional laye rs inside connected and concatenated so that the number o f f ilters on output will quadruple from the input. I n this way, the maximum information and gra dient flow are pr eserved in each dense block and eas y for training (Huang et al. 2017 ). The input fi rstly passed a convolutional la y er and max pooled followed by batch normalization (BN) for low level feature extraction. Then three dense blocks were followed with three transitional 3 layers in betw een to d ecrease the sp atial size so that a f eature map wit h a size of 20x20x512 was generated for embedded representation. Based on this scheme, the feature tends to be rep re sented by the lat ent variables. Before upscaling, th e intermediate la ye r and the dropout layer were used to against potential overfitting problem within convolution lay ers (Wu and Gu 2015 ). To propagate ba ck to original size as input, three upsca le blocks were used based on bilinear interpolatio n ( Long et al. 2015) for up-sampling followed by a 3x 3 convolutional lay er so the “end - to - e nd” scheme could be achieved for pixel-wise segmentation. Figure1. Developed architecture 4. EXPERIMENTS Design and Setup Experiment samples were designed to simulate the delamination in concrete slab with diff erent depths. Figure 2 (b) shows the layout of reinforcement and position of mimi cked delamination. Three slabs h ad be en constructed and heated outsi de from 10 am to 2 pm by the sun in three adjacent days (Figure 2c). The slab was then moved inside for heat releasing and data w as recorded by the thermal c amera at a sampling rate of 0.1Hz (Figure 2a). Four cas es were conducted to train 4 models for supervised learning in four situations (Table 1). Tra ining size (image height by width by frame) for each case wa s then determined b y the criterion of contrast limi ting and d ata augmentation which are discussed in the following sections. Figure2. (a) Data collection; (b ) slab with mim icked delamination; (c) sl ab heated at outdoor by the sun 4 Table 1 Case Slab Name Depth from top Slab Size Delamination Size Original size Training Size 1 A 1.75” 40’x45’ 10’x10’ 639x562x445 360x360x4005 2 B 2.75” 40’x45’ 10’x10’ 660x578x413 360x360x3717 3 C 3.75” 40’x45’ 10’x10’ 632x553x511 360x360x4599 4 A, B, and C all 40’x45’ 10’x10’ ? x? x1369 360x360x12321 Data Preparation and Description The inclusion of training da ta was determined b y using the contrast limits. He re we calculated the temperature contrast between th e delaminated and sound area s, a nd then the threshold of 0.5 ºC was chosen as the cut-o ff to determine the length of training data for each case (Fi gure 3). The selection of this thre shold was based on (1) value was recommended b y ASTM standard (ASTM 2007) ; (2) author’s ex perience that the feature variation of mimicked de lamination deviated too far awa y whe n it was smaller than the threshold. As the results, 445, 413, and 511 thermal image s were included for case 1, 2, and 3; and case 4 included a summation of previous three cases. Figure 3. Data se lection of thre e slabs for training b y contrast limit Figure 4 shows how the thermal feature of de lamination varied for each slab during the heat rel easing stage . At Figure 4 (a), the fea ture of delamination differs in contrast (temperature dif ference ) in three slabs. T he shallower depth of delamination located (slab A), the lar ger contrast could be obse rved. Also, a hi gh tempe rature zone on edge s was obse rved d ue to the closeness to the boundary o f the slab. Figure 4 (b ) shows the section profile of each slab at different time windows which the variation of shape could be observed. The solid red line in each subplot (Fig ure 4 b ) in dica ted the truth boundary loca tion for each delamination buried. Thus, the object o f this study could be furtherly refined to train the model to capture these variations so that the true shape and size of delamination could be represented. 5 Figure 4. Ra w thermal image o f three slabs at ti me 0 (a); and temperature va riations at different time windows during he at releasing (b) Data Augmentation and Labeling One of the issues with machine learning/deep learning models is over fitting when the training sample is limited in size and generality. To increase the training size and variet y, data augmentation is an often used technology ( Mikołajczyk and Grochowski 2018). Co nventional augmentation is to use the combination of transformations which i ncludes: crop, translation, rotation, reflection, scaling, and shearing. Here we focused on the c rop and translation a t the current stage of the stud y. Figure 5 shows the augmented data for a single image a nd its corresponding labels. After the augmentation, each raw thermal image had been divided into 9 cropped images with a size of 360x360 in pixels. This addressed the issue that the desig ned the delamination was always on center which may cause location sensitive of trained models. Then each ima ge was labeled in a wa y that 1 represents the delaminated area and 0 as the background in the pixel level. Thus, the sample size of training data increased dramatically and was presented in the last column of table 1. Figure 5. (a) Dat a augmentation th rough crop and translation; (b) th e corresponding labels Training and Parameter Setting Four models were tra ined individually b y using the training sets in four cases shown in table 1. For each case, th e data was split into 70% for training and 30% for testing. Total iteration (epoch) of tra ining was set to 40 and mi ni -batch was deplo y ed with the size of 32 (Li et al. 2014). The activation function of ReLu was selected for all embedded CNN la y ers. The loss function is define d b y softmax activation paired with cross-entropy pen alty at the output la yer for minimization. The optimization was then carried b y Adam optimizer with a learning rate of 0.0001. Additionally, the L2 6 regularization was implemented to handle the overfitting potential b y considering the weight of model complexity. All models were trained and tested by using Python with TensorFlow library on GPU. 5. RESULTS To evaluate the per formance of model trainin g, two metrics were used: true positive rate (TP) and false negative rate (FN). TP was defined to mea sure the rate of true delamination over predicted delamination. FN calculated the rate of false background over the pre dicted ba ckground. We would expect as higher as possible in TP for “precision”, and as lower as possible in FN as the “fa ll - out”. Table 2 shows the results after 40 iter ations of training. All cases r eturned TP rate ov er 99% and low FN rate which indicated a good performance of current models in terms of accurac y. Data visualization was combined with validation shown in Figure 6. Table 2 Case Epoch Train TP Train FN Test TP Test FN 1 40 0.9949 0.00006 0.9944 0.00011 2 40 0.9967 0.00004 0.9962 0.00006 3 40 0.9952 0.00006 0.9942 0.00009 4 40 0.9979 0.00003 0.9979 0.00004 Validation with Down Scaled Images The data was visualized and validated b y down scaling the original thermal images for all cases (sho wn in Figure 6). For all f our cases, the per formance degraded in the late r time window which the pattern of delamination was larg ely diffused in ra w images ( comparing result betwe en time 0 and time 67). Thus, the miss alarmed segmentation was more intended to occur at th e later image in case 1, 2, and 3. Comparing ca se 1 to 3, the depth affected significantly for shap e profiling e ven the architecture of th e three cases was the sa me. The deeper delamination buried, more degradation in performa nce for shape sketching ( e.g. case 3). Case 4 sho wed the best overall shape profiling performa nce due to the combination of training samples. 6. DISCUSSION AND CONCLUSIONS Although all cases showed a good perform ance in terms of TP and FN rates, the shape profiling in validation still needs improvement. One issue co uld be the unbalanced s amples due to the la yout of slabs s ince the tot al area (pix els) for the background were much larger than the area (pix els) assigned to the delaminat ion. Thus, there are more training samples for than delamination which mak es the model has more experience on lea rning background. Also, the unbalanced class es might also cause model bias during training which will be consider ed in the future developme nt. I t is our finding that increasin g the sample variety ind eed improved the model g enerality (better shape profiling) for case 4. It ma y reveal the potential power of deplo yed architecture and deep learning technolog y. In the future work, the parameter tunin g, architecture refining, more data augmentation will be conduct ed . 7 Figure 6 Validat ion and visualizat ion of results for four cases REFERENCES Abdel-Qader, I., Yohali, S., Abudayyeh, O., and Ye hia, S. (2008). "Segm entation of thermal image s for non -destructive evaluation of bridge decks." Ndt & E International , 41(5), 395-405. ASTM (2007). "D4788- 03,“Standard Test Method for Detecting Delaminations in Bridge Decks Using I nfrared Thermograph y ,” ASTM Int ernational, West Conshohocken, PA." Dabous, S. A., Yag hi, S., Alkass, S., and Moselhi, O. (2017). "Conc rete bridge deck condition assessment using I R Thermog raph y and Ground Penetrating Radar technologies." Automation in Construction , 81, 340-354. Ellenberg, A., Kontsos, A., Moon, F., and B artoli, I. (2016). "Bridge Deck delamination identification from unmanned aerial vehicle infrared imagery." Automation in Construction , 72, 155-165. Gucunski, N., and Coun cil, N. R. (2013 ). Nonde structive testing to ident ify concrete bridge deck deterioration , Transportation Research Board. Hiasa, S., B irgul, R., and Catbas, F. N. (2017). "Effec t of defect size on subsurface defect dete ctability and defe ct depth estimation for concre te structures by infrared thermography." Journal of Nondestructive Evaluation , 36(3), 57. 8 Hiasa, S., B irgul, R., and Catbas, F. N. (2017). "I nvestigation of effective utiliz ation of infrared thermography (IRT) through adv anced finite element modeling ." Construction and Building Materials , 150, 295-309. Huang, G., Liu, Z., Weinberger, K. Q., and va n der Maaten, L. "Densely connected convolutional networks." Proc., Proceedings of the IEEE conference on computer vision and pattern recognition , 3. Kee, S.-H., Oh, T., Popovics, J . S., Arndt, R. W., an d Zhu, J. (2011). "Nondestructive bridge d eck testing with air-coupled impact-echo a nd infrared thermography." Journal of Bridge Engineering , 17(6), 928-939. LeCun, Y., B engio, Y., and Hinton, G. (2015). "Deep learning." nature , 521(7553), 436. Li, M., Zhang, T., Chen, Y., and Smola, A. J . "Efficient mini -batch training for stochastic optimization." Proc., Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , ACM, 661- 670. Long, J., Shelhamer, E., and Darrell, T. "Fully convolutional networks f or semantic segmentation." Proc., Proceedings of the IEEE conference on computer vision and pattern recognition , 3431-3440. Maierhofer, C., A rndt, R., Röllig, M., Rieck, C., Walther, A., Scheel , H., and Hillemeier, B. (2006 ). "Application of impu lse-thermography for non- destructive assessment of concrete structures." Cement and Concrete Composites , 28(4), 393-401. Mikołajczyk, A., and Grochowski, M. "Data augmentation for improving deep learning in image classification p roblem." Proc., 2018 International Interdisciplinary PhD Workshop (IIPhDW) , IEEE, 117-122. Omar, T., and Nehdi, M. L. "Clusterin g-Based Threshold Model for Condition Assessment of Concrete Bridge Decks Usin g Infrare d Thermography." Proc., International Congress and Exhibition" Sust ainable Civil Infrastructures: Innovative Infrastructure Geotechnology" , Springer, 242-253. Pekala, M., Joshi, N., F reund, D. E., Bre ssler, N. M., DeB uc, D. C., and Burlina, P. M. (2018). "Deep Learning based Retinal OCT Segmentation. " arXiv prep rint arXiv:1801.09749 . Schlegl, T., Waldstein, S. M., Bogunovic, H., Endstraßer, F., Sadeghipour, A., Philip, A. -M., Podkowinski, D., Gerendas, B. S., Langs, G., and Schmidt -Erfurth, U. (2017). "Fully Automate d Detection and Quantification of M acular Fluid in OCT Using Deep Learning." Ophthalmology . Sultan, A. A., a nd Washer, G. ( 2017). "A pixel- by -pixel re liability analy sis of infrared thermography (IRT) for the de tection of subsurface delamination. " NDT & E International , 92, 177-186. Washer, G., Fenwick, R., Nelson, S., and R umbayan, R. (2013). "Guidelines for thermographic inspection of concrete brid ge components in shaded conditions." Transportation Research Record: Journal of the Transportation Research Board (2360), 13-20. Wu, H., and Gu, X. (2015). "Towards dropout training for convolutional neural networks." Neural Networks , 71, 1-10.