A Deep Regression Model for Seed Identification in Prostate Brachytherapy

A Deep Regression Mo del for Seed Lo calization in Prostate Brac h ytherap y Y ading Y uan, Ren-Dih Sheu, Luk e F u, and Y eh-Chi Lo Departmen t of Radiation Oncology Icahn School of Medicine at Moun t Sinai New Y ork NY 10029, USA yading.yuan@mssm.edu Abstract. P ost-implant dosimetry (PID) is an essential step of prostate brac hytherap y that utilizes CT to image the prostate and allow the loca- tion and dose distribution of the radioactiv e seeds to b e directly related to the actual prostate. How ever, it it a very challenging task to identify these seeds in CT images due to the sev ere metal artifacts and high- o verlapped appearance when m ultiple seeds clustered together. In this pap er, w e propose an automatic and efficien t algorithm based on 3D deep fully con volutional net work for iden tifying implan ted seeds in CT images. Our metho d models the seed localization task as a supervised regression problem that pro jects the input CT image to a map where each element represen ts the probabilit y that the corresp onding input vo xel belongs to a seed. This deep regression mo del significan tly suppresses image arti- facts and makes the p ost-pro cessing m uch easier and more controllable. The prop osed metho d is v alidated on a large clinical database with 7820 seeds in 100 patients, in which 5534 seeds from 70 patients we re used for mo del training and v alidation. Our method correctly detected 2150 of 2286 (94.1%) seeds in the 30 testing patien ts, yielding 16% impro vemen t as compared to a widely-used commercial seed finder softw are (V ariSeed, V arian, Palo Alto, CA). Keyw ords: 3D deep fully con volutional net w ork · seed lo calization · prostate brac hytherap y . 1 In tro duction With estimated 174,650 new cases and 31,620 deaths in 2019, prostate cancer remains the most common type of cancer diagnosed in men in the US [1]. Seed implan t brac hytherap y , whic h in volv es permanent implantation of radioactiv e sources (seeds) within the prostate gland, is the standard option for lo w and in termediate risk prostate cancer [2]. Despite v arious improv ements in planning and seed deliv ery , the actual radiation dose distribution ma y deviate from the plan due to v arious factors such as needle positioning v ariations, prostate de- formation, seed deliv ery v ariations and seed migration. Therefore, p ost-implan t dosimetry (PID) is recommended to assure the qualit y of the implantation and to establish the relationship betw een radiation dose and clinical outcomes [3]. 2 Y. Y uan et al. Fig. 1. An example of seed app earance in CT images in axial (left), sagittal (middle) and coronal (right) view, resp ectively . Y ellow arrows indicate the metal artifacts and red dots represen t mannually annotated seed locations. Clustered seeds can b e clearly seen in saggital and coronal views, as indicated by the blue arrows. PID is typically p erformed at day 30 following implantation that utilizes CT to image the implan ted area, from whic h prostate and surrounding organs at risk (O ARs) are outlined and seed lo cations are iden tified. Accurate lo calization of implanted seeds is essential to quantify the dose dis- tribution to those organs. How ev er, man ual identification of these seeds is time consuming giv en a large num b er of seeds implan ted, t ypically taking 10 - 20 min utes to identify 60 to 100 seeds p er patient. Therefore, accurate and auto- mated metho ds for seed localization are of great demand. While the radio-opaque seeds app ear with high contrast on the CT images, automatic seed lo calization is in practice a c hallenging task due to the following tw o unique characteristics, as shown in Fig. 1. Firstly , the presence of fudicial mark ers introduces sev ere metal artifacts on CT images, whic h significan tly increases the complexit y of seed identification. Secondly , due to seed delivery v ariations and seed migration, some implan ted seeds are v ery close to each other to form seed clusters. This highly-o verlapped app earance make it hard to identify individual seed on CT images. Sev eral automatic approac hes ha ve b een dev elop ed to lo calize seeds in CT images suc h as geometry-based recognition method [4] and Hough transform [5]. Recen tly , Nguyen et al. [6] prop osed a c ascaded metho d that inv olves threshold- ing and connected component analysis as initial detection of seed candidates, and follow ed b y a modified k-means method to separate groups of seeds based on a priori intensit y and v olume information. Zhang et al. [7] employ ed canny edge detection and an improv ed concav e p oints matc hing to separate touch- ing seeds after gra y-lev el-histogram based thresholding. All these methods use hand-crafted features that require sp ecialized domain knowledge. Mean while, sophisticated pre- and p ost-pro cessing steps are usually in tro duced to facilitate the seed lo calization procedure. As a result, the ev aluation of these metho ds w as mainly conducted with ph ysical phantom or small amount of clinical cases. Recen tly , deep con volutional neural net works (CNNs) ha ve b ecome p opular in medical image analysis [8] and hav e achiev ed state-of-the-art performance in v arious medical image computing tasks such as lung no dule detection [9], A Deep Regression Mo del for Seed Lo calization in Prostate Brach ytherapy 3 Fig. 2. (a) The target probabilit y maps created from the dot man ual annotations in Figure 1. (b) The corresp onding predicted probabilit y maps inferred from the prop osed deep regression netw ork. gland instance segmentation in histology images [10], liver and tumor segmen- tation [11], skin lesion segmentation [12] and classification [13]. Due to the ca- pabilit y of learning hierarchical features directly from raw image data, CNNs usually yield b etter generalization p erformance esp ecially when ev aluating on a large scale of dataset. Enligh tened b y the latest adv ances in deep learning research, we prop ose a no vel framework based on deep CNNs to automatically lo calize the implan ted seeds in 3D CT images. Our contributions in this pap er are three fold. Firstly , w e mo del seed lo calization as a regression problem and introduce a fully auto- mated solution b y lev eraging the discriminativ e pow er of deep CNNs. T o the b est of our knowledge, this is the first attempt of using deep neural net works to tac kle this c hallenging task. Secondly , instead of directly predicting the seed co- ordinates in 3D space, we design a probability map of se ed lo cations to accoun t for the uncertaint y of man ual iden tification, whic h impro ves the robustness of mo del prediction. Finally , w e ev aluated the prop osed metho d on a large clini- cal database with 7820 seeds in 100 patients, and compared the results with a commercial seed finder soft ware (V ariSeed, V arian, Palo Alto, CA). 2 Metho dology 2.1 Deep Regression Model As shown in Fig. 1, the ground truth is provided as dot annotations, where each dot corresp onds to one seed. Ho wev er, considering the seed has a finite dimension 4 Y. Y uan et al. (ab out 0.8 mm in diameter and 4.5 mm in length), any dot annotation should b e considered as correct as long as it’s lo cated on the seed. As a result, a large v ariation can b e observ ed in the ground truth in terms of the annotation p ositions on the seeds, which makes it unnecessarily challenging and prone to o verfitting if the exact annotation p ositions are directly used as learning target. Instead, w e conv ert the discrete dot annotations into a contin uous probability map P ( x ) ( x ∈ R 3 ) and cast the seed lo calization task as a sup ervised regression problem that learns a mapping betw een a 3D CT image set I ( x ) and P ( x ), denoted as ˆ P ( x , w ) = F ( I ( x ) , w ) for ˆ P ( x ) the inferred probability map and w the learned parameters (w eights). F or each training image I i ( x ) that is annotated with a set of 3D p oints C i = { C 1 , . . . , C N ( i ) } , where N ( i ) is the total n um b er of seeds annotated by the user, w e define the ground truth probabilit y map to b e a k ernel density estimation based on the pro vided p oin ts: ∀ x ∈ I i , P i ( x ) = X C ∈ C i N ( x ; C , Σ ) , Σ =   σ 2 x 0 0 0 σ 2 y 0 0 0 σ 2 z   . (1) Here x denotes the co ordinates of any vo xel in image I i , and N ( x ; C, Σ ) is the normalized 3D Gaussian k ernel ev aluated at x , with the mean at the user annotation C and a diagonal co v ariance matrix Σ . Considering the ph ysical seed dimension and the magnification effect during CT imaging, w e fixed σ x = σ y = 1 mm and σ z = 2 mm in our study . Figure 2 (a) shows several examples of the probabilit y map that are created from the dot man ual annotations in Fig. 1, and (b) are the corresp onding predicted maps inferred from the proposed deep regression mo del. W e train a deep regression netw ork (DRN) to map the input CT images to the probability map using a symmetric con v olutional enco ding-deco ding struc- ture, as sho wn in Fig. 3. Con v olution and max-p o oling are employ ed to aggregate con textual information of CT images in the encoding pathw ay , and transpose con volution is used to reco ver the original resolution in the deco ding pathw ay . Eac h con volutionl lay er is follow ed by batch normalization and rectified linear unit (ReLU) to facilitate gradient bac k-propagation. Long-range skip connec- tions, which bridge across the enco ding blo c ks and the decoding blo c ks, are also created to allow high resolution features from enco ding path wa y b e used as additional inputs to the con volutional lay ers in the deco ding path wa y . By explicitly assembling low- and high-level features, DRN b enefits from lo cal and global con textual information to reconstruct more precise probabilit y map of seed lo cations. Considering the target probability map is non-negativ e, w e use sof tplus as the activ ation function in the last con v olutional lay er to ensure a p ositiv e output of DRN, whic h approximates the ReLU function as: sof tplus ( x ) = 1 β · l og (1 + exp ( β · x )) . (2) In this study , we set β = 1. The conv olutional kernel size is fixed as 3 and stride as 1, except for transp ose conv olution where we set b oth kernel size and stride A Deep Regression Mo del for Seed Lo calization in Prostate Brach ytherapy 5 Fig. 3. Arc hitecture of the proposed deep regression net work (DRN). DRN is a fully 3D mo del that emplo ys conv olution and max-p ooling to aggregate contextual information, and uses transpose conv olution and long-range skip connection for b etter determination of seed lo cations. The num b ers under eac h block represen t the dimensions of its output, in which the first dimension denotes the feature channel. as 2 for upscaling purp ose. Zero-padding is used to ensure the same dimension during con volution. All the op erations are p erformed in 3D space. T raining DRN is ac hieved by minimizing a loss function b et ween the pre- dicted probabilit y map F (( I x ) , w ) and the target map P ( x ). Since the ma jority of v oxels in the target probabilit y map b elongs to bac kground, DRN tends to fo cus more on learning background rather than the Gaussian-shap ed seed an- notations. In order to account for this im balance b etw een bac kground and seed annotations, we use a weigh ted Mean Squared Error (MSE) as the loss function, with the w eight as the target map P ( x ): L ( w ) = 1 N · N X n =1 [ P ( x n ) · ( P ( x n ) − ˆ P ( x n , w )) 2 ] , (3) where N is the total num b er of vo xels in the training mini batch. 2.2 Implemen tation Our DRN was implemented with Python using Pytorch (v.0.4) pack age. T rain- ing DRN to ok 500 iterations from scratch using Adam sto c hastic optimization metho d with a batc h size of 4. The initial learning rate was set as 0 . 003, and learning rate deca y and early stopping strategies were utilized when v alidation loss stopped decreasing. In order to reduce o verfitting, we randomly flipp ed the input volume in left/righ t, superior/inferior, and anterior/posterior directions on the fly for data augmentation. W e used seven-fold cross v alidation to ev al- uate the performance of our mo del on the training dataset, in whic h a few h yp er-parameters were also experimentally determined via grid searc h. All the exp erimen ts w ere conducted on a workstation with four Nvidia GTX 1080 TI GPUs. 6 Y. Y uan et al. As for pre-pro cessing, w e simply truncated the vo xel v alues of all CT scans to the range of [ − 80 , 175] HU to eliminate the irrelev ant image information. The CT images were resampled to 0 . 5 mm isotropically and 128 × 128 × 96 v olume of in terest (VOI) centered on the prostate w as extracted from the entire CT image as input to DRN. During inference, the new CT images w ere pre- pro cessed follo wing the same pro cedure as training data preparation, then the trained DRN was applied to VOI to yield a 3D probability map. W e used a 3D w atershed segmentation algorithm to conv ert the probability map to the final seed lo cations. 3 Exp erimen ts W e assembled a database of 100 prostate cancer patien ts treated with seed im- plan t brach ytherap y from 2008 to 2019 in our institution. The num b er of im- plan ted seeds (Palladium 103) ranged from 48 to 156. Sev ent y patients with 5534 seeds w ere randomly selected for mo del training and v alidation, while the remaining 30 patien ts with 2286 seeds were reserved for indep enden t testing. A CT scan w as performed on each patient 30 da ys after implan tation, with in-plane resolution ranging from 0 . 6 × 0 . 6 to 1 . 4 × 1 . 4 mm and slice thickness from 2 . 5 to 3 . 0 mm. The ground truth w as obtained by a semi-automatic procedure, in which V ariSeed seed finder algorithm was first used to searc h implan ted seeds near prostate region in the CT images. Since this automatic pro cedure usually results in a few erroneous seed placemen ts, user in terv ention was required to correct these errors based on the seed lo cations in the CT images. The seed lo calization as well as the reconstructed radiation dose distribution were finally appro ved by a radiation oncologist. W e ev aluated the performance of the prop osed method b y comparing the pair-wise distance betw een the predicted seed locations and the ground-truth lo cations. F or a seed obtained from the automated metho d and one from ground truth, if their distance w as the shortest among the list of seeds that needed to b e paired, they were considered as a pair and remo v ed from the list. If a pair- wise distance was smaller than 3 mm, the corresp onding ground truth seed was considered as b eing correctly identified by the automated metho d. Figure 4 shows tw o examples of PID study in CT images in axial, sagittal and coronal views, resp ectiv ely , in which 77 seeds were implan ted in patient (a) and 143 seeds in (b). Also shown are the corresponding DRN predictions of the probabilit y map. It clearly shows that the metal artifacts and seed o verlap app earance are significan tly suppressed, which mak es the seed lo calization muc h easier. The plots on the right show the 3D distributions of the ground truth and the seeds iden tified by DRN. Overall, it to ok ab out 60 seconds for DRN to reco ver the n umber of implanted seeds on 30 testing patients. The median pair-wise distance w as 0 . 70 mm [25% − 75%: 0 . 36 − 1 . 28 mm]. T able 1 details the comparison betw een DRN and V ariSeed seed finder in seed detection, in whic h the first and fourth ro ws list the num b er of implan ted A Deep Regression Mo del for Seed Lo calization in Prostate Brach ytherapy 7 Fig. 4. Tw o examples of PID study in CT images in axial, sagittal and coronal views. The second and fourth rows are the corresp onding probability maps generated b y the prop osed DRN model. The right column sho ws the ov erall 3D distributions of the ground truth and seeds identified by DRN. In each figure, the red dots represent the ground truth while the cyan dots are seed lo cations identified b y DRN. seeds. F or a large range of n umber of implanted seeds (from 48 to 143), the prop osed DRN outp erformed V ariSeed by a big margin on almost every patien t. Ov erall, DRN correctly iden tified 2150 out of 2286 seeds (94 . 1%) in 30 testing patien ts, achieving 16% improv ement as compared to V ariSeed (81 . 0%). 4 Conclusion In this pap er, w e pioneered the application of deep learning in the task of iden- tifying radioactive seeds in CT-based p ost-implan t dosimetry study for patients undergoing prostate brach ytherapy . Despite the c hallenges in seed localization in CT images, the prop osed deep regression mo del achiev ed muc h higher detection accuracy as compared to a widely-used commercial softw are on a large clinical database. Also, our mo del was found to b e very efficien t, taking ab out 2 seconds on av erage for a new test case. Instead of manually dra wing 3D b ounding box or mask on each seed, our method only requires dot annotations as ground truth for model training, whic h greatly simplifies the data labe lling pro cedure. This 8 Y. Y uan et al. T able 1. Comparison b et ween DRN and V ariSeed in seed detection accuracy on the 30 testing patients. Bold v alues are the num b ers of implan ted seeds in each patien t. No. of seeds 48 50 52 52 58 58 60 62 66 66 66 69 69 71 71 DRN (%) 95.8 92.0 96.2 98.1 94.8 87.9 91.7 91.9 86.4 97.0 97.0 94.2 91.3 93.0 95.8 V ariSeed (%) 79.2 48.0 42.3 65.4 79.3 77.6 76.7 69.4 75.8 81.8 90.9 84.1 79.7 74.6 91.5 No. of seeds 72 72 74 77 78 79 82 84 88 95 99 100 108 117 143 DRN (%) 94.4 93.1 91.9 96.1 94.9 94.9 93.9 97.6 95.5 90.5 94.0 92.0 90.7 100.0 95.8 V ariSeed (%) 77.8 90.3 79.7 87.0 67.9 87.3 100.0 76.2 84.1 81.1 85.9 73.0 85.2 94.0 92.3 w eakly-sup ervised learning framework can b e easily generalized to other ob ject detection tasks suc h as fudicial marker trac king in 2D/3D real-time imaging and source/catheter p ositioning in high dose rate (HDR) brach ytherapy . Ac kno wledgment This w ork is partially supported by gran t UL1TR001433 from the National Cen- ter for Adv ancing T ranslational Sciences, National Institutes of Health, USA. References 1. Siegel, R. L., et al.: Cancer Statistics, 2019. Cancer J Clin. 69, 7–34 (2019) 2. Chin, J., et al.: Brach ytherapy for patients with prostate cancer: american so ciet y of clinical oncology/cancer care ontario joint guideline update. J. Clin Oncol. 35, 1737-43 (2017) 3. Sto c k, R., et al.: Imp ortance of p ost-implan t dosimetry in p ermanen t prostate brac hytherap y . Eur Urology 41, 434-439 (2002) 4. Liu, H., et al.: Automatic lo calization of implanted seeds from post-implant CT images. Phys Med Biol 48(9), 1191 - 1203 (2003) 5. Holupk a, E. J., et al.: An automatic seed finer for brac hytherap y CT postplans based on the Hough transform. Med Phys. 31(9), 2672-2679 (2004) 6. Nguy en, H. G., et al.: Automatic 3D seed lo cation and orien tation detection in CT image for prostate brach ytherapy . In: IEEE ISBI 2014, pp. 1320-1323. (2014) 7. Zhang, G., et al.: Automatic seed picking for brach ytherap y p ostimplan t v alidation with 3D CT images. Int J CARS. 12, 1985 - 1993 (2017) 8. Litjens, et al.: A survey on deep learning in medical image analysis. Med Imag Anal. 42, 60-88 (2017) 9. Setio, A., et al.: V alidation, comparison, and com bination of algorithms for au- tomatic detection of pulmonary no dules in computed tomograph y images: the LUNA16 challenge. Med Imag Anal. 42, 1 - 13 (2017) 10. Xu, Y., et al: Gland instance segmen tation using deep multic hannel neural net- w orks. IEEE T rans. Med. Imaging 64(12), 2901-2912 (2017) 11. Bilic, P ., et al. : The liver tumor segmentation benchmark (LiTS). arXiv preprint arXiv:1901.04056 (2019) 12. Y uan, Y., et al.: Automatic skin lesion segmen tation using deep fully conv olutional net works with Jaccard distance. IEEE T rans. Med. Imaging 36(9), 1876-1886 (2017) 13. Y u, L., et al. : Automated melanoma recognition in dermoscopy images via v ery deep residual netw orks. IEEE T rans. Med. Imaging 36(4), 994-1004 (2017)