Kymatio: Scattering Transforms in Python

Journal of Mac hine Learning Researc h 21 (2020) 1-6 Submitted 1/19; Revised 10/19; Published 1/20 Kymatio: Scattering T ransforms in Python Mathieu Andreux ma thieu.andreux@owkin.com T om´ as Angles tomas.angles@ens.fr Georgios Exarc hakis geor gios.exarchakis@ens.fr Rob erto Leonarduzzi robe r to.leonard uzzi@ens-l yon.fr Gaspar Rochette gasp ar.rochette @ens.fr Louis Thiry louis.thir y@outlo ok.fr John Zark a johnzarka@gmail.co m ´ Ec ole normale sup´ erieur e, CNRS, PSL R ese ar ch University, 45, ru e d’Ulm, 75005 Paris, F r anc e St´ ephane Mallat stephane.malla t@ens.fr ´ Ec ole normale sup´ erieur e, CNRS, PSL R ese ar ch University, 45, ru e d’Ulm, 75005 Paris, F r anc e Col l` ege de F r anc e, 11, plac e Mar c elin-Berthelot 75231 Paris, F r anc e Flatir on Institut e, 162 5th Av enue, New Y ork, NY 10010, USA Joakim And´ en janden@fla tir oninstitute.org Flatir on Institut e, 162 5th Av enue, New Y ork, NY 10010, USA Eugene Belilo vsky eugene.belilovsky@umontreal.ca Mila, Universit´ e de Montr´ eal, 6666 St Urb ain Str e et, Montr e al, Queb e c H2S 3H1, Canada Joan Bruna br una@cims.nyu. edu Vincen t Lostanlen vl1019@nyu.edu New Y ork University, 70 Washington Squar e South, New Y ork, NY 10012, USA Mua wiz Chaudhary chaudhm@ww u.edu Western Washington University, 516 H igh Str e et, Bel lingham, W A 98225, USA Matthew J. Hirn mhirn@msu.edu Michiga n State University, 426 Auditorium R o ad East L ansing, MI 4882 4, USA Edouard Oy allon edouard.oy allon@lip6.fr CNRS, LIP6, Sorb onne University, 4 plac e Juss ieu, 75252 Paris, F r anc e Sixin Zhang sixin.zhang@pku.edu. cn Peking University, N o. 5 Yiheyuan R o ad, Haidian Distr ict, Beijing 100871, China Carmine Cella carmine.cella@berkeley.edu University of California, Berkeley, 101 Spr oul Hal l, Berkeley, CA 94720, US A Mic hael Eic k en b e rg meickenber g@fla tir oninstitute.org Flatir on Institut e, 162 5th Av enue, New Y ork, NY 10010, USA Editor: Balazs Kegl Abstract The wav elet scattering transform is an inv ariant and stable signal repr esentation suitable for man y signal pro cessing and mac hine learning applications. W e presen t the Kymatio softw are pack age, an easy-to-use, high-p erfor mance Python implemen tation o f the scatter- ing transform in 1D, 2D, and 3D that is compatible with mo dern deep lea rning fra mew orks , including PyT o r ch and T ensorFlow / Ker as . The transforms are implemented on b oth CPUs © 2020 Mathieu Andr eux, T om´ as Angles, Georgios Exarchakis, Roberto Leonarduzzi, Gaspar Ro chett e, Louis Thi ry , John Zark a, St´ ephane Mall at, Joakim And´ en, Eugene Belil o vsky , Joan Bruna, V i ncen t Lostanlen, Muawiz Chaudhary , M atthew J. Hirn, Edouard Oyallon, Si xi n Zhang, Car mine Cell a, M ic hael Eick en berg. License: CC-BY 4.0, see https://creativ ecommons.org/licenses/by/4.0/ . At tribution requir ements are provided at http://jmlr.org/ papers/v21/19- 047.html . Andreux et al. and GPUs, the la tter o ffering a significant s p eedup ov er the for mer. The pack age also has a small memory footpr in t. So urce co de, documentation, a nd ex a mples ar e av aila ble under a BSD license at https ://www .kymat.io . Keyw ords: Scattering T rans fo rm; GPUs; W av elets; Conv olutional Net works; Inv ariance 1. In t ro duction Man y classification and regression tasks h av e a d egree of in v ariance to translations and de- formations, suc h as those r elating to images, audio recordings, and electronic d en sities. The scattering transform w as introduced in Mallat (2012) to build a signal represent ation that is in v ariant to suc h transf ormations wh ile p reserving as muc h as p ossible the in formation rel- ev an t to the task at h and. It is defined as a con v olutional net w ork whose filters are fi xed to b e wa vel et and lo w pass av eraging filters coupled w ith mo dulus nonlin earities. It has man y fa v orable theoret ical prop erties (Mallat, 20 12; Bruna et al., 201 5; W aldspur ger, 2017) and enjo ys considerable success as a p o w erful to ol in mo dern signal p ro cessing (Adel et al., 2017; Bruna and Mallat, 2013; An d´ en and Mallat, 2014; Chud´ a ˇ cek et al., 2014; Sifre an d Mallat, 2013; Eic k en b erg et al., 2017). It is also effectiv e in com bination with mo d ern representa tion learning app roac h es (Oyall on et al., 2018; Sainath et al. , 2014; Z eghidour et al., 2016). This article presents Kymatio , a scattering transform implemen tation that is user- friendly , well -do cumented, fast, and compatible w ith existing automatic d ifferen tiation li- braries. It brings together transform s in 1D, 2D, and 3D under a unifie d application pro- gramming in terface (API). T he scattering netw ork is also tr a v ersed d epth-first to reduce memory r equ iremen ts, enabling efficie nt pr o cessing in limited-memory en vironmen ts, su c h as GPUs. 2. Implemen ting t he Scattering T r ansform Definition W e consider signals defined on a grid of size N 1 × · · · × N d for d = 1 , 2 , 3. Give n t w o signals x [ n ] and y [ n ] on this grid, we den ote their p erio dic con v olution by x ⊛ y [ n ]. The second-order scattering transform is d efined using t w o w a v elet filter banks { ψ (1) λ 1 [ n ] } λ 1 ∈ Λ 1 and { ψ (2) λ 2 [ n ] } λ 2 ∈ Λ 2 , where λ 1 and λ 2 are frequency indices in the sets Λ 1 and Λ 2 . It also includes a lo wpass filter φ J [ n ], where the in tege r J > 0 sp ecifies the a v eraging scale 2 J of the fi lter. T ogether with a non-linearity ρ ( t ), these filters define the s cattering transform . The zeroth-order scattering co efficient S 0 x [ n ] is the lo cal a ve rage giv en by S 0 x [ n ] = x ⊛ φ J [ n ]. Conv olving x [ n ] w ith the first-order w a v elet filter bank { ψ (1) λ 1 [ n ] } λ 1 ∈ Λ 1 , applying ρ ( t ), and con v olving with φ J [ n ], we obtain the fir s t-order scattering co efficients S 1 x [ n, λ 1 ] = ρ  x ⊛ ψ (1) λ 1  ⊛ φ J [ n ] , λ 1 ∈ Λ 1 . (1) The mod ulus of th e first w av elet transform acts as a demo d ulation, shifting its energy to the lo w fr equencies. How eve r, only some of these f requencies are cov ered b y the lo w-pass filter φ J . W e r eco ver the remaining fr equencies by decomp osing ρ ( x ⊛ ψ (1) λ 1 [ n ]) using th e second filter bank, but this is d one only for a sub s et Λ 2 ( λ 1 ) of Λ 2 since ρ ( x ⊛ ψ (1) λ 1 [ n ]) is a lo w-frequency signal. Typically , w e ha v e Λ 2 ( λ 1 ) = { λ ∈ Λ 2 , | λ | > | λ 1 |} . Th e result is then 2 Kyma tio: Sca ttering Transf orms in P ython passed throu gh ρ ( t ) and a v erage d, yiel ding the second-order co efficien ts S 2 x [ n, λ 1 , λ 2 ] = ρ  ρ  x ⊛ ψ (1) λ 1  ⊛ ψ (2) λ 2  ⊛ φ J [ n ] , λ 1 ∈ Λ 1 , λ 2 ∈ Λ 2 ( λ 1 ) . (2) The energy of higher-ord er scattering coefficient s is typically small and do es n ot greatly influence results (W aldspurger , 201 7; Brun a and Mallat, 2013; And ´ en and Mallat, 2014). On the other hand they can b e compu tationally int ensiv e. W e ha v e thus c hosen to restrict our scop e to second-order co efficien ts, whic h is what is used in most works. Implemen tation Signals obtained by filtering and applying ρ ( t ) are low-frequency , so in- termediate results are do wnsampled to reduce computational load as in And´ en et al. (2014). In 1D and 2D, we use Morlet wa vele ts whic h a re close to analytic ( i.e . , complex-v alued with lo w energy in the negativ e frequencies) and the non-linearit y is th e complex mo dulus ρ ( t ) = | t | for t ∈ C (And´ en and Mallat, 2014; Bruna an d Mallat, 2013). The 3D trans- form is calculated using solid harmonic w a v elets ψ λ 1 = ψ j,ℓ,m , where j indexes the scale, and ℓ, m are the azim uthal and m agnetic qu an tum n umbers. In this case the n on-linearit y ρ : C 2 ℓ +1 → R is d efined, with a sligh t abuse of notation, as ρ ( x ⊛ ψ j,ℓ ) = p P m | x ⊛ ψ j,ℓ,m | 2 (Eic k enb erg et al., 2017). F ollo wing O y allon et al. (2018), we compute th e scatte ring trans- form in a depth-first manner, reducing the num b er of intermediate signals stored at a giv en time. S ince con volutions are all p erio dic, they ma y b e efficien tly calculated using fast F ou r ier transforms. 3. Pro ject vision Co de qualit y Adopting the p hilosoph y of scikit-le arn (Pe dregosa et al., 2011), the goal of Kymatio is not to maximize the num b er of features, but to pr o vide a stable and easy- to-use framew ork. T o this end, w e make hea vy use of u nit tests, minimize the num b er of dep end en cies, and striv e f or in tuitiv e int erfaces inspired by mo dern d eep learning paradigms. Kymatio also pr o vides an exte nsive user guide, includ in g an API reference, a tutorial, in- stallatio n instructions, and easy-to-understand examples, sev eral of which feature real-w orld applications. Comm unit y and bug trac king Kymatio is free and op en-source soft w are with a 3-clause BSD license. Th e mem b ers of its core d ev elopmen t team all ha v e exp erience implemen ting scattering transf orms in other pack ages. A k ey goal of Kymatio is to com bine these efforts and foster a communit y effort in order to pro d uce high-qualit y softw are and main tain a critical mass of con tributors for its mainte nance. The pac k age w as r eleased p ublicly on GitHub No v em b er 17th, 2018. T he main comm unication c h annel is the GitHub page f or questions, b ug rep orts, and feature r equests. There is also a dedicated Slac k c hannel. Relation to previous soft ware Aside from the emphasis on co de qualit y and usability , Kymatio pr o vides sev eral impro v emen ts ov er p r evious scattering im p lemen tations: • Python is the de facto standard for data science softw are, but most existing scattering pac k ages are implemen ted in MA TLAB. In con trast, Kymat io pro vides a completel y Pythonic imp lemen tation, enabling in tegration w ith the scien tific Py th on ecosystem. 3 Andreux et al. dimension gpu diff. core devs. license language ScatNet 1D, 2D 5 Apac he 2.0 MA T LAB ScatNetLigh t 2D 2 GPLv2 MA TLAB PyScatW av e 2D X 3 BSD-3 Python Scattering.m 1D 1 GPLv3 MA TLAB PyScatHarm 3D X 1 BSD-3 Python W a v elet T o olb o x 1D N/A Proprietary MA T LAB Kymatio 1D, 2D, 3D X X 15 BSD-3 Python T able 1: Comparison to existing scattering trans form pac k ages. • GPU c omp atibility is critical to many d ata science workloads. Kymatio offers an easy-to-use GPU implemen tation for scattering transforms in 1D, 2D, and 3D. • F r ontends are pr o vided for many fr ameworks, including NumPy , scikit-le arn , PyT or ch , and T ensorFlow / Ker as , allo wing for seamless in tegrating scattering transforms in a v ariet y of p ip elines. In particular, the PyT or ch , and T e nsorFlow / Ker as frontends allo w for inclusion into man y deep learning wo rkflows. • Differ entiability of the scattering transform sim p lifies applications in reconstruction and generativ e mo d eling, among others. T able 1 p ro vides a detailed comparison of existing implement ations: ScatNet (And´ en et al., 2014), ScatNetLigh t (Oy allon and Mallat, 2015), PyScatW a v e (Oy allon et al., 2018), Scat- tering.m (Lostanlen and Mallat, 2015), PyS catHarm (Eic k en b erg et al., 2018), and the scat- tering transf orm implemente d in the MA TL AB W a v elet T o olb o x. 4. User In terface and Do cumen tation In terface The in terface is designed to b e flexible and consisten t across in puts and fr on- tends. Let us consider the PyT or ch frontend. W e first create a sca ttering ob ject by s p eci- fying the a v eraging scale J an d the inp ut signal shap e. from kymat io .torc h im port Scatteri ng1D , Scatteri ng2D , Harmo nicSca ttering3D S = Scatte ring1D (J, shap e=(length ,)) S = Scatte ring2D (J, shap e=(height , wi dth)) S = Harmon icScat tering3D (J, shap e=(height, width, depth)) The resulting ob ject S ac ts like a nn.Mo dule ob j ect in PyT or ch . The scattering transf orm S is app lied throu gh calls of the form x = torch.randn((28, 28)) output = S(x) Switc hing f rom GPU or CPU fu nctionalit y also follo ws the API of nn .module . S.cuda() # Run on GPU S.cpu() # Run on CPU 4 Kyma tio: Sca ttering Transf orms in P ython Do cumen tation and examples Sev eral examples are p ro vided with the co d e, illustrat- ing the p ow er of K ymatio . T h ese in clude image reconstruction and generation fr om s catter- ing (Angles and Mallat, 201 8), hybrid scattering and CNN training on CIF AR and MNIST (Oy allon et al., 2018), regression of m olecular prop erties on QM7/QM9 u sing solid harmon ic scattering (Eic k en b erg et al., 2017), and classifying recordings of sp oken digits. 5. Conclusion Kymatio p ro vides a we ll do cumen ted, u ser-friendly , and fast implement ation for th e scat- tering transform. It can b e used with the PyT or ch and T ensorFlow / Ker as deep learning framew orks and su pp orts a v ariet y of applications th at hav e b een previously inaccessible to non-exp erts including hybrid deep learnin g, generativ e mod eling, and 3D c hemistry applica- tions. F utu re wo rk in clud es further optimization for sp eed, flexibility , and back end supp ort. Ac kno wledgmen ts W e th an k Laur en t Sifre, Sergey Zago ruyko and Gabriel Huang for their h elpful commen ts. The pro ject was s u pp orted b y ER C Inv ariantClass 32 0959. EB is fund ed by a Go ogle F o cused Researc h Aw ard a nd IV ADO. MJH is p artially sup p orted by Alfred P . Sloan F el- lo wship #F G-2016 -6607, D ARP A Y oung F acult y Aw ard # D16AP00 117, and NSF grant #162021 6. The Flatiron In stitute is a division of the S im on s F oundation. References T. Adel, T. Cohen, M. Caan, M. W elling, et al. 3D scattering trans forms for disease classification in neuroimaging. Neur oImage: Clinic al , 14:506 –517, 2017. doi: 10.1016/ j. nicl.2017 .02.004. J. And´ en et al. Scatnet. Computer Softwar e , 2014. URL http://w ww.di.ens .fr/data/software/scatnet . J. And ´ en and S. Mallat. Dee p scattering sp ectrum. IEEE T r ans. Sig nal Pr o c ess. , 62(16): 4114– 4128, Aug 2014. doi: 10.1109 /TSP .2014.2326991. T. Angles and S. Mallat. Generative net w orks as inv erse problems with scattering trans- forms. In P r o c. ICLR , 2018. J. Bruna and S. Mallat. Inv arian t scatt ering con v olution net w orks. IE EE T r ans. Pattern Ana l. Mach. Intel l. , 35(8):1 872–18 86, 2013. doi: 10.1109/TP AMI.2012.230. J. Brun a, S. Mallat, E . Bacry , and J .-F. Muzy . Intermitten t pr o cess analysis w ith scattering momen ts. Ann. Statist. , 43(1):323 –351, 02 2015 . doi: 10.1214/ 14- A OS1276. V. Chud´ a ˇ cek et al. Scattering trans form for intrapartum f etal heart r ate v ariabilit y f ractal analysis: A case-con trol study . IEE E T r ans. Biome d. Eng. , 61 (4):110 0–1108, 2014. doi: 10.110 9/TBME.201 3.2294324. 5 Andreux et al. M. Eick enberg et al. Solid h armonic wa v elet scattering: Predicting quan tum molecular energy fr om inv ariant descriptors of 3D electronic dens ities. In Pr o c. NIP S , pages 6540– 6549, 2017. M. Eic k en b erg et al. Solid harm on ic w a v elet s cattering for predictions of molecule pr op erties. The Journal of Chemic al Physics , 148(24):2 41732, 20 18. doi: 10.1063/ 1.50237 98. V. Lostanlen and S. Mallat. W a v elet scattering on the pitc h spiral. In Pr o c . DAFx , 2015. S. Mallat. Group in v arian t scat tering. Comm. Pur e Appl. Math. , 65(10) :1331–1 398, 2012. doi: 10.1002 /cpa.2141 3. E. Oy allon and S. Mallat. Deep r oto-translati on scatte ring for ob ject classification. In Pr o c. CVPR , Jun e 2015. E. Oy allon et al. Scattering net works for hybrid represen tation lea rning. IEEE T r ans. Pattern Anal. Mach. Intel l. , 41(9): 2208–2 221, 2018. doi: 10.1109/ TP AMI.2018.2855738. F. P edregosa et al. Scikit-learn: Mac hine learning in Python. J. M ach. L e arn. R es. , 12 (Oct):282 5–2830 , 2011. T. N. Sainath et al . Deep scattering sp ectra w ith deep neu ral net wo rks for L V C SR tasks. In Pr o c . Intersp e e ch , 2014. L. Sifre and S. Mal lat. Rota tion, scaling and deformation inv arian t scattering f or te xture discrimination. In Pr o c. CVPR , 2013. doi: 10.1109 /CVPR.2013.1 63. I. W aldspur ger. Exp onen tial deca y of scattering co efficient s. In Pr o c. SampT A , pages 143– 146, 2017. d oi: 10.110 9/SAMPT A.2017.8 024473. N. Zeghidour et al. A deep scattering spectrum –deep siamese netw ork pip eline for un- sup erv ised acoustic mo deling. In Pr o c. ICASSP , pages 4965 –4969. IEEE, 2016. doi: 10.110 9/ICASSP .2016.747 2622. 6 This figure "algorithm.png" is available in "png" format from: