A Multi-modal Deep Neural Network approach to Bird-song identification

We present a multi-modal Deep Neural Network (DNN) approach for bird song identification. The presented approach takes both audio samples and metadata as input. The audio is fed into a Convolutional Neural Network (CNN) using four convolutional layers. The additionally provided metadata is processed using fully connected layers. The flattened convolutional layers and the fully connected layer of the metadata are joined and fed into a fully connected layer. The resulting architecture achieved 2., 3. and 4. rank in the BirdCLEF2017 task in various training configurations.

💡 Research Summary

The paper presents a multimodal deep learning framework for bird‑song identification that jointly processes acoustic spectrograms and auxiliary metadata (geolocation, elevation, and time‑of‑day categories). The authors first convert raw field recordings into short‑time Fourier transform (STFT) based mel‑spectrograms (80 mel bands) using either a 256‑point or a 512‑point FFT window. After normalizing the spectrogram to the