Improved analysis of deep bioacoustic embeddings through dimensionality reduction and interactive visualisation
Francisco J. Bravo Sanchez, Nathan B. English, Md Rahat Hossain, Steven T. Moore
Abstract
Deep neural networks (DNN) are a popular tool to process environmental sounds and identify sound-producing animals, but it can be difficult to understand the decision-making logic, particularly when it does not produce the expected results. Here we describe a new and enhanced visual interactive analysis of embeddings and explore its application in bioacoustics. Embeddings are the output of the penultimate layer of a DNN, an N-dimensional vector that, only one step removed from the final output, represent the inner-workings of a DNN model. Using existing dimensionality reduction techniques we converted the N-dimensional embeddings into 2 or 3-dimensional arrays displayed in scatterplots. By incorporating sound samples into the scatterplots we developed a visual and aural interactive interface and demonstrate its utility in assessing the performance of trained bioacoustic models, facilitating post-processing of results, error detection, input selection and the detection of rare events, which the reader can experience in online examples with publicly available code.