Beyond the Naked Eye: Computer Vision for Detecting Brown Marmorated Stink Bug and Its Punctures
Lennart Almstedt, Francesco Betti Sorbelli, Bas Boom, Rosalba Calvini, Elena Costi, Alexandru Dinca, Veronica Ferrari, Daniele Giannetti, Loretta Ichim, Amin Kargar, Cătălin Lazăr, Lara Maistrello, Alfredo Navarra, David Niederprüm, P. Offermans, Brendan O’Flynn, Lorenzo Palazzetti, Niccolò Patelli, Cristina M. Pinotti, Dan Popescu, Aravind Krishnaswamy Rangarajan, Liviu Serghei, Alessandro Ulrici, Lars Wolf, Dimitrios Zorbas, Leonard Zurek
Abstract
In this article, we introduce machine learning (ML) techniques developed for the monitoring of the brown marmorated stink bug (BMSB), a significant agricultural pest responsible for considerable crop damage worldwide. The <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Haly.ID</small> project, initiated in early 2021, aims to enhance BMSB monitoring through the utilization of information and communication technology methods. We employ computer vision techniques on RGB images captured by drones and investigate the performance of deep neural networks to evaluate the impact of this invasive species on crop yields in orchards around Europe. Specifically, we evaluate the single shot multibox detector, detection transformer, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">YOLOv5</small> , <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">YOLOv9</small> , and <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">YOLOv10</small> architectures for full-level and patch-level image analysis, respectively. To improve detection accuracy, we experiment with shortwave infrared hyperspectral imaging (SWIR-HSI) in laboratory settings. Given that pheromone baited traps are the most accepted tools for pest detection by field operators, we also propose an Internet of Things sticky trap with an integrated camera equipped with lightweight convolutional neural networks models operating “on the edge” in this resource constrained system. In addition, we develop a client–server application for real-time bug detection, integrating the ML models to provide accessible results to farmers. Lastly, we explore effective postharvesting strategies using SWIR-HSI images to detect insect punctures invisible to the naked eye, thereby enhancing the quality of marketable fruit.