A dual-branch multimodal model for early detection of rice sheath blight: Fusing spectral and physiological signatures
Haiye Yu, Xiaokai Li, Yue Yu, Yuanyuan Sui, Junhe Zhang, Lei Zhang, Jiangtao Qi, Nan Zhang, Ranzhe Jiang
Abstract
• Multimodal methods use spectral and physiological signs to identify early symptoms. • Dual-branch MMCG-MHA model captures spatial and temporal disease features. • Using interval and individual variable combination strategies. • Transformed 1D data into 2D GASF images to capture underlying data structures. • Key physiological parameters correlated with disease detection metrics. Rice sheath blight (ShB) poses a major threat to global rice production. To address early disease detection, this study developed a multimodal predictive model identifying presymptomatic infections using data integration and processing techniques. Current detection methods often fail to capture these infections; the proposed model integrates disease-specific spectral features with cophysiological parameters and applies data transformation to detect early-stage ShB with high accuracy. Spectral and physiological data were collected using ASD HH2, Li-6800, and OS1P systems during the infection process. A 12-combination method of 5 interval and 2 individual variables was employed to extract sensitive spectral features, and Pearson correlation analysis was used to screen critical cophysiological parameters. One-dimensional (1D) data were transformed into two-dimensional (2D) Gramian Angular Summation Field images to reveal underlying data structures. The novel dual-branch multimodal deep learning model, MMCG-MHA, combines a gated recurrent unit (GRU) for 1D data analysis with a convolutional neural network (CNN) for 2D image analysis, integrating multimodal data spatiotemporal characteristics through a multi-head attention mechanism. Results revealed that the MMCG-MHA model significantly outperformed the single-modal approach, achieving a classification accuracy of 94.1667 % using the optimal feature band extraction method (IVISSA-CARS) and cophysiological parameters, representing 18.96 % and 11.65 % improvements over GRU and CNN, respectively. Key spectral bands in the 500–585 and 737–855 nm ranges, along with eight physiological parameters, were essential for accurate and early detection of ShB, correlating with changes in chlorophyll content and leaf internal structure. This study highlights the potential of multi-source data fusion and advanced processing for early crop disease detection, advancing agricultural monitoring systems.