Multimodal Deep Learning via Late Fusion for Non-Destructive Papaya Fruit Maturity Classification
Cinmayii Manliguez, John Y. Chiang
Abstract
Maturity of fruits significantly affects various areas of the agriculture industry such as the quality assurance of agricultural products, supply chain, and marketing. However, classifying papaya fruit maturity given six ripeness stages with precision remains a challenge since most changes happen inside the fruit rather than the external characteristics, which are quite similar between stages. Using internal properties in classification would require destructive and time-consuming laboratory tests. With the emergence of deep learning and imaging technologies, data with high dimensions, which correlates with internal and external characteristics of an object such as those produced by hyperspectral cameras, can be processed to perform a high-level intelligent classification task without impairing the fruit. In this paper, we present an AI-derived non-destructive approach that utilizes hyperspectral and visible-light images in estimating the papaya fruit maturity stage and implements multimodality via late fusion of imaging-specific networks. The proposed multimodal architecture is composed of imaging-specific deep convolutional neural networks as base learners and a meta-learner that executes late fusion of the dual unimodal networks. Multiclass logistic regression and averaging are explored as the meta-learners of the multimodal fused network that generates the final classifications. Experimental results of the proposed multimodal-late fused models are compared with the multimodal-feature concatenation approach for estimation of papaya fruit maturity, and our proposed model framework obtained an improved F1-score of up to 0.97. This indicates that multimodal-late fused architecture and multimodal imaging systems have great potential for agricultural and other industrial applications.