Multimodal Siamese Framework for Accurate Grade and Measure Estimation of Tropical Fruits
Misaj Sharafudeen, Vinod Chandra S S
Abstract
Fruits have always been a vital ingredient in a nutrient-rich diet. Ensuring the quality of crop production at the supplier level is equally essential to estimating the quantity measure of boxed fruits and their geographical origin at the consumer level. In this article, we propose an interlaced deep neural framework that aids in the accurate qualitative (freshness—good/bad) and quantitative (weights in kilograms) predictive analysis of tropical fruits and their origin in wholesale and retail fruit boxes. This methodology merges an object detection network (YOLOv7) and a deep multi-input multi-output siamese residual convolutional neural network (SRCNN), enabling simultaneous task accomplishment. Separate datasets were compiled to comprehend the initial tasks: Annotated FruitNet and FruiBox. The FruitNet360 dataset was reclustered based on the geographical origin of the fruit. A mean average precision score of 95.90% by YOLOv7 suggests a robust fruit quality detection and localization system. The interconnected siamese layers extract shared features from inputs, enhancing joint learning. The visual weight prediction system exhibited a marginal root-mean-squared error rate of a mere 0.157. The origin of the fruits was identified with 98.33% accuracy. The bottleneck layer of SRCNN facilitated simultaneous regression and classification, capturing the hidden dynamics of source data and contributing well to a combined regression and classification model. Our automation framework could surpass the drawbacks of conventional approaches and reduce the overhead expenses associated with a manual system. This framework could also be integrated into smart devices to assist vendors and consumers.