A novel Mask R-CNN-based tracking pipeline for oyster mushroom cluster growth monitoring in time-lapse image datasets
Christos Charisis, Sari Nuwayhid, Dimitrios Argyropoulos
Abstract
This paper introduces a novel deep learning-based instance segmentation tracking pipeline for the continuous monitoring of oyster mushroom cluster growth in a series of images at different time intervals. Multiple time-lapse photography trials were carried out in a mushroom farm under real-life operational conditions and a comprehensive dataset containing 6,382 high-resolution images with 37,004 mushroom clusters of various sizes were created. Two Mask R-CNN configurations in the feature extraction, a CNN-based (ConvNeXt) and a Transformer-Based (Swin), were examined for the detection, segmentation and localisation of individual clusters in the images. Then, the clusters recognized by the instance segmentation model, were linked across a series of timelapse images to track their growth across the consecutive images. A custom tracking algorithm, which uses object masks instead of bounding boxes was developed to accurately track mushroom instances even when they are highly occluded or temporarily hidden. This method considers the cluster orientation, size, shape and growth stage and accounts for the variance in object detection inherent to the instance segmentation model. The pipeline also integrates robust image post-processing filters to address illumination effects, occlusion and partial mushroom harvesting. Finally, a mushroom sizing method that converts pixel dimensions of segmented clusters into relative to the substrate’s real-size dimensions was proposed using the time-static substrate blocks as a reference. This enables estimation of cluster area, which is critical for monitoring the mushroom growth trajectories. The results revealed that Mask R-CNN with ConvNeXt as a backbone achieved the best performance for mushroom cluster instance segmentation (mAP = 0.876, recall = 0.858, F1-score = 0.867) and tracking (MOTA = 0.967). The proposed pipeline provides valuable information on cluster growth patterns and maturity levels and sets a solid basis for future smart farming applications in oyster mushroom cultivation such as yield prediction and robotic harvesting.