Plants meet machines: Prospects in machine learning for plant biology

Abstract

Machine learning approaches are affecting all aspects of modern society, from autocorrect applications on cell phones to self-driving cars to facial recognition, personalized medicine, and precision agriculture. Although machine learning has a long history, drastic improvements in these application areas recently have been driven by improvements to computational infrastructure; increased computing power; increased ability to collect, manage, and store very large amounts of data; and algorithmic advances. Multiple types of machine learning have been developed, each with its own techniques, strengths, and weaknesses, making certain approaches better matches for certain problems than others. Supervised machine learning and the use of neural networks (e.g., deep learning; Table 1) underlie much of the recent accelerated application of machine learning to many biological problems, including those across a range of scientific questions in plant science. For example, deep learning technologies have recently achieved impressive performance on a variety of predictive tasks, such as species identification (Unger et al., 2016; Carranza-Rojas et al., 2017), plant species distribution modeling (e.g., Zhang and Li, 2017; Botella et al., 2018), weed detection (Yu et al., 2019), and mercury damage to herbarium specimens (Schuettpelz et al., 2017). They are also being applied to questions of comparative genomics (e.g., Xu and Jackson, 2019) and gene expression (Mochida et al., 2018) and to conduct high-throughput phenotyping (e.g., Singh et al., 2016; Ubbens and Stavness, 2017) for agricultural and ecological research. Moreover, novel approaches are poised to revolutionize studies of plant phenology (e.g., Pearson et al., 2020) and functional traits through application to more than 30 million images of herbarium specimens now available at iDigBio (http://www.idigbio.org) as well as other digital repositories. The application of machine learning methods to extract data from herbarium specimens has grown and diversified in a few short years, beginning with species identification in a specific geographic region (e.g., Unger et al., 2016). Subsequent attempts to use deep learning to tackle the difficult taxonomic task of identifying species in large collections of herbarium specimens showed that convolutional neural networks trained on thousands of digitized herbarium sheets are able to learn highly discriminative patterns (e.g., Carranza-Rojas et al., 2017). These results are very promising for extracting a broad range of accurate annotations in a fully automated way. Such approaches are also being applied to identification of plant phenophase (i.e., bud, flower, fruit), which is important for assessing the effects of climate change on plant growth and reproduction and for comparing plant responses with those of pollinators, migratory birds, and other species that rely on plants for food and/or nesting sites (see, e.g., Lorieul et al., 2019; Pearson et al., 2020; Brenskelle et al., 2020; Goëau et al., 2020). Likewise, other evolutionary or ecological traits, such as leaf shape and size, leaf margins, and flower color, could also potentially be scored from images of herbarium specimens. However, despite the promise of applying deep learning to herbarium specimen images to address a range of questions, this emerging field also raises challenging methodological questions about how to avoid any bias and misleading conclusions when analyzing the produced data. Indeed, as for any statistical learning method, convolutional neural networks are sensitive to bias issues, including the way in which the training data sets are built. Moreover, as good as the prediction might be on average, the quality of the produced annotations can be very heterogeneous from one sample to another, depending on various factors such as the morphology of the species, the storage conditions in which the specimen was preserved, and the age of the specimen when imaged. Given both the opportunities and challenges, additional research into the application of machine learning approaches to herbarium specimen images is needed to enable greater applicability to a broad range of scientific questions. The field of machine learning is moving rapidly, with the development of alternative approaches that may be best suited to specific questions, data sources, and analytical techniques. This special collection of articles in Applications in Plant Sciences presents 16 papers, published across two issues of the journal, that explore methods and applications of machine learning to studies of plant ecology, morphology, genomics, and agriculture. The first issue comprises eight papers and focuses on applications to images of herbarium specimens, on topics from phenology to herbivory. The second issue includes papers that address a broader range of topics, data, and biological scale. We summarize the content of both issues here. Plant phenological research has seen major advances in recent years through the use of herbarium specimens (Willis et al., 2017). Herbarium specimens collected over the past three centuries provide insight into flowering, leaf-out, and fruit timing globally and across plant phylogeny (Davis et al., 2015). A major hurdle, however, is that to harness the full power of herbarium specimens for phenological research requires counting reproductive structures, which can be time consuming. Thus, automated recognition of reproductive structures on herbarium specimens is a key goal in current phenological research (Lorieul et al., 2019; Pearson et al., 2020). Two papers in this special issue address plant reproductive phenology. To make use of the extensive volume of herbarium specimens for examining angiosperm reproductive phenology, Goëau et al. (2020) applied a state-of-the-art segmentation approach (mask R-CNN) to automate locating, segmenting, and counting reproductive structures on images of herbarium specimens of Streptanthus tortuosus Kellogg (Brassicaceae). Phenological stages (i.e., buds, flowers, immature fruits, mature fruits) are distinct in S. tortuosus, and specimens were scored for phenophase. Evaluation of the performance of the method indicated that it shows particular promise in identifying the number of reproductive structures (accuracy was nearly 80%), but the accuracy of the results varied with respect to the training annotations, the type of reproductive structures scored, and the size of the reproductive structures. Although promising, these results suggest that further refinement is needed, and it is unclear how well the approach will scale to other species with different floral morphologies and perhaps less well-differentiated phenophases. To train machine learning algorithms to do this, however, will require massive input data to data-hungry machine learning algorithms. In this issue, Brenskelle et al. (2020) assess the conditions needed for volunteers to help gather these data. The authors test for the effects of training type (in person or online), career stage, plant taxon, and phenological stage scored on the accuracy of volunteer-provided phenological data from herbarium specimens. Regardless of expertise and training method, users provided highly accurate data, although data from people trained in person were more accurate than those trained online. This study provides a best practices guide for collecting annotation data. Importantly, the authors also demonstrate that online citizen science platforms might be able to provide accurate annotation data that can then be used downstream to train machine learning algorithms to recognize phenological stages. Morphological variation, coupled with variation in the quality of herbarium specimens, leads to noise and potential bias in automated coding of characters from specimen images. Image segmentation is a computer vision algorithm that groups together pixels of an image that have similar attributes and generates a mask for each focal object in the image, such as a flower in an image of a herbarium specimen. Application of masks, such as those applied to plant phenophases, can help to reduce noise and bias. White et al. (2020) developed a workflow to apply segmentation masks to plant images using deep learning. Focusing on ferns, they generated a model that could segment herbarium images automatically, efficiently, and accurately across the morphological diversity of this clade. Although their study was restricted to ferns, the workflow is generalizable to all herbarium images and, with modification, may be applicable to other clades of plants with highly different morphologies. Plants and insects have been interacting for 400 million years, and these interactions have likely driven diversification of both clades. The fossil record shows evidence of herbivory, providing a glimpse into long-term patterns of plant–herbivore interactions and evolution. However, how herbivory changes over shorter timescales and geography is much less clear. Despite the fact that botanists generally attempt to collect specimens that are free of herbivore damage, herbarium specimens offer a view of plant–herbivore interactions over the past three or four centuries, with the potential to infer spatial and temporal patterns of herbivory, including response to climate change (Meineke and Davies, 2018; Meineke et al., 2018). However, manual scoring of insect damage to herbarium specimens is extremely laborious, and the possibility of applying machine learning to quantify the patterns and extent of insect damage to plant specimens is appealing. Meineke et al. (2020) initiated machine learning methods to explore their ability to classify multiple types of herbivory (and its absence) across a pair of divergent plant species. Although herbivory could not always be classified with high accuracy, the use of hand-drawn boxes to locate areas of potential herbivory increased the accuracy of herbivory classification to 81.5%. The authors further identify ways to expand the accuracy of the models in future applications, potentially paving the way for exploring patterns of herbivory in relation to climate change, invasive species, and more. The contributions of machine learning to the plant sciences, especially for automated species identification from images of digitized herbarium specimens, is showing great promise (Schuettpelz et al., 2017; Wäldchen and Mäder, 2018). This is especially true for genera with only slight morphological variation among species, particularly when compounded by hybridization and the presence of infraspecific taxa. Pryer et al. (2020) have built on this work with Equisetum L., a distinctive genus with 15 extant species complicated by morphological plasticity and frequent hybridization events that have resulted in a disproportionately high number of misidentified herbarium specimens. Equisetum includes two relatively distinct species (E. hyemale L. and E. laevigatum A. Braun) and a widespread, sexually sterile hybrid between them (E. ×ferrissii Clute) (Rutz and Farrar, 1984; Des Marais et al., 2003). The challenges faced here result from the cylindrical nature of the stem, which results in dramatic differences in specimen images due to factors such as the geometry of the flattened stems, the number of stems included on a single sheet, stem colors, and imaging parameters. Compounding the variations among images is the fact that accurate identification has more to do with the appearance of stem nodes and strobili than other features. Through successive testing of several models, Pryer and colleagues discovered that, out of 30 test images, 27 were classified correctly. Although the number of specimens is probably too small to be broadly generalizable, E. hyemale images were correctly classified in nine of 10 cases, E. ×ferrissii images in eight of 10 cases, and E. laevigatum images were never confused, resulting in an accuracy of 90%. These results suggest strong potential for machine learning’s impact on the accurate determination of closely similar taxa. In their contribution, Ott et al. (2020) outline the development and output of GinJinn, object-detection software designed to extract leaf images from herbarium specimens based on the TensorFlow (Abadi et al., 2016) object-detection application programming interface (API), an API designed to make supervised deep learning object detection accessible for plant scientists. Although GinJinn makes heavy use of TensorFlow’s API, the authors maintain that GinJinn is not merely a wrapper for the API; it also provides data preprocessing, project set up, pretrained model download, simple model exporting, and the use of trained networks for the extraction of bounding boxes from newly acquired data. GinJinn was tested on a data set of 286 JPEG images of preserved plant herbarium specimens provided by the herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem, Berlin, Germany. The images were annotated using the free open-source tool LabelImg version 1.8.1 (https://github.com/tzutalin/labelImg), resulting in a total of 889 annotated intact leaves within 243 images of herbarium specimens of two species of Leucanthemum Mill. (the diploid L. vulgare Lam. and the tetraploid L. ircutianum DC.) known for their high variability in leaf shape. The task is complicated by the rare occurrence of intact leaves versus non-intact leaves in these species. Using 183 specimens as the training data set, the GinJinn pipeline extracted one or more intact leaves in 95% of 61 test images. A major challenge to cataloging and describing plant diversity lies in the development of high-throughput technologies that facilitate rapid discovery of new taxa hidden in the backlog of still-to-be processed herbarium specimens. The 400,000 plant species currently known to science have required more than 250 years to name and classify, and as many as 70,000 flowering plant species are likely yet to be discovered (Joppa et al., 2011). Many of these may well be among the estimated one million specimens currently backlogged in herbaria. From Little et al.’s (2020) perspective, this renders herbaria largely untapped resources for the new and rapidly developing use of artificial intelligence (AI) in taxonomic research (Wäldchen and Mäder, 2018). To capitalize on this enthusiasm and encourage an increasing number of AI specialists to devote attention to algorithms that can produce species identifications, these authors mounted a Kaggle competition platform to crowdsource effective machine learning algorithms for analyzing plant specimen images. The competition data set included 46,469 images representing 683 species of the family Melastomataceae (Tan et al., 2019). In just two months, 254 models were developed that automatically identified the taxa among these digital representations, with the top four models identifying specimens to species with >88% accuracy. Trait extraction from herbarium specimens can be laborious and time consuming, making the process an excellent candidate for the application of high-throughput machine learning protocols and algorithms. Here, Weaver et al. (2020) describe and test LeafMachine, an automated, open-source software tool for recognizing and measuring leaf dimensions from herbarium specimens and single leaf images across a wide range of largely woody taxa (trees, shrubs, lianas), although some herbaceous taxa were also included. The tests show varying results based on image resolution, specimen presentation, leaf condition, and whether leaf clumping was present. Of ~1000 images containing measurable leaves as confirmed through assessment, LeafMachine produced morphometric information for at least one leaf in 82.0% of high-resolution images and 60.8% of low-resolution images, suggesting positive results to the researchers but with a need for as machine learning technologies The second set of papers a broad range of topics, beginning with application of machine learning approaches to agriculture. The use of to in agricultural is both and and are needed, especially for the use of application of small of on a and application and methods of such as However, such approaches to precision require highly accurate methods of detection and identification of in agricultural et al. (2020) applied an segmentation convolutional neural to generated images of agricultural field to plants and then identify them as or Using this mask the authors were able to correctly identify and plants at precision of and identification of was generally more with precision as high as for but less than for the other Using these detection to of could be and plant were more than with alternative bounding of the models to for plant species, plant size, plant and interactions could accuracy for greater automated weed with of with interactions are et al., and can be highly for agricultural and et al., a plant traits with to insects is of broad in the such type of insect are small that as that insect herbivore and many such leaf traits, counting the required to address a research can be a et al. (2020) make advances of by images of making the leaves through a and applying novel and methods for counting They then results from these novel methods to manual counting and that the accurate novel method was input from the and was accurate at on leaf Although fully automated counting has not yet been this study an important and of a major in automated phenotyping for Given the ability to automate the of plant traits, a be whether the plant traits extracted could be used for plant species could the traits for species identification be using machine learning et al. (2020) the use of for plant identification using as well as identifying the traits between species. Using the Plant Trait et al., 2020) and a collection of species that and other they were able to correctly identify plant species with to accuracy in such as leaf fruit and flower were identified as being some of the more plant data are collected by automated methods as the type of approach in this can be used to guide and the data collection high-resolution images of plant for use in downstream and machine learning algorithms has a challenging current methods use that are to (e.g., et al., imaging using with into the of plants methods through which are into the et al., by et al., 2018), with the to and or use methods such as for and woody plants with in or et al., 2018) or imaging et al., 2017) for plants with et al. (2020) on to the of these images by two state-of-the-art deep learning the et al., 2016) and the method is designed to high-resolution output from low-resolution images to not by a of these demonstrate that these models the when trained with data Supervised machine learning methods are the methods used when applied to plant science. machine learning approaches are used to automate or reduce the and time needed to that were by These of well to supervised machine learning approaches also provide for data and of collected data. et al. (2020) and the use of an in discovery of species The authors to and that can to and in some problems and data recognition and are in a range of applications including the automated of specimen and Thus, the automated recognition of scientific can be particularly for some Little (2020) and developed an open-source approach for scientific name recognition using artificial neural The method on an approach that can recognize scientific across a range of (e.g., with high and precision and at of Plant are generally large and with and high amounts of over plant now published et al., 2018), many more and both and resources available for thousands of other plant species (e.g., et al., et al., 2019), data are now available for comparative of plant across Although methods for identifying are currently for gene and other attributes of plant require further Machine learning approaches are being applied to a range of problems in plant genomics, and et al. (2020) the promise of these They on supervised machine learning for gene from information as well as data. gene may and within a plant and have or effects on functional prediction a of at gene expression and and the authors machine learning methods at each of these problems as well as those designed to information across and biological these the authors identify current to more models and suggest Many machine learning methods have been developed for and as more and more methods are being developed and to such as and and et al., 2016; et al., 2018). et al. (2020) developed a segmentation and approach for leaf using The approach by the authors a small number of image to automate segmentation over more than with of greater than 90%. The approach is on leaf manual but the of being able to apply machine learning methods to automate the of a wide variety of data and image The application of machine learning to questions in plant is in its yet the promise of these methods to a broad range of problems is clear. From to of plant morphology, and and from assessing ecological interactions of plants with and their to use in new approaches machine learning have the potential to change how study plants and the questions can with from to all by new machine learning will further enable new in plant However, as the contributions to this special issue have methods with high accuracy for application are development and may require extensive in training data Thus, despite the promise and of machine learning certain problems may not be of in the model or the data needed for training sets are not available or not We that the papers in this collection encourage further on the emerging applications of machine learning to plant This collection of papers was by with from the

Topics & Concepts