Herbarium specimen label transcription reimagined with large language models: Capabilities, productivity, and risks

Abstract

Natural history collections balance immense scientific value with financial constraints. Despite their key role in biodiversity research, they often face resource and staffing shortages. Within this complex landscape, the rise of large language models (LLMs; see Table 1 for definitions of italicized terms), particularly ChatGPT, is notable (OpenAI, San Francisco, California). LLMs have quickly gained prominence, garnering praise as innovative tools while also facing criticism for biases, potential misuse, data privacy, and data ownership concerns. Despite the contrasting sentiments they provoke, LLMs and their future derivatives are poised to redefine workflows and management of natural history collections. In this essay, we focus on specimen label transcription as motivation to venture deeper into the rapidly changing world of LLMs and pose several pressing questions. (1) Are LLMs capable of handling the complexities of curatorial work? (2) Can the use of LLMs alleviate some of the strain on chronically underfunded collections by increasing productivity? (3) What risks might we encounter in this integration? A collection of key-value pairs encoded in the JavaScript Object Notation (JSON) format. For example: Specimen Label = { “ Genus ” : [ Genus ] , “ Species ” : [ species ] , … , “ Date ” : [ YYYY − MM − DD ] } ${Specimen\; Label}=\{ \mbox{\unicode{x00060}\unicode{x00060}} {Genus}\mbox{''}:[{Genus}], \mbox{\unicode{x00060}\unicode{x00060}} {Species}\mbox{''}:[{species}],\ldots , \mbox{\unicode{x00060}\unicode{x00060}} {Date}\mbox{''}:[{YYYY}-{MM}-{DD}]\}$ LLMs represent one of the latest breakthroughs in natural language processing; they can interpret, generate, translate, and transform human-like text. This capability is honed by training the models on extensive volumes of textual data, adjusting the weights of billions of parameters (often likened to digital neurons) in the process. This vast network empowers LLMs to generate coherent content by leveraging these learned patterns to probabilistically predict subsequent tokens that might correspond to letters, words, or extended text segments. LLMs are potent computational tools because of their generative and nondeterministic properties. When given the same input prompt multiple times, a LLM might produce varied outputs. This blend of adaptability and unpredictability, often colloquially compared to human creativity, enables LLMs to comprehend and produce content for virtually any scenario. However, taming LLMs to produce consistent responses that are useful for real-world problem solving can be challenging. Herbarium specimen digitization is often impeded by numerous bottlenecks, including transcribing the specimens’ labels into a searchable database and geolocating historical specimens. Currently, human transcribers begin with an empty spreadsheet or database form and manually type the contents of each label into the appropriate fields—a process that frequently involves reference searches to validate geographic locations or taxonomy. Then, some form of quality control takes place before transcribed text is uploaded to the database of record (e.g., Specify, Symbiota, BRAHMS). If appropriate, records can then be uploaded to public repositories. Therefore, the goal of an automated transcription tool is to automate the most time-consuming parts of the transcription process—typing and validating label content—to produce formatted data (rows in a spreadsheet) that closely matches or exceeds the quality of human transcription (Walton et al., 2020). While optical character recognition (OCR) is still an active field of research, several existing OCR implementations are capable of extracting text from herbarium labels, including EasyOCR (https://github.com/JaidedAI/EasyOCR/tree/v1.7.0), Tesseract (Kay, 2007), PaddleOCR (https://github.com/PaddlePaddle/PaddleOCR/releases/tag/v2.6.0), Google's Vision AI (https://cloud.google.com/vision), or FloraTraiter's custom OCR implementation (Folk et al., in press). Other tools like LeafMachine2, DeepText, or the methods from Thompson et al. (2023) can be used to isolate text from the surrounding specimen, reducing input noise for the OCR algorithm (Zhong et al., 2017; Weaver and Smith, 2023). Because there are numerous established procedures for OCR, we will turn our focus to the crux of the problem, putting label contents into the correct predefined column of a spreadsheet. OCR algorithms typically return unstructured text that includes useful information like genus, species, and geographic location, as well as extraneous words or meaningless symbols that result from the OCR algorithm's misinterpretation of the image. Placing each bit of information into the appropriate spreadsheet column (or ignoring gibberish) is a task for which LLMs are uniquely suited. If a user submits unstructured OCR text to a LLM without any additional context or instructions, then there is no way to anticipate what the LLM will produce because the unstructured OCR text alone is directionless—the output will be chaotic and inconsistent. To coax the LLM to correctly parse the unstructured OCR text, the prompt must be engineered in a way that increases the probability that the LLM will output the text in the desired format, which is a JSON dictionary where keys are the spreadsheet column names and the fields are the appropriate content. Prompting LLMs involves a series of systematic steps delineated in Figure 1. Despite careful engineering, the LLM might occasionally return an improperly formatted JSON dictionary or include irrelevant explanatory text. Constraining LLM output to a specified format can be challenging, but several refinement methods can guide the often inconsistent and unpredictable behavior of LLMs. Structured output parsers wrap the prompt within rigorous linguistic parameters—JSON notation in our case—to significantly enhance the consistency of the LLM's output. In cases where the method fails, or the JSON format is invalid, recursive prompting recycles the error-containing original response back into the LLM with a new corrective prompt that only instructs the LLM to fix the previous JSON format failure. Together, these strategies can effectively manage the unpredictable nature of LLM outputs and consistently return a properly formatted spreadsheet ready for curatorial review. To explore whether LLMs are capable of curatorial work in practice, we created a prototype tool called VoucherVision to test various LLM models, prompts, and techniques in the realm of herbarium label transcription (see https://huggingface.co/spaces/phyloforfun/VoucherVision). VoucherVision serves as a demonstration of the practical issues related to the intersection of LLMs and curation. Many of the processes we previously described could also be adapted to address a range of unsolved problems, e.g., any situation that involves the interpretation, translation, or creation of text is a suitable candidate. We integrated VoucherVision into our existing transcription workflow, allowing for a comparison with the conventional, fully manual approach. Rather than directly allocating images to transcribers, we first channel a batch (200–500 specimen images) through VoucherVision, which yields a completed spreadsheet within a few hours. Because this version of VoucherVision leverages OpenAI or Google LLM APIs, it requires minimal local computational resources and can be run on most internet-connected computers. A staff member then reviews and rectifies any evident errors using our editing tool (https://github.com/Gene-Weaver/VoucherVisionEditor) before uploading the LLM-generated spreadsheet into the database of record. An impressive 25% increase in the number of records processed per hour has been noted so far, suggesting promising potential advantages for collections’ workflows including the reallocation of funds and resources to other critical tasks such as georeferencing, which often get sidelined because of budget constraints. This shift could lead to more efficient utilization of grant funds and help to better meet project objectives. As we integrate LLMs into curatorial environments and beyond, we are confronted with three significant risks. The first concern pertains to potential job and volunteer losses. While LLMs produce competent results, we underscore that LLMs are likely to enhance workflow productivity, rather than replace human curators. Curatorial staff, student technicians, and volunteers can identify collection trends, decipher obscure markings, and apply insights from past digitization projects to improve label transcription accuracy—given the unpredictable nature of LLM outputs, human oversight is essential for quality assurance. In this light, curatorial staff, student technicians, and volunteers are likely to transition to more editorial and managerial roles, where they ensure correct data entry and effectively guide the LLM outputs. Curators, technicians, and volunteers excel at manually transcribing damaged or near-illegible specimen labels. By leveraging their expertise for these challenging labels and letting LLMs handle the more straightforward specimens, it is possible to create efficient workflows to optimize the specimen transcription process. The second concern involves copyright laws, data provenance, data sovereignty, and data protection. LLMs are trained with vast, often varied data sets, each with their specific origin and usage permissions. Many natural history collections have been gathered with the assistance of Indigenous peoples or through colonial exploitation (Park et al., 2023). Inadvertently publicizing sensitive indigenous knowledge and data will contribute to the accelerated exploitation of endangered species facilitated by the internet in recent decades (Lavorgna and Rekha, 2022). As the curatorial community continues to address these issues, there is a pressing need for consensus on best practices for handling data from such collections. Consequently, developers of LLM methods should diligently investigate each model to ensure that its training data does not breach any copyright, sovereignty, or intellectual property protections. Developers of new LLM methods should actively seek permission to use data to train new models and disclose all underlying data sets to the end user. Using APIs from large corporations necessitates scrutiny of their terms of service; data submitted through APIs might affect an institution's copyright standing or be used as training data for proprietary models. Cloud-based APIs also carry a risk of data leaks, potentially disclosing information about protected taxa (Convention on International Trade in Endangered Species of Wild Fauna and Flora (CITES), Federally Endangered) or other specimens (such as those that are the source of government-regulated illicit drugs or chemicals), which may violate data sharing agreements. Once data are submitted to the API, we have little certainty and limited control over the destination of this sensitive information. Given these uncertainties, locally hosted, bespoke curatorial LLMs have become an increasingly appealing option. The final concern centers on the trade-offs associated with different LLM models. APIs provided by corporations like OpenAI's GPT family of models are convenient and accessible to institutions lacking robust computational resources, but also have per-transaction costs and are susceptible to potential changes or deprecations that could disrupt digitization workflows. Note that while writing this essay, we refactored our GPT implementation twice because of changes to the API. Locally hosted open-source LLMs like Llama, Falcon, or Wizard, while potentially more challenging to implement, offer a more assured continuity of service, although their permanence is still not guaranteed. Proprietary LLMs also tend to offer state-of-the-art methods before they are available to the open-source community. For example, OpenAI released its GPT-4V multimodal LLM, which is capable of directly extracting and parsing text from specimen images without the need for separate OCR algorithms. The incorporation of LLMs into natural history collection workflows is a dynamic process, requiring constant vigilance and adaptability. Each stage of LLM integration into a natural history collection workflow is poised for further refinement and innovation, providing fertile ground for future research and enhancements. We firmly believe that by embracing these methodologies and fostering the growth of novel approaches, we can revitalize curatorial procedures and make significant strides toward reducing the extensive backlog of specimens yet to be digitized. As the field of LLMs and machine learning transcription methods continue to evolve rapidly, collaboration and knowledge sharing within the collections community are imperative. In alignment with this philosophy and building upon previous efforts (see Dillen et al., 2019), the University of Michigan Herbarium is curating transcription benchmarking data sets and fine-tuned LLM models, making them freely available on Hugging Face (https://huggingface.co/SLTP). On our GitHub page (https://github.com/Gene-Weaver/SLTP-Benchmarking-Tools), we provide simple Python scripts that can be used to compare LLM predictions against the manually generated ground truth transcriptions and to create maximally dissimilar benchmarking data sets from larger collections. This has given rise to the Specimen Label Transcription Project (SLTP), a growing international collaboration of herbaria and museums with the shared goal of exchanging prompts, data sets, and models. SLTP aims to streamline the creation of an efficient, cost-effective, end-to-end transcription workflow for natural history collections. We eagerly invite all interested researchers and institutions to join this synergistic journey toward efficient, comprehensive label transcription. We find ourselves both invigorated and startled by the untapped potential of Large Language Models in natural history collections. These flexible, innovative tools stand to reshape traditional curatorial processes, greatly aiding in the daunting task of tackling the massive backlog of specimens awaiting digitization. W.N.W. wrote the initial manuscript and designed the project. B.R.R. and K.L. coordinated the use of herbarium data for VoucherVision and validated the accuracy of the LLM-transcribed labels. W.N.W. and S.A.S. contributed to designing the LLM implementation in VoucherVision. B.R.R., K.L., and W.N.W. contributed to the design and testing of the VoucherVision Editor software. All authors contributed to revising and editing the text. All authors approved the final version of the manuscript. B.R.R. and K.L. are supported by NSF DBI 2101868. S.A.S. and W.W. are supported by NSF DEB 2217116. The authors sincerely thank Andrew Hipp, Deborah Paul, Cam Webb, Rob Guralnick, James Mickley, Robert Laport, and Rukaya Johaadien for their insightful discussions, shared experiences, and thoughtful considerations of potential risks, all of which enhanced the depth of this note. The authors thank the University of Michigan Herbarium staff for their help coordinating and supporting this project. The authors would like to extend their gratitude to the two anonymous reviewers for their insightful comments, which greatly improved the quality and clarity of the final manuscript. We also thank the editorial staff for ensuring a smooth and expedient publication process.

Topics & Concepts