Evaluation of Human-Bias in Labeling of Ambiguous Features in Electron Microscopy Machine Learning Models

Abstract

The analysis of defects within a microstructure makes up a sizable portion of the materials characterization process for advanced materials in extreme conditions. The procedure of analyzing defects involves taking electron microscopy images of the material during or after exposure and individually labeling the features in each image, which introduces inherent inconsistency and bias between scientists. Identifying biases and increasing consistency could provide the scientific community with standard guidelines for defect analysis in the microstructure of materials, as well as the potential for better use of experimental time and collaboration across groups. This work focuses on identifying and understanding these biases for radiation-induced cavities, a dominant defect in irradiated metals used for advanced nuclear reactor applications. By acquiring cavity labeling data from a significant population using Transmission Electron Microscopy (TEM) generated cavity images, the impact of cavity contrast, minimum cavity size, cavity density, and cavity size distribution on labeling statistics can be obtained. This information, in turn, provides guidance on the role of human influence on machine learning models developed from the labeled data. Specifically, a better understanding of the human bias itself and how these biases propagate through supervised training of machine learning models such as the You Only Look Once (YOLO) family of models can be directly linked to materials performance analysis. A round robin labeling effort including 6 scientists and a ground truth created by 2 scientists was conducted across six selected cavity images to quantify variance in how scientists within the same research group count cavities in irradiated materials. The images used were from an under focused TEM bright field condition to maximize contrast and had a wide range of cavity sizes, cavity density, and image quality. Note, cavities in under focused conditions typically show central bright contrast with a dark concentric ring [1], although the individual contrast variation varies depending on cavity size, degree of defocus, and material composition [2]. Scientists consistently disagreed on the labeling of cavities when the ratio of cavity diameter to image size fell below 5%, indicating that there is a threshold below which cavities become highly ambiguous. Additionally, images with a bimodal cavity distribution displayed a greater propensity for labelers to find the cavities with a greater size and miscount the smaller cavities. The images that contained predominantly small cavities with a diameter below 5% of the image size were consistently under counted, indicating cavity size relative to total image size has a more significant impact on human bias compared to overall factors such as image resolution and signal-to-noise ratio. The round robin results provide initial guidance on imaging factors contributing to labeling bias, but further analysis is needed to translate this understanding to its impact on machine learning models. Assigning a statistically significant population of experienced cavity labelers to additional work requires extensive coordination whereas Amazon Web Services (AWS) enables the crowdsourcing of this task to a much larger remote workforce through Amazon Mechanical Turk (MTurk). A recent study showed the labeling ability of MTurk workers is on par with students when identifying features in white light images [3]. The previous work of Zhou et al. demonstrates the viability of using MTurk, but had limited datasets with small features (<5% of the image width or height) and minimal impact of distracting background features. Here, an initial study using the round robin images was cast to the MTurk system to evaluate the ability to crowdsource labeling of electron microscopy images with known ambiguity as shown in Fig. 1. The MTurk data gathered through AWS resembles the round robin data in terms of F1 score across the relative size of the cavity with ambiguity below 5% of the relative cavity size. The images also perform similarly regarding how the distribution in cavity size impacts the statistics with preference for labeling larger cavities. This initial data indicates that MTurk data, when done at higher scale to account for lack of domain expertise, can replicate the inherent bias of domain experts. This work will present our round robin results and expand upon it using the MTurk workflow to rapidly iterate through cavity electron microscopy images generated synthetically to systematically evaluate factors such as background variances, contrast variances, and cavity-to-image size variances. The synthetic data then ensures the ability to train efficient ML models while also allowing for precise placement and labeling of cavities with increasing image complexity and ambiguity. The synthetic images have been developed using an in-house, lightweight simulation script where simplified 1D phase contrast simulations based on the theory of Rűhle and Wilkens [4] are stochastically integrated with modern TEM physics models, generating cavity images that are blended into real TEM images of unirradiated microstructures. Our synthetic datasets have a 0.06 reduction in F1 compared to experimentally generated datasets. These results indicate that the primary features of cavities is produced in the synthetic datasets enabling either labeled data generation method to be used in future human-bias evaluation studies. This work will present on the most recent findings based on this workflow and its impact on ML model performance within object detection tasks when trained with engineered bias based on the quantitative results from the MTurk-based experiments. A representative round robin image (a) with overlayed heat maps of the Round Robin results (b) and the MTurk results (c). The ground truth is labeled in orange, the AWS or round robin data in blue, and overlapping data in pink.

Topics & Concepts