Automated Segmentation of 3D Cytoskeletal Filaments from Electron Micrographs with TARDIS

Abstract

3D segmentation of cytoskeletal filaments and organelles is crucial for studying these structures in cellular (cryo-) electron microscopy (EM) and tomography (ET). Manual annotation remains the gold standard for labeling these objects, due to the limited accuracy of available tools. Existing semi-automatic [1] or fully automatic approaches (e.g. [2]-[4]) can speed up the process, but often require extensive case-by-case tuning by users or significant manual correction of their outputs. In order to scale analysis to the growing number of micrographs and tomograms and enable precise quantification of biological structures, high-accuracy automatic segmentation algorithms are required. Segmentation is commonly separated into two categories: semantic segmentation, in which objects of interest are separated from other uninteresting signals, and instance segmentation, in which multiple objects of interest are distinguished from each other. Existing segmentation methods primarily focus on the semantic segmentation problem and do not distinguish between individual instances such as microtubule (MT) filament or contiguous membrane. While simple approaches to filament instance segmentation have been proposed [5], they are inadequate in complex environments and sensitive to semantic segmentation errors. Therefore, there is a need not only for more accurate semantic segmentation algorithms but also instance segmentation methods. Here, we present TARDIS (Transformer and Rapid Dimensionless Instance Segmentation), a fully automatic segmentation workflow designed to overcome these challenges. TARDIS provides modular solutions for semantic and instance segmentation, enabling complete annotation of micrographs and tomograms. For semantic segmentation, we propose an Unet variant that enables fast and accurate pixel-level classification. For instance segmentation, we propose a graph formulation to identify filament-like structures from point clouds obtained from semantic segmentation. This network allows flexible geometry specifications and TARDIS will be extended in the future to include pre-trained instance segmentation networks for organelles. Semantic and instance segmentation modules allow for new semantic segmentation models for organelles, or other structures of interest to be easily inserted into the pipeline without the need to retrain the instance segmentation network, and vice versa. The TARDIS workflow consists of three main steps: semantic segmentation which produces a semantic mask, post-processing of the semantic mask into a point cloud representation of the objects, and instance segmentation of the point cloud (Figure 1A). We train convolutional neural networks to classify pixels containing objects of interest. We train these networks to annotate MTs and membranes using a manually labeled dataset. By including a variety of EM formats (plastic- and cryo-sections) in the training set, we find that these networks can generalize well to new tomograms collected on other microscopes and with diverse settings. Although our semantic segmentation networks are highly accurate, as with any algorithm, model prediction is not perfect. Semantic masks can be ambiguous. Filament intersections or nearby crossings may be labeled as one positive region, while small gaps or noisy areas labeled as negative may cause a single object to appear as multiple regions. To overcome this issue, we convert the predicted semantic mask into a point cloud representation, which enables detailed modeling of all objects. To do this, we skeletonize the semantic mask followed by a euclidean distance transformation to extract the backbone of all objects. We frame the problem of instance segmentation from the point cloud representation as a graph learning problem. Given the point cloud, we seek to find a graph, defined by edges between the points, where a subgraph consists of points belonging to the same instance. We introduce the Dimensionless Instance Segmentation Transformer (DIST) a novel neural network to solve the above problem. DIST uses an SO(n) invariant transformer layer architecture to operate on point clouds of arbitrary dimension and outputs, for each pair of points, the probability that an edge exists between them (Figure 1B). We then decoded the most likely set of instances using a graph cut. This allowed us to accurately predict individual instances from the point cloud without prior knowledge of the filament number, order, or position. We assess our instance segmentation algorithm on biological structures in electron microscopy (EM) data and compare it with current state-of-the-art algorithms for instance segmentation of filament-like structures. We evaluate DIST on a membrane segmentation task (2D Cryo-EM micrographs; Figure 2A) and an MT segmentation task (3D plastic section tomograms; Figure 2B). To benchmark instance segmentation performances we measured mean class coverage (mCov; [6]). This metric measures the intersection-over-union between the ground truth label and its matching predictions. We find that TARDIS drastically outperforms existing methods in terms of speed and accuracy on these tasks. In summary, our TARDIS workflow is a fast and accurate solution for semantic and instance segmentation of filament structures in EM and cryo-EM micrographs of cells. Our method outperforms current solutions in terms of accuracy and speed and is designed to be fully automatic, eliminating the need for retraining or parameter optimization. With its ability to perform high-throughput micrograph segmentation and analysis, TARDIS has the potential to greatly advance the field of biological and biomedical research. In the future, we envision that TARDIS will grow to include a library of semantic segmentation and instance segmentation models for other cellular structures. (A) High-level illustration of the TARDIS workflow. Micrographs are first processed by the semantic segmentation network to produce a semantic mask, which is then converted into a point cloud representation that is processed by our SO(n) invariant DIST model to output a predicted graph. Object instances are then identified by performing a graph cut. (B) Detail DIST workflow. Edge representations are initialized from the distances between points in the input point cloud, yielding an SO(n) invariant representation. The DIST layers then operate on edge features using triangular multiplicative update and axial attention modules to make geometrically-informed updates to the edge representations, which are finally decoded to edge probabilities. Arrows show information flow. TARDIS performance and example tomogram and micrographs. (A) Instance segmentation of MTs from EM tomograms compared to Amira and Multi-curve fitting. DIST produces substantially more accurate instance segmentations and runs faster. Scale bar, 500 nm. (B) Instance segmentation of membrane from 2D Cryo-EM micrographs compared with multi-curve fitting. Scale bar, 10 nm.

Topics & Concepts