Accelerating Electron Tomography Workflows Through Implicit Neural Representations

Electron tomography (ET) has emerged as a powerful method for resolving the three-dimensional (3D) structure and statistical analyses of materials. For example, tomographic quantification of nanoparticle size distribution, morphology, and composition has been used to explain electrocatalytic properties [1, 2, 3]. However, automated tomographic experiments remain constrained by acquiring a sufficient amount of tilt images needed to compensate for the missing wedge and by the feedback loop of evaluating alignment parameters with reconstruction quality. Reconstruction algorithms such as filtered backprojection (FBP), simultaneous iterative reconstruction technique (SIRT) [4], and real-space iterative reconstruction (RESIRE) [5] are fast but cannot compensate for missing information. More recently, novel deep learning techniques have been developed to address limited-angle sampling [6, 7, 8]. These methods rely on a semi-self-supervised approach that is effective when multiple datasets can be collected with sufficient sample heterogeneity. Most importantly, conventional and machine learning methods require time-consuming pre-processing to produce interpretable tomograms at high magnification and/or requiring multiple datasets to train sinogram prediction models, preventing on-the-microscope reconstruction. To overcome the limitations of the discussed algorithms and enable real-time reconstructions directly on the microscope, we develop a fully self-supervised tomographic reconstruction algorithm using implicit neural representations (INRs) [9] with well-defined metrics for early-stoppage and reconstruction quality evaluation. During acquisition, coarse alignments and background subtraction are applied to each tilt image which is continuously fed into the INR algorithm. During acquisition, the reconstruction algorithm refines the object and learns the full 3D pose, producing an accurate tomogram without manual preprocessing and enabling flexible acquisition sampling strategies, as shown in Figure 1a. Assuming acquisition of each projection direction takes ∼1 minute, our INR method can run 50 iterations on a single NVIDIA L40s GPU to incorporate the corresponding tilt image. Moreover, model regularization enables recovery of missing-wedge information without prior pretraining and allows reconstructions of dose-limited datasets. A flowchart of this workflow is shown in Figure 1b. Figure 2 shows results for a simulated tomography experiment using a phantom (Figure 2a). We first reconstruct a sparse tilt series consisting of five 200x200 images from -70° to 70° with 35° increments. In Figure 2b, we show the projected potential perpendicular to the missing wedge direction of the INR and SIRT. We then use a symmetric sampling scheme acquiring the positive and negative tilt angle in-between the original tilt angles (i.e, taking the -52.5° and 52.5° sequentially). In Figure 2c, after an additional 5 images the INR framework achieves a structural similarity index measure (SSIM) of 0.974 in Figure 2d, successfully reaching the ground truth structure of the sample. By contrast, SIRT is not yet fully converged even after including 62 projections in the reconstruction. We also compute a cross-validation error defined as the mean-squared error between the forward projection of the current volume at the next tilt angle and the acquired tilt image. Using this metric, we can determine when to stop the experiment to minimize total acquisition time without sacrificing reconstruction quality. This preprocessing and INR reconstruction framework accelerates tomographic reconstruction by producing high-quality tomograms and enables live on-the-microscope evaluation of dataset quality, reducing experimental acquisition time and enabling fast statistical analyses of materials. We will also demonstrate progress towards implementing this automated reconstruction pipeline for real experiments [10]. Overview of the tomography experiment, preprocessing pipeline, and reconstruction framework. a) A diagram of an electron tomography experiment using a symmetric sampling acquisition scheme, where the order of acquisition is shown. b) Flowchart of our pipeline. The tilt series is continuously acquired and is preprocessed where drift correction, background subtraction, and coarse alignments (cross-correlation and coarse rotational alignments) are applied. The processed tilt stack is then fed into the INR reconstruction loop, providing the relevant metrics (SSIM and cross-validation) for early stopping of the experiment. Comparison of simulated on-the-microscope reconstructions using INR and SIRT. a) A 3D render of the simulated phantom and the reference ground truth projected potential. INR and SIRT projected potentials with b) 5 and c) 10 tilt images. c) The SSIM and cross-validation metrics computed during acquisition for both INR and SIRT. The SIRT data is shown in the inset plot.