Deep learning in confocal fluorescence microscopy for synthetic image generation and processing pipeline

Abstract

Machine Learning has had a significant impact on microscopy, enabling faster and more accurate analysis of biological imaging data. In particular, Generative Adversarial Networks (GANs) and U-Net have emerged as powerful tools in this field. GANs (I. Goodfellow et al. 2020) are a type of deep learning model that consists of two neural networks, a generator and a discriminator. The generator creates synthetic images while the discriminator attempts to differentiate between the synthetic images and real images. Through this adversarial process, the generator improves its ability to generate realistic images, while the discriminator improves its ability to differentiate between real and synthetic images. In microscopy, GANs can be used to generate synthetic microscopy images or to fill in missing or degraded image data as we show in this work. U-Net (O. Ronneberger et al. 2017) is a type of convolutional neural network that is specifically designed for image segmentation tasks. The architecture of U-Net consists of an encoder and a decoder, with skip connections between corresponding layers in the encoder and decoder. In microscopy, U-Net has been used to segment objects of interest in microscopy images, such as cells or subcellular structures, enabling more accurate analysis of the images. Overall, the integration of machine learning techniques, particularly GANs and U-Net, into microscopy has enabled researchers to analyze imaging data more efficiently and effectively, leading to new insights and advances in the field of biology(K. Dunn 2019, F.Long 2020). In this work, a GAN architecture is trained to generate confocal fluorescence microscopy synthetic images from blood monocyte stacks from control individuals and patients, where nuclei and mitochondria were marked with different fluorescent probes. These images are then processed by an own implemented pipeline consisting of deconvolution, segmentation and feature extraction for mitochondria classification In the deconvolution stage, the methods implemented in the ImageJ plugin "DeconvolutionLab2" (D. Sage et al. 2017) are used, where their performance is analyzed based on the parameters used and their characteristics, such as whether they are regularized algorithms or if they are iterative or non-iterative. For segmentation, different approaches are evaluated, starting with traditional histogram-based thresholding methods (Otsu, Huang, Li, among others), non-supervised clustering methods such as K-Means (Lloyd 1957; MacQueen 1967), and Deep Learning methods such as the U-Net neural network. In feature extraction, morphological and connectivity features are obtained. The morphological characteristics obtained are the usual ones (volume, area, sphericity, among others). The connectivity characteristics are found from skeletonization, pruning and graph modeling (M. Zanin et al 2020). The parameters found are the number of nodes, the density of links and the efficiency, among others. Finally, for the mitochondrial classification, classical approaches such as Decision Tree, Logistic Regression and Support Vector Machine (SVM) were used. The work was done in the Python programming language. We are currently working on making this framework publicly available. The final result of the work is an end-to-end pipeline with different processing options in the deconvolution and segmentation stages usable for different microscopy data, a synthetic data generator that achieves performance when it comes to simulating the effect of fluorescence in binary masks, and an application of both products for the mitochondrial classification with an accuracy result greater than 70%. It is concluded that neural networks have a fundamental role in the processing of medical and biological images, and can be used for data augmentation, segmentation and classification.