University of Southern California Dissertations and Theses

Multiplexing live 5d imaging with multispectral fluorescence: Advanced unmixing through simulation and machine learning

(USC Thesis Other)

Multiplexing live 5d imaging with multispectral fluorescence: Advanced unmixing through simulation and machine learning

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content MULTIPLEXING LIVE 5D IMAGING WITH MULTISPECTRAL FLUORESCENCE: ADVANCED UNMIXING THROUGH SIMULATION AND MACHINE LEARNING By Hsiao-Ju (Rose) Chiang A Dissertation Presented to the FACULTY OF THE USC GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of Requirements for the Degree DOCTOR OF PHILOSOPHY (BIOMEDICAL ENGINEERING) May 2022 Copyright 2022 Hsiao Ju Chiang i Acknowledgements Over the course of my doctoral journey, I was nurtured with a tremendous amount of support and guidance from several mentors and colleagues. Words could not express how grateful I am to join the best lab I could ever imagine with the most supportive PI, Dr. Scott Fraser, and a father-like mentor, Dr. Francesco Cutrale. I thank Dr. Scott Fraser for his generous, humble, and supportive mindset. His heart is big enough for all the random things I accomplished: applying for NSF fellowship, getting other degrees in EE and technology commercialization, pursuing entrepreneurship, going on international conferences and travels, taking classes outside academics, to working remotely when I had to. Scott and his wisdom dad jokes encouraged me a lot over the years. I also like to thank Scott for providing this incubator with brilliant physics, biologists, and scientists from various fields, making me comfortable with always being the stupidest person in the room. Dr. Le Trinh, Dr. Masahiro Kitano, and Dr. Thai Truong provide numerous valuable suggestions from a different angles for my research. These fantastic people around me are the driving force to keep refining myself. It is crazy to look back at how much I have grown in every perspective over the past six years. I started with not much experience. Dr. Francesco Cutrale took a great risk on me and took me as his first Ph.D. student. Dr. Francesco Cutrale trained and challenged me in microscopy, imaging, algorithm development, and even how to present my work to the artistic level. I thank Dr. Francesco Cutrale for spending countless days and evenings refining my works over five years. Dr. Francesco Cutrale also allowed me to think outside academia and led projects to real-world products. I was genuinely proud to have such a wonderful mentor that pushed me to thrive and keep up with him. I also want to thank all the peers in Fraser lab, Shanshan, Wen, Valerie, Peiyu, Pu, and Marilena, who support each other through countless nights and weekends in labs. I would especially like to thank Daniel Koo, the main driven force in my many projects. It would have taken me decades to finish this work without his generously provided help. Daniel showed me how hard-working a talented, wise person can still be. Daniel put up with my millions of questions in the off hours and used his magical touch to fix my magmatic field of breaking things up and providing encouragement when I was self-doubt. Lastly, I want to thank all my family and friends. I thank my parents for the shelter and laughter, and my sister for being my most vigorous back. I thank Dr. Chang for inspiring me to pursue my Ph.D. I thank my USC cohorts explored this new experience with me, especially Angela, Kai, and Bing. I thank my Ph.D. gang, Dr. Peng, Dr. Sun, and Dr. Zheng. I’d also like to thank Yang and his family for their huge support. Big thanks to E. K. Ma for her companion while I’m writing the thesis. Lastly, I would like to thank myself for the hard-working and courage to get out of my comfort zone. ii Table of Contents ACKNOWLEDGEMENTS ................................................................................................................................... I LIST OF TABLES .............................................................................................................................................. IV LIST OF FIGURES ............................................................................................................................................. V ABSTRACT ...................................................................................................................................................... VIII CHAPTER 1 INTRODUCTION ......................................................................................................................... 1 1.1 FLUORESCENCE MICROSCOPY ENABLES 5D IMAGING ...................................................................................... 1 1.1.1 Biophysics of fluorescence and labeling ............................................................................................... 1 1.1.2 Confocal laser scanning microscope ..................................................................................................... 4 1.1.3 Two photon microscopy enables deep tissue imaging ........................................................................... 7 1.1.4 Hyperspectral fluorescent imaging ....................................................................................................... 8 1.1.5 5D imaging ............................................................................................................................................ 9 1.2 THE NEED AND CHALLENGES OF LIVE MULTIPLEXING ................................................................................... 10 1.2.1 The need of multiplexing ...................................................................................................................... 10 1.2.2 The challenges in multiplexing ............................................................................................................ 11 1.2.3 The challenges in live imaging ............................................................................................................ 13 1.2.4 The challenges of live multiplexing ..................................................................................................... 15 1.2.5 Current solution: Linear unmixing for hyperspectral data ................................................................. 16 1.2.6 Unmixing challenges in biological data .............................................................................................. 18 1.3 IMAGING METABOLISM FROM INTRINSIC FLUORESCENCE SIGNAL ................................................................. 21 1.3.1 What is autofluorescence? ................................................................................................................... 21 1.3.2 Imaging autofluorescence using FLIM and phasor approach ............................................................ 23 1.3.3 Challenges for autofl imaging ............................................................................................................. 25 1.3.4 Moving toward real world challenges ................................................................................................. 26 1.4 MACHINE LEARNING IN IN LIFE SCIENCE ....................................................................................................... 26 1.5 ALGORITHMICALLY OVERCOME CURRENT LIMITATIONS ............................................................................... 28 CHAPTER 2 HYU: HYBRID UNMIXING FOR LONGITUDINAL IN VIVO IMAGING OF LOW SIGNAL TO NOISE FLUORESCENCE ......................................................................................................... 33 2.1 HYBRID UNMIXING (HYU) PROVIDES A PLATFORM TO COMBINE PHASOR ANALYSIS WITH OTHER UNMIXING ALGORITHMS ....................................................................................................................................................... 33 2.1.1 HyU inherit phasor advantage from phasor approach ....................................................................... 33 2.1.2 HyU inherit linear unmixing advantage .............................................................................................. 35 2.1.3 Identification of unknown components using HyU .............................................................................. 35 2.1.4 HyU provide user friendly interface to visualize independent components ........................................ 40 2.2 THE HYBRID METHOD OUTPERFORMS THE TRADITIONAL METHOD ................................................................ 42 2.2.1 HyU shows higher accuracy in simulation data .................................................................................. 42 2.2.2 HyU overcomes the bleedthrough problem in LU in experimental data ............................................. 43 2.3 HYU HAS BETTER RESOLUTION IN LOW SNR DATA ....................................................................................... 47 2.3.1 Better spatial resolution ...................................................................................................................... 47 2.3.2 Longer timelapse imaging ................................................................................................................... 52 2.3.3 Extract information from intrinsic signal ............................................................................................ 54 2.4 DISCUSSION ................................................................................................................................................... 59 2.5 SUMMARY ..................................................................................................................................................... 62 2.6 SUPPLEMENTARY .......................................................................................................................................... 63 CHAPTER 3 A FRAMEWORK FOR MODELING SPECTRAL FLUORESCENCE .............................. 72 3.1 GENERATING 1D FLUORESCENT SIGNALS ...................................................................................................... 74 iii 3.1.1 How we simulate a fluorescent signal ................................................................................................. 74 3.1.2 How we simulate fluorescence signals collected in microscope ......................................................... 75 3.1.3 Adding noise step by step ..................................................................................................................... 76 3.1.4 Finding conversion rate ....................................................................................................................... 79 3.2 GENERATING 2D FLUORESCENT IMAGE ......................................................................................................... 81 3.2.1 The necessity of realistic-bio-pattern simulation ................................................................................ 81 3.2.2 .............................................................................................................................................................. 82 3.2.3 How image masks are generated ......................................................................................................... 84 3.2.4 Realistic bio-pattern image ................................................................................................................. 84 3.3 SIMULATION PROVIDES GROUND TRUTH QUANTIFICATION OF ALGORITHMS ................................................. 86 3.3.1 Comparison of microscopy signals and simulated signals .................................................................. 86 3.3.2 Comparison of images from experiment and simulation ..................................................................... 88 3.3.3 Quantification on algorithm performance ........................................................................................... 89 3.3.4 Summary .............................................................................................................................................. 92 CHAPTER 4 SPECTRAL DENOISE 1D-UNET FOR PREDICTION OF ULTRA-LOW SNR SIGNALS ............................................................................................................................................................................... 93 4.1 CURRENT NOISE REDUCTION METHODS AND CHALLENGES ........................................................................... 94 4.2 SD-UNET IS DESIGNED FOR DENOISING FLUORESCENT HYPERSPECTRAL ....................................................... 95 4.2.1 Neuron network background: CNN vs Unet ........................................................................................ 96 4.2.2 Spectral denoise 1D-Unet (Sd-Unet) structure .................................................................................... 98 4.2.3 Customized loss function ................................................................................................................... 100 4.2.4 Training and testing data .................................................................................................................. 101 4.2.5 Performance evaluation .................................................................................................................... 104 4.3 SD-UNET DENOISES NOISY SIGNALS ............................................................................................................ 105 4.3.1 Simulation data .................................................................................................................................. 105 4.3.2 Experimental data .............................................................................................................................. 108 4.4 SD-UNET RECOVERS DETAILS IN MICROSCOPY IMAGE ................................................................................. 111 4.5 SUMMARY ................................................................................................................................................... 112 CHAPTER 5 THE FUTURE OF DECODING LIVE USING HYPERSPECTRAL FLUORESCENCE IMAGING .......................................................................................................................................................... 113 5.1 POTENTIAL STUDY OF LABEL FREE METABOLISM ........................................................................................ 114 5.1.1 Experimental setup ............................................................................................................................ 114 5.1.2 Pre-identified autofluorescence cursors from solutions .................................................................... 115 5.1.3 Difficulty of imaging label free metabolism ...................................................................................... 116 5.1.4 Unmixing and identification of autofluorescence signatures using HyU .......................................... 120 5.1.5 Future direction ................................................................................................................................. 122 5.2 LIVER DEVELOPMENT AND REGENERATION ................................................................................................. 124 5.2.1 Background ........................................................................................................................................ 125 5.2.2 Experimental design .......................................................................................................................... 126 5.2.3 Imaging liver regeneration through laser ablation ........................................................................... 129 5.2.4 Experimental and analytical challenges ............................................................................................ 132 5.2.5 Preliminary results and discussion .................................................................................................... 134 5.2.6 Summary and future direction ........................................................................................................... 137 5.3 CONCLUSION ............................................................................................................................................... 138 REFERENCES .................................................................................................................................................. 140 iv List of Tables Table 1 Experimental setting exploration in the imaging side. ............................................................................. 32 Table 2 Five different input training and predicting set. .................................................................................... 102 Table 3 Network performance for different training and predicting pairs. ........................................................ 102 Table 4 Performance of Sd-Unet on simulation data .......................................................................................... 106 v List of Figures Figure 1.1 A simplified Jablonski diagram demonstrating fluorescence fundamentals. ........................................ 2 Figure 1.2 Distinct emission and excitation spectra in a fluorochrome chart. ...................................................... 3 Figure 1.3 Common fluorophores and their peak excitation and emission wavelengths. ...................................... 4 Figure 1.4 Simplified widefield fluorescence microscopy structure. ...................................................................... 5 Figure 1.5 Single excitation vs 2-photon excitation in microscopy ........................................................................ 7 Figure 1.6 Confocal microscopy improves the spatial resolution in thick samples compared to widefield fluorescence microscopy. ........................................................................................................................................ 7 Figure 1.7 Six genetic loci are observed at once from Visser et al ....................................................................... 12 Figure 1.8 Bleedthrough is inevitable when multiple fluorophores are in use. .................................................... 13 Figure 1.9 Signals collected from a microscope are distorted from expected emission spectra. ......................... 20 Figure 1.10 Common intrinsic signals phasor map from FLIM data. [32] .......................................................... 24 Figure 1.11 Standard machine learning framework. ............................................................................................ 28 Figure 1.12 Project pipeline schematic. ............................................................................................................... 29 Figure 1.13 The emission spectra for the fluorophores use in this thesis work. ................................................... 31 Figure 2.1 Schematic illustrating how Hybrid Unmixing (HyU) enhances analysis of multiplexed hyperspectral fluorescent signals in vivo. .................................................................................................................................... 34 Figure 2.2 Pre-identified cursors for common fluorophores on phasor map. ...................................................... 36 Figure 2.3 HyU enables identification and unmixing of low photon intrinsic signals in conjunction with extrinsic signals. ................................................................................................................................................................... 38 Figure 2.4 Residual maps facilitate identification of independent spectral components. .................................... 39 Figure 2.5 HyU interface showing how to obtain spectra info from raw data and save it for future use. ........... 42 Figure 2.6 Hybrid Unmixing outperforms standard Linear Unmixing (LU) in both synthetic and live spectral fluorescence imaging. ............................................................................................................................................ 45 Figure 2.7 Volumetric unmixing results of a quadra-transgenic zebrafish with HyU and LU highlights improvements in contrast and spatial features(Figure 2.6). ................................................................................. 46 Figure 2.8 Hybrid Unmixing enhances unmixing for low-signal in vivo multiplexing and achieves deeper volumetric imaging. ............................................................................................................................................... 49 Figure 2.9 Residual analysis of experimental data supports performance improvement of HyU. ....................... 52 Figure 2.10 HyU reveals the dynamics of developing vasculature by enabling multiplexed volumetric time- lapse. ..................................................................................................................................................................... 53 Figure 2.11 HyU pushes the upper limits of live multiplexed volumetric timelapse imaging of intrinsic and extrinsic signals. .................................................................................................................................................... 56 Figure 2.12 Application of denoising filters reveals improved results with lower residuals. .............................. 57 Figure 2.13 Comparison of residual images for LU and HyU highlights improved HyU performance. ............. 58 Figure 2.14 Schematic overview of residual calculation. ..................................................................................... 70 Figure 3.1 Schematic overview of simulation framework. .................................................................................... 73 Figure 3.2 Zeiss 488/561 bandpass filter shows how signal got affected by optics. ............................................ 77 Figure 3.3 Four emission spectra before and after adding 488/561 filter. .......................................................... 77 Figure 3.4 Point spread function (PSF) is a one of the critical noise greatly affect our signal in the experimental data. ....................................................................................................................................................................... 78 vi Figure 3.5 Compare experimental data and simulation data in different levels of poisson noise. ....................... 78 Figure 3.6 Intensity count distribution from a single detector. ............................................................................ 79 Figure 3.7 Two ways to calculate S_factor. .......................................................................................................... 79 Figure 3.8 block simulation of a non-repetitive ratio of combinations in gradient. ............................................. 81 Figure 3.9 Simple geometry simulation. ............................................................................................................... 82 Figure 3.10 realistic simulation from experimental data. .................................................................................... 83 Figure 3.11 Schematic of generating photon masks for getting a realistic ratio with ground truth. ................... 85 Figure 3.12 Simulation data matches experimental data. .................................................................................... 87 Figure 3.13 Four fluorophore simulation with ground truth in different intensity and combination ................... 88 Figure 3.14 2D simulation from biological realistic photon mask. ...................................................................... 88 Figure 3.15 Comparison of unmixing results for synthetic data at different SNR demonstrate improved HyU performance. ......................................................................................................................................................... 89 Figure 3.16 Quantification of HyU vs LU unmixing results for synthetic data highlight increased HyU performance. ......................................................................................................................................................... 90 Figure 3.17 Residual analysis for synthetic data identifies locations with reduced algorithm performance. ...... 91 Figure 4.1 The encode-decode structure of CNN. (Image source: Applied Deep Learning from Arden Dertat) 96 Figure 4.2 U-net structure (image source: Ronneberger et al. [74]) ................................................................... 97 Figure 4.3 Schematic overview of the SdU-net structure. A modification of Unet with 7 layers and 1D input. .. 98 Figure 4.4 Comparing Sd-Unet denoising with traditional denoising methods. .................................................. 99 Figure 4.5 Customized loss function improves the prediction of spectral shape. ............................................... 101 Figure 4.6 Network performance from different number of number of spectra and training range. ................. 104 Figure 4.7 The final Sd-Unet training convergence example with the best validation loss of 0.00049. ............ 105 Figure 4.8 Sd-Unet example on a super low photon (<40 counts) testing data. ................................................ 106 Figure 4.9 Sd-Unet recover an extreme example of highly noisy data. .............................................................. 108 Figure 4.10 Sd-Unet recovers signal in different SNR from experimental data. ................................................ 109 Figure 4.11 Sd-Unet denoising on different level of SNR on a four-fluorophores zebrafish embryo. ................ 110 Figure 4.12 Phasor analysis before and after applying Sd-Unet indicates the spectral shape distribution is converging toward the high SNR data. ............................................................................................................... 111 Figure 4.13 HyU unmixing results from Ch2 before and after applying Sd-Unet shows a spatial improvement after applying Sd-Unet. ....................................................................................................................................... 112 Figure 5.1 Two-photon action cross sections and emission spectra from biological molecules. ....................... 115 Figure 5.2 Autofluorescence signatures in experimental data compare to signatures obtained from solutions.117 Figure 5.3 Phasor analysis on different regions of wildtype zebrafish in different temperature condition. ...... 117 Figure 5.4 Metabolic signal distribution in 19 hpf zebrafish embryo from Casper fish. .................................... 118 Figure 5.5 Distribution of 6 metabolic signals from a 22 hpf quadra-transgene Gt(cltca-citrine);Tg(ubiq:lyn- tdTomato;ubiq:Lifeact-mRuby;fli1:mKO2 zebrafish embryo. ............................................................................ 118 Figure 5.6 FAD expressed in zoomin hindbrain region in 22 hpf zebrafish. ...................................................... 119 Figure 5.7 HyU analysis of 36 hpf Casper zebrafish of unmixing intrinsic signals. .......................................... 120 Figure 5.8 Identification of previously unknown components. ........................................................................... 121 Figure 5.9 Preliminary data and experimental design on NADH free bound metabolic changes. .................... 123 Figure 5.10 Retinoids validation using Raldah2 mutant fish from phasor. ........................................................ 123 vii Figure 5.11 Experimental design for liver development and regeneration. ....................................................... 127 Figure 5.12 Exogenous labels reveal liver outgrowth and vascularization. ...................................................... 128 Figure 5.13 2-Photon laser ablation (2-PLA) of the liver. ................................................................................. 131 Figure 5.14 Liver regeneration after laser ablation. .......................................................................................... 132 Figure 5.15 Liver microscopy images show imbalanced signal expression from inner and outer layer. .......... 134 Figure 5.16 Increasing metabolic activity in developing liver. .......................................................................... 136 Figure 5.17 Liver recovery over three days from laser ablation on 6 dpf liver. ................................................. 136 viii Abstract Two of the most challenging but compelling puzzles that human beings have been trying to solve are decoding the secrets of life and alleviating the suffering of people affected by diseases. Right now, traditional methods utilize 2D imaging, which cannot provide sufficient information to fully capture the biological mechanism. An ideal solution is to track the same specimen in the whole volume and multiple labels over time to decode the dynamic biological system. Therefore, we propose to use 5D imaging, which contains 3D (x, y, z) plus time and labels, as our solution. However, the full capability of 5D imaging is limited by current technologies. This thesis explores the state-of-the-art high- content imaging methods, their challenges, and novel algorithmic solutions. This dissertation proposal explains the development of three novel toolkits to advance multiplexing spectral fluorescent live imaging. The first tool is an unmixing algorithm for untangling overlapping signals. The second tool is a spectral fluorescence modeling framework used to enable the simulation of any fluorescence data without conducting complicated experiments. The third tool is a network that utilizes machine learning to denoise the microscopy noise in acquired spectral signals. These innovative tools have the potential to significantly advance our ability to study complex biological phenomena and open previously inaccessible windows of observation into biological systems. These toolkits are highly versatile. They can be applied to intrinsic signals, fluorescent labels, and the combination of the two. This can open the scientific community to a wide range of studies from clinical to model-based biological systems. 1 Chapter 1 Introduction To extract useful information from life science, it requires the capability to capture interested targets within specimens and to interpret the image by processing data. With its high contrast, high specificity, and multiple parameters, fluorescence microscopy has become the reference technique for imaging. In this chapter, I am going to review the basics of imaging and imaging analysis, the challenges, and the more advanced tools. 1.1 Fluorescence microscopy enables 5D imaging 1.1.1 Biophysics of fluorescence and labeling Scientists label different parts of specimens with different fluorophores. Fluorescent molecules or proteins capable of emitting fluorescence are known as fluorophores. When a fluorophore is hit with energy by a laser, it absorbs a high-energy photon. This high energy excites an electron within the molecule to an unstable high energy state, as shown in a simplified Jablonski diagram in Figure 1.1. A photon of light is emitted when the excited electron undergoes relaxation– this light is known as fluorescence. [1]–[5] 2 Figure 1.1 A simplified Jablonski diagram demonstrating fluorescence fundamentals. Jablonski diagram displays the energy states of a molecule. Typical molecules are mainly confined to the ground electronic state level S0 until absorbing energy. The blue arrow to the left represents the energy of an ultraviolet photon that can cause the molecule to transition from the ground state level S0 to either a singlet first electronic excited state S1 or to a second electronic excited state S2. The green arrow on the right shows the lowest energy photon that can be emitted by this molecule as it returns to S0. The emitted photon is fluorescent emission. The full process duration from a molecule absorbing a photon to a higher energy state before returning to the S0 by emitting fluorescence is fluorescence lifetime (τnsec). Image source: Lleres et all [6] Each fluorophore has its distinct excitation and emission spectrum, as shown in Figure 1.2. Like how fingerprints can be used to recognize individuals, different fluorophores can be distinguished based on the collected emission spectra. Researchers excite fluorophores at certain energy according to their excitation profile. The development of microscopy has driven researchers to design more fluorescent agents to look at multiple targets. Figure 1.3 shows some common fluorophores detected in different colors. Numerous researches have been done to record common fluorophores' excitation and emission signatures. [2] Fluorescent proteins are used to label structures in live cells and tissues and report on ph and temperature factors. They enable researchers to use molecular cloning methods to attach fluorophores to present enzymes and then use optical microscopy to examine cellular activity in live systems. GFP and its color-shifted variants have been employed in thousands of Bio-imaging experiments thanks to technological advances in wide-field fluorescence and confocal microscopies, such as ultrafast digital cameras and multi- tracking laser control systems. They have allowed researchers to see previously invisible processes such as creating nerve cells in the brain, the spread of cancer cells, the progression of Alzheimer's disease in nerve cells, and the formation of insulin-generating cells in an embryonic pancreas. [7]–[9] 3 Fluorescent molecules can be detected in either chemically fixed tissue or living imaging. In a fixed sample, the specimen is fixed in place by applying chemicals such as formaldehyde to maintain the structure. It can be labeled using non-transgenic labeling methods such as immunofluorescence, fluorescence in situ hybridization (FISH), a technique widely used to detect and localize specific DNA or RNA sequences in fixed samples. Live imaging, on the other hand, uses a fluorescent reporter attached to a specific protein through genetic-based labeling techniques such as injection of transgene endogenous tags (a protein being modified to attach fluorescent molecules to tag it). The binding protein can then be observed in real-time through fluorescence microscopy. Different research groups have been engineering different fluorophores to label more structures. Commonly fluorescent labeling methods include chemical labeling, enzymatic labeling, peptide/protein tag, and transgenesis labeling. [10], [11] Figure 1.2 Distinct emission and excitation spectra in a fluorochrome chart. In this chart, fluorescent dyes in different colors are presented with their excitation and emission spectra. Fluorescent dyes can be categorized in blue, green, yellow, orange, red, far red channels. Each dye has a different relative brightness. Note: Fluorochrome is any fluorescent dyes that are used to stain biological material prior to microscopic examination, whereas fluorophore is a molecule or functional group that is capable of fluorescence. (Image source: abcam) 4 Figure 1.3 Common fluorophores and their peak excitation and emission wavelengths. This image is the time exposure of fluorescence excited at different wavelengths and viewed through corresponding filters. (Image source: Roger Y Tsien Nobel Lecture)[8] 1.1.2 Confocal laser scanning microscope Microscopes are the indispensable tools to study the otherwise invisible biological world. A basic microscope comprises a light source, objective lens, and detectors. Microscopy is a rapid expansion field. Researchers have been working for decades to pursue more satisfactory resolutions. In the 18th century, cells were first seen through Leeuwenhoek’s microscope with 200x magnification. Since then, scientists have been trying to get a closer peep inside the cells– distinguishing different organelle, observing the interaction between objects, and tracking the process of mechanisms. This desire drove tons of breakthroughs in the microscopy field in the 20th century, especially after discovering fluorophores. Different cell types can be distinguished by collecting and identifying fluorophore-specific wavelengths. Filters are used in the microscope to discriminate between the various emission wavelengths emitted by the different 5 fluorescence molecules. In traditional widefield fluorescence microscopy, a dichroic mirror is placed between the laser and the specimen. This dichroic mirror reflects the laser light to the specimen and only allows the light of a particular wavelength emitted by the specimen to pass through to the detector, as shown in Figure 1.4. However, in the widefield fluorescence microscope, excitation light not selectively shines on the entire sample. Due to the detection of signals from the entire field of view, the target's contrast may be highly obscured due to the combination of signals from multiple sources, resulting in noisy images. Therefore, the widefield fluorescence microscope is good at observing thin samples but not thick samples.[2], [12]–[16] Figure 1.4 Simplified widefield fluorescence microscopy structure. A Dichroic mirror separates the excitation and emission wavelengths in a fluorescence microscopy. (image source: scientifica.uk) The invention of the confocal laser scanning microscope in 1987 greatly increased the spatial resolution. Confocal laser scanning microscope excites a specimen within a narrow plane of focus by passing light from a laser through the objective of a standard 6 light microscope. (Figure 1.5 b) The light source of a laser scanning microscope is the laser. The beam of the laser is directed by a pair of mirrors. The light is then focused on the sample through objective. By precisely collecting the emitted photons at each point of sample from changing the laser excitation position, an image is reconstructed pixel by pixel. While structurally similar to widefield fluorescence microscopy, a conjugated pinhole is used to block out-of-focus light before light reaches detectors in confocal microscopy. The pinhole, or confocal aperture, rejects any light emitted by out-of-focus planes. The design of the pinhole substantially improves spatial resolution by allowing only photons in the focal plane to pass through, as shown in Figure 1.6. In this way, scientists can look deeper into the tissue, realizing true three-dimensional analysis. 7 Figure 1.5 Single excitation vs 2-photon excitation in microscopy (a) Simplified Jablonski diagram shows the photon states in single photon excitation and two-photon excitation. (b) Microscopic structure for confocal and two photon microscopy. (Image source: Mostany et al.) [17] Figure 1.6 Confocal microscopy improves the spatial resolution in thick samples compared to widefield fluorescence microscopy. (a-c) images taken from widefield fluorescence microscope and (d-f) images taken from confocal fluorescence microscope. I(mage source: Olympus) 1.1.3 Two photon microscopy enables deep tissue imaging There are two types of excitation methods for fluorescence microscopy: single- photon excitation and multiphoton excitation. The first approach, as the confocal microscopy introduced in the previous section, makes use of single photons with sufficient energy to excite fluorescent molecules in the tissue as in Figure 1.5 (a). On the other hand, the second approach requires the employment of at least two photons. The probability of 8 two or more photons hitting together at the desired structural components and generating the desired energy is relatively low; therefore, multiphoton imaging creates high resolution images without a pinhole. In other words, only at the interested location, the two excitation photons are going to come together in the high enough probability such that an emission photon would occur. In this way, multi-photon microscopy excites precisely the interested small region of the sample. Comparing two photon excitation microscopy to confocal microscopy, the latter has greater depth penetration due to the longer excitation wavelength required and the ability to collect scattered emission photons as a meaningful signal. It also has the additional benefit of minimizing photodamage due to the use of lower energy photons and the confinement of fluorescence to the geometrical focus of the laser point. Two-photon excitation is consequently well suited for the imaging of dynamic processes in cell biology at high resolution in deep tissue across time. 1.1.4 Hyperspectral fluorescent imaging Multiple spectral imaging opens another door for differentiating different materials, chemical states, or multiple labels. Spectral imaging adds an extra wavelength dimension to the 2D image. Hyperspectral imaging, specifically referring to a high number of spectral channels, was first developed in the late 1970s by NASA’s Jet Propulsion Laboratory. It was originally used to pinpoint mineral deposits [using the reflection light from the object that was coming from the sunlight]. Instead of collecting images in traditional RGB channels (which compose the colorful images in human eyes), hyperspectral imaging acquires an image with a more detailed spectral dimension through tiny channels. Each channel collects a narrow range of light wavelengths. The adding spectral information 9 preserves the distinct optical properties of interest targets. Using the spectra from interested targets or individual components as a reference to detect or identify interested targets facilitates better classification or mapping of heterogeneous distribution, or unmixing overlapping emission spectra, which is extremely useful in multiple fields. Common applications include remote sensing, agriculture, food safety, quality control, sustainability, environmental monitoring, cosmonautic applications, medical diagnosis and life science. It was first used in biological research with fluorescence imaging in 2001 from Lansford et al. [18] Hyperspectral fluorescence imaging (HFI) is a novel way of imaging. A third dimension, lambda, is added to the traditional two spatial dimensions (x and y) microscopy image by recording the emission profile from fluorescence for each pixel. The collection can be done through multiple detectors at once or through a single detector with multiple scans where each scan photons in a different wavelength range are taken. The collected data is a 3D hyperspectral cube. For a volumic image, a 2D image in each z slice would become a 3D cube. This information-rich 3D cube for each slice greatly facilitates the capability to simultaneously follow multiple labels in molecules, cells or tissues across spatial and temporal resolution. Combined with the timelapse and volumic imaging, HSI allows scientists to gain a more comprehensive understanding of biological systems through accessing 5D information. 1.1.5 5D imaging Researchers have been designing a variety of microscopes with the goal of expanding the resolution in each dimension. Efforts are made toward a faster acquisition, 10 deeper volume, higher spatial resolution. Thanks to the development of fluorescent microscopes, standard 4D (3D volume plus time) information has been expanded to acquire five-dimensional (5D) information. Scientists can add cell type dimension information to the original 4D data because of the capability to recognize and discriminate between distinct structures inside biological systems. This 5D information can be obtained through 5D imaging. Here we define 5D in 5D imaging as spatial dimension (3D), time, and emission wavelength (also known as lambda channel.) 1.2 The need and challenges of live multiplexing 1.2.1 The need of multiplexing The capability to follow the cell interactions within the same specimen over time is essential for the biological field. There are various unsolved questions regarding the interaction among cell, tissue relationships. An emerging tool to these intertwined questions is multiplexing. Multiplexing is the capability to collect and process multiple information at the same time. In 15 century, in order for Leonardo da Vinci to understand the human body, he had to dissect each organ from a different body and put them together. Likewise, decades ago, researchers needed to label one target per fixed sample at a time and imaged them separately. Each image was saved separately, and it was challenging to understand the complex biology from the relative structures acquired from different samples. Recent years, researchers are keen to study biological phenomena within complex systems that require multiple labels to be followed in the same specimen. With the advancement of 11 fluorescence labeling techniques, multiple labels can be used to tag on interested structures. Now multiple structures can be captured at the same sample. [8], [19], [20] 1.2.2 The challenges in multiplexing In the standard multi-channel(label) fluorescence microscopy, to observe many labels, the maximum emission wavelengths of individual fluorophores are collected in separate channels using fixed samples. Signal bleedthrough, a phenomenon happening when signals crosstalk between channels, often prevents the reading of more than 3 colors. One of the biggest factors that restricts the number of parallel labels is the signal bleedthrough raised from overlapping emission spectra from fluorescent dyes. The fluorophores mentioned in 1.1.1 have very overlapping emission spectra, which would not be a problem when a single fluorophore is added to one specimen at a time. However, as more fluorophores are imaged in the same specimen at the same time, it becomes hard to distinguish the light that is being emitted from one fluorophore or from another. In this standard practice, before using fluorescent dye, researchers first need to check the available instrument because the type, number of lasers, filters and detectors dictate the fluorophores that can be used. Researchers may choose three well-separated fluorophores based on the maximum emission wavelength, such as GFP, YFP, and RFP, to reduce bleedthrough effects. Then use either sequential or parallel excitation plus corresponding filters to collect desired signals, which is the optical filtering method. Even so, the signal crosstalk exists.[4], [21] One example, Visser et. al explored the multiplexing capability of observing six genetic loci at once through FISH (see 1.1.1 for labeling), shown in Figure 1.7. To the left of Figure 1.7, fluorescence are expressed in blue under the traditional single color setting. 12 The image on the right shows 6 colors from different genetic loci. However, fluorescence signals in the right image are observed where the left image is shown as background(gray). This is because of the signal bleedthrough and high background noise. The chosen fluorophores are highly overlapping as in Figure 1.8. The group used narrow bandpass excitation and emission filters to reduce bleedthrough. Note here, the bleedthrough is inevitable even with the well-chosen narrow bandpass filters. In brief, two major drawbacks to using current standard methods are shown here: Strong bleedthrough occur with high noise when multiple fluorophores labels are in used, compromising the image quality, and limiting the interpretable number of labels in use. Fixed samples miss the information from the dynamic of biological processes which is vital to life science. Figure 1.7 Six genetic loci are observed at once from Visser et al The left image is where the fluorescence signal express while the right is the observation of six different fluorophores using FISH and optical filter methods. (Image source: Visser et al.)[22] 13 Figure 1.8 Bleedthrough is inevitable when multiple fluorophores are in use. (A) absorption and emission spectra for 6 fluorophores in Fig 1.7 (B) the optical filter used to separate the signals. The emission range are well separate in the optical filter. But the signal crosstalk in the emission profiles leading to the bleedthoguh in the image. [22] 1.2.3 The challenges in live imaging In life science, to observe a dynamic process, spatial and temporal resolution are equally important. Thanks to the advent of genetically tagging labels and fluorescence microscopy, the standard practice has evolved from fixed sample to live imaging. Researchers can now look at multiple targets at once while they are alive and functioning. Live imaging is a critical analytical technique in biomedical research laboratories 14 researching cell biology, neurobiology, pharmacology, and developmental biology. Yet, live imaging has its own constraints. To begin with, the difficulty in imaging life science is to have enough photons required for a decent quality image but not so many that may induce photobleaching or photodamage to the live sample. When imaging fixed samples, a high intensity light and a lengthy exposure time are often used to overcome photobleaching, a process that fluorophores lose their ability to fluoresce effectively when fluorophore is excited for too many times; however, these parameters must be avoided when imaging live cells in order to reduce photodamage, where the energy of light stresses cells and they eventually die. That precise number of photons fulfilling the above requirement is referred to as the photon budget per image acquisition. Photon budget is the number of detectable photons that a chosen fluorophore can contribute to the experiment at a given period of time. In practical terms, it sets a limit to how many image acquisitions can be performed on a sample. Typically, live imaging requires a trade-off between image quality and cell viability. As a result, spatial and temporal resolutions are frequently reduced in experiments to avoid using a high light intensity and a lengthy exposure time. Moreover, one big challenge of live imaging is the uncertainty in biology makes it hard to acquire a ground truth leading to analysis challenge. First of all, imaging live samples often encounters scattering effects and autofluorescence. As a result, live imaging suffers from higher noise from signals other than targeting signals. For example, most plant and animal tissues present some autofluorescence when exposed to short wavelength light. The autofluorescence, or intrinsic fluorescence, naturally occurs in compounds. (see more in 1.3.1) The autofluorescence normally have very wide emission spectra and are 15 thus sometimes difficult to distinguish from that generated by fluorescently tagged tags using conventional filtering techniques. Therefore, they interfere with the detection of fluorescent signals, particularly when the signals of interest are dim – it causes structures other than the ones of interest to become apparent. This results in a low signal from the fluorophores but high noise from the background image. To extract information from the low SNR data under the fact no ground truth present makes it an extremely challenging task. Another layer of difficulty to extract information from live imaging stems from the biological variability, which complicates the experimental design from image acquisition to analysis. When dealing with model organisms, there is always an inherent variability between samples even if they are genetically the same. This hinders the ability to acquire the specified target information over multiple samples. For example, in zebrafish after genotyping, a process to sort desirable fluorescence expression, the same batch transgenic sample may have different levels of expression of the same fluorophore expression. This may also occur when dealing with wildtype samples and other model organisms such as mouse. Therefore, live imaging, while providing more information for better understanding of dynamic systems, further adds extra complications which hinders the acquisition and analysis process. 1.2.4 The challenges of live multiplexing Live multiplexing is getting popular for 5D information extraction. However, current standard methods combine the challenges in live imaging and multiplexing, reaching the ceiling of getting interpretable results. Adding to the challenge in live imaging, the variation increases significantly when multiple fluorophores are used in parallel. As a 16 result, each acquisition becomes somewhat of a trial-and-error process to determine, for example, the optimal laser power or setting balance for the required photon budget, making imaging and image processing a difficult procedure at times. On top of the overlapping emission spectra, live multiplexing imaging is a time sensitive event. In standard imaging, sequential excitation, as previously mentioned, is more popular than parallel excitation because sample bleaches faster when parallel lasers are in use. Yet, in the live multiplexing image, the acquisition time in sequential excitation is in direct proportion to the label counts, with the high number of labels in use either increasing the acquisition time or energy load. While changes in a biological system can occur from seconds to months, given the current experimental setting (common sample used for biological research, mice, zebrafish, cells … etc ) for cell or small organ imaging, events might be missing if the acquisition time took too long. Therefore, even with a limited number of labels in use, it is not ideal to conduct time sensitive live imaging using the standard microscopy under this sequential acquisition setting. 1.2.5 Current solution: Linear unmixing for hyperspectral data Hence, to overcome the overlapping emission spectra in collecting signals, one of the biggest obstacles for live multiplexing in fluorescence microscopy. As mentioned in 1.1.4, hyperspectral imaging techniques can be a potential solution. The data is acquired in a lambda mode where extra information on emission spectra is collected and could potentially help unmixing the intertwined signals. However, further analysis is required to decode this hyperspectral cube. 17 Each pixel in a hyperspectral image is a blend of more than one material. One explanation for a mixed signal is because of the low resolution of the sensor. Distinct materials may coexist in a single pixel, and the resulting spectral measurement will be some composite of the distinct spectra. This is common for remote sensing systems flying at high altitudes or doing wide-area surveillance. Another explanation is that when different materials are blended into a homogenous mixture in the pixel image from. This situation can arise regardless of the sensor's spatial resolution. Therefore, there is a need to quantitatively decompose, or "unmix," these mixtures from pixels of a combination of multiple components. Linear unmixing is the standard practice for unmixing spectral data. The assumption is that each acquired spectrum in the hyperspectral cube is a linear combination of independent components, or their spectra, mixed in different ratios. These independent components are called endmembers and they have their own distinct emission spectra. The hypothesis for linear unmixing in this work is that given i independent spectral fingerprints (fp), each collected spectrum (I(λ)) is a linear combination of fp, and the sum of each fp contribution (𝑅) is 1. 𝐼(𝜆) =𝑊 ! 𝑅 ! 𝑓𝑝 ! +𝑊 " 𝑅 " 𝑓𝑝 " +⋯+𝑊 # 𝑅 # 𝑓𝑝 # +𝑁 (1) 𝛴𝑅 # =1 (2) where 𝑅 # is the ratio, 𝑊 # the weight, and 𝑁 the noise. The acquired spectra are collected in the original spectral cube with shape (t,z,c,y,x), with t as time, c as channel and x,y,z spatial dimensions. 18 Therefore, combining spectral imaging with linear unmixing is a highly useful technique that can be used to untangle spectral overlap that would be otherwise difficult to separate. Dickinson et al. first combined and applied these techniques to fluorescence microscopy in 2001. Valm et al utilized spectral imaging with linear unmixing to reveal six organelle interactomes. However, collecting spectral data increases the data size in proportion to the number of channels acquired. In this case, acquiring 32 channels increased their single cell data to 1T. LU is a pixel by pixel based method, leading their analysis took a whole week to analyze this single cell data. Moreover, a strong threshold was applied to remove background noise, losing some information within the cell connecting tissues. These limitations in applying LU on spectral data prevent a wide usage in unmixing overlapping fluorescent emission in cells and tissues. [19], [23] 1.2.6 Unmixing challenges in biological data Standard approaches for separating spectrally overlapping labels, such as linear unmixing, were developed for remote sensing, where sunlight acts as the light source and strong reflectance signals are collected with reduced noise. However, in live fluorescence microscopy signals are often photon starved and affected by both Poisson and instrumental noise, shown as Figure 1.9. Contrary to the requirement of good signal for LU, the intrinsic difficulty of live imaging as previously mentioned 1. low SNR data from the low photon regime disrupts the analysis 2. unknown biological noise plus autofluorescence breaks linearity of the system. Standard unmixing strategies generally underperform in noisy environments and yield limited results when applied to the low Signal-to-Noise fluorescent spectra. These factors, combined with the statistical requirement in biological experiments 19 and the high computational costs of these standard strategies, have hampered both the performance and the popularity of multi-spectral fluorescence multiplexing. In general, despite the availability of a range of technologies and considerable work invested in biological information extraction, existing setups do not fully use the capabilities of 5D imaging. 20 Figure 1.9 Signals collected from a microscope are distorted from expected emission spectra. (A) expected emission spectrum for mCherry (B) collected signal under photon starvation (C) Signals collected suffer from both multiply types of noises and phot on starvation 21 1.3 Imaging metabolism from intrinsic fluorescence signal Autofluorescence has been widely used in studying cellular metabolism. Cellular metabolism is a key regulator of cell function and plays an essential role in the development of numerous diseases. Understanding of cellular metabolic pathways is critical for the development and assessment of novel therapies and diagnostics. The use of autofluorescence for neuron science and disease diagnosis was reported as early as 1911 by Stubel. With the advent of microscopy including 2-photon excitation and increased sensitivity of CCD, the study of label-free autofluorescence signal is gaining popularity. However, fluorescence microscopy still mainly uses exogenous markers because of the low intensity and spectral complexity nature of autofluorescence leading to the challenge of detecting and interpreting autofluorescence signals. [24]–[29] 1.3.1 What is autofluorescence? "Autofluorescence'' is a term used to differentiate intrinsic fluorescence of cells and tissues from exogenous fluorescent markers that attach to cell and tissue components. Although autofluorescence is notorious for impairing the capacity to observe and quantify fluorescent markers in vivo, it actually provides a wealth of information about cell activity and structure. A number of metabolites have been reported in literature to be fluorescent and to change their spectra or lifetime according to their biochemical configurations. Scientists have identified endogenous fluorophores associating cellular metabolic processes or structural matrix of tissues, such as NADH, Flavins, Retinoids, collagen/elastin proteins etc. NAD(P) is a major electron acceptor in the energy metabolism pathways. The reduced form, NAD(P)H, is fluorescent. When NAD(P)H is 22 bound to the proteins, the fluorescent quantum yield increases. Therefore, scientists often use the NAD(P)H free bound ratio as an indicator of redox state. Similarly, flavins, such as riboflavin, flavin mononucleotide (FMN) and flavin adenine dinucleotide (FAD) are also involved in a wide range of biological processes and known as redox-active coenzymes. Another example, in the extracellular matrix, collagen and elastin are also recognized as important autofluorescence molecules (from cross-linking between amino acids) to identify cell structures. [4], [28], [30]–[33] 23 Figure 1.10 Autofluorescence excitation emission spectra. [34] 1.3.2 Imaging autofluorescence using FLIM and phasor approach Researchers today use Fluorescence Lifetime Imaging Microscopy(FLIM) to study these intrinsic signals. As mentioned in section 1.1.1, the time difference from a fluorophore getting excited to releasing the energy is lifetime. FLIM measures the difference of this lifetime decay from fluorescence and differentiate different fluorescence with their distinct lifetime. By using FLIM, researchers avoid the difficulty of interpreting the highly overlapping spectra from autofluorescence. For example, NADH free and bound have highly overlapping emission spectra with maximum emission spectrum peak at approximately 460 nm and 445 nm respectively, but their lifetimes 0.3 ns and 2-3 ns are very different; therefore many researches have been studying the cell metabolism by conducting FLIM.[7] Phasor approach Phasor approach is widely used to analyze fluorescence lifetime data since 2004 Enrico Gratton’s group introduced Frequency Domain FLIM. One main challenge is extracting information from the complex lifetime information. Phasor Approach to FLIM overcomes these challenges by using Fourier transformation (FT) of the decay curve to a vectorial representation on a 2-dimensional phasor plot. FT decomposes the image into its real and imaginary components. Phasors are the representation of real and imaginary components of the signal on-to a plot. As shown in Fig. 1.10. Phasor-FLIM has been utilized to discern between distinct metabolic states in specific cells and tissues with separate lifetimes. FLIM has been effectively used to determine the condition of NADH in both mice and, more recently, the human cornea. Phasor-high FLIM's precision 24 measurements have a limit on the number of metabolites that may be seen. NADH (bound versus free) are the primary biomarkers that can be detected with this approach at the moment, leaving a substantial portion of the biological mystery unsolved. The advantages of this approach include 1. dimensionality reduction: each 2D signal in the image is converted into a single point in Fourier space. 2. geometrical-analysis: phasor space is linear. 3. easy classification: points are statistics and similar points cluster together. [35]– [41] Figure 1.10 Common intrinsic signals phasor map from FLIM data. [32] 25 1.3.3 Challenges for autofl imaging There are several unique challenges in relation to analyzing autofluorescence signal. Autofluorescent signals have broad and extremely overlapping emission spectra. Traditional sequential excitation and collection would not work in this case. Spectral imaging and standard unmixing algorithms also failed due to these overlapping spectra. Therefore, some groups are using FLIM for this type of analysis. FLIM setting are relatively uncommon. Even though the technique exists, it is still limited to a reduced number of labs. In addition, interpretation of FLIM data is very complex. Although phasors simplify the FLIM data analysis, it still requires experienced users who are familiar with both biology and signal processing, and have exposure to a large amount of FLIM phasor data to perform a meaningful analysis. These metabolic changes are highly heterogeneous. Metabolic pathways are complex and changes are highly dynamic: changes can occur in the order of seconds or minutes or occur over a period of days, weeks, months and years. The FLIM and phasor approach work well for cells, but not so well for a more complicated sample such as tissues or organs. In more complicated samples, more other signals may interfere with the analysis. Weak signals and lower intensities compared to extrinsic fluorescence signals. High biological background noise from tissue plus the intrinsic lower intensity nature of autofluorescence make these signals in extremely low SNR. None of the current analyses performs decently with low SNR data. The above reasons make the extremely low SNR data almost inaccessible. 26 1.3.4 Moving toward real world challenges Autofluorescence imaging can be a powerful tool in fundamental biological research ranging from cellular energy metabolism, differentiation, to tissue analysis, and diagnosis. These signals are intrinsically fluorescent, which potentially opens the door to real-time diagnosis of pathological disorder from the label-free imaging. In standard pathological diagnosis, stains and biopsy are required, which can cost a lot of time and money. The possibility of real-time analysis and diagnosis for living sample analysis from intrinsic metabolic signals greatly reduces the time of long surgery where normally a huge chunk of tissue is cut off from the patient and taken to labs, and couple weeks for researchers to use expensive stains and conduct diagnosis. The patient may need to suffer from a second surgery depending on the result. However, additional work is required to fully utilize it; 1. Strong unmixing analysis tool to unmix the extremely overlapping emission spectra 2. More understanding of individual components presents with the given imaging setting. 3. A way to handle low SNR data or a way to increase SNR. 1.4 Machine Learning in in life science With the refinement of next-generation imaging tools, microscopy and image analysis need high-throughput and high-content approaches to understand the dynamic cell behaviors. The resolution and quality of the images acquired through microscope is fundamentally constrained by the optics, photophysics of molecular probes, and sensors. Scientists have been developing analysis tools or algorithms to overcome the limitations. It is not easy because a deep knowledge in fluorescence microscopy is required to build or use an imaging processing algorithm such as Richardson–Lucy deconvolution. Another 27 challenge of analysis tools for microscopy data is normally they are designed for specific usage for specific types of data. Machine learning, especially deep learning could be one promising solution to generalize and facilitate the process. Machines have been known to surpass human performance for solving complex problems because of its capability to recognize the pattern and extract features from a large volume of data. With greatly expanded informational dimensions and potentially reaching terabytes in size per experiment, there is a need for automated solutions like machine learning for both processing and interpretation of such multidimensional datasets. Efforts have been made toward image restoration, deconvolution, super-resolution, object detection, translation between label-free and fluorescence images and motion analysis, as well as image segmentation, classification, and phenotyping. [42]–[48] However, it is not trivial to build a machine learning network. Generally, two steps are required for building a machine learning network: 1. training on a large amount of data so that the computer can learn from underlying patterns and relationships within this data. 2. testing on data that researchers are going to predict or classify. Data becomes a critical part of the concentration of the network. A lot of open questions require more study and understanding including what is a good amount of data needed? What is required for the data? How to efficiently generate data and make the most use of it? When applying deep learning, how is the result obtained and how realizable is that? Lastly, back to the difficulty of imagining life science, how do researchers quantify the results when ground truth does not even exist? 28 Figure 1.11 Standard machine learning framework. The core idea of machine learning network is using a tons of data to find a pattern can be used to recognize for prediction or classification. Features need to be extracted to train the network. Raw data would go through a preprocessing to reduce noise or background correction for better detecting of individual objects for feature extraction. [49] 1.5 Algorithmically overcome current limitations In Chapter 1, we review the state-of-art technology from microscopes, fluorescent labels to imaging methods and their corresponding challenges. In this thesis work, the goal is to develop imaging processing tools to algorithmically solve the mentioned limitations from overlapping emission spectra to disrupted signals. Therefore, consistent imaging acquisition methods and instruments are applied. We used commercial laser scanning microscopes (LSM 780, LSM 880, Carl Zeiss) with both confocal and multiphoton excitation, collected in 32 channel lambda mode (hyperspectral imaging), imaged on transgenic or wildtype zebrafish embryos. The transgenic zebrafish line is designed for solving problems with highly overlapping emission spectra. With the constraints from instruments and samples, we aim to push the limit of 5D imaging utilizing low photon budget to balance among spatial, temporal, and cell type dimensions. Here we define the general experimental conditions as following. 29 Figure 1.12 Project pipeline schematic. The thesis work is developing algorithms to solve the spectral overlapping and signal disruption problems with the hyperspectral data. The animal model is zebrafish for the following experiment. But the input data for the algorithms do not limit to zebrafish. Zebrafish fishline Adult fish were raised and maintained in strict accordance with the recommendations in the Guide for the Care and Use of Laboratory Animals by the University of Southern California, where the protocol was approved by the Institutional Animal Care and Use Committee (IACUC) (Permit Number: 12007 USC). Upon crossing appropriate adult lines, the embryos obtained were raised in Egg Water (60 μg/ml of Instant Ocean and 75 μg/ml of CaSO4 in Milli-Q water) at 28.5 o C with addition of 0.003% (w/v) 1-phenyl-2-thiourea (PTU) around 18hpf to reduce pigment formation. Transgenic Gt(cltca-Citrine) ct116a line is a genetrap of clathrin, heavy polypeptide a, labeling transport vesicles with heightened expression in the vasculature. 2 Tg(kdrl:mCherry) labels the vasculature and was a kind gift from Ching-Ling Lien 30 (Children’s Hospital Los Angeles). Tg(fli1:mKO2) ct641ca labels pan endothelial cells in both blood vessels and lymphatics as previously reported. 3 Tg(ubiq:lyn-tdTomato) labels all cell membrane by expression of lyn-tdTomato from the ubiquitin promoter, while Tg(ubiq:Lifeact-mRuby) labels actin by expression of LifeAct-mRuby fusion from the ubiquitin promoter. mpv17a9/a9;mitfaw2/w2 (casper) line was purchased from Zebrafish International Resource Center (ZIRC) and csf1rj4e1/j4e1 (panther) line 4 was a kind gift from David Parichy (Univ. Virginia). We crossed casper with panther to produce triple heterozygote mpv17a9/+;mitfaw2/+;csf1rj4e1/+ F1 generation fish, which were subsequently incrossed to produce F2 generation with 27 combinations of mutational state of these genes. Since csf1rj4e1 phenotype was not clear in F2 adult with casper phenotype, we outcrossed these fish with panther fish to determine the zygocity of csf1rj4e1 mutation based on the frequency of larva with xanthophores (heterozygote and homozygote produced 50%- and 0%-fraction of xanthophore-positive larva, respectively) by fluorescence microscopy. The casper;csf1rj4e1/j4e1 line is viable and reproducible; we outcrossed either casper;csf1rj4e1/j4e1 line or casper;csf1rj4e1/+ line with other fluorescent transgenic lines over several generations to obtain fish harboring multiple transgenes on casper background either in the presence or absence of xanthophores. Sample preparation 31 Figure 1.13 The emission spectra for the fluorophores use in this thesis work. Gt(cltca-Citrine), Tg(ubiq:lyn-tdTomato), Tg(fli1::mKO2), and Tg(ubiq:Lifeact- mRuby). Signals are collected from 32 channel Zeiss LSM 780 with 960 nm excitation and 790+ bandpass filter. Spatial and spectral overlapping labels are purposely generated with the animal sample. Transgenic zebrafish lines were intercrossed over multiple generations to obtain embryos with multiple combinations of the transgenes. All lines were maintained as heterozygous for each transgene. Embryos were screened using a fluorescence stereo microscope (Axio Zoom, Carl Zeiss) for expression patterns of individual fluorescence proteins before imaging experiments. A confocal microscope (LSM 780, Carl Zeiss) was used to isolate Tg(ubiq:Lifeact-mRuby) lines from Tg(ubiq:lyn-tdTomato) lines by distinguishing spatially- and spectrally-overlapping signals. For in vivo imaging, 5–6 zebrafish embryos at 18 to 72 hpf were immobilized and placed into 1% UltraPure low- melting-point agarose (catalog no. 16520-050, Invitrogen) solution prepared in 30% Danieau (17.4 mM NaCl, 210 M KCl, 120 M MgSO47H2O, 180 M Ca(NO3)2, 1.5 mM HEPES buffer in water, pH 7.6) with 0.003% PTU and 0.01% tricaine in an imaging dish with no. 1.5 coverglass bottom, (catalog no. D5040P, WillCo Wells). Following 32 solidification of agarose at room temperature (1–2 min), the imaging dish was filled with 30% Danieau solution and 0.01% tricaine at 28.5 °C. Image Acquisition. Table 1 Experimental setting exploration in the imaging side. Images were acquired on a Zeiss LSM 780 laser confocal scanning microscope equipped with a 32-channel detector using 40x/1.1 W LD C-Apochromat Korr UV-VIS- IR lens at 28ºC. Samples of Gt(cltca-Citrine), Tg(ubiq:lyn-tdTomato), Tg(fli1::mKO2), and Tg(ubiq:Lifeact-mRuby), were simultaneously imaged with 488 nm and 561 nm laser excitation, for citrine, tdTomato, mKO2, and mRuby. A narrow 488 nm/561 nm dichroic mirror was used to separate excitation and fluorescence emission. Samples were imaged with a 2-photon laser at 740 nm to excite autofluorescence, using a 690 nm low pass filter to separate excitation and fluorescence. For all samples, detection was performed at the full available range (410.5-694.9nm) with 8.9nm spectral binning. 33 Chapter 2 HyU: Hybrid Unmixing for longitudinal in vivo imaging of low signal to noise fluorescence 2.1 Hybrid unmixing (HyU) provides a platform to combine Phasor analysis with other unmixing algorithms 2.1.1 HyU inherit phasor advantage from phasor approach Phasor approaches as mentioned in section 1.3.2 reduce the computational load because of the characteristics as a compressor and an encoder, where 5D data can be mapped into a 2D plane. With the collected microscopy data from Zeiss 780 spectral mode, using phasor reduces the 32 channels of an HFI spectral plot into a position on a 2D- histogram, representing the real and imaginary Fourier components of the spectrum (Fig. 2.1 A, B). Different 32 channel spectra are represented as different positions on the 2D phasor plot, and mixtures of the two spectra will be rendered at a position along a line connecting the pure spectra. Because the spectral content of an entire 2D or 3D image set is rendered on a single phasor plot, there is a dramatic data compression - from a spectrum for each voxel in an image set (up to or even beyond Gigavoxels) to a histogram value on the phasor plot (Megapixels). [32], [41], [50]–[53] 34 Figure 2.1 Schematic illustrating how Hybrid Unmixing (HyU) enhances analysis of multiplexed hyperspectral fluorescent signals in vivo. (A) Multicolor fluorescent biological sample (here a zebrafish embryo) is imaged in hyperspectral mode, collecting the fluorescence spectrum of each voxel in the specimen. (B) HyU represents spectral data as a phasor plot, a 2D histogram of the real and imaginary Fourier components (at a single harmonic). (C) Spectral denoising filters reduce the Poisson and instrumental noise on the phasor histogram, providing the first signal improvement. (D) The phasor acts as an encoder, where each histogram-bin corresponds to a number n of pixels, each with a relatively similar spectrum (E). Summing these spectra effectively averages the spectra for that phasor position.This denoising results in cleaner average spectrum for this set of pixels, which are ideally suited for analytical decomposition through unmixing algorithms (F). (G) Unmixing results in images that separated into spectral components. Here, linear unmixing (LU) is used for unmixing, but HyU is compatible with any unmixing algorithm. Note that HyU offers a major reduction in data size and complexity of the LU (or any other unmixing) computation, because the calculation is applied to the 10 4 histogram bins (D), rather the the ~10 7 voxels in the specimen (A). This reduces the number of calculations required for LU dramatically. In addition, because each “bin” on the phasor plot histogram corresponds to multiple voxels with highly similar spectral profiles, the binning itself represents spectral averaging, which reduces the Poisson and instrumental noise (Fig. 2.1C-E). Poisson noise in the collected light is unavoidable in HFI unless the excitation is turned so high that the 35 statistics of collected fluorescence creates hundreds or thousands of photons per spectral bin. This binning improves the pixelated result in linear unmixing since the processing is now working on spectra coming from similar combination. The clear separation of the spectral phasor plot and its referenced imaging data, permits denoising algorithms to be applied to phasor plot with minimal degradation of the image resolution. 2.1.2 HyU inherit linear unmixing advantage Here, we use the standard unmixing algorithm, linear unmixing (LU) as an example of how HyU combines with other algorithms. HyU combines the best features of hyperspectral phasor analysis and LU, resulting in faster computation speeds and more reliable results, especially at low light levels. LU or other unmixing approaches can be applied to the spectra on the phasor plot, offering a dramatic reduction in computational burden for large image data sets (Fig. 2.1D). To understand this saving, consider the conventional approach of LU applied to image data at the voxel level (Fig. 2.1A,F). A timelapse volumetric dataset of 512x768x17 (x, y, z) pixels, over 6 timepoints , would require 40 million operations. HyU’s requires only ~18 thousand operations to unmix each bin on the phasor plot, representing more than a thousand-fold saving (Fig. 2.1F,G). [23], [54]–[57] 2.1.3 Identification of unknown components using HyU Identification of independent spectral components has been an adversity for unmixing hyperspectral data. First, the collected spectra may be distorted by reduced SNR. 36 Secondly, excitation of intrinsic signals causes uncertainty of the biological sample. Favorably, HyU simplifies this process by adapting the Phasor approach and achieving a semi- or full- automation process for spectra identification and selection. In HyU spectra can be loaded from an existing library, virtually automating the analysis process. Pre- identified cursors are generated from common fluorophores such as mKO2, tdTomato, mRuby, Citrine. (Fig. 2.2) For fluorescent signals we used in this paper were obtained by imaging single labeled samples in areas morphologically and physiologically known to express the specific fluorescence. Those fingerprints were compared with literature fingerprints and manually corrected to reduce noise. Figure 2.2 Pre-identified cursors for common fluorophores on phasor map. A) Extrinsic and intrinsic labels displaying in second harmonic phasor and (B) intrinsic labels in first harmonic phasor. Typically, second harmonic phasor has a more spread out phasor distribution. However, the intrinsic signals are more spread out when first harmonic FFT is applied. In the presence of unexpected fluorescent signals, spectra can also be selected and visualized directly from the phasor. The “tails” on the phasor distributions are independent components. In our HyU graphical interface, clicking on the phasor visualizes the spectra within a small area (9x9 bins by default, the size is adjustable from the interface) of the phasor histogram (Figure 2.1 D). In the example in Figure 2.3(A-D), 2.4, we identify 5 37 distinct endmembers on the Phasor Figure 2.3 D), visualize their spectra identifying Citrine, mRuby, Td-Tomato, mKO2, and one strong autofluorescence signature. Residual is a metrics we designed to quantify the performance without knowing the ground truth, more detail can be found in section 2.2.1 and method. Here, we are able to map residual within the phasor and create a residual phasor map. The use of Residual Phasor Map (Figure 2.4) allows for identification of areas in the phasor with high amount of residuals, likely corresponding to a missing endmember in the unmixing. Residual Image Maps provide a rapid overview of residuals in the image data, for identification of location in the dataset of the missing endmember. 38 Figure 2.3 HyU enables identification and unmixing of low photon intrinsic signals in conjunction with extrinsic signals. (A) HyU results of a whole zebrafish embryo provide a frame of reference not only for the improved unmixing of extrinsic signals, but also its increased sensitivity which enables identification and unmixing of intrinsic signals which inherently exist in a low- photon environment. (B) HyU results of the head region (box in A) reveal the simplicity of identifying an unknown autofluorescent signal among multiple extrinsic signals using the phasor method for a quadra-transgenic zebrafish Gt(cltca-citrine);Tg(ubiq:lyn- 39 tdTomato;ubiq:Lifeact-mRuby;fli1:mKO2) imaged over multiple tiles. Scale bar: 80 µm. (C) The input spectra required to perform the unmixing are easily identified on (D) the phasor plot when visualizing each spectrum as a spatial location. Phasors offer a simplified identification and selection of independent and unexpected spectral components in the encoded HyU approach. Intrinsic signals are notoriously low in emitted photons leading to an inability to unmix using traditional unmixing algorithms. (E) The zoomed-in acquisition of the head region of the embryo (box in A) displays HyU’s unmixing results of many intrinsic and extrinsic signals when in an environment of very low photon output, a previously highly difficult experimental condition to unmix. Scale bar: 70 µm. (F) The phasor plot representation provides the easily identifiable eight independent fluorescent fingerprint locations. (G) The spectra corresponding to each of the eight independent spectral components are also provided a reference. Colors in (F) match renderings in (E) and (G): NADH bound (red), NADH free (yellow), retinoid (magenta), retinoic acid (cyan), reflection (green), elastin (purple) and extrinsic signals: mKO2 (blue), and mRuby (orange). All signals were excited with a (A-D) single photon laser at both 488 nm and 561 nm or a (E-G) two photon laser at 740 nm. Figure 2.4 Residual maps facilitate identification of independent spectral components. Experimental fluorescence microscopy data often includes unexpected autofluorescence signals. Residual Maps (Methods) provide additional information to account for these signals and properly adjust HyU analysis. (A) Average intensity image with pixels pseudo-colored in cyan (autofluorescence) and magenta (background) according to the ROIs selections on the (B) Residual Phasor Map, computed from the HyU of 4 input spectra with a threshold of zero. The pseudo-colored areas of the image match those presenting high residual values in the (C) Residual Image Map. Changing the unmixing input to include the unexpected autofluorescent spectrum (cyan) and performing the Residual Phasor map selections produces the (D) background pseudo-colored (magenta) 40 image. The inclusion of autofluorescence as an independent spectral component in the unmixing decreases the number of pixels corresponding to the autofluorescent signal (cyan ROI) in the (E) Residual Phasor Map, thereby matching with the (F) Residual Image Map, which no longer presents high residuals in the center portion of the image. Increasing the threshold to 250 removes the pixels with high residuals corresponding to the background, removing them from the (G) average intensity image, (H) Residual Phasor Map, and (I) Residual Image Map. 2.1.4 HyU provide user friendly interface to visualize independent components Visualizing independent signatures is easy within the developed software. Figure 2.5 shows a step-by-step protocol of how to load data, for example, from a single label Td- Tomato raw data and how to visualize the corresponding spectrum and how to save the spectrum for future use. Step 1: Calculate Phasor after load data. Click on the “Calculate Spectral Phasor” button which is now active in the main initial window. A new window with the Phasor plot will appear. Step 2: Filter Phasor and go to Hybrid Unmixing tab. (1) Select the multiplier next to “Filter Data” and set it to 5x. (2) Click on “Filter Data”. (3) Click on the rightward arrow next to the “Combination Tools” tab twice until the Hybrid Unmixing tab appears. Then select the Hybrid Unmixing tab. Step 3: Unmix hyperspectral dataset. (1) In the Hybrid Unmixing tab, click the “Select Spectra” button and wait for the progress bar on the bottom of the window to finish; the “Select Spectra” should turn into a “Double click to finish” button. Step 4: Then click the “Load fp txt file” button. Select the text file with the spectra values. (3) After selecting the text file, click the “Double click to finish” button; the “Hybrid Unmixing [beta]” button should become active. (4) Finally, click the “Hybrid 41 Unmixing [beta]” button. The unmixed files should be created in the same directory as the input file. 42 Figure 2.5 HyU interface showing how to obtain spectra info from raw data and save it for future use. (Step1) denoise data (Step2) identify components using preidentified cursor (Step3) select independent components (Step 4) visualize spectrum of it 2.2 The hybrid method outperforms the traditional method 2.2.1 HyU shows higher accuracy in simulation data To quantitatively assess the relative performance of LU and HyU, we analyzed them on synthetic hyperspectral fluorescent datasets, created by computationally modelling the biophysics of fluorescence spectral emission and microscope performance (Fig 2.6 A, B). We used this synthetic dataset to evaluate LU and HyU algorithm performance quantitatively by using metrics such as Mean Square Error (MSE) and unmixing residual (see Methods; for both metrics, a lower value indicates better performance). In addition to the computational efficiency mentioned above, HyU analysis shows better ability to capture spatial features over a wide dynamic range of intensities, when compared with standard LU, in large part due to the denoising created by processing in phasor space (Fig. 2.6 A, B). The improved accuracy is demonstrated by a lower MSE, in comparing the results of LU and HyU to the image ground truth. The absolute MSE for HyU is consistently up to 2x lower than that of LU, especially at low and ultra-low fluorescence levels (Fig. 2.6 C). MSE can be further decreased by the use of denoising filters on the phasor plot, resulting in superiority of HyU relative to LU for HFI at low (5- 20 photons/spectrum) and ultralow (2-5 photons/spectrum) levels (Fig. 2.6 D). To better characterize the performance in the experimental data without ground truth, we also define the unmixing residual as the difference between the original 43 multichannel hyperspectral images and their unmixed results. Residuals provide a measure of how closely the unmixed results reconstruct the original signal (Fig. S3, Methods). Unmixing residuals are inversely proportional to the performance of the algorithm, with low residuals indicating high similarity between the unmixed and the original signals. Analysis of unmixing residuals in the synthetic data highlights an improved interpretation of the spectral information in HyU with an average unmixing residual reduction of 21% compared to the standard (Fig. S5C). The reduction in both MSE and average unmixing residual for synthetic data demonstrates the superior performance of HyU, and provides a baseline comparison when demonstrating performance improvements for experimental data. 2.2.2 HyU overcomes the bleedthrough problem in LU in experimental data We support the enhanced performance of HyU with analysis of experimental data, which reveals comparatively lower unmixing residuals and a higher dynamic range as compared to LU. Data was acquired from a quadra-transgenic zebrafish embryo Tg(ubiq:Lifeact-mRuby);Gt(cltca-citrine);Tg(ubiq:lyn-tdTomato);Tg(fli1:mKO2), labelling actin, clathrin, plasma membrane, and pan-endothelial cells, respectively (Figs. 2.6 E-L). HyU unmixing of the data shows minimal signal cross-talk between channels while LU presents noticeable bleed-through (Fig. 2M-P). Consistently with synthetic data, we utilize the unmixing residual as the main indicator for quality of the analysis in experimental data, owing to the absence of a ground truth. The residual images (Fig. 2.6 F, G) depict a striking difference in performance between HyU and LU. The average relative residual of HyU denotes a 7-fold improvement compared to LU (Fig. 2.6 H) in 44 disentangling the fluorescent spectra. We visualize the unmixed channels independently (Fig. 2.6 I to L), zooming in on details (Fig.2.6 I to P) to highlight areas affected by bleed- through and which are difficult to unmix. HyU, with contrast 2-fold higher than standard LU, reduces bleed-through effects and produces images with sharper spatial features, leading to better interpretation of the experimental data (Fig. 2.6 K, L, Fig. 2.7). 45 Figure 2.6 Hybrid Unmixing outperforms standard Linear Unmixing (LU) in both synthetic and live spectral fluorescence imaging. (A) Hybrid Unmixing (HyU) and (B) Linear Unmixing (LU) tested using a hyperspectral fluorescence simulation that was generated from four fluorescent signatures (emission spectra, Sup fig 5E). (C) Absolute Mean Squared Error (MSE) shows that HyU offers a consistent reduction in error across a broad range of photons per spectra (#photons/independent spectral components, here resulting from 4 reference spectra combined). (D) The performance differences in the MSE of HyU relative to LU persists when applying multiple phasor denoising filters (0 to 5 median filters). The analysis of this synthetic data shows the consistent improvement of HyU at low photon counts with over a 2-fold improvement when 5 denoising filters are applied at a signal level of 16 photons per spectrum. Shaded regions for line plots denote the 95% confidence interval around the mean. (E) Unmixing of experimental data from a 4-color 46 zebrafish shows increased contrast for HyU (left) compared to LU (right). Scale bar = 50 µm. (F, G) The increased accuracy is revealed by residual images of HyU and LU, showing the spatial distribution of unassigned signals after the analysis of data in E. The results show consistently lower residual values for HyU (F) compared to LU (G). (H) Box plots of the residuals in F and G presents values of 11% for HyU compared to 77% for LU with *(p < 10 -10 ). Box plot elements are defined in Methods. (I-L) Enlarged rendering of HyU results (E, white box) clearly shows low levels of bleed-through between labels (M-P) Similar enlargement of LU results show noticeably worse performance. Note that regions with bright signals (membrane J, N white arrow) bleed through other channels (M) and (O). Scale bar: 20 µm. Tetra-labeled specimen used here was Gt(cltca-citrine);Tg(ubiq:lyn-tdTomato; ubiq:Lifeact-mRuby;fli1:mKO2) Figure 2.7 Volumetric unmixing results of a quadra-transgenic zebrafish with HyU and LU highlights improvements in contrast and spatial features(Figure 2.6). Volumetric zoom-in view of the somites within the trunk region of a 10-dpf Gt(cltca- citrine);Tg(ubiq:lyn-tdTomato; ubiq:Lifeact-mRuby;fli1:mKO2) zebrafish merging all channels in (A) HyU and (B) LU. (A-E) HyU presents a wide dynamic range of intensities with average contrast 1.11-fold higher than LU compared to (F-J) LU. In LU, bleed-through from the (H) membrane label (arrow) is observed in the (G) lymphatic vasculature channel (arrow) and the (I) actin channel (arrow). This incorrect re- assignment of intensities is not present in the corresponding HyU channels for (B) 47 vasculature and (D) actin, where fibers (arrow) are cleanly unmixed. (K) Phasor Residual Distribution shows the distribution of relative residual (%) and photon counts in phasor histogram bins. Residual distribution shows the distribution of relative residual (%) and photon counts in histogram pixels for both (L) HyU and (M) LU. 2.3 HyU has better resolution in low SNR data 2.3.1 Better spatial resolution Applying HyU to another HFI dataset further highlights HyU’s improvements in noise reduction and reconstitution of spatial features for low-photon unmixing. (Figs. 2.8- 2.9). In the zoomed-in image of a single slice of the embryo skin surface, acquired in the trunk region, the HyU image correctly does not display pan-endothelial (magenta) signal in the periderm, an area which should be devoid of endothelial cells and mKO2 signal (Fig. 2.8 C). In contrast, the result from LU shows visually distinctive pan-endothelial signal throughout the tissue plane (Fig. 2.8 D). This incorrect estimation of the relative contribution of mKO2 fluorescence for LU is possibly due to the presence of noise, corrupting the spectral profiles. This is further delineated in the intensity profiles of the mKO2 signal between HyU and LU with much higher individual peaks from noise demonstrated for LU (Fig. 2.8 G, lower left). Intensity profiles for both magnified cross- sections of the volume (Fig. 2.8 C-F) provide a striking visualization of the improvements of HyU. The line intensity profiles in HyU present reduced noise and represent more closely the expected distribution of signals (Fig. 2.8 G,H). The visible micro patterns of actin on the membrane of the periderm suggest that the improvements quantified with synthetic data are maintained in live samples’ signals and geometrical patterns of microridges 22 . By contrast, noise corruption and the presence of misplaced signals are characterized in the results from LU, with high frequency intensity variations that mis- match both the labeling and biological patterns. 48 Another advantage of using phasor is the phasor denoising filter [52] can now be easily applied and greatly improve the ability to work with low SNR data. In Figure 2.9, lower residual can be observed in the unmixing results. 49 Figure 2.8 Hybrid Unmixing enhances unmixing for low-signal in vivo multiplexing and achieves deeper volumetric imaging. (A) Hybrid Unmixing (HyU) volumetric renderings compared to those of (B) Linear Unmixing (LU) for the trunk portion in a 4-color zebrafish demonstrate an increased 50 contrast and reduced residual in HyU results, especially over deeper parts of the sample. The 4 labels in the fish are Gt(cltca-citrine);Tg(ubiq:lyn-tdTomato;ubiq:Lifeact- mRuby;fli1:mKO2), respectively labeling clathrin-coated pits (green), membrane (yellow), actin (cyan) and endothelial (magenta). (C,E) HyU results have increased spatial resolution and less bleed though comparing to those of (D,F) LU. Scale bar: 20 µm. When observing the zoomed-in visualization of the surface region of the sample, the yellow signal distinctly marks the membrane and the cyan signal clearly labels the actin in (C) HyU. The same signals are not distinct in (D) LU because of multiple incorrectly assigned magenta pixels that bleed through compromising the true signal in other channels. Similarly, for the zoomed-in visualization of the Perivascular region of the embryo, in (E) HyU, the yellow and magenta signals clearly distinguish the membrane and vasculature while in (F) LU, the results are corrupted by greater noise. (G,H) Intensity line plots of each of the four results signals for HyU (solid) and LU (dashed) demonstrate the improved profiles with greatly reduced noise peaks in HyU as compared to LU. Intensities are scaled by the maximum of each unmixed channel. DL: digital level. (I) Box plots of the relative residual values as a function of z depth for HyU and LU highlight the improvements in the unmixing results. HyU has an unmixing residual of 6.6% ± 5.3% compared to LU’s 58% ± 17%. The average amount of residual is 9-fold lower in HyU with narrower variance of residual. Box plot elements are defined in Methods. 51 52 Figure 2.9 Residual analysis of experimental data supports performance improvement of HyU. Residual analysis for multispectral fluorescent data of a 5dpf quadra-transgenic zebrafish Gt((cltca-citrine);Tg (ubq:lyn-tdTomato);(ubiq:Lifeact-mRuby);(fli1:mKO2)) in Figure 3. Unmixing results for (A) HyU and (B) LU, respectively. Residual Image map of the z- averaged dataset for (C) HyU and (D) LU show lower residual values for HyU, suggesting improved quality of unmixing. (E) Residual distribution relative to the original intensity in each pixel as a function of estimated photon counts per spectrum for LU and (F) HyU. (G) Residual Phasor Map presents increased residual values in the background region (arrow). Jet colormap scale refers to C, D and G. (H) Residual Phasor Histogram for HyU shows distribution of residuals in the broad dynamic range of photons for experimental data. (I) Raw phasor with 0 threshold applied and 5 denoising filters, the ROI (yellow circle) highlights the background pixels in the (J) average spectral image (showing the first z slice). HyU is more accurate and results in more reliable unmixing results across the depth of sample with greatly reduced unmixing residuals. The average residual for HyU is 9- fold lower than that of LU with a 3-fold narrower variance. (Figs. 3I, S8). This reduction in the residual is consistent with increasing z-depth where HyU unmixing results stably maintain both lower residuals and variance on average. These reduced residuals correspond both to a mathematically more precise and more uniform decomposition of signals as illustrated by the distribution of residuals versus photons (Figs. S8E,F, S14). 2.3.2 Longer timelapse imaging 53 Figure 2.10 HyU reveals the dynamics of developing vasculature by enabling multiplexed volumetric time-lapse. Hybrid Unmixing (HyU) overcomes challenges in performing multiplexed volumetric time-lapse in vivo imaging of a developing embryo. Here we present this (A) HyU rendering for the trunk portion of a 3-color zebrafish Gt(cltca- citrine);Tg(kdrl:mCherry;fli1:mKO2) at timepoint 0. (B) HyU unmixed results allow for quantitative analysis and segmentation, here an example representing the time evolution of the segmented volumes of mCherry (vasculature, magenta) mKO2(endothelial- lymphatics, yellow) and citrine (clathrin-coated pits, cyan). Box and line plots were generated using ImarisVantage as described in Methods. (C1-4) Time lapse imaging of the formation of the vasculature over 300 mins (zoomed-in rendering of the box in A) at 0, 100, 200, 300 minutes. This show that HyU provides good unmixing at low light levels to permit multiplexing to be used in the observation of development of a live embryo. 54 We utilized HyU’s increased sensitivity to overcome common challenges of multiplexed imaging such as poor photon yield and spectral cross-talk and were able to visualize dynamics in a developing zebrafish embryo. We used a triple-transgenic zebrafish embryo with labeled pan-endothelial cells, vasculature, and clathrin-coated pits (Tg(fli1:mKO2); Tg(kdrl:mCherry); Gt(cltca-Citrine)). Multiplexing these spectrally close fluorescent proteins is enabled by HyU’s increased sensitivity at lower photon counts. The increased performance at lower SNR allowed us to maintain high quality results (Fig. 2.10) while performing faster acquisitions and reducing photon-damage through lower excitation laser power and pixel dwell time. Decreased experimental requirements allow for tiling of larger volumes, extending the field-of-view while still providing enough time resolution for developmental events, even with a high number of multiplexed fluorescent signals. The time-lapses visualize the formation of ventral vasculo-endothelial protrusions acquired in parallel to the development of clathrin and kdrl. HyU enables comparative quantifications of spatio-temporal features, allowing for the determination of volumetric changes over lengthy timelapses, in this case, over the course of 300 minutes (Fig. 2.10 B). [13], [58] 2.3.3 Extract information from intrinsic signal HyU is well posed for the analysis of intrinsically low autofluorescence owing to its ability to operate at low SNR. In Figure 2.3 E-G we visualize unmixing of multiple autofluorescent signals based on spectra acquired from in vitro solutions. For example, the measurement of NADH in its free and bound state is possible thanks to a shift in the emission spectra when NADH is bound to enzymes such as Lactate Dehydrogenase (LDH). Likewise, retinol and retinoic are known to have different autofluorescent spectra. 55 HyU provides the ability to combine the information from intrinsic and extrinsic signals during live imaging of samples, at both single (Fig. 2.3) and multiple time points (Fig. 2.11). The graphical representation of phasors allows identification of unexpected intrinsic fluorescence signatures in a quadra-transgenic zebrafish embryo Gt(cltca- citrine);Tg(ubiq:lyn-tdTomato;ubiq:Lifeact-mRuby;fli1:mKO2), imaged with single photon (488 and 561nm excitation) (Fig. 2.3 A-D). The elongated distribution on the phasor (Fig. 2.3 C) highlights the presence of an additional, unexpected spectral signature, related to strong sample autofluorescence (Fig. 2.3 D blue). HyU analysis of the sample, inclusive of this additional signal, provides separation of the contributions of 5 different fluorescent spectra with residual 3.9%±0.3%. HyU allows for reduced energy load, tiled imaging of the entire embryo without perturbing its development or depleting its fluorescence signal (Fig. 2.3 A). The higher speed, lower power imaging allows for subsequent re-imaging of the same sample, as we report in the zoomed high-resolution acquisitions of the head section (Fig. 2.3 B,E). With the ability to unmix low photon signals, HyU enables imaging and decoding of intrinsic signals, which are inherently low light. Two photon lasers are ideal for exciting and imaging blue-shifted intrinsic fluorescence from samples. Here, the same quadra- transgenic sample is imaged using 740 nm excitation to access both intrinsic and extrinsic signals (Fig 2.3 E-G). HyU enables unmixing of at least 9 intrinsic and transgenic fluorescent signals (Fig. 2.3), recovering fluorescent intensities from labels illuminated at a sub-optimal excitation wavelength (Fig. 2.3 E). The spectra for intrinsic fluorescence were obtained from in vitro measurements and values reported in literature (Methods). For this sample the intrinsic signals arise from events related mainly with metabolic activity (NADH and Retinoids), tissue structure (elastin), and illumination (laser 56 reflection) (Fig. 2.3E). These results confirm our conclusion that HyU is a powerful tool for allowing the imaging and analysis of endogenous labels.[13], [30], [32], [59], [60] Figure 2.11 HyU pushes the upper limits of live multiplexed volumetric timelapse imaging of intrinsic and extrinsic signals. HyU’s increased sensitivity provides a simple solution for the challenging task of imaging timelapse data at 6 time points (125 mins) for both intrinsic signals and extrinsic signals of a quadra-transgenic zebrafish: Tg((cltca-Citrine);(ubiq:lyn- tdTomato);(ubiq:Lifeact-mRuby);(fli1:mKO2)). (A) – (F) Volumetric renderings of HyU results for time points acquired at 25 min intervals reveal the high-contrast and - multiplexed labels of NADH bound (red), NADH free (yellow), retinoid (magenta), retinoic acid (cyan), mKO2 (green), and autofluorescence from blood cells (blue) when excited @740nm. Further extrinsic signals for mKO2 (yellow), tdTomato (magenta), mRuby (cyan), Citrine (green) and blood cells autofluorescence (blue) are also readily unmixed using HyU when exciting the sample @ 488/561nm. HyU provides the 57 capacity to simultaneously multiplex 9 signals in a live sample over long periods of time, a previously unexplored task. Scale bar: 50 µm. Figure 2.12 Application of denoising filters reveals improved results with lower residuals. (A) Residual Image Map of HyU unmixing of a quadra-transgenic zebrafish Gt((cltca:);Tg(Citrine);(ubiq:); (ubiq:lyn-tdTomato); (-ubiq:Lifeact- mRuby);(fli1:mKO2)) inclusive of one strong autofluorescent signal. Residual values are calculated with different number of denoising filters 6 . The average relative residual visibly decreases with the increase of denoising filter numbers. (B) Residual Phasor Map shows a major decrease in values between 0 and 1 denoising filters, maintaining statistically similar values at higher denoising filter applications. (C) Phasor plots for different denoising filters. Phasor of raw data prior to denoising or thresholding, presents noise connected to each of the detectors, as well as a high count area corresponding to the background region. The phasor plot distribution highlights areas with higher pixel counts coming from the background noise, which correspond with lower Residual Phasor Map values in B. (D) Average residual values for Residual Image Map (A) and Residual Phasor Map (B) highlight that the improvement on residuals mostly focuses on the first application of the denoising filter. In this initial denoising, the average relative residual decreased from 69.8% to 46.8%, further decreasing to 42.6% after 5 denoising. Average relative residual for phasor decreased from 33.9% to 7.1% after 1 denoising filter was applied, further decreasing to 2.1% after 5 denoising applied. With standard processing threshold of 250 digital levels applied (bottom 0.38% intensities of 16-bit format), the average relative residual decreased from 7.2% to 4.6%, further decreasing to 4.1% after 58 5 denoising filters. Average relative residual for phasor decreases from 10.1% to 2.6%, further decreasing to 1.1% after 5 denoising filters. Bars denote the variance of the relative residual values. Figure 2.13 Comparison of residual images for LU and HyU highlights improved HyU performance. Residual Image projection for LU and HyU of a 3D dataset of 3 dfp quadra-transgenic zebrafish GtTg(cltca-citrine);Tg(:Citrine);(ubiq:); (ubiq:lyn-tdTomato);(-ubiq:Lifeact- mRuby);(fli1:mKO2)) with an intensity threshold of 250. (A) LU Residual Image Map for a single slice (z=8 of 17 in a z-stack) provides average relative residual of 35.9%, while the (B) corresponding map for HyU averages at 7.1%. (C) Residual Phasor Map for the z-stack presents average relative residual of 1.1%. The reduction in residuals for HyU is maintained across the z-stack, as shown in (D) the average LU and (E) HyU Residual Image Maps built from the average of residuals across all z-slices. The average residual improvement for HyU at 4% compared to LU at 21% is 5.3-fold. Finally, we exploited the HyU capabilities to multiplex volumetric timelapse of extrinsic and intrinsic signals by imaging the tail region of the same quadra-transgenic zebrafish embryo. We excite extrinsic labels at 488/561 nm and the intrinsic signals with 740 nm two photon, collecting 6 tiled volumes over 125 mins (Figs. 2.11-2.13). HyU unmixing in this sample allows for distinction of 9 signals, separating their contributions with sufficiently low requirements to allow repeated imaging of notoriously low SNR intrinsic fluorescence. 59 2.4 Discussion Our results reveal the advantages of Hybrid Unmixing (HyU) over more conventional Linear Unmixing (LU) in performing complex multiplexing experiments. HyU overcomes the significant challenges of separating multiple fluorescent and autofluorescent labels with overlapping spectra while minimally perturbing the sample with excitation light. The chief advantage of HyU is its multiplexing capability when imaging in the presence of biological and instrumental noise, especially at low signal levels. HyU increased sensitivity improves multiplexing in photon limited applications, in deeper volumetric acquisitions and in signal starved imaging of autofluorescence. Our simulation demonstrate that HyU improves unmixing of spatially and spectrally overlapping fluorophores excited simultaneously. The increased robustness at low photon imaging conditions reduces the imaging requirements for excitation levels and detector integration time, allowing for imaging with reduced photo-toxicity. Live imaging on multi-color samples performed at high sampling frequency enables improved tiling to increase the field-of-view while maximizing the usage of the finite fluorescent signals over time. Two- photon imaging of intrinsic and extrinsic signals suggests the ability of HyU to multiplex signals with large dynamic range differences extending multiplexed volumetric imaging into the time dimension. Although improved, images with particularly low signal still present corruption, setting a reasonable range of utilization above 8 photons/spectrum. Simplicity of use and versatility are other key advantages of HyU, inherited from both the phasor approach 33 and traditional unmixing algorithms. Phasors here operate as a spectral encoder, reducing computational load and integrating similar spectral signatures 60 in histogram bins of the phasor plot. This representation simplifies identification of independent spectral signatures through both phasor plot selection and phasor residual mapping, accounting for unexpected intrinsic signals in a semi-automated manner, while still allowing fully-automated analysis by means of spectral libraries. The simplicity of this approach is especially helpful in live imaging where identifying independent spectral components remains an open challenge, owing to the presence of intrinsic signals. High-SNR reference spectra can be derived from other experimental data or identified directly on the phasor. Selection of portions on the phasor plot allows for visualization of the corresponding spectra in the wavelength domain. This intuitive versatility allows for identification of both the number of unexpected signatures and their spectra, a task previously difficult to perform due to noise and lack of global visualization tools. In single photon imaging, HyU phasor allowed identification of a fifth distinct spectral component arising from general autofluorescent background, thereby improving the unmixed results. In two photon imaging, HyU enabled identification and multiplexing of 8 highly overlapping signals possessing a wide dynamic range of intensities, between intrinsic and extrinsic markers. Combination of single and two photon imaging increased the number of multiplexed fluorophores to 9, considering some of the extrinsic labels being excited at two photons. Multiplexing of signals may be further improved by implementing HyU on fluorescent dyes. HyU performs better than standard algorithms both in the presence and absence of phasor noise reduction filters. Compared with LU, the unmixing enhancement when such filters are applied is demonstrated by a decrease of the MSE of up to 21%, with a reduction of the average amount of residuals by 7-fold. Even in the absence of phasor denoising 61 filters, HyU performs up to 7.3% better than the standard based on Mean Squared Error of synthetic data unmixing. This base improvement is due to the averaging of similarly shaped spectra in each phasor histogram bin, which reduces the statistical variability within the spectra used for the unmixing calculations. This averaging strategy works well for general fluorescence spectra owing to their broad and mostly unique spectral shape. In the absence of noise, for example in the ground truth simulations, LU produces an MSE 6-fold lower than HyU. In these noiseless conditions, the binning and averaging of spectra in the phasor histogram, without denoising, provides statistically indifferent values of error respect to LU, suggesting results of similar quality. HyU can interface with different unmixing algorithms, adapting to existing experimental pipelines. We successfully tested hybridization with iterative approaches such as non-negative matrix factorization, fully constrained and non-negative least- squares. Speed tests with iterative fitting unmixing algorithms demonstrate a speed increase of up to 500-fold when the HyU compressive strategy is applied. Due to the initial computational overhead for encoding spectra in phasors, there is a 2-fold speed reduction for HyU in comparison to standard LU. However, this may be improved with further optimizations of the HyU implementation. One restriction of HyU derives from the mathematics of linear unmixing, where linear equations representing the unmixed channels need to be solved for the unknown contributions of each analyzed fluorophore. To obtain a unique solution from these equations and to avoid an underdetermined equation system, the maximum number of spectra for unmixing may not exceed the number of channels acquired, generally 32 for commercial microscopes. This number could be increased; however, due to the broad and 62 photon-starved nature of fluorescence spectra, acquisition of a larger number of channels could negatively affect the sample, imaging time and intensities. Depending on the number of labels in the specimen of interest, extending the number of labels to simultaneously unmix beyond 32 will likely require spectral resolution upsampling strategies. HyU improvement is related to the presence of various types of noise in microscopy images, such as Gaussian, Poisson and digital as well as unidentified sources of spectral signatures. In the multiplexing of fluorescent signals, HyU offers improved performance, quality- and speed-wise in the low-signal regime. HyU is poised to be used in the context of in vivo imaging, harvesting information from samples labeled at endogenous-level. 2.5 Summary In conclusion, we quantitatively show that HyU, a phasor based, computational unmixing framework, is well suited for tackling the many challenges present in live imaging of multiple fluorescence labels. HyU’s reduced requirements in the amount of fluorescent signal permit a reduction of laser excitation load and imaging time. These factors enable multiplexed imaging of biological events with longer duration, higher speed and lower photo-toxicity while providing access to information-rich imaging across different spatio-temporal scales. The reduced requirements of HyU make it fully compatible with any commercial and common microscopes capable of spectral detection, facilitating access to the technology. Our analysis demonstrates HyU’s robustness, simplicity and improvement in identifying both new and known spectral signatures, and vastly improved unmixing outputs, providing a much-needed tool for delving into the many questions still surrounding studies with live imaging. 63 2.6 Supplementary Hybrid Unmixing - Linear Unmixing In the Hybrid Unmixing implementation, Jacobian Matrix Inversion is applied on the average spectrum of each phasor bin with dimensions (c,s,g) where g and s are the phasor histogram sizes and c is the number of spectral channels acquired. The average spectra in each bin is calculated by using the phasor as an encoding, to reference each original pixel spectra to a bin. Resulting ratios for each component channel are assembled in the form of a phasor bin-ratio cube with shape (i,s,g) where i is the number of input independent spectra fp (Linear Unmixing section). This phasor bin-ratio cube is then referenced to the original image shape, forming a ratio cube with shape (t,z,i,y,x) where x, y, z, t are the original image dimensions. We multiply the ratio cube with the integral of intensity over channel dimension of the original spectral cube, with shape (t,z,y,x), obtaining a final result dataset with shape (t,z,i,x,y). HyU Algorithm The pseudo-code utilized for the HyU algorithm is as follows: Input: I(x,y,c,z,t) [5D hyperspectral image] U(i,c) [Reference spectra (n spectra))] Output: I_U(x,y,i,z,t) [Multi-channel unmixed image] 64 Procedure: HYU(I(x,y,c,z,t), U(n, c)) // Single Harmonic Fourier Transform G(x,y,z,t), S(x,y,z,t) = phasor_transform(I(x,y,c,z,t)) // 2D Histogram of G and S values H(g,s) = histogram2d(G(x,y,z,t) S(x,y,z,t)) // Averaging of hyperspectral image over phasor histogram I_H(g,s,c) = phasor_average(I(x,y,c,z,t), H(g,s)) // Linear Unmixing of averaged hyperspectral image I_U(g,s,i) = LU(I_H(g,s,c), U(i,c)) // Reference unmixed phasor image back to original image dimensions I_U(x,y,i,z,t) = reverse_phasor_reference(I_U(g,s,i) return I_U(x,y,i,z,t) Other unmixing algorithms Unmixing algorithms utilized for speed comparisons with the HyU algorithm (Supplementary Figure 13) were plugged in the unmixing step of the analysis pipeline and sourced as follows. Non-negative Constrained Least Squares and Fully Constrained Least Squares from pysptools.abundance_maps (https://pysptools.sourceforge.io/abundance_maps.html). Robust Non-Negative Matrix Factorization[61] python implementation was obtained from (https://github.com/neel- dey/robust-nmf) 65 Data visualization Rendering of final result datasets were performed using Imaris 9.5-9.7. In Figures 2 and 3, contrast settings (minimum, maximum, gamma) for each channel were set to be equal to provide reasonable comparison between HyU and LU results. Gamma was set to 1, no minimum threshold was applied, and the maximum for each channel was set to 1/3 of the maximum intensity. The images were rendered using Maximum Intensity Projection (MIP), and for improving display, they were digitally resampled in the z-direction, maintaining a fixed xy ratio to attenuate the gap generated from sparse sampling z-wise on the microscope. Box plot generation All box plots were generated using standard plotting methods. The center line corresponds to the median, the lower box border corresponds to the first quartile, and the upper box border corresponds to the third quartile. The lower- and upper- line extensions correspond to one and a half times the interquartile range below and above the first and third quartiles respectively. Timelapse registration A customized python script (Supplementary Code) was first utilized to pad the number of z slices across multiple time points, obtaining equally sized volumes. The “Correct 3D drift” plugin[62] (https://imagej.net/Correct_3D_Drift) in FIJI[63] (https://imagej.net/Fiji) was used to register the data. Timelapse statistics 66 Box plots and line plots for timelapses were generated using ImarisVantage in Imaris 9.5-9.7. Box plot elements follow the same guidelines as described above. Line plots are connected box plots for each time point with the solid line denoting the median values, and the shaded region denoting the first and third quartiles. Code availability All the relevant code is available from the corresponding author upon reasonable request. Software and instructions can be downloaded from http://bioimaging.usc.edu/software.html#HySP. Data availability All the relevant data are available from the corresponding author upon reasonable request. Datasets for Figs.1–6 and simulations are available for download at http://bioimaging.usc.edu/software.html#sampledatasets in the samples section. Performance quantification Mean Square Error For synthetic data, a ground truth is available for comparison of unmixing fidelity between HyU and LU. fp contributions, or ratios, were used for quantification, owing to the arbitrary nature of intensity values in microscopy data. We utilize Mean Square Error (𝑀𝑆𝐸) for determining the quality of the ratios in synthetic data. We define MSE as the square difference of the ratio recovered by an unmixing algorithm (r unmixed ) and the ground truth ratio (r) divided by the total number of pixels (n). 𝑀𝑆𝐸 = ! $ |𝑟 %$&#'() −𝑟| " (3) 67 To simplify comparison between different unmixing algorithms, we define Relative Mean Square Error (RMSE) as: 𝑅𝑀𝑆𝐸 =6 *+, !" *+, #$" −17∗100% (4) RMSE measures the improvement in MSE when using HyU as compared to LU. Residuals For experimental data, in the absence of ground truth, we quantify the performance of the results returned by the unmixing algorithms with the following measurements: Average Relative Residual, Residual Image Map, Residual Phasor Map, and finally, Residual Intensity Histogram. Residual (𝑅) is calculated as: For image: 𝑅(𝑥,𝑦,𝑐,𝑧,𝑡)=𝐼 -./ 1&.2( (𝑥,𝑦,𝑐,𝑧,𝑡)−𝐼 3$&#'() 1&.2( (𝑥,𝑦,𝑐,𝑧,𝑡) (5) For phasor: 𝑅(𝑔,𝑠,𝑐)=𝐼 -./ 1&.2( (𝑔,𝑠,𝑐)−𝐼 3$&#'() 1&.2( (𝑔,𝑠,𝑐) (6) The spectral intensity difference between the unmixed image and original image for each pixel or phasor bin depend on the following descriptions of the intensity image (I), where: 𝐼 -./ 1&.2( =∑ 𝑟∗𝑓𝑝 # # +𝑁 (7) 𝐼 3$&#'() 1&.2( = ∑ 𝑟 %$&#'() ∗𝑓𝑝 # # (8) The original spectrum (𝐼 -./ 1&.2( ) is the combination of each independent spectral component (𝑓𝑝) with its ratio (𝑟) plus noise (𝑁). The recovered spectrum is obtained by the multiplication of recovered ratios (𝑟 %$&#'() ) with each corresponding individual component. 68 Relative Residual (RR) is calculated as the sum of the residual values over C channels and normalized to the sum of the original intensity values over C channels (with C = 32 in our instrument). 𝑅𝑅(𝑥,𝑦,𝑧,𝑡)= ∑ -(',7,8,9,:) % &'( ∑ 1 )*+ -.*/0 (',7,8,9,:) % &'( (9) 𝑅𝑅(𝑔,𝑠)= ∑ -(2,<,8) % &'( ∑ 1 )*+ -.*/0 (2.<.8) % &'( (10) The Average Relative Residual (Figure 2.13) provides a single comparison value for evaluating the performance of different processing methods on the same data, such as the application of multiple filters, applying of various threshold values, and variations in the number of components estimated. Average Relative Residual (𝑅𝑅 .>2 ) is defined as the average of the relative residual for every pixel in the image or every phasor bin in the phasor histogram. For image: 𝑅𝑅 .>2 = ∑ ∑ ∑ ∑ -- 1 $ 2 3 '79: (11) For Phasor: 𝑅𝑅 .>2 = ∑ ∑ -- / 4 2< (12) The Residual Image Map visualizes the residual values for each pixel of the image. Regions with higher residual values appear to characterize portions of the dataset with increased amount of noise or where an unexpected spectral signature is present. Residual Image Maps (𝑅 #&2 &.? (𝑥,𝑦)) project the Relative Residual (RR) cube to the 2D image shape for each voxel, providing an estimated visualization of an algorithm ratio recovery performance in the spatial context of the original image. 𝑅 #&2 &.? (𝑥,𝑦)= ∑ 𝑅𝑅(𝑥,𝑦,𝑧,𝑡)∗100 9,: (13) Residual Phasor Map visualizes residuals for each bin of the phasor histogram (Figure 2.13). These maps allow for insights on where HyU unmixing results have 69 reduced performance in phasor domain and indicate phasor locations of unexpected additional spectral components. 𝑅 ?@ &.? (𝑔,𝑠)=𝑅𝑅(𝑔,𝑠)∗100 (14) The Residual Intensity Histogram 𝑅 1$: A#<: (𝑝,𝑟𝑟) calculates the distribution of the relative residual in relation to intensity overall all pixels or all phasor bins. Higher residuals appear to be present in regions with lower signal intensity and SNR, providing degraded performance. For image: 𝑅 1$: A#<: (𝑝,𝑟𝑟) =𝑐𝑜𝑢𝑛𝑡G𝑃(𝑥,𝑦,𝑧,𝑡),𝑅𝑅(𝑥,𝑦,𝑧,𝑡)I ?,BB (15) For phasor: 𝑅 1$: A#<: (𝑝,𝑟𝑟) =𝑐𝑜𝑢𝑛𝑡G𝑃(𝑔,𝑠),𝑅𝑅(𝑔,𝑠)I ?,BB (16) 𝑃 = ∑ 1 )*+ -.*/0 % &'( C∗<E (17) Where p is a bin of the histogram P, rr is a bin of RR, and sf is the factor which converts the number of photons to digital intensity levels. 70 Figure 2.14 Schematic overview of residual calculation. Image residual is the residual for image (x,y). (A) Raw hyperspectral data cube in dimension of (x,y,λ). x,y is the spatial dimension. λ is the wavelength range from the spectral channel on the detector. (D) Recovered model in dimension of (𝑥,𝑦,𝜆) comes from (B) the product of Recover ratio (𝑥,𝑦,𝑐ℎ) and Independent spectra (𝑐ℎ,𝜆). 𝑐ℎ is the number of independent spectra or unmixing component. (C) Residual is the difference of Recovered model and Raw data. Same logic for Phasor residual, but instead of (𝑥,𝑦) the dimension of Phasor is composed from the real and imaginary Fourier components (𝐺,𝑆). Image Contrast 71 Image contrast measures the distinguishability of a detail against the background. Here we use percent contrast to refer the relationship between the highest and lowest intensity in the image. 𝐶𝑜𝑛𝑡𝑟𝑎𝑠𝑡 = 1 4 F1 5 1 5 Where the Intensity of signal average (I s ) is the average of top 20% intensities in the image. The Intensity of background average (I B ), the average of bottom 20% image intensities. 72 Chapter 3 A framework for modeling spectral fluorescence Microscopy imaging has moved toward more than just the resolution but the information extraction. Novel image processing tools facilitate the information extraction from these data. A simulation framework that provides plenty of data with ground truth is missing. The motivations for building this simulation framework are mainly for two reasons: 1. Without a known ground truth for biological data, it is challenging to compare results or quantify the improvement of the fast-developing algorithms. 2. The new trend applies machine intelligence, especially neuron networks or deep learning, to recognize the pattern from complex biological data. These methods require a large number of training data. A better way to generate the microscopic data without sample preparation and image acquisition would facilitate the field to move faster. 73 Figure 3.1 Schematic overview of simulation framework. 74 3.1 Generating 1D fluorescent signals To generate a bio-pattern-realistic simulation, we replicated the biophysics of how fluorophores are excited in a microscope through 3 steps. First, we generated a realistic- photon-distribution image and ground truth ratio from multiple single label experimental data. Next, we simulated the signal in each pixel based on the type and number of fluorophores assigned from step one. Lastly, we simulated the process of how signals are collected in a microscope from photon to digital level and the corresponding noise adding through each path. Overall, simulation can be broken down into 3 aspects -- the simulation of fluorescence emission, the simulation of acquired spectra from microscope, a realistic image. [64], [65] 3.1.1 How we simulate a fluorescent signal Creating an ideal simulation for spectral emission datasets requires a proper modelling of how fluorophores signals are generated and collected within a microscope. A fluorophore is excited by a laser at a specific wavelength and emits a photon in relation to the energy imparted onto the fluorophore. This photon emission is a stochastic process, with the emission of many photons from the same excitation energy having differing levels of energy and therefore different emission wavelengths. This emission profile is called the emission spectrum of the fluorophore, and it is a probability distribution function where photons are distributed at different energy levels (wavelengths). The ideal spectrum of a fluorophore considers the case where an infinite number of photons are emitted. However, the limitation of energy input during live imaging where laser power must be kept low in order to protect samples from photo-toxicity and prevent photo-bleaching of fluorophores. This low input energy translates to a limited number of excited photons. 75 Since the spectral emission profile is a probability distribution arising from stochastic emission, having a low number of photons causes a large deviation from the ideal emission profile of the fluorophore. In contrast, the measured spectrum of the fluorophore at any one point (experiment, sample) will be a distribution of the PDF, which is limited by the finite number of photons emitted for the specified laser power. 3.1.2 How we simulate fluorescence signals collected in microscope In laser scanning microscopy systems, as mentioned in Chapter 1, photons emitting from the sample travel through objective, dichroic mirrors and finally reach detectors. Detectors acquire these photons either by collecting all photons going through a specific optical filter. Each channel or detector collects a certain number of photons within the given timeframe, known as pixel dwelling time. Finally, the counted photon is converted to digital level as intensity of the image. Hence, the simulation begins with an input value representing the total number of photons (TP) emitted by a virtual fluorophore. The TP is fed into a random number generator constrained by the corresponding PDF of the virtual fluorophore. The random number generator produces a distribution of numbers ranging from one to the total number of detector channels (NDC). This distribution is then organized into a 1D histogram with the number of photon counts recorded for each detector channel. This process simulates the stochastic emission of a limited number of photons for a fluorophore, so we call this the stochastic spectrum (SS) which is an array of integer values of size NDC. 76 3.1.3 Adding noise step by step Unlike the literature emission profile of a fluorophore, the factors mentioned above in the microscope affect the acquiring signal. In other words, any output collected through a microscope suffers from various amounts of noise which obscure the original signal input. The ultimate signals collected in the microscope are at the digital level. Various forms of noise such as poisson noise from the dark current, sensor noise, and salt and pepper noise, accompany the conversion of photon detection to digital levels when photons are collected at the microscope detector. Hence, the mentioned SS is converted from photon values to digital levels in detectors. This process is first accompanied by the addition of Poisson noise, which occurs when the incidence rate of the photons is being counted by the detector. This signal is then converted from photon values to digital levels by multiplying by the conversion rate. Since the conversion from photons to digital levels is also accompanied by noise, Gaussian noise with zero mean and standard deviation from the sqrt of the digital level is then added to the converted spectrum resulting in a digital spectrum (DS). Next, a dichroic mirror, which is used to block the excitation laser light, in the microscope blocks photons in the specific wavelength range of the excitation. Hence, a portion of the emission profile is also removed. This application of the optical filter is simulated by multiplying the DS with a float value for each channel corresponding to the transmission rate of the photons through the optical filter resulting in an optically reduced digital spectrum (ODS). Note this operation is taking place here because of how the filter multiplication affects photon integer level vs digital level. 77 Finally, gaussian noise with mean and standard deviation for each channel taken from calibration measurements corresponding to the specific confocal microscope is added to the ODS. Figure 3.2 Zeiss 488/561 bandpass filter shows how signal got affected by optics. Different filter setting in experimental data affect the signals in different ways. In our example here, a 488/561 filter is used to block the parallel lasers in use, these filters block light in the corresponding wavelength in different percentage. Figure 3.3 Four emission spectra before and after adding 488/561 filter. The emission spectra are citrine, mKO2, tdTomato, mRuby respectively in orange, green, blue and red. Here the spectra are collected with and without applying filter during imaging. 78 Figure 3.4 Point spread function (PSF) is a one of the critical noise greatly affect our signal in the experimental data. After adding PSF, intensity count distribution starts showing gaussian distribution (right) from original discrete signals (left). Figure 3.5 Compare experimental data and simulation data in different levels of poisson noise. The level of poisson noise applied would affect the intensity count distribution. 79 3.1.4 Finding conversion rate Figure 3.6 Intensity count distribution from a single detector. In a normal detector, the detector collects photons and converts them into digital levels. Instrumental noise, or readout noise also happens in this process. Therefore, detector offset, readout noise, and conversion rate can be calculated from the intensity count histogram distribution. Figure 3.7 Two ways to calculate S_factor. S_factor can be found by collecting dark current distribution of DL or measuring the ratio of variance versus intensity. We failed to use the first method because in the 80 commercial microscope we used, the collected signals are corrected already. Single channel distribution is shown in (A). The detectors are very sensitive and less noise is observed and therefore the conversion rate is hard to observe. (B) Second method is to measure the variance versus intensity in each channel. Here we calculate the s_factor in different gains . One big challenge when simulating on a digital level is determining the conversion rate. Conversion rate helps to convert from photon to digital. The conversion rate is a property inherent to the detector of any microscope and is key to matching the correct digital levels when simulating the output of the user’s specific setup. The method we used to measure the conversion rate (s_factor) was to determine the ratio of the signal variance versus signal intensity of images collected by the microscope as described in the work by Dalal et.al. This method requires the collection of stable fluorescent spectra signals from multiple chroma slides covering different emission ranges at a specific gain and different laser powers. By varying the laser power, multiple points can be plotted for the variance vs average digital levels, leading to a value for the s_factor for each channel of the detector. Finally, each s_factor was multiplied by the quantum efficiency of each channel to determine an approximately constant conversion rate. For this work, we determined the conversion rate for a commercial LSM 880 microscope.[40], [50],[66] For this work, since our experimental data was collected from a commercial LSM 880 microscope, we determined the conversion rate for this specific set up. With the Zeiss LSM commercial microscopes, preprocessing is performed on the output images to reduce noise which remove the count vs intensity information necessary for calculation of the s_factor; therefore, to avoid calculating the detector correction factor from build-in software of microscopes 81 3.2 Generating 2D fluorescent image After we demonstrated the ability to simulate a single emission spectrum with a given combination of fluorophores and account for a variety of noises. Next step is to assign them to a 2D image and form a 3D cube that makes sense. 3.2.1 The necessity of realistic-bio-pattern simulation Figure 3.8 block simulation of a non-repetitive ratio of combinations in gradient. The true distribution in biology is missing in the most intuitive blocks simulation. Here, we simulated Six C(4 2) blocks with four single fluorophore emission and combination of 2 and 3 and in consideration of the effect from noise. Each pixel has a different ratio combination where we fail to account the power of grouping pixels with similar spectra. 82 3.2.2 Figure 3.9 Simple geometry simulation. In the simple geometry simulation, a visualizable unmixing result is clear but the uniform intensity fail to replicate the realistic case. We then simulated overlapping geometry simulation, where the simulation has patterns and is easier for visualization. This simulation demonstrates the advantage of grouping pixels to replicate reality. However, this has the disadvantage of 1. less combination ratios 2. constant photon counts, which is not the realistic case. 83 Figure 3.10 realistic simulation from experimental data. Even with experimental image mask, the ratio is limited to the designed combination and a true ratio is missing in the region signal is not presented in the original mask. We then used experimental data to generate the photon mask, which would account for the dynamic intensity distribution of real biological data and have complicated patterns that are similar to reality. We overlay different masks in different ratios. However, because of the dynamic intensity distribution of each image, each pixel has a different number of photons, leading to the ratio we put not the ground truth from real data. Therefore, we need to come up with a solution to simulate a photon mask where we have control of how the photon is assigned. 84 3.2.3 How image masks are generated To generate biologically relevant fluorescent images, input images containing the proper spatially organized photon values are necessary as a template for each simulated fluorescent biological marker. The values for each pixel in these input images are the total number of photons (TP) as described in the previous section. Experimental hyperspectral datasets are used to generate these input images, which we call photon masks. For constructing photon masks with more signal compared to background, some experimental datasets were first zoomed-in and cropped around distinct patterns. Starting with a single image slice or hyperspectral cube (x,y,lambda), a summation of intensity values is performed across the lambda dimension to create a proper distribution of intensities across x and y (referred to as the intensity mask). This intensity mask is then thresholded using a manually determined threshold value before being max scaled to a predetermined photon value. This output is the photon mask, and an individual photon mask is generated for each fluorophore that will be simulated. With these photon masks, we have the exacting mapping of photon number to pixel for each simulated signal, allowing for an easy calculation of the ratio for each independent component independent component needs to be defined somewhere by dividing the number of photons per label by the total number of photons from all labels. 3.2.4 Realistic bio-pattern image With the photon masks and spectral simulation, it is possible to then generate the necessary biologically relevant fluorescent simulation images fully. Each pixel of the photon mask undergoes the spectral simulation process described in section 2.4. However, 85 once the stochastic spectrum is generated, first a random combination of background biological signals is generated and added to the stochastic spectrum for each pixel. Then, we consider the effect of the point spread function (PSF), coming from background noise of the sample, photons intruding from other z slices, would also affect the final image by simulating a two-dimensional Gaussian kernel is applied for each of the 32 channel images, emulating the interaction of photons across neighboring locations (pixels). Finally, salt and pepper noise occurs randomly on the image in the photon to digital conversion process due to the fluctuations in the detector. We simulated this noise with a constantly set SNR (0.92). Likewise, the rest of the spectral simulation takes place, creating the optically reduced digital spectrum for each of the pixels in the photorealistic simulated hyperspectral dataset. Figure 3.11 Schematic of generating photon masks for getting a realistic ratio with ground truth. 86 Spectra image cube was taken from experimental data. We sum over intensity channels to get a total intensity channel from all the contributed photons. Then by dividing the photon conversion rate, a photon mask with single channel saved the number of photons that is going to assign to individual simulated signal. 3.3 Simulation provides ground truth quantification of algorithms 3.3.1 Comparison of microscopy signals and simulated signals With this project, we proposed a framework to build a simulation with any given combination of fluorophores. We demonstrate a simulate with a single signal and four signals in the example here. We first use calibration Chroma slide with 405 emission spectrum to compare simulation and experimental data. Chroma slide has uniform and consistent emission signals. We plot out random 5 spectra from simulation with ground truth in different photon fluctuation and experimental data. Here we showed that we are able to simulate the poisson noise very similar to experimental data with different levels of intensity and a known ground truth, as shown in Fig. 3.12. The average of spectra from the whole experimental dataset is shown in red line in Fig 3.12. D while the average of spectra from the whole noisy simulation and ground truth are shown in green and blue lines respectively. The green line and blue line completely overlap. The average of experimental data and simulation data show no distinct difference. This indicates our simulation strongly interprets the experimental data with the biophysics property from the microscope. 87 Figure 3.12 Simulation data matches experimental data. Here we test the simulation on chroma slide 405. Five random spectra are displayed from(A) Simulated data for Chroma slide 405 (B) Simulated ground truth (C) Chroma slide 405 collected from Zeiss LSM 880 (D) average of whole dataset from (A)(B)(C). Now, we move from a single fluorophore simulation to a four-fluorophore simulation using Citrine, mKO2, tdTomato, mRuby emission spectra. We plotted 10 random spectra in Figure 3.13. We are able to simulate simulation in different combinations and the many noises including optical filter that disrupt the signal. 88 Figure 3.13 Four fluorophore simulation with ground truth in different intensity and combination 3.3.2 Comparison of images from experiment and simulation We first demonstrate the simulation from 1D signal perspective, now we are comparing the simulation result with experimental data in 2D image. Here we replicated the intensity dynamic distribution from our biological realistic photon mask. From the histogram distribution, our simulation is slightly noisier than experimental data. Figure 3.14 2D simulation from biological realistic photon mask. Simulation (left) replicates the pattern in experimental data(right) with a nosier photon distribution plot from a single channel(ch22). The gaussian shape in the experimental data is the true signal while the signal in simulation is blend in with noise. 89 3.3.3 Quantification on algorithm performance As mentioned in chapter 1, it is challenging to define a ground truth in a biological sample, which makes it difficult to quantify the performance of image processing tools. Now with the proposed simulation, we are able to precisely control the type, intensity level (or number of photons in each pixel) and quantify the errors as shown in Figure 3.15-3.17. Figure 3.15 Comparison of unmixing results for synthetic data at different SNR demonstrate improved HyU performance. Ground truth photon mask of the four independent fluorescent signals, (A) mKO2, (B)Citrine, (C) mRuby, and (D) tdTomato for synthetic data. (E) The maximum intensity 90 projection (MIP) of the simulated 32 channel hyperspectral image generated from the four ground truth masks at low signal-to-noise ratio (SNR). In this case, a maximum of 10 photons are simulated for each fluorescent component. (F-I) Grayscale representation of the maximum emission channel of each component, based on the respective spectra. Unmixing result of (J) LU and (K) HyU for simulations with a maximum of 10, report decreased performance. In the ultra-low SNR simulation (5 photon at most for each component), both LU (I) and HyU (M) results are deteriorated, however HyU maintains a 1.5x lower averageMSE compared to LU. Figure 3.16 Quantification of HyU vs LU unmixing results for synthetic data highlight increased HyU performance. HyU performance is evaluated under several algorithmic parameters and experimental conditions. (A) Relative MSE between HyU and LU was calculated as a function of max input photons/spectrum over 5 denoising filters for HyU. The improvement increases both with the number of photons and the number of denoising filters, showing significant differences above 7 photons/spectra with peak at 124%. Shaded regions denote the 95% confidence interval around the mean. (B) Absolute MSE from LU and HyU algorithms for the same synthetic dataset with and without beam splitters. The addition of optical filters causes the MSE of LU to increase on average by 8% , compared to an average increase of 5% for HyU. (C) Average relative residual of synthetic data with and without beam splitters with increasing level of denoising. The average relative residual without beam splitters with denoising (HyU-filt1x – HyU-filt5x) is 83%, compared to 109% for LU. In the absence of denoising filters (filt0x) the average relative residual is 92.9%. Beam splitters were applied in this simulation and both Mean Squared Error (MSE) and residual values were calculated with and without beam splitters. (D) Simulated spectral with beam splitter and (E) simulated spectral without beam splitter are shown. 91 Figure 3.17 Residual analysis for synthetic data identifies locations with reduced algorithm performance. Simulated data in Figure 2.2 A,B for four fluorescent labels (Citrine, mKO2, tdTomato, mRuby) is analyzed with LU and HyU. Unmixing results for (A) HyU and (B)LU. 92 Residual Image Map for (C) HyU and (D) LU results presents regions with higher residual (red) along the boundary between the sample’s labelled features and the background, where signal-to-noise drops. The average residual values for LU (118%) are higher than HyU (94%). (E) Residual Phasor Map shows higher residual for the background region (arrow), consistently with the results in C, D. The Jet colorbar scale corresponds to C, D and E. (F) Phasor Residual Intensity Histogram maps the average photon counts in each histogram bin in E between 0 and 50 photons and presents a trend of decreasing relative residuals with photon number. The (G) Average Relative Residual plot shows higher values for LU compared to HyU with different denoising filters applied. Ground truth values are also included for comparison. Box plot elements are defined as described in Methods. An (H)original phasor plot with 0 threshold and 5 denoising filters applied is presented. The ROI (yellow circle) highlights the background pixels in yellow in the (I) average spectral intensity image. The noise from background and residual can be decreased considerably with an intensity threshold. 3.3.4 Summary We create a simulation framework that is able to simulate any fluorescence microscopy data given the emission spectra from fluorophore and conversion rate from the microscope in use. We consider the biophysics in fluorescence microscopy and account for noises including Poisson noise, salt and pepper noise, readout noise, and PSF. We can simulate data in 1D spectra, a biological realistic 2D image, and the combination 3D cube. This simulation framework has the flexibility in fluorophore type, number of fluorophores, spectral channels (here we use 32 channels for Zeiss microscopes), level of different noises, filters in use. This simulation provides a method to quantify performance of developing analysis tools and save researchers time and energy from conducting imaging experiments including generating transgenic samples, phenotyping them, figuring out the imaging setting and spending days or months to get data for the purpose of developing image or signal processing tools. In addition, the capability of generating large amounts of data with ground truth enables the utilization of machine learning in related fields. 93 Chapter 4 Spectral Denoise 1D-Unet for prediction of ultra-low SNR signals Multiple spectral imaging opens another door for differentiating different materials, chemical states, or multiple labels. Spectral imaging adds an extra wavelength dimension to the 2D image. The adding spectral information preserves the distinct optical properties of interest targets and allows better classification or mapping of heterogeneous distribution or unmixing overlapping emission spectra. Applications include remote sensing, agriculture, quality control, sustainability, and life science. Standard spectral data analysis includes linear decomposition, phasor analysis, support vector machine, or other machine learning methods. However, these approaches have limited generalizability or suffer from information loss. Moreover, these analyses have compromised performance with the presence of noises. Noises are inevitably present in imaging, especially in life science, where understanding complex biological systems require tracking dynamic interactions in live samples. In fluorescence microscopy, noises vary from biological noise, readout noise to environmental noise. These noises greatly limited the information extraction of spectral data. For example, noise disrupted the shape of collected signals and broke the linearity of the system, which is a vital assumption for the standard unmixing algorithm, Linear Unmixing (LU). The results of LU are highly dependent on the input spectra and independent components, where a slight change due to the noise may return inaccurate results. Hence, a decent spectral noise reduction method could potentially fully unleash the power of extracting information from spectral imaging. 94 4.1 Current noise reduction methods and challenges Multiple groups attempt to computationally increase the SNR or decrease the noise, so-call denoise. Given the ultimate goal of imaging live samples, the use of laser light is limited in low power to prevent photon bleaching and photon toxicity, leading to an incredibly high noise environment. Traditional image denoising methods including filtering methods, wavelet transforms, wiener filter, total variation minimization. After 21th century, the gaussian scalar mixture algorithm and nonlocal mean algorithm and, more advanced, adaptive nonlocal means algorithm were developed and widely used. Recent days machine learning or deep learning denoising have become a popular method to increase image quality, reduce noises, or conduct recognition and classification. However, current approaches train ML and NN directly on images, using 2D convolutional layers which embed the spatial characteristics of the sample. While this approach has succeeded for published applications including disease classification, segmentation spatial features, improving image quality, and predicting imaging modalities, it has some major drawbacks. First, it cannot be generalized when applied to fluorescence microscopy, where the large range of intrinsic and genetically encoded labels offers a wide range of spatial patterns, requiring that a new network be re-trained for each new experiment. Second, it cannot predict new patterns, as the image-based training aspect of the current ML/NN training process biases the system against the discovery of new patterns (e.g. from genetic mutations), as it forecasts learned information onto new imaging data. Moreover, in the standard CNN or deep learning, it is not clear what happens in the steps in the middle, making it a black box unable to interpret. Therefore, current machine learning or deep learning networks are very specific to certain tasks and are mostly image-based so they learn spatial features that limit to the input image. Most of the work is about fine tuning the hyper-parameter and structure. The applications are not versatile. [44]–[47], [66]–[73] 95 4.2 Sd-Unet is designed for denoising fluorescent hyperspectral We designed a versatile deep learning network which enables recovering low SNR fluorescence spectral data based on biophysical properties. Firstly, our model is a 1D convolutional network. It is free of the 2D spatial features in the input sample. This pixel- based structure leads to a more general type of input and has a wide range of applications. Secondly, we innovatively incorporate a customized loss function to check the grouping loss of the multi-channel pixels. Inspired by the concept of Phasor, this proposed structure groups the pixels with similar spectral shapes which encode biological information. In other words, the design takes the biological signatures within each pixel into account. Lastly, the proposed network is more adaptable to the existing unmixing computations in fluorescent microscopy by adding on prior fluorescent information. This flexible, biologically interpretable, and efficient network achieves fast denoising of any type of fluorescent spectra data with myriad biological targets or applications. 96 4.2.1 Neuron network background: CNN vs Unet Figure 4.1 The encode-decode structure of CNN. (Image source: Applied Deep Learning from Arden Dertat) Convolution neutron network (CNN) is an encoder-decoder structure, one encoder to encode the features to lower dimensional features map and one decoder to map the features into several probabilistic maps (number of classes) of same width and height as input using transposed convolutions. The output size of the convolutional layer first shrinks then expands. 97 Figure 4.2 U-net structure (image source: Ronneberger et al. [74]) Unet is a special case of the encode-decode structure of CNN, following the encode-decode structure. In addition, it adds cross paths between the corresponding encoder and decoder layers to increase the capability of detail expression for the model. U-net was originally designed for image segmentation. Because of its fine designed structure, it is now also widely applied in label free prediction and denoising. The architecture is dedicated to encoding multiple spatial feature channels starting from a single spatial channel image. 98 4.2.2 Spectral denoise 1D-Unet (Sd-Unet) structure Figure 4.3 Schematic overview of the SdU-net structure. A modification of Unet with 7 layers and 1D input. Unet was originally designed for 2D image segmentation on spatial features rather than on spectral features. We modified a 1-dimensional U-net and designed a 7 blocks architecture with 4-layer encoder and 3-layer decoder, suitable for denoising spectral data. The architecture is divided into two structures: encoder and decoder. The encoder summarizes the input information on different scales. The decoder remaps the features to generate an output map using transposed convolutions. The output shape for each 99 convolutional layer first reduces then expands, increasing the network’s robustness to features from edges to semantics. This 1D-Unet creates cross paths between encoder and decoder to increase the model’s capability for synthesis of detail. The additional dropout layers together with other regularization methods prevent the overfitting. The tunable parameters include lambda loss, layers, kernel size, filter size and dropout, which are around 153k parameters in total. The initial test on using 1D-Unet shows a bigger improvement than traditional denoising methods such as median filter or PCA, as shown in Figure 4.4. Figure 4.4 Comparing Sd-Unet denoising with traditional denoising methods. Sd-Unet shows a six times improvement comparing to Median filter and a three times improvement comparing to PCA. We utilized Keras57 in combination with Hyperopt (http://jaberg.github.io/hyperopt/) for creating the ML architecture and performing distributed asynchronous algorithm configuration and hyperparameter optimization. We 100 utilized Tree-structured Parzen Estimator (TPE) that considers the dependency between hyperparameters. 4.2.3 Customized loss function Typical loss function, determine how close the output is to the original ground truth which helps converge the network to get closer to the ground truth, calculates the mean square error(MSE) of the prediction and ground truth data, as equation (4-1) Traditional loss function = MSE (spectrum) (4-1) Our network is designed for fluorescent microscopy data, where individual shape of the spectrum is an important feature. In order to account for that, we customized an optimized loss function to integrate the Phasor information G/S value while reconstructing the spectrum shape. Therefore, for our loss function, we calculated not only the mean square error of the spectra intensity over channel space but also the mean square error of how G, S values have been changed between the ground truth and prediction. The equation is as follow(4-2). Sd-Unet loss function = MSE (spectrum) + γ"MSE(G/S)" (4-2) By using this customized loss function on our initial testing data, the best validation loss of epoch decreased to 0.00119 compared to 0.00135 when MSE is the only loss function, which is a 1.1 fold improvement. 101 By constructing the network in this way, incorrect spectra shape error in the prediction is better predicted as shown in Figure 4.5. Figure 4.5 Customized loss function improves the prediction of spectral shape. Purple circle highlights a previously incorrect spectral shape which is corrected after applying customized loss function. 4.2.4 Training and testing data We generated fluorescent spectra simulation at different SNR conditions following the framework in Ch3. We simulated data of photon range from 3 -300 counts per spectra in all ratio combinations with multiple points for each combination. One thing special about the fluorescent microscopy data as previously mentioned, there is a dichroic mirror in the microscope that blocks light with certain wavelengths. We are curious to see if machine learning can learn to recover the signal loss because of the optical filters. Here 102 we test five different inputs of testing and training data. Fluorescent data with full emission spectra, fluorescent data account for imaging filter setting, intrinsic fluorescence data, combination of all, as shown in the table 2. Table 2 Five different input training and predicting set. According to our test result the best training input and ground truth pair is without any preprocessing and postprocessing to recover the signal loss from optical filters. As shown in Table3, the network trained with ground truth and output with optical filters returned the lowest MSE and gs error among all the pairs. The metrics used are reported in the next section. Table 3 Network performance for different training and predicting pairs. 103 We also test on the best training range to train the network- training by giving 1500 spectra in each photon range (15000 spectra in total), training with a total of 1500 spectra, training with 150000 spectra in total. Then we plot out the MSE performance and GS performance in different photon range for 3 combination of training number and range. Training with each batch of range returns the lowest performance in average. However, in 104 the lowest photon range (below 30 counts) it has the greatest MSE improvement but not with GS performance. The training set with 1500 total on the whole range test if a lower number of training set return a more generalized network and would prevent overfitting. Note in the 90-120 and 210-240 photon range, its GS improvement outperforms the other two methods. Overall, training set with 150000 in total on the whole range gives the highest improvement. Figure 4.6 Network performance from different number of number of spectra and training range. MSE performance is calculate from the MSE improvement after applying the network. GS performance is calculating the GS MSE improvement after applying the network. Blue line is training with 1500 spectral on each range. Orange line is a total of 1500 spectra train directly on the network. White line is 150000 spectra in total. 4.2.5 Performance evaluation To evaluate the performance, we compare the MSE from raw intensity values and phasor transform values (GS values) of spectra in each pixel between the original noisy data and the Sd-Unet prediction. Moreover, we calculated G/S shift error to get an extra evaluation of how the emission spectra change. We are able to convert the intensity value with photon counts and measure the performance of photon range in relation to MSE of intensity and phasor as well as GS shift error. [52] 105 Figure 4.7 The final Sd-Unet training convergence example with the best validation loss of 0.00049. The best validation loss 0.00049 from our final network. 4.3 Sd-Unet denoises noisy signals 4.3.1 Simulation data We use a newly generated simulation where we have the noisy inputs and their ground truth as described in Ch3 to test out the performance of the network. We test on simulation with photons randomly distributed in the range of 30-300 and print out the performance in each range shown in table 4. The network demonstrates an average of 7- fold improvement in intensity MSE. Worthy notice, the GS improvements, which represent the shape of spectra, are more distinct in the lower photon range. The GS MSE improvement is at average of 2 and 3.8 in the lower SNR(photon < 40). GS shift error improvement of at least 1.4-fold across all photon range and an especially high 106 improvement of 5.5 for the low SNR (photon < 40). Here we plot out 10 random spectra with the original input noisy simulation, Sd-Unet prediction and the ground truth shown in Figure 4.8. Figure 4.8 Sd-Unet example on a super low photon (<40 counts) testing data. Table 4 Performance of Sd-Unet on simulation data 107 The previous simulation was done on the standard condition; however, we also want to see what the limitation of the network is. Here we examine the network in an 108 extreme condition with extremely low SNR data, intensity lower than 1600 (signals with less than 3 number of photons per channel) and high salt and pepper, and readout noise, as shown in Figure 4.9 (left). The predicted result perfectly removes the noise and recovers a more reasonable spectral signal. This proves the network works in an extreme condition. Figure 4.9 Sd-Unet recover an extreme example of highly noisy data. 4.3.2 Experimental data Next, we validate the network on experimental data. In this experiment, we acquired an image from a transgenic zebrafish with Td-Tomato label within the same image frame in different SNR. We changed the average frame of each acquisition: one average for lowest SNR, two averages for low SNR, four averages for normal SNR, and lastly eight averages for the high SNR. As previously mentioned, the ground truth does not exist in real data. But since a single-color transgenic line is used here. The Td-Tomato emission spectra from literature can be found online as described in Ch1. We plot a single spectrum from the same voxel before and after applying Sd-Unet on these four different SNR data as shown in Figure 4.10. The spectra from normal and high SNR are clearly in the shape of mCherry spectrum and even with the lowest SNR recover data, the shape looks closer than mCherry comparing to before applying the network. 109 Figure 4.10 Sd-Unet recovers signal in different SNR from experimental data. Experimental data was collected in Zeiss LSM 880 from a Tg(Td-Tomato:Lifeact) zebrafish embryo. Multiple acquisition with different averaging frame and laser power were applied on the same position to get data in different SNR. The highest SNR data supposed to serve as the ground truth. However, even with 8 averaging the spectrum still suffer from noise from the biophysics. The Sd-Unet recovery result on the right shows a perfect Td-Tomato profile. Next, we test the network with more challenging experimental data following the same manner described previously in different SNR levels. We use the transgenic zebrafish with three fluorophores (Critine, Td-Tomato, and mRuby) and acquire images with different averaging frames, from 1, 2, 4, 8. The network was originally trained with four fluorophores so we would like to see if it can recover signals from not only one single color but also multiple colors from not the exact trained data. 110 Qualifying errors and performance are difficult under the circumstance of no ground truth present in multicolor experimental data. We compare the spectra and phasor in order to understand the network performance. We plot out a couple random spectra within the same positions across four images before and after applying Sd-Unet, as shown in Figure 4.11. The high SNR original signals share higher similarity with Sd-Unet denoised data. Moreover, the denoised data share similar spectra shape after normalization (not shown here), indicating consistent results. Figure 4.11 Sd-Unet denoising on different level of SNR on a four-fluorophores zebrafish embryo. Experimental data was collected in Zeiss LSM 880 from a quadra-transgene zebrafish Gt(cltca-citrine);Tg(ubiq:lyn-tdTomato;ubiq:Lifeact-mRuby;fli1:mKO2zebrafish 111 embryo. Multiple acquisition with different averaging frame and laser power were applied on the same position to get data in different SNR. We conduct phasor analysis before and after applying Sd-Unet without applying any compressive denoising filter. Phasor analysis provides an easy visualization of spectral shape distribution of the whole data. (Method is described as in Ch2 supplementary.) The phasor shows Sd-Unet’s ability to greatly denoise the data, where noise are the fuzzy region on the phasor as shown in Figure 4.12 low SNR, and converge of the spectral shape toward the ground truth. Figure 4.12 Phasor analysis before and after applying Sd-Unet indicates the spectral shape distribution is converging toward the high SNR data. 4.4 Sd-Unet recovers details in microscopy image Finally, we visualized the performance of Sd-Unet on hyperspectral data by comparing the unmixing result of this image labeled with mKO2, Citrine, mRuby and Td- Tomato as described in previous sections. As shown in Figure 4.13, the spectra before and after applying Sd-Unet are as what we have seen in previous sections. Smoother signals with less poisson noise and less disrupted signals are observed. Looking at the unmixing composite images, the result after applying the network is clearer with better contrast. Zoom in to a single channel with Td-Tomato marker expressed in membrane, the disrupted 112 edges are recovered in the denoised image and the micro pattern from member fibers are now obviously seen. Figure 4.13 HyU unmixing results from Ch2 before and after applying Sd-Unet shows a spatial improvement after applying Sd-Unet. 4.5 Summary We have created a network that can denoise or recover signals of spectral data that is typically suffer from biophysics constrain of the microscope, down to five photons. We have proven and shown Sd-Unet works well with data acquired from single label or multiple labels sample. We also discover by improving the spectral feature we also improved the spatial feature for different analysis tools. The network is a highly versatile network, which means by slightly changing the architecture or setting, we could utilize it for another project. The network is trained on a large number of commonly used fluorescent labels and metabolic biomarkers. Our network learned the process of biophysics instead of learning fixed patterns. This novel approach will allow the use of the trained network in a generic scenario, as it is independent of the spatial pattern of the fluorescence, while leveraging the common spectral and biophysical properties of the fluorescent signals. This generalized network will allow prediction of new patterns with absence of bias. 113 Chapter 5 The future of decoding live using hyperspectral fluorescence imaging In the following sections, I list two projects utilizing the proposed tools to study the intertwined biology. I have initiated these works and collected preliminary data; however, did not have the opportunity to finish. These high impactful biological applications would be interesting research projects for future students to take over. 114 5.1 Potential study of label free metabolism As previously mentioned, intrinsic signals with abundant metabolism information have been a hot research topic. These signals come from endogenous fluorescence (EF), also known as autofluorescence. The standard method uses FLIM to avoid working with the highly overlapping emission signature. With all the tools developed with this thesis work, decoding the label free intrinsic fluorescent signals using spectral imaging is now possible. We have been exploring non-invasive in vivo imaging of intrinsic metabolic signatures. Some initial results have been shown in section 2.3.4. Here, I am going to show more preliminary data that could be useful to understand the potential of using label free imaging to study metabolism. [7], [41], [51], [74], [75] 5.1.1 Experimental setup Two photon excitations could excite metabolites with intrinsic fluorescence such as Riboflavin, NADH, Folic acid, Cholecalciferol, Pyridoxine, Retinol. Their cross section and emission spectra are highly overlapping as shown in Figure5.1 b and Figure 5.2 b. As previously mentioned, these intrinsic signals are highly varied in terms of expression and dynamic, so that different excitation wavelength or power or regions may lead to different observation. (See more in 5.1.3) With this work, we use Zeiss LSM 780 with 740 nm 2- photon excitation with 690 nm filter to study these metabolites. Images were taken on zebrafish samples, from wildtype, transgenic lines, to Casper lines (See Chapter 1.5). In this way, we capture the intrinsic signals under different conditions separately of no interference from extrinsic signals, with interference from extrinsic signals, no interference from pigment cells. We take images on whole zebrafish embryo, tissue/organ 115 level and single cells to observe the metabolism in different level of the system within the fish. (Figure 5.5- Figure 5.7) Figure 5.1 Two-photon action cross sections and emission spectra from biological molecules. (a) Action cross sections (absorption cross section multiplied by the fluorescence quantum yield) of six molecules that contribute much of the intracellular 2PE fluorescence. All compounds were measured in solutions. (b) Emission spectra of the compounds in a (measured in the same solvents). (Image source: Zipfel et al)[59] 5.1.2 Pre-identified autofluorescence cursors from solutions To help users with any level of experience with fluorescence microscopy data to use HyU for studying autofluorescence, we have pre identified cursors measuring from solution as shown in Figure 2.2. For autofluorescent signals, the spectrum for Elastin was obtained experimentally and compared with literature. (see 5.1.3) Spectra for Nicotinamide Adenine Dinucleotide (NADH) free, NADH bound, Retinoic acid, Retinol and Flavin Adenine Dinucleotide (FAD) were acquired from in vitro solutions using the microscope. NADH free from B-Nicotinamide Adenine Dinucleotide (Sigma-Aldrich, St. Luis, MO, #43420) in Phosphate Buffered Saline (PBS) solution. NADH bound from B- Nicotinamide Adenine Dinucleotide and L-Lactic Dehydrogenase (Sigma-Aldrich, #43420, #L3916) in PBS. Retinoic acid from a solution of Retinoic Acid (Sigma-Aldrich, #R2625) in Dimethylsulfoxide (DMSO). Retinol from a solution of Retinol synthetic (Sigma-Aldrich, #R7632) in DMSO. FAD from Flavin Adenine Dinucleotide Disodium Salt Hydrate (Sigma-Aldrich, #F6625) in PBS. 116 5.1.3 Difficulty of imaging label free metabolism As mentioned in section 1.3.3, imaging label free intrinsic signals has been extremely challenging. First of all, these intrinsic signals have highly overlapping and broad emission spectra, as in Figure 5.1 b, Figure 5.2 b. Secondly, the collected signal is low SNR in nature. In order to avoid photodamage to the sample and maintain the normal metabolism, the laser power is turned low by purpose. Therefore, the number of emission photons is low, causing high photon starvation. The lower intensity plus high noise, the ultra-low SNR data is hard to extract information from. Moreover, the data collected through the microscope is disrupted. In figure 5.3, even with the phasor denoise filter applied, where standard low SNR data would greatly denoised and gain a improved spectra shape, the spectra shape is not aligned to solution spectra. Uncertainty of biology also adds another layer of difficulty. Literature does not report a complete profile of signatures present in the data, as in Figure 5.1 and Figure 5.2. There are multiple reasons. Different pH levels or temperature may shift the spectra and metabolism. NADH bounds with different proteins may also shift the emission. In addition, the endogenous fluorophore distribution in larger samples is not yet fully studied. Current studies focus on single cell or tissue level. Endogenous fluorophores play different roles in the developmental functions and show different levels of expression in different developmental stages or regions within the same sample. As shown in Figure 5.4 to Figure 5.8, what signals presented in the data may change according to imaging regions at different stages. Moreover, the excitation power also determines the detectable intrinsic signals. The signal may not be detectable with given energy. In Figure 5.8, we showed the same sample with different zoom in and same 740 nm two photon excitation in different power. Different detectable signatures can be observed. While in Figure 5.8, FAD signals are strongly 117 expressed. The image is taken in 0.078x0.078 µm with 3% laser power, which is 10 times higher power per pixel compared to data in Figure 5.3. Figure 5.2 Autofluorescence signatures in experimental data compare to signatures obtained from solutions. Both types of signatures were obtained from Zeiss LSM 780 with 740 nm 2 photon excitation. The experimental data were obtained from 3dpf zebrafish embryo shown in solid line. The spectra shapes are disrupted even with compressive denoising filter applied. The spectra are different from solution spectra shown in dash line. In experimental data, a blue previously unidentified signature was observed. This turned out to be elastin that wasn’t shown in the literature we originally take as an example. Figure 5.3 Phasor analysis on different regions of wildtype zebrafish in different temperature condition. 118 Metabolism changes according to different temperature. 28 degrees Celsius is the normal developmental environment for zebrafish. We observed the metabolism change in a lower and higher temperature environment in different regions in two different sample. What EF presented is more in relation to the region of the sample than the environment. Figure 5.4 Metabolic signal distribution in 19 hpf zebrafish embryo from Casper fish. Signals are highly expressed in yolk, which indicates the metabolites may be more active in the yolk at early stage development. (Note: No pigment cells are presented in Casper fish.) (Preliminary data from HyU) Figure 5.5 Distribution of 6 metabolic signals from a 22 hpf quadra-transgene Gt(cltca- citrine);Tg(ubiq:lyn-tdTomato;ubiq:Lifeact-mRuby;fli1:mKO2 zebrafish embryo. NADH free bound express across all cells in the embryo but shows a different level of intensity based on different region. Before the larvae can feed by themselves, carotenoids are stored in the yolk sac and transferred and metabolized to retinoic acid in the body of the embryo by active mechanisms. Elastin is highly express in the notochord. (Preliminary data from HyU) 119 Figure 5.6 FAD expressed in zoomin hindbrain region in 22 hpf zebrafish. A distinct component is found in the (A) Phasor analysis highlighting in cyan circle. (B) Corresponding emission spectra selected from the phasor. (C) Cyan circle highlight the cyan region from raw data. (D) Unmixing results. 120 Figure 5.7 HyU analysis of 36 hpf Casper zebrafish of unmixing intrinsic signals. Casper zebrafish is a transgenic zebrafish characterized by the absence of pigments. Dataset was acquired in the two-photon spectral mode @740 nm excitation. HyU Unmixing was performed utilizing 5 pure intrinsic signals measured in solution (A) Merged overview of all signals (B) NADH bound (C) NADH free (yellow), (D) retinoid (magenta), (E) retinoic acid (cyan) which appears mainly in the yolk sac, known location where carotenoids are stored, transferred and then metabolized to retinoic acid (F) elastin (green) has a similar distribution with in the zebrafish floorplate at this developmental stage (G) Phasor (H) average spectra from selection in G. 5.1.4 Unmixing and identification of autofluorescence signatures using HyU The collected hyperspectral autofluorescence data can be processed using HyU as introduced in Chapter 2. Unlike extrinsic signatures with a spread-out distribution on 2nd harmonic phasor, intrinsic signatures are found highly centralized on 2nd harmonic phasor. 121 In this case, the 1st harmonic phasor is used for analyzing autofluorescence data. With the pre-identified cursors (see 5.1.2), we can easily recognize if the specific endmember is present in the collected data. Spectra visualization function within the software allows to show and save the emission signatures. These signatures can be double checked with literature. Unmixing results would be saved in separate channels and the metabolism distribution within the live sample be visualized. (Figure 5.5 -5.7) Figure 5.8 Identification of previously unknown components. (A) two distinct tails were observed on the phasor plot, highlighting by blue and yellow circles (B) visualizing the corresponding spectra helps to identify the peak emission in 420 nm and 550 nm respectively. (C) conduct literature research and found the corresponding signature from Elastin and Flavins. [26] As in section 2.1.3, one of the biggest advantages of using HyU to analyze fluorescence data is the capability to identify previously unknown endmembers in the data on phasor. As mentioned in the previous section, a thorough understanding of autofluorescence signatures does not yet exist. We started studying autofluorescence with only five pre-identified cursors: NADH free, NADH bound, folic acid, Retinoid and Retinoic acid. In our collected data, however, two distinct endmembers consistently show up on phasor (Figure 5.9 A). With the spectra visualization tool, we recognized one with 122 maximum emission at 420 nm and another at 550 nm. Finally, we determined they are Elastin and FAD from the study from 1998 Andersson et al. [26] HyU unmixing results visualize the distribution of these metabolic activities in zebrafish. For example, at 19 hpf stage shown in figure 5.6, high expression of metabolites is displayed in the yolk region, showing the yolk as an important organ that provides the essential nutrient at early embryonic stage. In the later embryonic stage in figure 5.6, retinoids and retinoic acid are highly clustered in the hindbrain, align to their importance for neuron patterning and differentiation. 5.1.5 Future direction In this section, we summarize the condition for autofluorescence imaging and potential problems due to biological variance. We demonstrate how to use HyU to study intrinsic signals from visualizing the metabolic distribution in biological samples to reading phasor. Here, I list a couple future directions that would continue to enrich the field. One big next step is autofluorescence validation. Current visualization methods are using emission signatures as reference. It would add robustness to the approach to validate the signature from the metabolism biology perspective. Pioneering experiments were conducted by changing the temperature to monitor the redox ratio changes, as in Figure 5.10 and Figure 5.4. We also conducted experiments on a Raldh2 mutant zebrafish where the retinoic acid was prohibited to generate. From the phasor analysis, retinoids signals are missing in the mutant fish, indicating our expectation of the miss of signal detection from proposed methods. FLIM can also be a good validation for NADH free and bound signatures. Fewer information regarding retinol and retinoic acid with FLIM are available. 123 Figure 5.9 Preliminary data and experimental design on NADH free bound metabolic changes. Metabolism changes according to different temperature. 28 degree Celsius is the normal developmental environment for zebrafish. We observed the metabolism change. Metabolism changes according to different temperature. 28 degree Celsius is the normal developmental environment for zebrafish. We observed the metabolism change in a lower and higher temperature environment in different regions in two different sample. Phasor analysis can be found in figure 5.4. Figure 5.10 Retinoids validation using Raldah2 mutant fish from phasor. In Raldh2 mutant fish the formation of retinoid is prohibited. Retinol and retinoic acid signals can be found from the phasor in the (A) wildtype (WT) fish hindbrain region but not in the (B) mutant fish. 124 An interesting next step would be training the Sd-Unet as mentioned in Chapter 4 to recover the disrupted autofluorescence signals. This requires a preliminary test for constructing the network either with prior knowledge of known autofluorescence signatures or a more unbiased spectral denoising from any signature. To validate the network requires a simulation with ground truth that can be constructed from the Ch3 methods. This would greatly enhance signal from the ultralow SNR intrinsic signal and facilitate the study of these metabolism activity in the low disturbance imaging condition. Finally, understanding of the interaction between tissue scattering and tissue- specific emission would be important to fully study the intrinsic signals. This includes a thorough understanding of the changes caused by different excitation light and power, an appropriate selection of the measurement environment, and extreme caution in maintaining suitable environmental conditions during the sample monitoring. This approach has great potential to help the diagnosis of pathological conditions in cell/tissue or even whole organ microstructure in patients without extra cost of time and money for labeling or surgery. This would require optimization of diagnostic sensitivity and specificity by utilization of specific excitation emission wavelength for different tissues or diseases and verification on the performance of new techniques by comparison with the standard ones. It is important to understand the correlation of autofluorescence signals in normal and pathological conditions. 5.2 Liver development and regeneration Numerous studies in model species indicate critical relationships between cellular metabolism, liver development, and regeneration. Determining the nature of these 125 intertwining relationships would significantly enhance developmental research and significantly expedite the development of treatment methods for liver diseases, lesions, and transplantation. This requires an approach that goes beyond standard investigations of cellular metabolism, liver development, and regeneration, and characterizes the dynamics of hepatocytes' metabolic response in vivo during development and regeneration. 5.2.1 Background Liver regeneration is a dynamic process that occurs in all vertebrates, from fish to humans. Extensive research in experimental animal models demonstrates that the regeneration process is tightly controlled into three distinct stages: A first response to injury is characterized by a decrease in the individual's systematic metabolic output; this is followed by the recruitment of dormant hepatocytes and other cells within the liver to proliferate; and finally, the proliferative program is terminated once the organ has been restored to its original size. While it is obvious that liver regeneration is a dynamic process, there is a lack of in vivo dynamic data and critical cellular interactions are poorly understood.[76]–[78] Cellular metabolism has been demonstrated to be a significant component of liver regeneration in experimental animal models. Understanding the dynamics of this metabolic response at the cellular level inside the regenerating liver is crucial for progress in developing treatment options to enhance liver regeneration in patients with hepatic disorders as well as for managing living donors for liver transplantation. However, there are few commonly available technologies for imaging cellular metabolism non-invasively. [41], [44]–[48] 126 In contrast to other regenerative processes, liver regeneration appears to be a compensatory growth process rather than a regeneration process. After partial hepatectomy (PH), the liver increases in bulk to compensate for the missing tissue by recruiting quiescent hepatocytes to re-enter the cell cycle. In other regeneration systems, such as the fin and limb, a blastema composed of undifferentiated cells develops and proliferates in order to recreate the regenerated tissue. The compensatory enlargement of the liver in response to cell proliferation is a conserved response that appears to be comparable in development and regeneration of the liver. Cell cycle regulators are necessary for both embryonic hepatic outgrowth and liver regeneration following partial hepatectomy in rats and fish (PH). In mice, hepatocyte growth factor (HGF) and epidermal growth factor (EGF) can stimulate hepatocytes to enter the S phase of the cell cycle and are key mitogens in liver regeneration. uhrf1 (ubiquitin-like protein with PHD and ring finger domains) is necessary for hepatic outgrowth, embryonic survival, and liver regeneration in fish. Although heterozygous uhrf1 mutants are alive, they demonstrate abnormal liver expansion following PH. The common genetic components that control cell growth and proliferation during development and regeneration raise doubts about the two processes' similarity: Are cell growth and proliferation rates identical during development and regeneration? Do the two procedures involve the growth of the same cell populations? [76], [81], [84]–[87] 5.2.2 Experimental design 127 Figure 5.11 Experimental design for liver development and regeneration. We employ both spectral imaging and lifetime imaging to perform longitudinal fluorescent imaging of the metabolites with endogenous fluorescence (such as depicted in Figure X) and metabolic enzymatic proteins (FlipTrap) of cellular metabolism in developing liver from 3-7dpf. Proliferation drives liver expansion throughout these phases. At 3 days post fertilization, the early liver bud is evident and differentiated hepatocytes have formed. Between 3-5dpf, hepatocytes continue to proliferate and differentiate into the various cell types of the liver (biliary ducts and stellate cells). Additionally, the vascular endothelium innervates the liver throughout these phases, forming the hepatic arteries and veins. By 5dpf, all digestive organs, including the liver, are developed. Imaging longitudinally of endogenous and exogenous fluorescence will provide key data and establish a framework for single-cell resolution of metabolites, metabolic rates and changes of enzymatic protein expression of all cell-types within the developing liver, including hepatocytes, as they grow and differentiate. [ref] 128 To visualize metabolic enzyme expression and location using exogenous labels, we employ double transgenic FlipTrap lines labeling components of the Krebs cycle and the vasculature (Tg(kdrl:mCherry)). These two transgenes enable the incorporation of numerous endogenous labels into the liver's cellular environment, as the FlipTrap line marks hepatocyte mitochondria (Fig. 6). We can then separate these labels and gain a better understanding of how numerous spectral signatures of metabolites contribute to the metabolic phenotype of hepatocytes, vasculature, ducts, and stellate cells during development of the liver. We perform FLIM on the same samples in addition to hyperspectral imaging. To do this, we may use the fluorescent specificity of FlipTrap labels in conjunction with the activity of intrinsic metabolites as determined by their fluorescence lifetime signatures. Figure 5.12 Exogenous labels reveal liver outgrowth and vascularization. Blend renderings of unmixed volumetric confocal multispectral dataset of liver from live zebrafish embryo (3 dpf- 5 dfp). Left shows where the liver is from the whole fish 3D volumetric image. Zoomin is composite view with two labels: suclg2-citrine(green) and 129 kdrl-mCherry(red) from 3dpf to 5dpf. Single mitochondria can be observed in the unmixing results. 5.2.3 Imaging liver regeneration through laser ablation Numerous experimental paradigms exist for examining zebrafish liver regeneration. In zebrafish, liver resection has been performed surgically by removing the ventral lobe. Alternatively, conditional targeting of tissues to trigger cell death has been devised in zebrafish. This method utilizes the Nitroreductase (NTR) enzyme to transform the prodrug Metronidazole (Mtz) into a cytotoxic DNA cross-linking agent. These procedures have proved difficult to execute precisely and efficiently. Due to the sample's tiny size, manual surgical resection for PH is often inaccurate and taxing for the animal, resulting in imprecise lesions and post-resection morbidity. Chemical ablation is successful in some tissues, such as the heart and pancreas, but not as effective in the liver. To overcome these limitations, increase precision and perform highly targeted ablation without injury to non-hepatic surrounding tissues, we perform 2-Photon laser ablation (2-PLA) of liver, as shown in Figure 5.13. With 2-PLA we can precisely create incisions and resections with micrometric precision inside the liver, without the need for surgically accessing the site. Because the fish do not have to recover from having their abdominal wall cut open, we can directly perform SpectraFLIM imaging after 2-PLA to assess the repair process and provide immediate longitudinal data of the regeneration response. FlipTrap technology permits us to identify hepatocytes by their fluorescence (Figure 5.13C) and target these cells for ablation with a high intensity focused laser (Figure 5.13D). Preliminary results confirm that 2-PLA is non-invasive, performed contact-free and has an extremely low mortality rate (Figure 5.13D). 130 Stage dependent plasticity in the larvae liver In testing the 2-PLA approach, we focused on resecting the liver in the larvae (5- 10dpf) and found stage dependent differences in recovery rates. Resection at 5dpf, when the liver is in the growth phase, resulted in rapid and complete recovery within 24hours. However, resection at 6dpf, when the liver begins to function, resulted in a longer recovery phase that occurred over three days. This stage dependence has not been observed in the adult liver. The rapid regrowth and plasticity of the liver at 5dpf may reflect the developmental potential of hepatocytes at this stage. Time-dependent release of metabolites in resected liver Hyperspectral imaging over time of the resected liver detected a time-dependent release of metabolites in the ablated tissue. Immediately following tissue ablation, high levels of free NADH and retinoid acid were detected in the cell-free region of the ablated liver. In addition, a cell death signature was detected in the phasor plot immediately following 2-PLA. Subsequent to the release of free NADH and retinoid acid, high levels of bound NADH was detected in the cell-free region 1day post-ablation (dpa). The release of metabolites into the cell-free region ceased by 2dpa. Interestingly, no differences in metabolic signatures were detected in the cells surrounding the site of ablation. 131 Figure 5.13 2-Photon laser ablation (2-PLA) of the liver. (A) Using conventional lasers, single- photon microscopy, the energy of the beam (red) is focused throughout the whole specimen causing damage to all tissues it crosses. (B) 2- Photon with its nonlinear effects are capable of focusing pulsed beam energy in a narrow volume in space (red). This contactless approach allows us to first locate fluorescently labeled liver (C), then to create a highly localized injury (white dash line) in the liver (green) without affecting surrounding areas. 132 Figure 5.14 Liver regeneration after laser ablation. Confocal microscopy data of laser ablation on 5 dpf liver and the injury recovery at 6 dpf with three labels ubiqH2B- tdtomato, suclg2-citrine, kdrl-mCherry. Zebrafish shows a high regeneration ability where the damage created from laser ablation nearly fully recovered the next day. 5.2.4 Experimental and analytical challenges Studying liver development and regeneration through tracking zebrafish in different stages and correlating their metabolism information is not trivial. First of all, a consistent imaging region is hard to get. Zebrafish show a steep growth rate from embryo stage (1 dpf to 7 dpf) making it hard to use the same mold to mount the sample. Liver starts growing from the left side of the sample on day 3 and displays a variance in terms of size and shape among different samples. The swimming bladder and fins also grow within the experiment timeframe, making it hard to mount the sample. 133 While we have been successful at applying the 2-PLA approach in the larvae (5- 10dpf), we aim to extend our analysis into the adult liver. Even with PTU to prevent the proliferation of pigment, pigments still grow within the aging sample. Pigments often absorb the energy from the laser and cause damage in the sample, leading to the difficulty of imaging adult fish samples. Therefore, a transgenic fish without pigment cells is needed for adult fish imaging. Casper fish, a transparent zebrafish, would be a good solution to avoid the photodamage due to the pigment. For resection and imaging of the adult liver, Casper fish enable easy imaging of the internal organs as the adults that are devoid of pigment and have transparent skin and scales. Another layer of challenge, as previously mentioned, comes from the complexity of biology. Variance exists within the same species, even in the same organ in one sample, different zoom in and different excitation may return different results. For example, in Figure 5.15, unbalanced signals are detected in the single slice of the inner layer and outer layer of the liver, which may be due to scattering effect or metabolism activity. This even signal distribution happens for both extrinsic and intrinsic signals. The phasor analysis for intrinsic signal is especially difficult since the metabolites are distributed unevenly across liver. More research needs to be done to fully understand the complexity of the sample. 134 Figure 5.15 Liver microscopy images show imbalanced signal expression from inner and outer layer. Images from a single image slice of extrinsic(A) and intrinsic B) signals are taken from a 6dpf Gt(suclg2-citrine) Tg(kdrl:mCherry) zebrafish embryo. (C) (D) shows the phasor analysis from intrinsic signal. A different center of mess (red point) is shown for the inner and outer layer. 5.2.5 Preliminary results and discussion In constructing the imaging pipeline, we performed experiments to optimize imaging conditions, multi-photon-based liver resection and analysis of FLIM and hyperspectral data. For imaging, we identified conditions for FLIM and hyperspectral time-lapse microscopy to limit phototoxicity and damage while being able to collect sufficient fluorescence signals from the developing and regenerating liver. To develop the liver resection assay, we tested the timing when liver ablation would lead to a regenerative phenotype in the fish larvae. For data analysis, we discovered that both FLIM and 135 hyperspectral imaging detected multiple novel autofluorescence signatures in tissues surrounding the liver that contributed to the phasor plot, a Fourier transformation of the lifetime and spectral signals. The multiple components shifted the FLIM and spectral signatures and were difficult to decompose as a number of the novel autofluorescence signatures were of unknown origins. We overcame this challenge by focusing the phasor analysis on subregions of the liver. Combined these optimizations led to the following initial observations: Shift from a glycolytic to oxidative phenotype in the developing liver For dynamic imaging of the developing liver, we focused on stages during hepatocyte differentiation and liver growth, 3 to 5 days postfertilization (dpf). Imaging of the gene trap in succinate-CoA ligase, GDP-forming beta subunit (Gt(suclg2a-citrine)) and a transgenic label of the vasculature revealed increasing vascularization of the liver during these stages. Hyperspectral imaging of intrinsic fluorescence detected four dominant metabolites in the developing liver: retinol, retinoid acid, free and bound NADH. The levels of retinol and retinoid acid increased as the liver differentiated and grew (3- 5dpf). This increase is reflective of the maturation of the liver as retinol and retinoid acids are known to be synthesized in the liver and can interact with retinoid receptors to control expression of a large number of genes involved in hepatic processes. The ratio of free to bound NADH shifted from a high to low level, indicating a shift from a glycolytic to oxidative phenotype. Subregion analysis revealed that the glycolytic phenotype associated with the vasculature innervating the liver. The multiple components detected by spectral imaging were corroborated by FLIM imaging. 136 Figure 5.16 Increasing metabolic activity in developing liver. Spectra analysis of autofluorescence signals and resulting maximum intensity projection renderings from 3 dpf to 5 dpf (left to right.) (A) spectral phasor of liver autofl signals. (B) Absolute intensities for NADH bound(red), NADH Free(yellow), Retinol(blue), Retinoic acid(magenta), H2B-tdtomato(green), kdrl-mcherry(cyan), laser reflection(dark red.) (C) Relative intensities. (D,E,F)Imaris rendering. Figure 5.17 Liver recovery over three days from laser ablation on 6 dpf liver. 137 (A) Spectral phasor. (B) Confocal microscopy data renderings. (C) Corresponding autofluorescence signals. The white arrow highlights the laser ablation region on day 6. Recovery can be observed in day 7 and day 8. 5.2.6 Summary and future direction The preliminary work used a new experimental pipeline as in Figure 5.12 to extract the dynamic relationships between cellular metabolism, liver development and liver regeneration. We acquired both Hyperspectral and FLIM data, both non-invasive in vivo imaging of intrinsic metabolic signatures. We combined this with multi- photon laser ablation, improved genetic labeling and conditional mutagenesis of metabolic enzymes (FlipTrap), to quantitatively characterize cellular metabolic phenotypes of cells within their in vivo context. Combining these technologies, we had initial data access to cellular metabolism over time in developing, and regenerating liver in zebrafish. Imaging and analyzing a conditional metabolic mutant zebrafish would be an interesting follow-up experiment. The genetic fluorescence labels to detected the liver and vasculature, Gt(suclg2-citrine) and Tg(kdrl:mCherry) needed to be bred into the Casper strain, a double homozygous recessive (roy orbison (roy a9 ); mitfa w2 ) mutation. We have initiated the three generation inbreeding scheme to create quadruple transgenic Gt(suclg2- citrine);Tg(kdrl:mCherry); roy a9 ; mitfa w2 fish to enable unambiguous identification of the liver for resection and provide readout of vasculature innervation, as in Figure 5.12 left and Figure 5.14. It would require more crossing for a stable line. More experiments and analysis are required to better understand and quantify the result. This includes looking at not only tissue level but also single cell levels. Next step is to understand the best experimental setting including region of interest and laser power aligned with cell activities from metabolism to morphology for both FLIM and spectral data. This project would potentially reveal new cellular mechanisms driving liver development and 138 regeneration and allow the identification of metabolic phenotypes and signatures that may be predictive of regenerative capabilities of hepatocytes. 5.3 Conclusion In this thesis, we developed three tools to improve sensitivity of hyperspectral fluorescence microscopy. These tools enable 1. recovery of signals with low photon emission, 2. easy identification of endmember, 3. unmixing overlapping emission spectra, 4. quantification algorithm performance, and 5. generation of biological realistic microscopy data. From the signal recovery point of view, we introduced machine learning denoising framework. With that, hyperspectral fluorescent data can easily be analyzed with traditional classification or unmixing algorithms. We also proposed HyU as an alternative unmixing algorithm that facilitates the understanding of complex biological hyperspectral data. With HyU, endmembers can be easily visualized in phasor plane and their compressively denoised spectra can be selected, visualized, compared with literature, and used for unmixing. Instead of performing tedious experiments, we proposed a flexible simulation framework to create large datasets of any fluorophore combination to avoid going through the generation of sample and exploring imaging setting. All three tools can be used individually and are adaptable with other analytic tools. For example, HyU can use other compression approaches other than phasor as the compressor to enable fast analysis and compressively denoise. Other unmixing algorithms can be used to replace linear unmixing. Simulation is also highly tunable according to interested patterns, fluorophores, instruments, and imaging settings. It can be used for quantification of analysis, testing algorithms, or training other machine learning networks. 139 Sd-Unet can be trained on any type of fluorophores and are highly versatile. The denoised result can be used for classification, mapping, or unmixing. In conclusion, we designed a better unmixing algorithm validated through simulation and boosted through machine learning denoising. Three proposed tools fundamentally enhance the information extraction in 5D imaging. From simulation, we quantified HyU unmixing results showing at least 2 times better performance in terms of MSE. From the machine learning denoising algorithm, we are now able to work with data with only 5 photons per pixel, which is 15 times lower than the number of photons of standard minimum requirement. The improved sensitivity enables a low light imaging condition that is suitable for live imaging. We are able to perform deeper, longer, live multiplexed imaging. I hope this thesis work is just the beginning of this field and these approaches can enable high impactful future biological research. 140 References [1] J. R. Lakowicz, Principles of Fluorescence Spectroscopy Principles of Fluorescence Spectroscopy. 2006. doi: 10.1007/978-0-387-46312-4. [2] J. W. Lichtman and J. A. Conchello, “Fluorescence microscopy,” Nature Methods, vol. 2, no. 12. pp. 910–919, Dec. 2005. doi: 10.1038/nmeth817. [3] J. Enderlein, “Advanced Fluorescence Microscopy,” in Comprehensive Biomedical Physics, vol. 4, 2014. doi: 10.1016/B978-0-444-53632-7.00409-3. [4] W. Becker, “Fluorescence lifetime imaging - techniques and applications,” Journal of Microscopy, vol. 247, no. 2, 2012, doi: 10.1111/j.1365-2818.2012.03618.x. [5] S. Kredel et al., “mRuby, a bright monomeric red fluorescent protein for labeling of subcellular structures,” PLoS ONE, vol. 4, no. 2, pp. 1–7, 2009, doi: 10.1371/journal.pone.0004391. [6] D. Llères, S. Swift, and A. I. Lamond, “Detecting protein-protein interactions in vivo with FRET using multiphoton fluorescence lifetime imaging microscopy (FLIM).,” Current protocols in cytometry / editorial board, J. Paul Robinson, managing editor ... [et al.], vol. Chapter 12, 2007, doi: 10.1002/0471142956.cy1210s42. [7] J. R. Lakowicz, H. Szmacinski, K. Nowaczyk, and M. L. Johnson, “Fluorescence lifetime imaging of free and protein-bound NADH,” Proceedings of the National Academy of Sciences of the United States of America, vol. 89, no. 4, 1992, doi: 10.1073/pnas.89.4.1271. [8] H. Tsurui, H. Nishimura, S. Hattori, S. Hirose, K. Okumura, and T. Shirai, “Seven- color fluorescence imaging of tissue samples based on fourier spectroscopy and singular value decomposition,” Journal of Histochemistry and Cytochemistry, vol. 48, no. 5, pp. 653–662, 2000, doi: 10.1177/002215540004800509. [9] T. Zimmermann, J. Rietdorf, and R. Pepperkok, “Spectral imaging and its applications in live cell microscopy,” FEBS Letters, vol. 546, no. 1, pp. 87–92, 2003, doi: 10.1016/S0014- 5793(03)00521-0. [10] H. Sahoo, “Fluorescent labeling techniques in biomolecules: A flashback,” RSC Advances, vol. 2, no. 18. 2012. doi: 10.1039/c2ra20389h. [11] C. P. Toseland, “Fluorescent labeling and modification of proteins,” Journal of Chemical Biology, vol. 6, no. 3. 2013. doi: 10.1007/s12154-013-0094-5. [12] S. W. Paddock, “Confocal Microscopy, Methods and Protocols, Second Edition,” Humana Press, vol. 1075, p. 388, 2014, doi: 10.1007/978-1-60761-847-8. [13] W. Denk, J. H. Strickler, and W. W. Webb, “Two-photon laser scanning fluorescence microscopy,” Science, vol. 248, no. 4951, 1990, doi: 10.1126/science.2321027. [14] C. Stringari, J. L. Nourse, L. A. Flanagan, and E. Gratton, “Phasor Fluorescence Lifetime Microscopy of Free and Protein-Bound NADH Reveals Neural Stem Cell Differentiation Potential,” PLoS ONE, vol. 7, no. 11, 2012, doi: 10.1371/journal.pone.0048014. 141 [15] Z. Lavagnino, J. Dwight, A. Ustione, T. U. Nguyen, T. S. Tkaczyk, and D. W. Piston, “Snapshot Hyperspectral Light-Sheet Imaging of Signal Transduction in Live Pancreatic Islets,” Biophysical Journal, vol. 111, no. 2, pp. 409–417, 2016, doi: 10.1016/j.bpj.2016.06.014. [16] P. T. C. So, “Two-photon Fluorescence Light Microscopy,” Life Sciences, pp. 1–5, 2002, doi: 10.1038/npg.els.0002991. [17] K. Svoboda and R. Yasuda, “Principles of Two-Photon Excitation Microscopy and Its Applications to Neuroscience,” Neuron, vol. 50, no. 6. 2006. doi: 10.1016/j.neuron.2006.05.019. [18] R. Lansford, G. Bearman, and S. E. Fraser, “Resolution of multiple green fluorescent protein color variants and dyes using two-photon microscopy and imaging spectroscopy,” Journal of Biomedical Optics, vol. 6, no. 3, 2001, doi: 10.1117/1.1383780. [19] A. M. Valm et al., “Applying systems-level spectral imaging and analysis to reveal the organelle interactome,” Nature, vol. 546, no. 7656, pp. 162–167, 2017, doi: 10.1038/nature22369. [20] F. Amat et al., “Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data,” Nature Methods, vol. 11, no. 9, pp. 951–958, 2014, doi: 10.1038/nmeth.3036. [21] Y. Sun, H. Wallrabe, C. F. Booker, R. N. Day, and A. Periasamy, “Three-color spectral FRET microscopy localizes three interacting proteins in living cells,” Biophysical Journal, vol. 99, no. 4, 2010, doi: 10.1016/j.bpj.2010.06.004. [22] B. J. Visser, M. C. Joshi, and D. Bates, “Multilocus imaging of the E. coli chromosome by fluorescent in situ hybridization,” in Methods in Molecular Biology, vol. 1624, 2017. doi: 10.1007/978-1-4939-7098-8_16. [23] S. Paddock, “Multi-Spectral Imaging and Linear Unmixing Add a Whole New Dimension to,” BioTechniques, vol. 31, no. 6, pp. 1272–1278, 2001. [24] W. Baschong, R. Suetterlin, and R. Hubert Laeng, “Control of autofluorescence of archival formaldehyde-fixed, paraffin-embedded tissue in confocal laser scanning microscopy (CLSM),” Journal of Histochemistry and Cytochemistry, vol. 49, no. 12, 2001, doi: 10.1177/002215540104901210. [25] M. Neumann and D. Gabel, “Simple method for reduction of autofluorescence in fluorescence microscopy,” Journal of Histochemistry and Cytochemistry, vol. 50, no. 3, 2002, doi: 10.1177/002215540205000315. [26] H. Andersson, T. Baechi, M. Hoechl, and C. Richter, “Autofluorescence of living cells,” Journal of Microscopy, vol. 191, no. 1, 1998, doi: 10.1046/j.1365-2818.1998.00347.x. [27] R. C. Benson, R. A. Meyer, M. E. Zaruba, and G. M. McKhann, “Cellular autofluorescence. Is it due to flavins?,” Journal of Histochemistry and Cytochemistry, vol. 27, no. 1, 1979, doi: 10.1177/27.1.438504. [28] J. E. Aubin, “Autofluorescence of viable cultured mammalian cells,” Journal of Histochemistry and Cytochemistry, vol. 27, no. 1, 1979, doi: 10.1177/27.1.220325. 142 [29] A. W. Knight and N. Billinton, “Distinguishing GFP from cellular autofluorescence,” Biophotonics International, vol. 8, no. 7, 2001. [30] D. K. Bird et al., “Metabolic mapping of MCF10A human breast cells via multiphoton fluorescence lifetime imaging of the coenzyme NADH,” Cancer Research, vol. 65, no. 19, 2005, doi: 10.1158/0008-5472.CAN-04-3922. [31] Z. Liu et al., “Mapping metabolic changes by noninvasive, multiparametric, high- resolution imaging using endogenous contrast,” Science Advances, vol. 4, no. 3, 2018, doi: 10.1126/sciadv.aap9302. [32] C. Stringari, A. Cinquin, O. Cinquin, M. A. Digman, P. J. Donovan, and E. Gratton, “Phasor approach to fluorescence lifetime microscopy distinguishes different metabolic states of germ cells in a live tissue,” Proceedings of the National Academy of Sciences of the United States of America, vol. 108, no. 33, 2011, doi: 10.1073/pnas.1108161108. [33] C. Stringari, R. A. Edwards, K. T. Pate, M. L. Waterman, P. J. Donovan, and E. Gratton, “Metabolic trajectory of cellular differentiation in small intestine by Phasor Fluorescence Lifetime Microscopy of NADH,” Scientific Reports, vol. 2, 2012, doi: 10.1038/srep00568. [34] M. Monici, “Cell and tissue autofluorescence research and diagnostic applications,” Biotechnology Annual Review, vol. 11, no. SUPPL. 2005. doi: 10.1016/S1387-2656(05)11007- 2. [35] C. Stringari, “Label-free separation of human embryonic stem cells and their differentiating progenies by phasor fluorescence lifetime microscopy,” Journal of Biomedical Optics, vol. 17, no. 4, 2012, doi: 10.1117/1.jbo.17.4.046012. [36] A. W. Browne et al., “Structural and functional characterization of human stem-cell- derived retinal organoids by live imaging,” Investigative Ophthalmology and Visual Science, vol. 58, no. 9, 2017, doi: 10.1167/iovs.16-20796. [37] G. Weber, “Resolution of the fluorescence lifetimes in a heterogeneous system by phase and modulation measurements,” Journal of Physical Chemistry, vol. 85, no. 8, 1981, doi: 10.1021/j150608a006. [38] A. H. A. Clayton, Q. S. Hanley, and P. J. Verveer, “Graphical representation and multicomponent analysis of single-frequency fluorescence lifetime imaging microscopy data,” Journal of Microscopy, vol. 213, no. 1, 2004, doi: 10.1111/j.1365-2818.2004.01265.x. [39] G. I. Redford and R. M. Clegg, “Polar plot representation for frequency-domain analysis of fluorescence lifetimes,” Journal of Fluorescence, vol. 15, no. 5, 2005, doi: 10.1007/s10895-005-2990-8. [40] M. A. Digman, R. Dalal, A. F. Horwitz, and E. Gratton, “Mapping the number of molecules and brightness in the laser scanning microscope,” Biophysical Journal, vol. 94, no. 6, 2008, doi: 10.1529/biophysj.107.114645. [41] F. Fereidouni, A. N. Bader, A. Colonna, and H. C. Gerritsen, “Phasor analysis of multiphoton spectral images distinguishes autofluorescence components of in vivo human skin,” Journal of Biophotonics, vol. 7, no. 8, pp. 589–596, 2014, doi: 10.1002/jbio.201200244. 143 [42] M. Weigert et al., “Content-aware image restoration: pushing the limits of fluorescence microscopy,” Nature Methods, vol. 15, no. 12, 2018, doi: 10.1038/s41592-018-0216-7. [43] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” Journal of Machine Learning Research, vol. 13, 2012. [44] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P. A. Muller, “Deep learning for time series classification: a review,” Data Mining and Knowledge Discovery, vol. 33, no. 4, 2019, doi: 10.1007/s10618-019-00619-1. [45] W. Ouyang, A. Aristov, M. Lelek, X. Hao, and C. Zimmer, “Deep learning massively accelerates super-resolution localization microscopy,” Nature Biotechnology, vol. 36, no. 5, 2018, doi: 10.1038/nbt.4106. [46] C. Belthangady and L. A. Royer, “Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction,” Nature Methods, vol. 16, no. 12, 2019, doi: 10.1038/s41592-019-0458-z. [47] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in 32nd International Conference on Machine Learning, ICML 2015, 2015, vol. 1. [48] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, 1998, doi: 10.1109/5.726791. [49] C. Sommer and D. W. Gerlich, “Machine learning in cell biology-teaching computers to recognize phenotypes,” Journal of Cell Science, vol. 126, no. 24, 2013, doi: 10.1242/jcs.123604. [50] M. A. Digman, V. R. Caiolfa, M. Zamai, and E. Gratton, “The Phasor Approach to Fluorescence Lifetime Imaging Analysis,” Biophysical Journal, vol. 94, no. 2, pp. L14–L16, 2008, doi: 10.1529/biophysj.107.120154. [51] S. Ranjit, L. Malacrida, D. M. Jameson, and E. Gratton, “Fit-free analysis of fluorescence lifetime imaging data using the phasor approach,” Nature Protocols, vol. 13, no. 9, pp. 1979–2004, Sep. 2018, doi: 10.1038/s41596-018-0026-5. [52] F. Cutrale et al., “Hyperspectral phasor analysis enables multi- plexed 5D in vivo imaging,” Nature Publishing Group, no. July 2016, 2017, doi: 10.1038/nmeth.4134. [53] L. Scipioni, A. Rossetta, G. Tedeschi, and E. Gratton, “Phasor S-FLIM: a new paradigm for fast and robust spectral fluorescence lifetime imaging,” Nature Methods, vol. 18, no. 5, pp. 542–550, May 2021, doi: 10.1038/s41592-021-01108-4. [54] N. Dobigeon, Y. Altmann, N. Brun, and S. Moussaoui, “Linear and Nonlinear Unmixing in Hyperspectral Imaging,” in Data Handling in Science and Technology, vol. 30, 2016. doi: 10.1016/B978-0-444-63638-6.00006-1. [55] C. Zeiss and M. Online, “Introduction to Spectral Imaging and Linear Unmixing,” Imaging, vol. 1, pp. 1–13, 2010. 144 [56] T. Zimmermann, “Spectral imaging and linear unmixing in light microscopy,” Advances in Biochemical Engineering/Biotechnology, vol. 95, pp. 245–265, 2005, doi: 10.1007/b102216. [57] M. E. Dickinson, G. Bearman, S. Tille, R. Lansford, and S. E. Fraser, “Multi-spectral imaging and linear unmixing add a whole new dimension to laser scanning fluorescence microscopy,” BioTechniques, vol. 31, no. 6, 2001, doi: 10.2144/01316bt01. [58] S. Isogai, N. D. Lawson, S. Torrealday, M. Horiguchi, and B. M. Weinstein, “Angiogenic network formation in the developing vertebrate trunk,” 2003, doi: 10.1242/dev.00733. [59] W. R. Zipfel, R. M. Williams, R. Christiet, A. Y. Nikitin, B. T. Hyman, and W. W. Webb, “Live tissue intrinsic emission microscopy using multiphoton-excited native fluorescence and second harmonic generation,” Proceedings of the National Academy of Sciences of the United States of America, vol. 100, no. 12, pp. 7075–7080, 2003, doi: 10.1073/pnas.0832308100. [60] M. C. Skala et al., “In vivo multiphoton microscopy of NADH and FAD redox states, fluorescence lifetimes, and cellular morphology in precancerous epithelia,” Proceedings of the National Academy of Sciences of the United States of America, vol. 104, no. 49, pp. 19494– 19499, 2007, doi: 10.1073/pnas.0708425104. [61] C. Févotte and N. Dobigeon, “Nonlinear hyperspectral unmixing with robust nonnegative matrix factorization,” IEEE Transactions on Image Processing, vol. 24, no. 12, 2015, doi: 10.1109/TIP.2015.2468177. [62] A. Parslow, A. Cardona, and R. J. Bryson-Richardson, “Sample drift correction following 4D confocal time-lapse Imaging,” Journal of Visualized Experiments, no. 86, 2014, doi: 10.3791/51086. [63] J. Schindelin et al., “Fiji: An open-source platform for biological-image analysis,” Nature Methods, vol. 9, no. 7. 2012. doi: 10.1038/nmeth.2019. [64] O. Krichevsky and G. Bonnet, “Fluorescence correlation spectroscopy: The technique and its applications,” Reports on Progress in Physics, vol. 65, no. 2, 2002, doi: 10.1088/0034- 4885/65/2/203. [65] T. Ha and P. Tinnefeld, “Photophysics of fluorescent probes for single-molecule biophysics and super-resolution imaging,” Annual Review of Physical Chemistry, vol. 63. 2012. doi: 10.1146/annurev-physchem-032210-103340. [66] A. A. Green, M. Berman, P. Switzer, and M. D. Craig, “A Transformation for Ordering Multispectral Data in Terms of Image Quality with Implications for Noise Removal,” IEEE Transactions on Geoscience and Remote Sensing, vol. 26, no. 1, 1988, doi: 10.1109/36.3001. [67] J. B. Lee, A. S. Woodyatt, and M. Berman, “Enhancement of High Spectral Resolution Remote-Sensing Data by a Noise-Adjusted Principal Components Transform,” IEEE Transactions on Geoscience and Remote Sensing, vol. 28, no. 3, 1990, doi: 10.1109/36.54356. 145 [68] Y. Qian and M. Ye, “Hyperspectral imagery restoration using nonlocal spectral-spatial structured sparse representation with noise estimation,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 6, no. 2, 2013, doi: 10.1109/JSTARS.2012.2232904. [69] N. Dobigeon, J. Y. Tourneret, and A. O. Hero, “Bayesian linear unmixing of hyperspectral images corrupted by colored Gaussian noise with unknown covariance matrix,” 2008. doi: 10.1109/ICASSP.2008.4518389. [70] H. Othman and S. E. Qian, “Noise reduction of hyperspectral imagery using hybrid spatial-spectral derivative-domain wavelet shrinkage,” IEEE Transactions on Geoscience and Remote Sensing, vol. 44, no. 2, 2006, doi: 10.1109/TGRS.2005.860982. [71] C. Ounkomol, S. Seshamani, M. M. Maleckar, F. Collman, and G. R. Johnson, “Label- free prediction of three-dimensional fluorescence images from transmitted-light microscopy,” Nature Methods, vol. 15, no. 11, 2018, doi: 10.1038/s41592-018-0111-2. [72] K. McDole et al., “In Toto Imaging and Reconstruction of Post-Implantation Mouse Development at the Single-Cell Level,” Cell, vol. 175, no. 3, 2018, doi: 10.1016/j.cell.2018.09.031. [73] T. v Truong, W. Supatto, D. S. Koos, J. M. Choi, and S. E. Fraser, “Deep and fast live imaging with two-photon scanned light-sheet microscopy,” vol. 8, no. 9, 2011, doi: 10.1038/nmeth.1652. [74] J. T. Sharick, P. F. Favreau, A. A. Gillette, S. M. Sdao, M. J. Merrins, and M. C. Skala, “Protein-bound NAD(P)H Lifetime is Sensitive to Multiple Fates of Glucose Carbon,” Scientific Reports, vol. 8, no. 1, 2018, doi: 10.1038/s41598-018-23691-x. [75] M. E. Gosnell, A. G. Anwer, J. C. Cassano, C. M. Sue, and E. M. Goldys, “Functional hyperspectral imaging captures subtle details of cell metabolism in olfactory neurosphere cells, disease-specific models of neurodegenerative disorders,” Biochimica et Biophysica Acta - Molecular Cell Research, vol. 1863, no. 1, 2016, doi: 10.1016/j.bbamcr.2015.09.030. [76] G. K. Michalopoulos, “Liver regeneration after partial hepatectomy: Critical analysis of mechanistic dilemmas,” American Journal of Pathology, vol. 176, no. 1, 2010, doi: 10.2353/ajpath.2010.090675. [77] B. C. Chen et al., “Lattice light-sheet microscopy: Imaging molecules to embryos at high spatiotemporal resolution,” Science, vol. 346, no. 6208, 2014, doi: 10.1126/science.1257998. [78] M. T. Guerra et al., “Mitochondrial calcium regulates rat liver regeneration through the modulation of apoptosis,” Hepatology, vol. 54, no. 1, 2011, doi: 10.1002/hep.24367. [79] G. K. Michalopoulos and M. C. DeFrances, “Liver regeneration,” Science, vol. 276, no. 5309. 1997. doi: 10.1126/science.276.5309.60. [80] I. J. Marques et al., “Metastatic behaviour of primary human tumours in a zebrafish xenotransplantation model.,” BMC cancer, vol. 9, p. 128, 2009, doi: 10.1186/1471-2407-9-128. 146 [81] S. Kiso et al., “Liver regeneration in heparin-binding EGF-like growth factor transgenic mice after partial hepatectomy,” Gastroenterology, vol. 124, no. 3, 2003, doi: 10.1053/gast.2003.50097. [82] A. Abu Rmilah, W. Zhou, E. Nelson, L. Lin, B. Amiot, and S. L. Nyberg, “Understanding the marvels behind liver regeneration,” Wiley Interdisciplinary Reviews: Developmental Biology, vol. 8, no. 3. 2019. doi: 10.1002/wdev.340. [83] J. Huang and D. A. Rudnick, “Elucidating the metabolic regulation of liver regeneration,” American Journal of Pathology, vol. 184, no. 2. 2014. doi: 10.1016/j.ajpath.2013.04.034. [84] C. McCusker, S. V. Bryant, and D. M. Gardiner, “The axolotl limb blastema: cellular and molecular mechanisms driving blastema formation and limb regeneration in tetrapods,” Regeneration, vol. 2, no. 2, 2015, doi: 10.1002/reg2.32. [85] C. Pfefferli and A. Jaźwińska, “The art of fin regeneration in zebrafish,” Regeneration, vol. 2, no. 2, 2015, doi: 10.1002/reg2.33. [86] K. Nejak-Bowen, A. Orr, W. C. Bowen, and G. K. Michalopoulos, “Conditional Genetic Elimination of Hepatocyte Growth Factor in Mice Compromises Liver Regeneration after Partial Hepatectomy,” PLoS ONE, vol. 8, no. 3, 2013, doi: 10.1371/journal.pone.0059836. [87] S. Verma, J. S. Purohit, A. Arora, S. Sinha, and M. M. Chaturvedi, “Liver regeneration: Metabolic and epigenetic regulation,” Hepatoma Research, vol. 7. 2021. doi: 10.20517/2394- 5079.2020.122.

Abstract (if available)

Abstract Two of the most challenging but compelling puzzles that human beings have been trying to solve are decoding the secrets of life and alleviating the suffering of people affected by diseases. Right now, traditional methods utilize 2D imaging, which cannot provide sufficient information to fully capture the biological mechanism. An ideal solution is to track the same specimen in the whole volume and multiple labels over time to decode the dynamic biological system. Therefore, we propose to use 5D imaging, which contains 3D (x, y, z) plus time and labels, as our solution. However, the full capability of 5D imaging is limited by current technologies. This thesis explores the state-of-the-art high-content imaging methods, their challenges, and novel algorithmic solutions.
This dissertation proposal explains the development of three novel toolkits to advance multiplexing spectral fluorescent live imaging. The first tool is an unmixing algorithm for untangling overlapping signals. The second tool is a spectral fluorescence modeling framework used to enable the simulation of any fluorescence data without conducting complicated experiments. The third tool is a network that utilizes machine learning to denoise the microscopy noise in acquired spectral signals.
These innovative tools have the potential to significantly advance our ability to study complex biological phenomena and open previously inaccessible windows of observation into biological systems. These toolkits are highly versatile. They can be applied to intrinsic signals, fluorescent labels, and the combination of the two. This can open the scientific community to a wide range of studies from clinical to model-based biological systems.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Extending multiplexing capabilities with lifetime and hyperspectral fluorescence imaging

PDF

Hyperspectral phasor for multiplexed fluorescence microscopy and autofluorescence-based pathologic diagnosis

PDF

Analytical tools for complex multidimensional biological imaging data

PDF

Machine learning and image processing of fluorescence lifetime imaging microscopy enables tracking and analysis of subcellular metabolism

PDF

Democratizing optical biopsy with diffuse optical imaging

PDF

Optimization methods and algorithms for constrained magnetic resonance imaging

PDF

Sampling theory for graph signals with applications to semi-supervised learning

PDF

Simulation and machine learning at exascale

PDF

A computational framework for diversity in ensembles of humans and machine systems

PDF

Metabolic profiling of single hematopoietic stem cells for developing novel ex vivo culture strategies

PDF

Analog and mixed-signal parameter synthesis using machine learning and time-based circuit architectures

PDF

Machine learning techniques for outdoor and indoor layout estimation

PDF

Advanced machine learning techniques for video, social and biomedical data analytics

PDF

Decoding memory from spatio-temporal patterns of neuronal spikes

PDF

Learning logical abstractions from sequential data

PDF

Multimodal and self-guided clustering approaches toward context aware speaker diarization

PDF

Fair Machine Learning for Human Behavior Understanding

PDF

Fluorescent imaging probes of nitrogen-containing bone active drugs: design, synthesis and applications

PDF

Efficient and accurate object extraction from scanned maps by leveraging external data and learning representative context

PDF

Machine learning based techniques for biomedical image/video analysis

Asset Metadata

Creator Chiang, Rose (author)

Core Title Multiplexing live 5d imaging with multispectral fluorescence: Advanced unmixing through simulation and machine learning

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Biomedical Engineering

Degree Conferral Date 2022-05

Publication Date 04/27/2022

Defense Date 03/01/2022

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag autofluorescence,computational tool,developmental biology,fluorescence,hyperspectral imaging,image processing,imaging,label-free imaging,machine learning,metabolism,metabolism imaging,microscopy,OAI-PMH Harvest,signal process,simulation,spectral imaging,unmixing

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Fraser , Scott (committee chair), Cutrale, Francesco (committee member), White, Kate (committee member), Zavaleta, Cristina (committee member)

Creator Email hsiaojuc@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC111117820

Unique identifier UC111117820

Document Type Dissertation

Format application/pdf (imt)

Rights Chiang, Rose

Type texts

Source 20220427-usctheses-batch-933 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu