University of Southern California Dissertations and Theses

Extending multiplexing capabilities with lifetime and hyperspectral fluorescence imaging

(USC Thesis Other)

Extending multiplexing capabilities with lifetime and hyperspectral fluorescence imaging

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content

EXTENDING MULTIPLEXING CAPABILITIES WITH LIFETIME AND
HYPERSPECTRAL FLUORESCENCE IMAGING

by

Wen Shi

A Dissertation Presented to the
FACULTY OF THE GRADUATE SCHOOL
UNIVERSITY OF SOUTHERN CALIFORNIA
In Partial Fulfillment of the
Requirements for the Degree
DOCTOR OF PHILOSOPHY
(BIOMEDICAL ENGINEERING)

May 2023

Copyright 2022 Wen Shi
ii
Acknowledgments
I have encountered a great number of people that supported me and guided me
throughout my Ph.D.
First and foremost, I’d like to express my deepest appreciation to my advisors Dr.
Scott Fraser and Dr. Francesco Cutrale for being so supportive, understanding, and
encouraging during a journey full of change, challenges, and growth. They have
been instrumental in giving me advice and guidance.
Scott is always open to the unknown and encourages me to explore. He always
prioritizes my growth over research results. It’s his open mind that allows me to
challenge myself constantly. I see a great scientist with great vision, an honest
heart, and a brilliant brain in him, facing great challenges but always creating values
and making an impact! Every one of our meetings has been inspiring and I’ve
learned so much from him. He encourages me to find the sheer joy of learning while
working on the project and making peace with research outcomes. He motivates me
to be an inner-driven, independent thinker and I believe that’s something deeply
profound. He combines academic rigor and outstanding academic skills in
mentoring my scientific writing through examples, instructions, and rubrics, which is
an invaluable lesson that helped me learn to think deeply, create meaning for
myself, and become aware of my own learning process. He showed me a lifelong
passion for learning and an attitude of always trying to better oneself, and I am
forever grateful for what he gave me.
iii
I want to thank my mentor Francesco for sticking with me during all the ups and
downs of the project throughout my Ph.D. Knowing how risky and difficult research
is and how often I fail daily, he always has faith in me. From lots and lots of great
meetings to spontaneous calls whenever I have questions, from multiple practice
talks in a row to multiple edits of the presentation slides, I was beyond lucky to have
him on my side. He showed me what I could do with my skills and translate them
from textbook knowledge into technical solutions to solve real-world problems.
I’ve met lots of great professors and mentors during my Ph.D. They might not have
had a direct impact on the project in this thesis, but they helped me in all sorts of
ways. I want to thank Dr. Richard Leahy, who supervised my rotation project and
served on my supervisory committee from rotation to my qualifying exam and then
to my dissertation defense. I spent a summer at Takeda Pharmaceuticals Company
for an internship as an amateur imaging scientist and I received lots of help from Dr.
Charles Glaus. He opened my eyes to the industry and guided me through a
challenging project that now I am proud of.
This also all happened thanks to the tireless effort of our collaborators, Dr. Gianluca
Truncate and David Warburton from CHLA, and Dr. Benjamin Steventon from
Cambridge University. They taught me that science is not the work of one individual,
but it is the combination and evolution of work and ideas from many.
I would like to express my gratitude to my lab members, Dr. Le Trinh, Dr. Masahiro
Kitano, Dr. Daniel Koo, Dr. Rose Chiang, Shanshan Cai, Dr. Peiyu Wang, Dr.
Valerie Komatsu, and Peter Luu. We had lots of discussions and they provided
iv
great insights and helped form this project in its early stages into what you see in
this thesis today. I have also made a handful of the most amazing, passionate, and
supportive folks I am happy to call my lifelong friends. They are the people who give
constructive suggestions for my research projects, go with me on food adventures,
share all my tears and laughter, and are there with me through thick and thin. They
are the type of people who give you a hand when they see you drowning and in the
next second share ridiculous Ph.D. memes with you. They are both my
cheerleaders and life mentors. We had so many adventures in the past few years,
both food-related and science-related. They make it seem shorter than it is.
Last but not least, I would like to thank my dearest parents. I went to boarding
school at the age of 11, and since then I have always lived distantly from my family.
While I was growing up, the chances to see my parents were continuously
becoming less frequent. I couldn’t fathom making new memories and growing
through new experiences without my parents by my side. Seven years ago, I came
across the Pacific Ocean to the United States for graduate school, which makes
visiting them even more difficult. My parents are so loving and supportive, and they
made lots of sacrifices. Even when my parents are not geographically close to me,
they never stop being my source of strength and guidance. I remember all they
gave me and all they are not able to see. They are the biggest reason that got me
into this place in my life.
I am proud of the work I’ve done to cope with all the stress, frustration, and failures
throughout my Ph.D. by bolstering my self-esteem and minimizing my need for
external validation. After all these years and experiences, I genuinely believe in
v
science and myself as if I had never failed. For that, I want to thank this Ph.D.
journey and all the ups and downs in it that empower me, inspire me, and
encourage me.

vi
Table of Contents
ACKNOWLEDGMENTS ---------------------------------------------------------------------------------- II
LIST OF TABLES ------------------------------------------------------------------------------------------ IX
LIST OF FIGURES ----------------------------------------------------------------------------------------- X
ABSTRACT ------------------------------------------------------------------------------------------------ XII
CHAPTER 1 INTRODUCTION -------------------------------------------------------------------------- 1
1.1 Motivation ------------------------------------------------------------------------------------------------------------------ 1
1.2 Fluorescence microscopes allow for observing biological processes -------------------------------------- 3
1.2.1 Introduction to fluorescence --------------------------------------------------------------------------------------- 3
Fluorescence ----------------------------------------------------------------------------------------------------------------- 3
The Jablonski Diagram ---------------------------------------------------------------------------------------------------- 5
Fluorescence lifetime ------------------------------------------------------------------------------------------------------- 7
1.2.2 Fluorescence spectroscopy ---------------------------------------------------------------------------------------- 8
1.3 Fluorescence Hyper-spectral Imaging (fHSI) -------------------------------------------------------------------- 13
1.4 Fluorescence Lifetime Imaging Microscopy (FLIM) ------------------------------------------------------------ 17
1.5 The imaging challenge of harvesting information in biomedical specimens ----------------------------- 21
1.5.1 Live fluorescence imaging trade-offs --------------------------------------------------------------------------- 21
1.5.2 Microscopy data suffer from low signals ----------------------------------------------------------------------- 22
1.5.3 Multiplexing information in higher dimension ----------------------------------------------------------------- 25
1.6 Projects and opportunities to access highly multiplexed information for fluorescent signals -------- 27
CHAPTER 2 SPECTRALLY ENCODED ENHANCED REPRESENTATIONS (SEER) 30
2 SPECTRALLY ENCODED ENHANCED REPRESENTATIONS (SEER) ------------- 30
2.1 Problems in visualizing hyperspectral data ---------------------------------------------------------------------- 30
2.2 Limitations of current approaches ---------------------------------------------------------------------------------- 31
2.3 Improving spectral compression with Phasor -------------------------------------------------------------------- 32
2.4 Improving visualization with Spectrally Encoded Enhanced Representations (SEER) --------------- 34
2.4.1 SEER workflow ------------------------------------------------------------------------------------------------------- 34
2.4.2 SEER colormap design -------------------------------------------------------------------------------------------- 37
vii
Standard Reference Maps ----------------------------------------------------------------------------------------------- 37
Tensor map ------------------------------------------------------------------------------------------------------------------ 43
Modes (Scale and Morph) ----------------------------------------------------------------------------------------------- 44
2.4.3 Applications in overlapping fluorescent data ----------------------------------------------------------------- 52
Color maps enhance different spectral gradients in biological samples ------------------------------------- 52
Spectral differences can be visualized in combinatorial approaches ----------------------------------------- 66
2.4.4 Quantitative validation of SEER --------------------------------------------------------------------------------- 68
2.5 SEER methods, simulations, and quantifications --------------------------------------------------------------- 72
2.5.1 Simulated hyperspectral test chart ------------------------------------------------------------------------------ 72
2.5.2 Standard RGB visualizations ------------------------------------------------------------------------------------- 73
2.5.3 Spectral separation accuracy calculation ---------------------------------------------------------------------- 75
2.5.4 Compressive spectral algorithm and map reference design --------------------------------------------- 76
Phasor calculations -------------------------------------------------------------------------------------------------------- 76
Standard map reference ------------------------------------------------------------------------------------------------- 77
Scale mode ------------------------------------------------------------------------------------------------------------------ 79
2.5.5 Color image quality calculation ----------------------------------------------------------------------------------- 86
Colorfulness ----------------------------------------------------------------------------------------------------------------- 86
Sharpness -------------------------------------------------------------------------------------------------------------------- 87
Contrast ----------------------------------------------------------------------------------------------------------------------- 88
CHAPTER 3 CLASSIFICATION OF MULTIMODEL 5-D BIOMEDICAL DATA WITH
SPECTRAFLIM-LSTM ---------------------------------------------------------------------------------- 90
3.1 SpectraFLIM enables multiplexing --------------------------------------------------------------------------------- 90
3.1.1 Single-cell resolution SpectraFLIM autofluorescence imaging ------------------------------------------ 90
3.1.2 Multi-dimensional SpectraFLIM data --------------------------------------------------------------------------- 92
3.2 4D-imaging classification to separate metabolic states ------------------------------------------------------- 94
3.2.1 Overview of current deep learning approaches -------------------------------------------------------------- 94
3.2.2 Deep learning approaches for investigating 5-D SpectraFLIM data ------------------------------------ 96
SpectraFLIM-LSTM -------------------------------------------------------------------------------------------------------- 99
2-D CNN -------------------------------------------------------------------------------------------------------------------- 101
3.2.3 Exploring subsections of SpectraFLIM data ---------------------------------------------------------------- 101
3.3 Improving classification by integrating spectral and FLIM information into deep learning --------- 103
3.3.1 The impact of spectrally resolved spatial information ---------------------------------------------------- 103
3.3.2 Comparing classification in intensity and lifetime domain ----------------------------------------------- 107
3.3.3 Classification across spectral, lifetime, and spatial dimensions --------------------------------------- 109
3.3.4 Effects of signal quality on classification --------------------------------------------------------------------- 109
3.4 Methods ----------------------------------------------------------------------------------------------------------------- 112
3.4.1 Image acquisition -------------------------------------------------------------------------------------------------- 112
3.4.2 Data standardization ---------------------------------------------------------------------------------------------- 112
3.4.3 Data Binning -------------------------------------------------------------------------------------------------------- 113
3.4.4 2-D CNN Model ---------------------------------------------------------------------------------------------------- 113
3.4.5 ConvLSTM Model ------------------------------------------------------------------------------------------------- 114
3.4.6 Models for binned data ------------------------------------------------------------------------------------------- 117
viii
3.4.7 Performance Metrics ---------------------------------------------------------------------------------------------- 117
CHAPTER 4 THE FUTURE OF MULTI-DIMENSIONAL FLUORESCENCE
MICROSCOPY DATA ANALYSIS ------------------------------------------------------------------ 119
4.1 Enhancing current techniques for multiplexing fluorescence data --------------------------------------- 121
4.1.1 Improve visualization with denoising ------------------------------------------------------------------------- 122
4.1.2 Improve classification by increasing signal strength ------------------------------------------------------ 123
4.1.3 Data augmentation to expand data set size for classification ------------------------------------------ 124
4.1.4 Expand image analysis to volumetric or time-lapse with SpectraFLIM ------------------------------ 125
4.1.5 Outlier detection with large in-class variance and imbalanced classification ----------------------- 126
Large in-class variance hampers the classification accuracy ------------------------------------------------- 127
Class imbalance hampers classification accuracy --------------------------------------------------------------- 127
Outlier detection of unseen conditions ------------------------------------------------------------------------------ 128
4.2 Potential applications of these techniques --------------------------------------------------------------------- 129
4.2.1 SEER for fast and enhanced visualization ------------------------------------------------------------------ 129
Speed improvement ----------------------------------------------------------------------------------------------------- 129
Enhanced visualization in low signal-to-noise regimes --------------------------------------------------------- 129
Highlighting subtle spectral differences ---------------------------------------------------------------------------- 130
4.2.2 Near real-time visualization for sequential data ------------------------------------------------------------ 132
Integrate SEER into hyperspectral imaging instruments ------------------------------------------------------- 132
Extend SEER for sequential analysis with various data types ------------------------------------------------ 133
4.2.3 SpectraFLIM for multi-dimensional and multimodal classification ------------------------------------ 133
REFERENCES ------------------------------------------------------------------------------------------ 135

ix
List of Tables
Table 3.1 Results for all input strategies with data in original shape ....................................................... 104
Table 3.2 Results for all input strategies with data after binning ............................................................ 105

x
List of Figures
Figure 1.1 The Jablonski diagram illustrating electronic states and their transitions of fluorescence ......... 6

Figure 1.2 Fluorescence intensity decay over time to calculate fluorescence lifetime. ............................... 8

Figure 1.3 Schematic of the light path on an epifluorescence microscope. ............................................... 10

Figure 1.4 Frequency-domain FLIM measured by phase shift and modulation decrease ......................... 20

Figure 1.5 Embedded noise in spectra extracted from fluorescence images ............................................ 23

Figure 2.1 Spectrally Encoded Enhanced Representations (SEER) conceptual representation. .............. 36

Figure 2.2 Simulated Hyperspectral Test Chart I rendered in TrueColor .................................................. 38

Figure 2.3 Effect of spectral shape in absence of background on Radial Map .......................................... 40

Figure 2.4 Effect of spectral intensity in presence of background on Radial Map. .................................... 41

Figure 2.5 Spectrally Encoded Enhanced Representation (SEER) designs. ............................................ 42

Figure 2.6 Enhanced contrast modalities. ................................................................................................. 45

Figure 2.7 Radial and Angular reference map designs and modes on SHTC I ......................................... 47

Figure 2.8 Gradient Ascent and Descent reference map designs and modes on SHTC I ........................ 48

Figure 2.9 Simulated Hyperspectral Test Chart II and its standard overlapping spectra. .......................... 49

Figure 2.10 Radial and Angular reference map designs and modes on SHTC II ...................................... 50

Figure 2.11 Gradient Ascent and Descent reference map designs and modes on SHTC II ..................... 51

Figure 2.12 Autofluorescence visualization comparison for tracheal explant ............................................ 52

Figure 2.13 Phasor Fluorescence Lifetime Imaging Microscopy (FLIM) of tracheal explant. .................... 53

Figure 2.14 Visualization of a single fluorescence label against multiple autofluorescences. ................... 56

Figure 2.15 Single-label fluorescence visualization comparison in presence of autofluorescence. .......... 57

Figure 2.16 Triple-label fluorescence visualization .................................................................................... 60

Figure 2.17 Triple-label fluorescence visualization comparison ................................................................ 61

xi
Figure 2.18 SEER in Maximum Intensity Projection (MIP) and Shadow Projection .................................. 62

Figure 2.19 Visualization of combinatorial expression on Zebrabow samples. ......................................... 65

Figure 2.20 Visualization comparison for combinatorial expression .......................................................... 65

Figure 2.21 Morph mode algorithm pictorial abstraction. ........................................................................... 83

Figure 3.1 Overview of SpectraFLIM data. ................................................................................................ 91

Figure 3.2 Anatomy of 5-D multimodal SpectraFLIM data ......................................................................... 93

Figure 3.3 Workflow of SpectraFLIM classification with multiple neural network strategies. ..................... 97

Figure 3.4 Inner structures of 2-D convolution and ConvLSTM. ................................................................ 98

Figure 3.5 2-D CNN and ConvLSTM architectures. .................................................................................. 98

Figure 3.6 Results for all the input strategies with data in original shape and data after binning ........... 106

Figure 4.1 Spectral denoising effect on Angular and Radial maps visualization of SHTC II ................... 122

xii
Abstract
Imaging the metabolism of intact cells and tissues would offer an important means
to assay health and injury. Fluorescence multi-spectral (hyperspectral) imaging
(fHSI) imaging has emerged as a great tool to capture spatio-temporal dynamics
across scales for molecules, cells, and tissues with multiple fluorescent labels.
The resulting data are dense in information and high in dimensionality, and often
require lengthy analyses to interpret the complex biological information for our eyes
to understand. We developed Spectrally Encoded Enhanced Representations
(SEER) for improved and computationally efficient color visualization of
hyperspectral images with overlapping spectra. This visualization technique
enhances subtle spectral differences in multi-labeled fluorescent and
autofluorescent images, providing a fast, intuitive, and mathematical way to interpret
hyperspectral images during collection, preprocessing, and analysis.
We furthermore combined fluorescence multi-spectral imaging with fluorescent
lifetime imaging (SpectraFLIM) to assay the metabolism as well as the spatial and
temporal dynamics of living samples. We demonstrated that SpectraFLIM offers
improved insights into complex biology at cellular resolution. Our SpectraFLIM
workflow can image metabolism and distinguish injured tissue from healthy tissue.
To better analyze SpectraFLIM data, we employed deep learning methods using a
2-dimensional Convolutional Neural Network (2-D CNN) with inputs from spectral
and/or lifetime domains. For comparison, we developed a convolutional long short-
term memory (ConvLSTM) network to classify complex SpectraFLIM data and find
xiii
that it extracts spatial features while preserving spectral correlations across all
channels. We find that either deep learning approach improves classification
accuracy, which should make imaging-based diagnostic tools more robust and
easier to apply in diverse settings. Our methods in this thesis combine analytical
tools with novel microscopy to understand complex biological processes in living
organisms.
1
Chapter 1 Introduction
1 Introduction
1.1 Motivation
The first step to developing solutions to cure diseases that are currently not being
tackled is to understand the biological events in the human body from the smallest
scale up to how it functions as a whole system. Scientists are trying to observe
where cells, tissues, and organs are in our body and how they are connected and
functioning. We combine these elements at different scales, from molecular,
cellular, and organ to human scales, and use this information to guide us towards
finding the underlying mechanisms in disease development and the right treatments
for when development abnormity happens or diseases progress. The driving force
of this type of research is to develop the technologies that allow us to image the
biological structure and molecular signals, and to interpret these images to better
understand complex events in living organisms.
We hope to have a sneak peek at the biological processes as they take place within
intact specimens. Advanced imaging techniques enable us to watch the cells as
they interact and follow these dynamic events as the disease progresses. Modern
microscopes evolve to collect images at deeper locations, with a faster speed, and
with a cellular resolution that helps scientists understand life sciences at the
molecular level. For example, scientists can watch cells take shape to form an
2
organ during embryogenesis, tissue metastasis, and cell migration in which cells
move their cell diameter within minutes. Advanced microscopes allow us to
understand biological structures and temporal dynamics by collecting the entire
volume at a robust speed and fine resolution.
The need for accessing meaningful biological information has grown with the
massive production of digital images of all kinds. Image analysis is always desirable
to extract useful information from complex biological image data. It is the
quantitative or qualitative characterization of 2-dimensional, 3-dimensional, or
higher-dimensional images, as it translates volumes of data into representative
features that give us highlights and conclusions. It is to bring the entire fragmented
world of imaging for a comprehensive and acknowledged interpretation. Complex
biological data provide comprehensive information to understand life sciences but
also demand smart analytical tools for interpretation. Human vision has its
limitations when it comes to high-dimensional data, and it is aggravated by noise.
Our visual cortex is good for extracting high-level information, but intuitive
observations do not provide enough information for scientists to quantitatively or
qualitatively measure biological events. Constructing a dynamic model of
specimens at different scales with cellular resolution could help scientists better
assess the development of all types of biological events, for example, how diseases
progress. The images we acquire using modern microscopes contain complex
biological information that is not simply RGB images our eyes can understand. We
need smart algorithms that allow us to push beyond the capabilities of the human
eyes and interpret the complex information captured by advanced imaging
3
techniques. Our goal is to develop such algorithms for high-dimensional microscopy
data that help understand complex biological processes.
1.2 Fluorescence microscopes allow for observing biological
processes
1.2.1 Introduction to fluorescence
Fluorescence
Fluorescence is the emission of electromagnetic radiation, usually in the form of
photons, and occurs from electronically excited states. Fluorescence lifetime and
fluorescence emission spectra are the most important intrinsic properties of a
fluorophore and have been leveraged extensively to study biomolecules, their
microenvironment, and their interactions within living organisms in biomedical
research. As will be described in more detail in the later section of this chapter, the
lifetime of a fluorophore is the average time for the fluorophore to emit photons once
excited. Fluorescence lifetime is typically nanoseconds scale. Because
fluorescence lifetime is sensitive to the local environment of the fluorophore, it has
been used to monitor intracellular biochemical reactions. The fluorescence emission
spectrum of a molecule is the fluorescence intensity measured as a function of the
emission wavelength after being excited by a source of light. This intensity and
wavelength distribution of the emission may change with variants such as
temperature, concentration, or interactions with other molecules around it. Some
fluorophores are also sensitive to microenvironment properties such as pH, polarity,
4
and ion concentrations, therefore fluorescence emission spectra can be used to
identify specific molecules and investigate these biological changes in a sample.
Many fluorescent substances are used for a range of biological molecules in
biomedical research by leveraging the properties of emission spectra and lifetime.
Some of these fluorescent stains are small molecules that are intrinsically
fluorescent and bind a biological molecule of interest. For example, some of these
are nucleic acid stains such as DAPI, which label the nuclei of cells. Some
fluorophores bind specific cellular structures and have been derivatized with a
fluorescent reporter. Fluorophores are fluorescence molecules that can be linked to
a different molecule that binds the target of interest within the sample, such as
fluorescein, and Alexa Fluor. The quest for fluorescent probes with a high specificity
allows for investigating biological processes in live imaging.
Fluorescence is extensively used in life sciences. There has been a remarkable
growth in the use of fluorescence for cellular and molecular imaging to characterize
and localize molecules at a subcellular level. Fluorescence detection has a high
sensitivity in monitoring dynamic processes in biological samples. Scientists take
advantage of such sensitivity and use fluorescence as probes for labeling and
locating cells and tissues in living organisms in biological research. Scientists track
cells or tissues at a specific region in their biological sample to monitor the temporal
dynamics and spatial structures by tagging specific proteins with fluorophores.
These advances in fluorescence studies decrease the cost and complexity in
biological research, for example, in flow cytometry, DNA sequencing, and genetic
analysis.
5

The Jablonski Diagram
The Jablonski diagram (Figure 1.1) is a tool used to visualize the principles of
fluorescence. The Jablonski diagram illustrates the processes that occur between
the absorption and emission of light. Energy states are labeled Sn, with a larger n
indicating a higher energy level. To generate fluorescence, the fluorophore is
excited from the lowest energy state (S0) to a higher energy state (Sn with n>0)
when it absorbs energy. Internal conversion (i.e., S2 to S1) is a process where
fluorophores relax to the low vibrational level of the first excited state (S1). Internal
conversion often involves energy dissipation through non-radiative processes such
as emitting heat. At each of the electronic energy levels, the same fluorophore
species can exist in several vibrational energy levels during the process of internal
conversion. The excited molecules then return to the ground state S0 rapidly by the
emission of photons. The photons during emission are the fluorescent signals the
detector captures to formulate the image using a microscope. Molecules in the S1
state could undergo a spin conversion to triplet state T1. This conversion is called
intersystem crossing. The emission from T1 is called phosphorescence and is
generally at lower energy and hence at a longer wavelength compared to
fluorescence. Lifetime is the average time the fluorophore spends in the excited
state before returning to the ground state. This time duration helps us differentiate
fluorophore species, microenvironments, or interactions in living organisms.
Emission energy is typically lower than excitation energy, this phenomenon was
observed initially by Sir G. G. Stokes and is referred to as the “Stokes shift”, which
6
describes the fact that the peak emission has a longer wavelength (lower energy)
than the peak excitation wavelength. This energy shift allows us to observe the light
that is originating from the fluorophores other than the excitation light from the
microscope as we can separate the emission wavelength from that of the excitation.

Figure 1.1 The Jablonski diagram illustrates the electronic states of a fluorescence
molecule and the transitions between them. The singlet ground, first, and second
electronic states are depicted by S0, S1, and S2 respectively. At each of these electronic
energy levels, fluorophores can exist in several vibrational energy levels. The transitions
between states are depicted as vertical lines to illustrate light absorption and emission.
For example, a population of the same kind of fluorescence molecules is excited from S0
to a higher energy level and the fluorophores reach the excited state S2, and part of that
energy is dissipated through internal conversion to the S1, and possibly relaxes to several
vibrational energy levels during the process of internal conversion. The excited
fluorescence molecules relax to the ground state S0 by emitting photons. The emission
happens not instantly after the excitation but usually takes several nanoseconds. The time
between the excitation and the emission is called the fluorescence lifetime. Molecules in
7
the S1 state undergo a spin conversion to the first triplet state T 1. This conversion is
intersystem crossing and the emission from T1 is called phosphorescence.

Fluorescence lifetime
Fluorescence lifetime is one of the essential characteristics of a fluorophore that we
can take advantage of in identifying fluorescent species. It is the time a molecule
remains in its excited state before returning to the ground state. Figure 1.2
illustrates the fluorescence intensity versus the time of a population of excited
fluorophore molecules to decay to zero intensity. The fluorescence lifetime 𝜏 is the
time at which the intensity has decayed to 1/e of the original value. As fluorescence
emission is a random process and very few molecules emit their photos at precisely
𝑡 = 𝜏, the lifetime of the a population of fluorophores is the time measured for the
number of excited molecules to decay exponentially to 1/e of the original population.
Fluorescence lifetime ranges in the order of 10
-9
to 10
-7
seconds in general.
Fluorescence detection can be mostly divided into two categories of measurements:
steady-state and time-resolved fluorescence. A steady state is commonly referred
to as “fluorescence intensity”, which is to detect the fluorescent emission signal
upon excitation where the excitation and emission processes are treated as
simultaneous. Fluorophores emit photons at a specific wavelength within
nanoseconds upon excitation. This very short wavelength gap, between excitation
and emission, described as the “stokes shift” in the previous section, allows the
detection of fluorescence intensity. Time-resolved fluorescence (TRF) is to record
the time-dependent intensity profile of the emitted light upon excitation. It is based
8
on the detection of intensity decay of the emission signal. In time-resolved
fluorescence measurements, the excitation pulse (shown in black) must be shorter
than the decay time of the fluorescent signal.

Figure 1.2 Fluorescence intensity decay over time is described by an exponential function
to calculate fluorescence lifetime. The fluorescence lifetime 𝜏, is the time at which the
intensity has decayed to 1/e of the original value. The excited state lifetime of fluorophores
can be used for the quantitative interpretation of fluorescence measurements.
Fluorescence lifetime of a fluorophore depends on its molecular environment but not on
local fluorophore concentration. Changes in the intracellular biochemical environment of
the fluorophores, such as temperature, pH, ions, and polarity, lead to changes in
fluorescence lifetime. Fluorescence lifetime is an important parameter that provides
information related to the microenvironment, biomolecular species of fluorescent proteins,
or their interactions within living samples. Fluorescence 1 and fluorescence 2 in this
diagram present different lifetime decay curves, which could be used as indicators to
separate two fluorescent species if they exist, or one fluorophore at different metabolic
stages or microenvironments.

1.2.2 Fluorescence spectroscopy
Visualizing dynamic processes in living organisms is significantly more challenging
than in static, fixed tissues. It requires a balance of imaging speed, spatial
9
resolution, penetration depth, and prevention from photobleaching and
phototoxicity. Biological processes in life sciences, take place at different scales,
from the atomic, molecular, cellular, organ, and whole living system, are dynamic in
time and space by nature. Our knowledge of complex biological processes is
strongly correlated with our ability to observe these events taking place in intact
specimens.
Fluorescence microscopes (Figure 1.3) are great tools for investigating biological
processes in living organisms. Fluorescence imaging reveals the localization and
measurements of molecules within the subcellular resolution, and some even at the
level of single-molecule level detection. Scientists can observe spatial structures
and monitor temporal dynamics in living organisms by tagging a specific fluorescent
protein at a specific region. These fluorescent signals are captured utilizing a
fluorescence microscope while biological events take place. The principles of a
fluorescence microscope in life sciences can be illustrated using the
epifluorescence microscope design in Figure 1.3. The biological sample is
illuminated with light of a specific wavelength. This specific wavelength is the
excitation light and can be separated from a light source of multiple wavelengths
using a set of excitation filters. Excitation light illuminates the specimen through an
objective lens. The fluorescence emitted by the specimen is then focused onto the
detector using the same objective. The dichroic mirror serves as a beam splitter that
transmits fluorescent signals to the detector but reflects the remaining excitation
light back towards the light source. The emission wavelength is then separated from
the excitation by emission filters placed before the detector. This microscope design
10
is commonly used in biology and is the basis for more advanced fluorescence
microscope designs.

Figure 1.3 Schematic of the light path on an epifluorescence microscope. The instrument
is composed of a set of excitation filters and emission filters to selectively pass through
the light, an objective to focus light, a dichroic mirror to reflect and transmit light, and a
detector to capture the emission light. The excitation wavelength, separated by an
excitation filter from a light source that could have various wavelengths, illuminates the
specimen through the objective lens. The fluorescent signals emitted by the specimen are
focused onto the detector by the objective and eyepiece. The dichroic mirror is a beam
splitter that transmits the fluorescent light through to the detector and reflects the
11
remaining excitation light back to the light source. The emission light can be separated
from the illumination light by an emission filter placed before the detector in the light path.

There is a wide palette of different color fluorophores, each one characterized by
unique excitation and emission spectra, which could enable scientists to
simultaneously follow differently labeled molecules, cells, or tissues
spatiotemporally. Among the advantages of using multiple fluorophores is the
capability to simultaneously follow differently labeled molecules, cells, or tissues
space- and time-wise. Fluorescence microscopes enable imaging of subcellular
imaging through fluorescence molecules with high specificity and sensitivity.
Fluorescent probes are extremely sensitive to detecting biological molecules such
as DNA, RNA, and proteins. Multi-labeled fluorescent samples can be detected
within the samples and allow for an easy way to monitor the interactions between
biological molecules and understand development in living organisms as a whole
system.
The limitation of current fluorescence microscopes is the number of fluorescent
proteins that can be followed at the same time, which is one of the hampering
factors in the progress of research in the life sciences. These fluorophores have
overlapping emission spectra, which limits the number and choice of fluorescent
proteins when using multiple labels. In conventional imaging techniques, a detector
captures the integral of overlapping spectra in the same wavelength band. Emission
filters (Figure 1.3) are optical band-pass filters that are placed before detectors such
that detectors capture only portions of emission spectra in a wavelength band. The
12
number of wavelength bands, often called channels, is usually limited as these
spectral ranges of minimal overlap diminish as the number of labels increases.
Standard optical multi-channel fluorescence imaging differentiates fluorescent
protein reporters through bandpass emission filters, collecting signals based on
wavelength. Imaging of multiple labels often involves sequential excitation of these
fluorophores, with narrow emission optical filters placed within the imaging path of
the fluorescence microscope to collect target emission wavelength within the filter
bandwidth at a time (Figure 1.3). Traditionally, researchers use up to three labels,
choosing well-separated spectra, based on the maximum emission wavelength. For
example, green fluorescent protein (GFP), yellow fluorescent protein (YFP), and red
fluorescent protein (RFP) are commonly used fluorescent probes, and scientists
have a good chance to differentiate them in fluorescence imaging because of the
difference in their peak emission wavelength. However, there isn’t a narrow
wavelength band that just includes a complete spectrum of one single label.
Crosstalk of signals between channels, generally called bleed-through, makes the
extraction of information from affected datasets challenging. Bleed-through occurs
when signals from different fluorophores appear in the same spectral channel as
their emission wavelengths overlap. When imaging multiple labels with overlapping
spectra, the strategy of sequential scanning and optical filters does not overcome
the crosstalk of signals between wavelength channels. The captured signals in
practice are composed of target emission wavelength and unwanted signals from
other excited fluorescent labels. It is difficult to separate individual overlapping
spectra and often requires post-processing methods to separate them.
13
Many experimental demands, in fields ranging from genomics and cell biology to
digital pathology, have motivated the development of better ways to image many
different labels in the same specimen, in a photon-efficient and time-efficient
manner. Needs for multiplex imaging, using a large number of fluorescent labels
simultaneously, have driven the development of more fluorescent labels, of new
imaging instruments tailored to capture information from multi-labeled samples, and
better ways to interpret the information, separating the signal from many different
fluorescent species. Many of these methods benefit significantly from the high
sensitivity and specificity of genetically encoded labels, such as fluorescent proteins
in live imaging, and some of them have become efficient enough to interpret the
intrinsic fluorescence of biological tissues. This thesis introduces fluorescence
hyperspectral imaging (fHSI) in section 1.3 and fluorescence lifetime imaging
microscopes (FLIM) in section 1.4 as techniques that make it easier to perform
multi-label imaging.

1.3 Fluorescence Hyper-spectral Imaging (fHSI)
Spectral overlap is one of the biggest challenges in multi-label imaging in
fluorescence microscopes. As mentioned above (section 1.2.1), one of the most
important characteristics of fluorescence molecules that we can use to identify them
is their emission spectra. Each fluorophore has a particular fluorescent emission
14
spectrum, the visible electromagnetic wavelength of the fluorescent emission
spectrum is approximately from 400 to 700 nm. Fluorescent spectra are broad
across the visible wavelength such that spectral overlap is a common phenomenon
between multiple fluorescent labels. Only a limited number of fluorophores can be
simultaneously imaged in this region without bleed-through artifacts. When imaging
multiple fluorescent labels in widefield fluorescence microscopy, experiments are
also subject to artifacts arising from intrinsic autofluorescent
3
signals. In these
cases, a technique known as multi-spectral (or hyperspectral) imaging can be
employed to provide an extra spectral dimension to unmix the overlapping signals
and resolve the spatial contribution of each fluorophore.
Fluorescence Hyperspectral Imaging (fHSI)
20–24
is a technique where the entire
visible spectrum is collected for each pixel in an image, producing a spectral cube
with dimensions (x, y, wavelength) for each imaging plane. fHSI overcomes the
limitations of imaging multi-labeled samples in widefield fluorescence microscopy,
enabling the separation of fluorescent proteins with overlapping spectra from the
endogenous fluorescent contribution
7–9
. The imaging instrument captures
successive images of a sample at different excitation or emission bandwidths to
achieve hyperspectral imaging. One simple strategy is to capture the image one
wavelength band at a time, usually with a collection of narrow bandpass filters. In
this configuration, the bandpass filter width determines the spectral channel width in
each scan. The filter-based approach has some significant drawbacks, including the
potential photobleaching and phototoxicity caused by repetitive scanning of the
sample, photon inefficiency as most of the emission photons are rejected, and time
15
inefficiency as this requires many images. A more versatile configuration we used in
our setup, is to employ a spectrally resolved detector array (multichannel
photomultiplier in our configuration) and diffraction gratings to split the emission
wavelength into multiple spectral bands. It acquires fluorescent images in a large
number of spectral bands simultaneously, thus greatly enhancing the acquisition
speed. The emission light is separated into component colors using a diffraction
grating and the selected spectral range of emission is obtained in each spectral
channel by the multichannel photomultiplier. In our configuration, the
electromagnetic wavelength of detection is split into 32 bands with a bandwidth of
8.9 nm, where each spectral channel only captures signals within a small range of
emission wavelength.
fHSI empowers scientists with a more complete insight into biological systems with
multiplexed information deriving from observation of the full spectrum for each point
in the image
7
. fHSI has been used to identify a large number of fluorophores and
autofluorescent signals. This is especially important in the field of biology where
tissues, proteins, and their functions within organisms are deeply intertwined, and
there remain numerous unanswered questions regarding the relationship between
individual components6. However, fHSI has been limited in the past by its
complexity of acquisition and by both the considerable expertise and the
computational time required for its analytical approaches. fHSI requires special
optical and hardware components for the instrument to capture images at different
excitation or emission bandwidths. A spectral image cube is often composed of a
large number of spectra in a high dimension, with essentially the whole
16
electromagnetic spectrum being represented at each pixel location. The data are
large in size and complex as it is impossible to interpret by visual inspection.
Analysis of lambda stacks can be both time-consuming and computationally
expensive.
Dimensional reduction strategies to mitigate bleed-through artifacts and extract
spectral information are commonly used to analyze multidimensional fHSI data. One
strategy is to construct fixed spectral envelopes from the first three components
produced by principal component analysis (PCA) or independent component
analysis (ICA), converting a hyperspectral image to a three-band visualization
10–14
.
The main advantage of spectrally weighted envelopes is that they can preserve the
human-eye perception of hyperspectral images. Each spectrum is displayed with
the most similar hue and saturation for tri-stimulus displays for the human eye to
easily recognize details in the image
15
. A drawback to approaches such as Singular
Value Decomposition to compute PCA bases and coefficients, or generate the best
fusion weights is that it can take numerous iterations for convergence
21
.
Considering that fHSI datasets easily exceed the Gigabytes range and cross the
Terabytes threshold
6
, such calculations will be both computationally and time
demanding. Furthermore, most visualization approaches have focused more on
interpreting spectra as RGB colors and not on exploiting the full characterization
that can be extracted from the spectral data.
17
1.4 Fluorescence Lifetime Imaging Microscopy (FLIM)
Time-resolved measurements provide insights to resolve dynamic biological
processes and therefore are great tools for cellular imaging in biological studies.
Fluorescence lifetime, rather than its intensity, is an intrinsic characteristic of
fluorescence and different fluorophores can have unique lifetimes. Scientists can
use lifetimes to characterize fluorophore species and separate labels.
Fluorescence Lifetime Imaging Microscopy (FLIM) is a technique that collects
lifetime decay information for each pixel of an image, producing a three-dimensional
data cube with dimensions of space and lifetime (x, y, lifetime). As mentioned in
section 1.2.1, an excited fluorophore relaxes to the ground state with a probability
based on the decay rates through a number of different decay pathways. The
fluorescence we observe is the excited fluorescent molecules emitting photons
during the process of relaxation to the ground state from singlet state. The
fluorescence emitted will decay over time according to 𝐼(𝑡)=𝐼
!
𝑒
"#/%
. The signal
intensity 𝐼 of each pixel in FLIM data is related to the lifetime 𝜏 of fluorophores,
which allows one to view contrast with different fluorescence decay rates. Thus,
FLIM data is independent of their emission spectra and can be of great use when
spectra overlap.
Intrinsic fluorescent signals, often called autofluorescence, are widely used as
indicators of cellular metabolism in fluorescence imaging and often linked to disease
progression. Autofluorescence is the natural emission of fluorescence by biological
structures so it doesn’t require extrinsic fluorescent labels in the sample. There are
18
a number of molecules emitting autofluorescence that are known to be reporters for
metabolic activity in tissues. These include Nicotinamide adenine dinucleotide
(NAD) in its free and bound states
4–7
, FAD8, Folic acid, as well as retinoids
9
.
Different types of tissue, and components of the extracellular matrix, such as
collagen, are also known to emit intrinsic fluorescence
10
. During organ
development, tissue injury and regeneration, tumorigenesis, and aging, the balance
of these autofluorescent signals changes, reflecting cellular processes such as
proliferation, differentiation, apoptosis, and migration
3,11
. FLIM to measure intrinsic
signals, offers insights into the metabolic and bio-molecular status of cells in real-
time and in their natural 3D environment, non-invasively. This technique has been
widely used to measure NADH/NADPH, and FAD and to identify cellular
proliferation or migration for cancer applications
5,7,8,13–19
. Most cellular pathways are
oxidation and reduction reactions (redox reactions), which are biochemical reactions
with the transfer of electrons. Electrons are carried through by coenzymes and one
important coenzyme in this process is Nicotinamide adenine dinucleotide (NAD).
NAD has two separate forms, reduced Nicotinamide adenine dinucleotide (NADH)
and oxidized Nicotinamide adenine dinucleotide (NAD+). NAD+ is not fluorescent
but NADH is autofluorescent. Scientists use this knowledge to image
autofluorescent signals in vivo for metabolic mapping of redox reactions. NADH has
sensitive lifetimes when their microenvironments change while the changes in their
excitation and emission wavelength are not substantial. Measuring NADH provides
scientists great insights to measure metabolism in vivo.
19
The time-domain FLIM and frequency-domain FLIM are two commonly used
methods to measure the lifetime information of every pixel in a fluorescence image.
The time-domain methods measure the time it takes for fluorophores to emit
photons once excited. A sample is excited with a pulse of light, the time-dependent
intensity is measured rapidly following the excitation pulse. The decay time 𝜏 is
calculated from the time at which the intensity decreases to 1/𝑒 of the intensity at
𝑡 =0 (Figure 1.2). A popular approach for time-domain measurements is time-
correlated single-photon counting (TCSPC). TCSPC measures the time between
excitation and the arrival of the emitted photon at the detector. It is usually achieved
by a pulsed laser source, a single photon sensitive detector, optical filters to
separate fluorescence signal from excitation light, and a TCSPC unit to measure the
time between fluorescence excitation and emission. Typically, a fast-gated image
intensifier of the TCSPC module serves as the detector to measure the time
difference between the laser pulse and the arrival of fluorescence photons. As
fluorescence emission is a stochastic process, it is important to collect as many
photons as possible to obtain a decent resolution of the parameter values to resolve
the decay time.
Another approach to measuring the decay time is the frequency-domain method.
The principles of frequency-domain FLIM are to extract the difference between the
modulation and the phase of emitted fluorescence with respect to excitation to
calculate lifetime. In frequency-domain FLIM, the sample is excited with a
modulated laser with a sinusoid shape. The emission of the fluorophore responds at
the same modulation frequency as the excitation frequency. The fluorophore lifetime
20
introduces a delay in time in emission with respect to excitation, which appears as a
shift to the right and a decrease in modulation depth as shown in Figure 1.4. This
shift is measured as a phase shift and can be used to calculate the decay time.
Fluorophores with shorter lifetimes have a smaller phase difference and a smaller
modulation decrease, while fluorophores with larger lifetimes will have a larger
phase difference and a more significant modulation decrease. Phase shift and
modulation depth change can be derived from the detectors, and with this
information, we can calculate fluorescence lifetime.

Figure 1.4 Frequency-domain FLIM measured by detecting the delay in time (phase shift)
and modulation decrease with respect to the excitation. Fluorophores are excited by a
modulated laser, which then introduces an emission at the same frequency but with a
delay in time (phase shift) and a decrease in modulation depth. Phase shift and
modulation depth can be extracted from the detectors, and with this information, we can
calculate fluorescence lifetime. Fluorophores with large lifetimes always have a large
phase shift and a big modulation decrease, fluorophores with smaller lifetimes have a
smaller phase shift and a bigger modulation decrease.

21
1.5 The imaging challenge of harvesting information in biomedical
specimens
1.5.1 Live fluorescence imaging trade-offs
Live-cell imaging brings advantages as well as challenges throughout the processes
from acquisition to analysis. Images with high signal-to-noise ratios often benefit
analysis but the ability to acquire such images is limited by several parameters
including fluorescent probe brightness, the pixel dwell time for each pixel, and
illumination power. Dwell time is the length of time that the illumination laser
remains in a unit of space corresponding to a single pixel in the image. The longer
the dwell time, the more photons can be collected from the pixel and thus yields
better signal-to-noise ratios. Live cells demand appropriate environments during
acquisition. Cells are susceptible to phototoxicity and photobleaching.
Photobleaching could be caused by the cleaving of covalent bonds between the
fluorophores and surrounding molecules. These irreversible modifications in
covalent bonds are caused by the transition from a singlet state to the triplet state of
the fluorophores. The excitation laser used in fluorescent microscopes, even with
brief illumination, can potentially damage the cells. High-intensity illumination and/or
prolonged light exposure cause photobleaching and phototoxicity during live
fluorescence imaging. Using a high illumination power or a long pixel dwell time can
increase the fluorescence signal strength. However, long pixel dwell time in
confocal imaging, strong laser power, and slow imaging speed increase the chance
of phototoxicity and photobleaching in live fluorescence imaging.
22
Live imaging requires a constant balance between achieving the necessary
resolution, and signal-to-noise ratios, and not damaging the cells. Scientists need to
balance the best resolution, imaging speed, imaging depth, duration, and best cell
health to obtain optimal results.

1.5.2 Microscopy data suffer from low signals
Fluorescence microscopy data suffer from low signals. Scientists need to limit the
illumination power and dwell time to prevent photobleaching and phototoxicity, but
by limiting the excitation power and the time that the scanned laser beam allowed to
remain in the pixel, only a limited number of fluorescence photons are collected to
generate the fluorescence images.
Fluorescent data are inherently noisy and fluorescence noise cannot be omitted.
The captured lifetime signals at each pixel present a fluctuation behavior when
photon counts are low in the image. A simple example of this behavior is as tossing
a fair coin and counting the heads and tails with only a handful of trials. These
fluctuations are from counting error, which is approximately 𝑛±√𝑛 (n is the count).
This type of error can be approximated using Poisson distribution and therefore is
known as Poisson noise or counting error. Poisson noise in fluorescence data is not
additive but is embedded in the original signal, which displays a non-linear
23
relationship between the noise components and the original data. Figure 1.5a
shows what fluorescence spectra look like in fluorescence data. The single
spectrum is contaminated by stochastic noise, and it requires sufficient data points
to obtain a smooth spectral curve (Figure 1.5b). The setting of acquisition
parameters on the instrument and the choice of analysis is often challenging for live
samples.

Figure 1.5 Spectra extracted from fluorescence images show fluorescence data are
inherently noisy and noise is embedded in the spectra. (a) Examples of single spectrum
extracted from fluorescence data. The three spectra are from three single-labeled
fluorescence images and each spectrum is extracted from one single pixel of the image.
Each “single” spectrum has fluctuations in the spectral curve due to noise. (b) Examples
of the averaged spectrum of each single-labeled fluorescence data. Each spectrum is the
average of 1,000 spectra in each image. Noise is embedded in original signals as shown
in each “single” spectrum and it requires a large number of data points to obtain a smooth
spectrum (as shown in the “average” spectrum).

In general, measured fluorescent signal S [photon/pixel] will depend on Quantum
Efficiency QE, background 𝐼
&
, noise factor 𝐹
'
, readout noise 𝑁
(
and gain 𝐺 of the
system. The noise will be proportional to:
24

Where the ideal fluorescent signal (𝑄𝐸∗𝑆) is embedded in the noise. The only
additive component is the background noise. The remaining parts of the noise
follow Poisson distribution and is not correlated to the fluorescent
signal(Hamamatsu Photonics K.K.Photomultiplier Technical Handbook (1994)
Hamamatsu Photonics K.K). It is true that for large numbers the Poisson distribution
approaches normal distribution, becoming closer to the Gaussian noise described in
the Independent Component Analysis (ICA) modeling in Chapter 2. However, this is
not the general case in fluorescence. In our instrument (Zeiss LSM 780) at 800
gain, the conversion factor photons to Digital Levels (DL): for autofluorescence the
high DL intensities peak around 4000 DL, hence ~20 photons or less. For
fluorescent data, we averaged spectra from 1000 pixels to achieve a smooth
spectrum in figure 1.5. Hence the gaussian distribution approximation of a Poisson
distribution is generally not achieved.
Photon-starved signals combined with inherent noise make fluorescence data
analysis extremely challenging. This photon-starved signal has an energy
distribution which is the fluorescence emission spectrum of the protein/dye.
Consequently, photon-starved spectra appear different from the expected.
Decomposition methods that require statistical independence of components, such
as Principal Component Analysis (PCA), do not properly account for these factors
and perform a noise reduction with decreased quality. We need a smart way to
25
denoise fluorescent spectra and enhance the wanted signals for image data
analysis.

1.5.3 Multiplexing information in higher dimension
Until the late 20th century, optical microscopy was limited to only capturing spatial
and temporal dimensions. 3D volumetric imaging allows scientists to acquire a stack
of 2D optical slices to better map the spatial structures of living specimens in their
natural 3D space. Video microscopy collects time-lapse images to observe the
development of biological processes and changes that are time sensitive.
Volumetric and time-lapse imaging enables the observation of spatio-temporal
dynamics in the life sciences. However, such imaging techniques become
insufficient to understand complex phenomena at molecular, cellular, and tissue
levels. For instance, the number of labels that scientists can image is limited even in
3D or furthermore 3D evolution over time. It remains an open challenge to unravel
the complexity of biological systems. There is a need for a high-dimensional
acquisition of biological events with advanced microscopy techniques.
Advanced microscopes allow scientists to capture information that characterizes
fluorophore properties, such as spectrum, lifetime, etc. These extra dimensions
capture fluorophores’ properties in biological processes and contain underlying
fundamentals that need to be unraveled to understand life sciences. In our
microscope setup, an optimized characterization of fluorescence lifetime signals is
26
captured using a multispectral approach, splitting the collected signal into separate
wavelength bands between 400 to 700 nm to detect different tissue components.
The drawback of acquiring vast multidimensional spectral or lifetime information is
an increase in complexity and computational time for the analysis, showing
meaningful results only after lengthy calculations. The multi-dimensional multimodal
datasets are high in information density, therefore visualizing and understanding
these large multi-dimensional datasets can be challenging.
In the current fluorescence microscopy world, data sets are large, easily in the tens
of Gigabytes. It is common to see a hyperspectral set with spectra numbering in the
billions (1.85 x 109 for Chapter 2 Figure 2.5 for example) with a lower number of
spectral bands compared to remote sensing. Given the intrinsic biological variability
and the usually low SNR, it is common to acquire several samples (in our
experiments ~10) for each condition and each control, positive and negative, which
makes the data size even larger.
While advances in imaging techniques promise a deeper understanding of
biological processes, current image acquisition and analysis pipelines are not
designed to properly interpret and extract information from this type of data.
Scientists need to decode this complex information captured in the spectral, lifetime,
or other domains. Therefore, there is a need to evaluate mechanisms in biological
processes with analysis tools that permit the extraction of fluorophore properties.

27
1.6 Projects and opportunities to access highly multiplexed
information for fluorescent signals
Analysis of large-scale multi-dimensional datasets is another pain point in
understanding microscopy data, especially when data has complex information and
suffers from low signal-to-noise ratios. A common preprocessing step to guide the
scientists during acquisition before performing a full-fledged analysis can optimize
the computational time. It is also advantageous to perform an informed
visualization of the spectral data during acquisition to optimize experimental time,
especially for lengthy time-lapse recordings. Such preprocessing visualization
allows scientists to evaluate image collection parameters within the experimental
pipeline as well as to choose the most appropriate processing method. However,
the challenge is to rapidly visualize subtle spectral differences with a set of three
colors, compatible with displays and human eyes, while minimizing the loss of
information. As the most common color model for displays is RGB, where red,
green, and blue are combined to reproduce a broad array of colors, hyper- or
multispectral datasets are typically reduced to three channels to be visualized.
Thus, spectral information compression becomes the critical step for the proper
display of image information.
In Chapter 2, we developed Spectrally Encoded Enhanced Representations
(SEER), an approach for improved and computationally efficient simultaneous color
visualization of multiple spectral components of hyperspectral fluorescence images.
Exploiting the mathematical properties of the phasor method, we transform the
28
wavelength space into information-rich color maps for RGB display visualization.
We present multiple biological fluorescent samples and highlight SEER’s
enhancement of specific and subtle spectral differences, providing a fast, intuitive
and mathematical way to interpret hyperspectral images during collection, pre-
processing, and analysis. This means that we can push beyond the capabilities of
the human eye and capture the infinitesimally small differences in the spectrum.
This technique has great potential to benefit surgical practices, disease diagnostics,
and food and health services.
Another pain point we want to address in this work is the analysis of complex
biological data. Biomedical imaging data is rich in information. Biological processes
can be more accurately understood by considering fluorophore properties, such as
fluorescence lifetime and emission spectra. This information maps the data to the
next level of dimensionality. Multi-dimensional biomedical imaging data contains
information more than just spatial structures we usually see in a general RGB
image. However, most analyses of biomedical image data focus only on the spatial
localization of fluorescent signals, thereby ignoring alternative dimensions that could
expand comprehension of disease states.
In Chapter 3, We developed a workflow using SpectraFLIM imaging to distinguish
injured tissues from healthy tissues. Fluorescence Lifetime Imaging and
Fluorescence Hyperspectral/multi-spectral Imaging are powerful, orthogonal
imaging techniques to monitor spatial and temporal dynamics in living organisms.
Spectral and fluorescent lifetime multimodal imaging (SpectraFLIM) combines these
two techniques to unravel the complexity of biology at cellular resolution.
29
We explored the SpectraFLIM data with deep learning methods using a 2-
dimensional Convolutional Neural Network (2-D CNN) with inputs from spectral
and/or lifetime domains. In addition, we developed a convolutional long short-term
memory (ConvLSTM) network to classify complex SpectraFLIM data, which extracts
spatial features while preserving spectral correlations across all channels. Our
results show that our method significantly improves classification accuracy when
integrating spectral and FLIM data into a deep learning model, which is of particular
interest for in vivo imaging-based diagnostics.
The current trend in data acquisition is to collect data larger and larger as the
resolution of instruments upgrades and image quality demands. Other than
SpectraFLIM, there are a number of existing tools that allow us to acquire
information with combined imaging modalities, which increases the dimensionality
of the image data to the next level. In Chapter 4, we explore the potential
improvements with our approaches in complex image analysis.

30
Chapter 2 Spectrally Encoded Enhanced
Representations (SEER)
2 Spectrally Encoded Enhanced Representations
(SEER)
2.1 Problems in visualizing hyperspectral data
The drawback of acquiring vast multidimensional spectral information is an increase
in complexity and computational time for the analysis, showing meaningful results
only after lengthy calculations. To optimize experimental time, it is advantageous to
perform an informed visualization of the spectral data during acquisition, especially
for lengthy time-lapse recordings, and before performing analysis. Such
preprocessing visualization allows scientists to evaluate image collection
parameters within the experimental pipeline as well as to choose the most
appropriate processing method. However, the challenge is to rapidly visualize subtle
spectral differences with a set of three colors, compatible with displays and human
eyes, while minimizing the loss of information. As the most common color model for
displays is RGB, where red, green, and blue are combined to reproduce a broad
array of colors, hyper- or multi-spectral datasets are typically reduced to three
channels to be visualized. Thus, spectral information compression becomes the
critical step for the proper display of image information.
31
2.2 Limitations of current approaches
Dimensional reduction strategies are commonly used to represent multidimensional
fHSI data
10
. One strategy is to construct fixed spectral envelopes from the first three
components produced by principal component analysis (PCA) or independent
component analysis (ICA), converting a hyperspectral image to a three-band
visualization
10–14
. The main advantage of spectrally weighted envelopes is that they
can preserve the human-eye perception of hyperspectral images. Each spectrum is
displayed with the most similar hue and saturation for tri-stimulus displays for the
human eye to easily recognize details in the image
15
. Another popular visualization
technique is pixel-based image fusion, which preserves the spectral pairwise
distances for the fused image in comparison to the input data
16
. It selects the
weights by evaluating the saliency of the measured pixel with respect to its relative
spatial neighborhood distance. These weights can be further optimized by
implementing widely applied mathematical techniques, such as Bayesian
inference
17
, by using a filters-bank18 for feature extraction
19
, or by noise
smoothing
20
.
A drawback to approaches such as Singular Value Decomposition to compute PCA
bases and coefficients, or generate the best fusion weights is that it can take
numerous iterations for convergence
21
. Considering that fHSI datasets easily
exceed the Gigabytes range and cross the Terabytes threshold
6
, such calculations
will be both computationally and time demanding. Furthermore, most visualization
approaches have focused more on interpreting spectra as RGB colors and not on
32
exploiting the full characterization that can be extracted from the spectral data. Our
approach is based on the belief that preserving most spectral information and
enhancing the distinction of spectral properties between relevant pixels, will provide
an ideal platform for understanding biological systems. The challenge is to develop
tools that allow efficient visualization of multi-dimensional datasets without the need
for computationally demanding dimensionality reduction, such as ICA, prior to
analysis.
2.3 Improving spectral compression with Phasor
In this work, we build maps based on Phasors (Phase Vectors). The Phasor
approach to fluorescence microscopy has multiple advantages deriving from its
properties for fluorescent signals22–25. After transforming the spectrum at each
pixel into its Fourier components, the resulting complex value is represented as a 2-
dimensional histogram where the axes represent the real and imaginary
components. Such a histogram has the advantage of providing a representative
display of the statistics and distributions of pixels in the image from a spectral
perspective, simplifying the identification of independent fluorophores. Pixels in the
image with similar spectra generate a cluster on the phasor plot. While this
representation is cumulative across the entire image, every single point on the
phasor plot is easily remapped to the original fluorescent image26–29.
Exploiting the advantages of the phasor approach, Hyper-Spectral Phasors (HySP)
have enabled the analysis of 5D hyperspectral time-lapse data semi-automatically
as similarly colored regions cluster on the phasor plot. These clusters have been
33
characterized and exploited for simplifying interpretation and spatially lossless
denoising of data, improving both collection and analysis in low signal conditions29.
Phasor analysis generally explores the 2d-histogram of spectral fingerprints utilizing
geometrical selectors22,23,25,27,28,30, which is an effective strategy but requires
user involvement. While capable of imaging multiple labels and separating different
spectral contributions as clusters, this approach is inherently limited in the number
of labels that can be analyzed and displayed simultaneously. Prior works directly
utilize phase and modulation for quantifying, categorizing, and representing features
within Fluorescence Lifetime and Image Correlation Spectroscopy data31–33. Our
method differs from the previous implementations22,23,25,27,28,30, as it focuses
instead on providing a mathematically constructed, holistic pre-processing
visualization of large hyperspectral data.
The solution we propose is to extract from both the whole denoised phasor plot and
image to reconstruct a “one-shot” view of the data and its intrinsic spectral
information. Spectrally Encoded Enhanced Representations (SEER) is a
dimensionality reduction-based approach, achieved by utilizing phasors, and
automatically creating spectrally representative color maps. The results of SEER
show an enhanced visualization of spectral properties, representing distinct
fluorophores with distinguishable pseudo-colors and mathematically highlighted
differences between intrinsic signals during live imaging. SEER has the potential of
optimizing the experimental pipeline, from data collection during acquisition to data
analysis, greatly improving image quality and data size.
34
2.4 Improving visualization with Spectrally Encoded Enhanced
Representations (SEER)
2.4.1 SEER workflow
The execution of SEER has a simple foundation. Each spectrum is assigned a
pseudo-color, based on its real and imaginary Fourier components, through a
reference color map. This concept is illustrated in detail in Figure 2.1 using an
example of the Zebrabow34 embryo dataset, where cells within the sample express
different ratios of cyan, yellow, and red fluorescent proteins, resulting in a wide-
ranging pallet of discrete spectral differences. The data is acquired as a
hyperspectral volume (x, y, z, λ) (Figure 2.1A), providing a spectrum for each voxel.
The spectra obtained from multiple regions of interest are complex, showing both
significant overlap and the expected difference in ratios (Figure 2.1B).
Discriminating the very similar spectra within the original acquisition space is
challenging using standard multispectral dataset visualization approaches (Figure
2.1C).
SEER was designed to create usable spectral contrast within the image by
accomplishing five main steps. First, the Sine and Cosine Fourier transforms of the
spectral dataset at one harmonic (usually 1st or 2nd owing to Riemann surfaces)
provide the components for a 2D phasor plot (Figure 1D). The phasor
transformation compresses and normalizes the image information, reducing a multi-
35
dimensional dataset into a 2D-histogram representation and normalizing it to the
unit circle.
Second, the histogram representation of the phasor plot provides insight into the
spectral population distribution and improvement of the signal through the
summation of spectra in the histogram bins. Pixels with very similar spectral
features, for example expressing only a single fluorophore, will fall within the same
bin in the phasor plot histogram. Because of the linear property of the phasor
transform, if an image pixel contains a mixture of two fluorophores, its position on
the phasor plot will lie proportionally along the line connecting the phasor
coordinates of those two components. This step highlights the importance of
geometry and distribution of bins in the phasor representation.
Third, spatially lossless spectral denoising, previously presented29, is performed 1-
2 times in phasor space to reduce spectral error. In short, median filters are applied
on both the Sine and Cosine transformed images, reducing the spectral scatter
error on the phasor plot while maintaining the coordinates of the spectra in the
original image (Figure 2.1E). Filters affect only the phasor space, producing an
improvement of the signal.
Fourth, we designed multiple SEER maps exploiting the geometry of phasors. For
each bin, we assign RGB colors based on the phasor position in combination with a
reference map (Figure 2.1F). Subtle spectral variations can be further enhanced
with multiple contrast modalities, focusing the map on the most frequent spectrum,
36
the statistical center of mass of the distribution, or scaling the colors to the extremes
of the phasor distribution (Figure 2.1G).
Finally, colors in the original dataset are remapped based on SEER results (Figure
2.1H). This permits a dataset in which spectra are visually indistinguishable (Figure
2.1A-C) to be rendered so that even these subtle spectral differences become
readily discernible (Figure 2.1I). SEER rapidly produces 3 channel color images that
approximate the visualization resulting from a more complete spectral unmixing
analysis.

Figure 2.1 Spectrally Encoded Enhanced Representations (SEER) conceptual
representation. (a) A multispectral fluorescent dataset is acquired using a confocal
instrument in spectral mode (32-channels). Here we show a Tg(ubi:Zebrabow)
34
dataset
where cells contain a stochastic combination of cyan, yellow and red fluorescent proteins.
(b) Average spectra within six regions of interest (colored boxes in a) show the level of
overlap resulting in the sample. (c) Standard multispectral visualization approaches have
37
limited contrast for spectrally similar fluorescence. (d) Spectra for each voxel within the
dataset are represented as a two-dimensional histogram of their Sine and Cosine Fourier
coefficients S and G, known as the phasor plot. Spatially lossless spectral denoising is
performed in phasor space to improve signal29. (f) SEER provides a choice of several
color reference maps that encode positions on the phasor into predetermined color
palettes. The reference map used here (magenta selection) is designed to enhance
smaller spectral differences in the dataset. (g) Multiple contrast modalities allow for
improved visualization of data based on the phasor spectra distribution, focusing the
reference map on the most frequent spectrum, on the statistical spectral center of mass of
the data (magenta selection), or scaling the map to the distribution. (h) Color is assigned
to the image utilizing the chosen SEER reference map and contrast modality. (i) Nearly
indistinguishable spectra are depicted with improved contrast, while more separated
spectra are still rendered distinctly.

2.4.2 SEER colormap design
Standard Reference Maps
Biological samples can include a multitude of fluorescent spectral components,
deriving from fluorescent labels as well as intrinsic signals, each with different
characteristics and properties. Identifying and rendering these subtle spectral
differences is the challenge. We found that no one rendering is sufficient for all
cases, and thus created four specialized color map references to enhance color
contrast in samples with different spectral characteristics. To simplify the testing of
the color map references, we designed a Simulated Hyperspectral Test Chart
(SHTC), in which defined areas contain the spectra we obtained from CFP, YFP,
and RFP zebrafish embryos. Each section of the test chart offers different image
contrast, obtained by shifting the CFP and RFP spectra maxima position with
respect to the YFP spectrum (figure 3). We render the SHTC in a grayscale image
and with SEER for comparison (Figure 2.5a). These representations can be rapidly
shown separately to determine which has the highest information content.
38

Figure 2.2 Simulated Hyperspectral Test Chart I rendered in TrueColor shows nearly
indistinguishable spectra. The simulation is represented here in “TrueColor RGB”
(Appendix). 𝑆
!
, 𝑆
"
, and 𝑆
#
spectra acquired respectively from CFP, YFP, and RFP
zebrafish embryos are used to generate an (a-i) 3-by-3 Simulated Hyperspectral Test
Chart. In each panel (a-i) of the chart, three spectra (𝑆
!
to 𝑆
#
) are represented as
concentric squares (see panel a) outer: –1 - blue, intermediate: –2 - yellow, middle: –3 -
red spectra respectively). The spectrum 𝑆
"
(intermediate square in each panel) is kept
unchanged in all panels. The maximum of spectrum 𝑆
!
is shifted by d1 (-2 wavelength
bins, -17.8 nm steps) with respect to the fixed spectrum 𝑆
"
maximum. 𝑆
#
max value is
shifted by d2 (2 wavelength bins, 17.8 nm steps) respect to 𝑆
"
maximum. The changes are
applied for 2 steps along the vertical (d1) and horizontal (d2) axis of the center panel
assembly (a-i), starting from d1=d2=0 (panel a). The spectra utilized in each panel (a-i)
are represented in panels j-r. Each plot (j-r) represents the averaged normalized 𝑆
!
- 𝑆
#

spectra as 32 wavelength bins, 8.9nm bandwidth, 410-695 nm detection. Each panel has
different visual contrast but is generally difficult to distinguish by eye due to significant
overlap in spectra. (s) R, G, B channels used in the Gaussian Kernel for True color
representation (red, green, blue lines) and average spectrum for panels (a-i) (yellow line)
for reference.

39
A reference map is defined as an organization of the palette where each spectrum
is associated with a color based on its phasor position. The color distribution in each
of the reference maps is a function of the coordinates of the phasor plot. In the
Angular map (Figure 2.5b), hue is calculated as a function of angle, enhancing
diversity in colors when spectra have different center wavelengths (“phases”) on the
phasor plot. For the Radial map (Figure 2.5c), we assign colors with respect to
different radii, highlighting spectral amplitude and magnitude. The radial position is,
in general, related to the intensity integral of the spectrum, which in turn can depend
on the shape of the spectrum, with the null intensity localizing at the origin of the
plot (Figure 2.4). In our simulation (Figure 2.5c), the colors obtained with this map
mainly represent differences in shape, however, in a scenario with a large dynamic
range of intensities, colors will mainly reflect changes in intensity, becoming
affected, at low signal-to-noise, by the uncorrelated background (Figure 5). In the
Gradient Ascent and Descent models (Figure 2.5d, e), the color groups differ
according to angle as seen in the Angular map with an added variation of the color
intensity strength in association with changes in the radius. Gradient maps enhance
similar properties as the Angular map. However, the Gradient Ascent (Figure 2.5d)
map puts a greater focus on distinguishing the higher-intensity spectra while de-
emphasizing low-intensity spectra; whereas the Gradient Descent (Figure 2.5e)
map does the opposite, highlighting the spectral differences in signals with low
intensity. The complementary attributes of these four maps permit renderings that
distinguish a wide range of spectral properties in relation to the phasor positions. It
is important to note that the idea of Angular and Radial maps have been previously
40
utilized in a variety of applications and approaches32,33 and are usually introduced
as “Phase” and “Modulation”, respectively. Here, we have recreated and provided
these maps for our hyperspectral fluorescence data as simpler alternatives to our
more adaptable maps.

Figure 2.3 Effect of spectral shape with constant intensities on the radial map in absence
of background. This simulation shows spectra with Gaussian shape and different standard
deviations using 32 wavelength bins, 8.9nm bandwidth, and a 410-695 nm range in the
absence of background. All spectra are centered on 543 nm (channel 16) and the integral
of intensities is kept constant. (a-l) For each value of the standard deviation, a grayscale
image and SEER visualization are presented. The map used is the Radial map centered
on the origin and extended to the border of the phasor plot. A color reference is added in
41
the phasor plot (m). Clusters on the phasor plot are distributed along the radius, with
distance from the origin inversely proportional to the standard deviation.

Figure 2.4 Effect of spectral intensity in presence of background on radial map. In this
simulation, the first panel (top-left) of the Simulated Hyperspectral Test Chart (Figure 2.2)
is reduced in intensity by a factor of 10
1
- 10
4
(panels 1-4 respectively) in the presence of a
constant background. Background with an average intensity of 5 digital levels was
generated in MATLAB; Poisson noise was added using the 𝑝𝑜𝑖𝑠𝑠𝑟𝑛𝑑() function.
Grayscale images (a, d, g, j) are scaled by (a) factor of 10, (d) factor of 10
2
, (g) factor of
10
3
, and (j) factor of 10
4
. Radial map (original) visualization shows a shift of panel colors
toward blue with decreasing intensities (b, e, h, k). The phasor plots (c, f, i, l) (harmonic
n=2) show a radius shift of the clusters toward the origin. Radial map reference is added
in (c). (m) The absolute intensities plot shows the average spectrum for the four panels,
maximum peak values are 1780, 182, 23, and 7 digital levels (panels 1-4 respectively).
The normalized intensity spectra (n) show an apparent broadening of the shape of spectra
with the decreasing signal-to-noise.

The Standard Reference Maps simplify comparisons between multiple fluorescently
labeled specimens as the palette representation is unchanged across samples.
42
These references are centered at the origin of the phasor plot; hence their color
distributions remain constant, associating a predetermined color to each phasor
coordinate. Fluorophore positions are constant on the phasor plot29, unless their
spectra are altered by experimental conditions. The ability of the Standard
Reference Maps to capture either different ratios of labels or changes in a label,
such as a calcium indicator, offers the dual advantage of providing a rapid,
mathematically improved overview and simplifying the comparison between multiple
samples.

Figure 2.5 Spectrally Encoded Enhanced Representation (SEER) designs. A set of
Standard Reference Maps and their corresponding result on a Simulated Hyperspectral
Test Chart (SHTC) designed to provide a gradient of spectral overlaps between spectra.
(a) The Standard phasor plot with the corresponding average grayscale image provides
43
the positional information of the spectra on the phasor plot. The phasor position is
associated with a color in the rendering according to a set of Standard Reference Maps,
each highlighting a different property of the dataset. (b) The angular map enhances
spectral phase differences by linking color to changes in angle (in this case, with respect
to origin). This map enhances changes in maximum emission wavelength, as phase
position in the plot is most sensitive to this feature and largely agnostic to changes in
intensity. (c) The radial map, instead, focuses mainly on intensity changes, as a decrease
in the signal-to-noise generally results in shifts toward the origin on the phasor plot. As a
result, this map highlights spectral amplitude and magnitude and is mostly insensitive to
wavelength changes for the same spectrum. (d) The gradient ascent map enhances
spectral differences, especially within the higher-intensity regions in the specimen. This
combination is achieved by adding a brightness component to the color palette. Darker
hues are localized in the center of the map, where lower image intensities are plotted. (e)
The gradient descent map improves the rendering of subtle differences in wavelength.
Color bars for b, c, d, and e represent the main wavelength associated with one color in
nanometers. (f) The tensor map provides insights into statistical changes in spectral
populations in the image. This visualization acts as a spectral edge detection on the
image and can simplify the identification of spectrally different and infrequent areas of the
sample such as the center of the SHTC. The color bar represents the normalized relative
gradient of counts.
Tensor map
The SEER approach provides a straightforward means to assess statistical
observations within spectral images. In addition to the four Standard Reference
Maps, we designed a tensor map that recolors each image pixel based on the
gradient of counts relative to its surrounding spectra (Figure 2.5f). Considering that
the phasor plot representation is a two-dimensional histogram of real and imaginary
Fourier components, then the magnitude of each histogram bin is the number of
occurrences of a particular spectrum. The tensor map is calculated as a gradient of
counts between adjacent bins, and each resulting value is associated with a color
based on a color map (here we use a “jet” color map).
The image is recolored according to changes in spectral occurrences, enhancing
the spectral statistical fluctuations for each phasor cluster. The change in frequency
44
of appearance can provide insights into the dynamics of the population of spectra
inside the dataset. A visible result is a type of spectral edge detection that works in
the wavelength dimension, facilitating the detection of chromatic changes in the
sample. Alternatively, the tensor map can aid in identifying regions that contain less
frequent spectral signatures relative to the rest of the sample. An example of such a
case is shown in the upper left quadrant of the simulation (Figure 2.5f) where the
center part of each quadrant has a different spectrum and appears with a lower
frequency compared to its surroundings.
Modes (Scale and Morph)
We have implemented two different methods to improve our ability to enhance
spectral properties: scaled mode and morphed mode.
Scaled mode provides an adaptive map with increased color contrast by
normalizing the Standard Reference map extreme values to the maximum and
minimum phasor coordinates of the current dataset, effectively creating the smallest
bounding unit circle that contains all phasor points (Figure 2.6b). This approach
maximizes the number of hues represented in the rendering by resizing the color
map based on the spectral range within the image. Scaled mode increases the
difference in hue and the contrast of the final false-color rendering. These
characteristics set the scaled mode apart from the Standard Reference Maps
(Figure 3.3a), which constantly cover the full phasor plot area and ease
comparisons between datasets. Scaled mode sacrifices this uniformity but offers
spectral contrast stretching that improves contrast depending on the values
45
represented in individual image datasets. The boundaries of the Scaled mode can
be set to a constant value across different samples to facilitate comparison.

Figure 2.6 Enhanced contrast modalities. For each SEER standard reference map design,
four different modes can provide improved contrast during visualization. As a reference,
we use the gradient descent map applied on a Simulated Hyperspectral Test Chart
(SHTC). (a) Standard mode is the standard map reference. It covers the entire phasor
plot circle, centering on the origin and anchoring on the circumference. The color palette is
constant across samples, simplifying spectral comparisons between datasets (b) Scaled
mode adapts the gradient descent map range to the values of the dataset, effectively
performing a linear contrast stretching. In this process the extremities of the map are
scaled to wrap around the phasor representation of the viewed dataset, resulting in the
largest shift in the color palette for the phase and modulation range in a dataset. (c) Max
Morph mode shifts the map center to the maximum of the phasor histogram. The
boundaries of the reference map are kept anchored to the phasor circle, while the colors
inside the plot are warped. The maximum of the phasor plot represents the most frequent
spectrum in the dataset. This visualization modality remaps the color palette with respect
to the most recurring spectrum, allowing insights into the distribution of spectra inside the
sample. (d) Mass Morph mode, instead, uses the histogram counts to calculate a
weighted average of the phasor coordinates and uses this color-frequency center of mass
as a new center for the SEER map. The color palette now maximizes the palette color
differences between spectra in the sample.

46
Morph mode exploits the dataset population properties captured in the image’s
phasor representation to enhance contrast. From the phasor histogram, either the
most frequent spectral signature or the center of mass (in terms of histogram
counts) is used as the new center reference point of the SEER maps. We call this
newly calculated center the apex of the SEER. The result is an adaptive palette that
changes depending on the dataset. In this representation mode, the edges of the
reference map are held anchored to the phasor plot circular boundary, while the
center point is shifted, and the interior colors are linearly warped (Figure 3c, d). By
shifting the apex, contrast is enhanced for datasets with off-centered phasor
clusters. A full list of the combination of Standard Reference Maps and modes is
reported (figure 6, 7) for different levels of spectral overlap in the simulations and
different harmonics. The supplement presents results for SHTC with very similar
spectra (figure 3), using the second harmonic in the transform (Appendix), and for
an image with a frequently encountered level of overlap (figure 8,9) using the first
harmonic. In both scenarios, SEER improves the visualization of multispectral
datasets (figures 6,7,9,10) compared to standard approaches (figure 3,8). A
detailed description of the choice of harmonic for visualization is presented in
Supplementary Note 1. Implementation of 1x to 5x spectral denoising filters29
further enhances visualization (figure 11,12).
47

Figure 2.7 Radial and Angular reference map designs and modes differentiate nearly
indistinguishable spectra (Simulated Hyperspectral Test Chart I). We present 4 different
modes that can be applied to each map. Here the second harmonic is utilized for the
calculations. Angular map (a) and Radial map (b) in Standard mode, Scaled mode, Max
Morph mode, and Mass Morph mode. In Standard mode, the reference map is centered at
the origin and limited by the phasor unit circle. In Scaled mode, the reference map adapts
to the phasor plot histogram, changing its coordinates to wrap around the edges of the
phasor clusters and enhancing the contrast of the chosen map properties. In Max Morph
mode, the map is centered on the spectrum with the highest frequency of appearance in
the phasor histogram. This mode improves sensitivity by using statistical frequency bias.
In Mass Morph mode, the map is centered on the weighted center of the phasor,
enhancing sensitivity for multiple small spectra. Visualizations are presented after 1x
spectral denoising.
48

Figure 2.8 Gradient Ascent and Descent reference map designs and modes differentiate
nearly indistinguishable spectra. Here the second harmonic is utilized for SEER. Gradient
Ascent map (a) and Gradient Descent map (b) in Standard mode, Scaled mode, Max
Morph mode, and Mass Morph mode. The two maps place a focus on very different
(Ascent) and similar (Descent) spectra by fading the reference map to dark at the center
and edges of the phasor plot unit circle respectively. Visualizations are presented after 1x
spectral denoising.
49

Figure 2.9 Simulated Hyperspectral Test Chart II and its standard overlapping spectra.
Simulated SHTC II was generated from the same zebrafish embryo datasets and the
same design used in SHTC1 (Figure 2.2) utilizing CFP, YFP, and RFP labeled samples
and a 3-by-3 block chart, with each block subdivided into 3 regions corresponding to
spectra 𝑆
!
, 𝑆
"
, and 𝑆
#
. The aim is to test scenarios with less overlapping spectra. We
change the shifting distance in this simulation to be d1 (-3 wavelength bins, -26.7nm
steps) and d2 (3 wavelength bins, 26.7nm steps). The channels used in the Gaussian
Kernel for TrueColor RGB representation here were 650nm, 510 nm, and 470nm which
respectively represent R, G, and B. The concentric squares in the lower right side of the
simulation are separated by a peak-to-peak distance of 53.6nm, with outer and inner
concentric squares well separated by 106.8nm. This distance is similar to the emission
gap between CFP (475 nm EM) and tdTomato (581 nm). Under these spectral conditions,
most methods are expected to perform well.
50

Figure 2.10 Radial and Angular reference map designs and modes rendering standard
overlapping spectra (Simulated Hyperspectral Test Chart II). Here the first harmonic is
utilized for SEER Angular map (a) and Radial map (b) in Standard mode, Scaled mode,
Max Morph mode, and Mass Morph mode are applied to the standard overlapping spectra
simulation. The reference maps show improved contrast consistently among different
modalities. Visualizations are presented after 1x spectral denoising.

51

Figure 2.11 Gradient ascent and descent reference map designs and modes
differentiation of standard overlapping spectra (Simulated Hyperspectral Test Chart II).
Here the first harmonic is utilized for SEER Gradient Ascent map (a) and Gradient
Descent map (b) in Standard mode, Scaled mode, Max Morph mode, and Mass Morph
mode. The reference maps provide enhanced visualization even in the scenario of spectra
overlapping at similar levels to commonly used fluorescent proteins. Visualizations are
presented after 1x spectral denoising.

52
2.4.3 Applications in overlapping fluorescent data
Color maps enhance different spectral gradients in biological samples
To demonstrate the utility of SEER and its modes, we present four example
visualizations of images taken from unlabeled mouse tissues and fluorescently
tagged zebrafish.

Figure 2.12 Autofluorescence visualization comparison for unlabeled freshly isolated
mouse tracheal explant. The sample was imaged using multi-spectral two-photon
microscopy (740nm excitation, 32 wavelength bins, 8.9nm bandwidth, 410-695 nm
detection) to collect the fluorescence of intrinsic molecules including folic acid, retinoids,
and NADH in its free and bound states. These intrinsic molecules have been used as
reporters for metabolic activity in tissues by measuring their fluorescence lifetime, instead
of wavelength, due to their closely overlapping emission spectra. This overlap increases
the difficulty in distinguishing spectral changes when utilizing an (a) TrueColor image
display (Zen Software, Zeiss, Germany) (b) The gradient descent morphed map shows
differences between apical and basal layers, suggesting different metabolic activities of
cells based on the distance from the tracheal airway. Cells on the apical and basal layers
(dashed boxes) are rendered with distinct color groups. The color bar represents the main
wavelength associated with one color in nanometers. (c) The tensor map image provides
insight into statistics in the spectral dataset, associating image pixels’ colors with the
corresponding gradient of phasor counts for pixels with similar spectra. The spectral count
gradients in this sample highlight the presence of fibers and edges of single cells. The
color bar represents the normalized relative gradient of counts. (d) Average spectra for the
cells in dashed boxes (1 and 2 in panel c) show a blue spectral shift in the direction of the
apical layer. (e) Fluorescence Lifetime Image Microscopy (FLIM) of the sample, acquired
using a frequency domain detector validates the interpretation from panel b, Gradient
Descent Map, where cells in the apical layer exhibit a more Oxidative Phosphorylation
53
phenotype (longer lifetime in red) compared to cells in the basal layer (shorter lifetime in
yellow) with a more Glycolytic phenotype. The selections correspond to areas selected in
phasor FLIM analysis (e, top left inset, red and yellow selections) based on the relative
phasor coordinates of NAD+/NADH lifetimes35.

Figure 2.13 Phasor Fluorescence Lifetime Imaging Microscopy (FLIM) of unlabeled freshly
isolated mouse tracheal explant. (a) Phasor FLIM representation of fluorescence lifetime
data for unlabeled freshly isolated mouse tracheal explant acquired in frequency domain
utilizing a 2-photon fluorescence microscope (LSM 780, Zeiss, Jena) tuned at 740 nm,
coupled with an acquisition unit with Hybrid Detectors (FLIM Box, ISS, Urbana-
Champaign). The selected regions correspond to a more Oxidative Phosphorylation
phenotype (red circle) and more Glycolytic phenotype (yellow circle). (b) FLIM segmented
image corresponding to the selection performed on phasor (a) where cells in the apical
layer exhibit Oxidative Phosphorylation phenotype compared to cells in the basal layer
with a Glycolytic phenotype. (c) The line joining free and bound NADH in the phasor plot
is known as the “metabolic trajectory”, and a shift in the free NADH direction is
representative of a more reducing condition and glycolytic metabolism, while a shift
towards more bound NADH is indicative of more oxidizing conditions and more oxidative
phosphorylation, as described in previous studies2–5. The extremes of the metabolic
trajectory are the lifetimes for NADH free and bound. The parameters for a lifetime (𝜏
phase and modulation) are in line with those reported in the literature (0.4ns free and 1.0-
3.4 ns bound)
3–8
.
54

In live samples, several intrinsic molecules are known to emit fluorescence,
including NADH, riboflavin, retinoids, and folic acid36. The contribution of these
generally low signal-to-noise intrinsic signals to the overall fluorescence is generally
called autofluorescence7. Hyperspectral imaging and HySP29 can be employed to
diminish the contribution of autofluorescence to the image. The improved sensitivity
of the phasor, however, enables autofluorescence to become a signal of interest
and allows for the exploration of its multiple endogenous molecules’ contributions.
SEER is applied here for visualizing multispectral autofluorescent data of an explant
of freshly isolated trachea from a wild-type C57Bl mouse. The tracheal epithelium is
characterized by a very low cellular turnover, and therefore the overall metabolic
activity is attributable to the cellular function of the specific cell type. Club and
ciliated cells are localized in the apical side of the epithelium and are the most
metabolically active as they secrete cytokines and chemically and physically
remove inhaled toxins and particles from the tracheal lumen. Contrarily, basal cells
which represent the adult stem cells in the upper airways, are quiescent and
metabolically inactive37,38. Because of this dichotomy inactivity, the tracheal
epithelium at homeostasis constituted the ideal cellular system for testing SEER
and validating with FLIM imaging. The slight bend on the trachea, caused by the
cartilage rings, allowed us to visualize the mesenchymal collagen layer, the basal
and apical epithelial cells, and the tracheal lumen in a single focal plane.
The explant was imaged with 2-photon laser scanning microscopy in multispectral
mode. We compare the state-of-the-art “true color” image (Figure 2.12a,), and
55
SEER images (Figure 2.12b, c). The Gradient Descent morphed map (Figure 2.12b)
enhances the visualization of metabolic activities within the tracheal specimen,
showing different metabolic states when moving from the tracheal airway apical
surface toward the more basal cells and the underlying collagen fibers (Figure
2.12b). The visualization improvement is maintained against different
implementations of RGB visualization (Figure 2.13) and at different depths in
volumetric datasets (Figure 2.28). The tensor map increases the contrast of cell
boundaries (Figure 2.12c). Changes in autofluorescence inside live samples are
associated with variations in the ratio of NAD+/NADH, which in turn is related to the
ratio of free to protein-bound NADH39. Despite very similar fluorescence emission
spectra, these two forms of NADH are characterized by different decay times (0.4ns
free and 1.0-3.4 ns bound)35,40–44. FLIM provides a sensitive measurement for
the redox states of NADH and glycolytic/oxidative phosphorylation. Metabolic
imaging by FLIM is well established and has been applied for characterizing
disease progression in multiple animal models, in single cells, and humans as well
as to distinguish stem cell differentiation and embryo development41–48.
Previous work has shown that both Hyperspectral Imaging and FLIM correlate with
metabolic changes in cells from retinal organoids49. Here, the dashed squares
highlight cells with distinct spectral representation through SEER, a difference that
the FLIM image (Figure 2.4d, Figure 2.14) confirms. The improvement of SEER in
visualizing intrinsic signals is clear when compared to standard approaches.
56

Figure 2.14 Visualization of a single fluorescence label against multiple autofluorescence.
Tg(fli1:mKO2) (pan-endothelial fluorescent protein label) zebrafish was imaged with
intrinsic signals arising from the yolk and xanthophores (pigment cells). Live imaging was
performed using a multi-spectral confocal (32 channels) fluorescence microscope with
488 nm excitation. The endothelial mKO2 signal is difficult to distinguish from intrinsic
signals in a (a) maximum intensity projection TrueColor 32 channels Image display
(Bitplane Imaris, Switzerland). The SEER angular map highlights changes in spectral
phase, rendering them with different colors (reference map, bottom right of each panel).
(b) Here we apply the angular map with scaled mode on the full volume. Previously
indistinguishable spectral differences (boxes 1,2,3 in panel a) are now easy to visually
separate. The color bar represents the main wavelength associated with one color in
57
nanometers. (c-h) Zoomed-in views of regions 1-3 (from a) visualized in TrueColor (c, e,
g) and with SEER (d, f, h) highlight the differentiation of the pan-endothelial label (yellow)
distinctly from pigment cells (magenta). The improved sensitivity of SEER further
distinguishes different sources of autofluorescence arising from the yolk (blue and cyan)
and pigments.

Figure 2.15 Visualization comparison for a single fluorescent label with other RGB
standard visualizations in presence of autofluorescence. Visualization of Tg(fli1:mKO2)
(pan-endothelial fluorescent protein label) zebrafish with intrinsic signal arising from the
yolk and xanthophores (pigment cells) (Figure 5) is here shown with different standard
approaches. Details for these visualizations are reported in the Methods section. (a)
SEER RGB mask for a single z-plane, obtained using gradient angular map in scaled
mode, this mask shows the colors associated by SEER to each pixel, without considering
intensity. (b) SEER maximum intensity projection (MIP) for the entire volume (c)
TrueColor 32 channels volume MIP (d) Peak wavelength volume MIP. (e) Gaussian
Default Kernel with RGB centered respectively at 650nm, 510 nm, and 470 nm. (f)
Gaussian Kernel at 10% threshold, RGB values centered at 686 nm, 588 nm, and 499nm.
(g) Gaussian Kernel at 20% threshold, RGB values centered at 668 nm, 579 nm, and
499nm. (h) Gaussian kernel at 30% threshold, RGB values centered at 641 nm, 570 nm,
and 499nm. (i) wavelength-to-RGB color representation for Peak Wavelength mask in
panel d. A representation of the RGB visualization parameters is reported in (j) Average
spectrum (blue plot) for the entire dataset with boundaries used for TrueColor 32ch MIP in
panel c. (k) Kernel used for panel e, average spectrum of the dataset (yellow plot), (l)
kernel used for panel f, average spectrum of the dataset (yellow plot), (m) kernel used for
58
panel g, average spectrum of the dataset (yellow plot), (n) kernel used for panel h, the
average spectrum of the dataset (yellow plot).

Microscopic imaging of fluorophores in the cyan to orange emission range in tissues
is challenging due to intrinsic fluorescence. A common problem is the bleed-through
of autofluorescent signals into the emission wavelength of the label of interest.
Bleed-through is the result of two fluorophores overlapping in emission and
excitation profiles so that photons from one fluorophore fall into the detection range
of the other. While bleed-through artifacts can be partially reduced with a stringent
choice of emission filters, this requires narrow collection channels, which reject any
ambiguous wavelength and greatly decreases collection efficiency. This strategy
generally proves difficult when applied to broad-spectrum autofluorescence.
mKusabira-Orange 2 (mKO2) is a fluorescent protein whose emission spectrum
significantly overlaps with autofluorescence in zebrafish. In a fli1:mKO2 zebrafish,
where all of the vascular and endothelial cells are labeled, the fluorescent protein,
mKO2, and autofluorescence signals due to pigments and yolk are difficult to
distinguish (Figure 5a, boxes). Grayscale renderings (figure 15) provide information
on the relative intensity of the multiple fluorophores in the sample but are not
sufficient for specifically detecting the spatial distribution of the mKO2 signal. True
color representation (Figure 5a, figure 16) is limited in visualizing these spectral
differences. SEER’s angular map (Figure 5b) provides a striking contrast between
the subtly different spectral components inside this 4D (x, y, z, λ) dataset. The
angular reference map enhances changes in phase on the phasor plot which nicely
discriminates shifts in the center wavelength of the spectra inside the sample
59
(Supplementary Video Movie 1). Autofluorescence from pigment cells is
considerably different from the fli1:mKO2 fluorescence (Figure 5c-h). For example,
the dorsal area contains a combination of mKO2 and pigment cells (Figure 5e-f) not
distinct in the standard approaches. The Angular map permits SEER to discriminate
subtle spectral differences. Distinct colors represent the autofluorescence from the
yolk and pigment cells (Figure 5g-h), enriching the overall information provided by
this single-fluorescently labeled specimen and enhancing the visualization of mKO2
fluorescently labeled pan-endothelial cells.

60

Figure 2.16 Triple label fluorescence visualization. Zebrafish embryo
Tg(kdrl:eGFP); Gt(desmin-Citrine);Tg(ubiq:H2B-Cerulean) labeling respectively
vasculature, muscle, and nuclei. Live imaging with a multi-spectral confocal
microscope (32-channels) using 458 nm excitation. Single plane slices of the tiled
volume are rendered with TrueColor and SEER maps. (a) TrueColor image display
(Zen, Zeiss, Germany). (b) Angular map in the center of mass morph mode
improves contrast by distinguishable colors. The resulting visualization enhances
the spatial localization of fluorophores in the sample. (c) Gradient Descent map in
61
max morph mode centers the color palette on the most frequent spectrum in the
sample, highlighting the spectral changes relative to it. In this sample, the
presence of skin pigment cells (green) is enhanced. 3D visualization of SEER
maintains these enhancement properties. Color bars represent the main
wavelength associated with one color in nanometers. Here we show (d, e, f)
TrueColor 32 channels Maximum Intensity Projections (MIP) of different sections
of the specimen rendered in TrueColor, Angular map center of mass mode, and
Gradient Descent max mode. The selected views highlight SEER’s performance in
the (d) overview of the somite, (e) zoom-in of somite boundary, and (f) lateral view
of the vascular system.

Figure 2.17 Visualization comparison for triple label fluorescence with other RGB standard
approaches. Visualization of Tg(kdrl:eGFP); Gt(desmin-Citrine); Tg(ubiq:H2B-Cerulean)
labeling respectively vasculature, muscle, and nuclei (Figure 6) is shown here with
different standard approaches. Details for this visualization are reported in the Methods
section. The same slice (here z=3) is shown as a maximum intensity projection (MIP)
using: (a) SEER gradient descent map in max morph mode, (b) SEER MIP angular map
mass morph mode, (c) TrueColor 32 channels, (d) Peak wavelength, (e) Gaussian Default
Kernel with RGB centered respectively at 650nm, 510 nm, and 470 nm. (f) Gaussian
Kernel at 10% threshold, RGB values centered at 597 nm, 526 nm, and 463 nm (g)
62
Gaussian Kernel at 20% threshold, RGB values centered at 579nm, 517 and 463nm (h)
Gaussian kernel at 30% threshold, RGB values centered at 561 nm, 526 nm and 490nm.
A representation of the RGB visualization parameters is reported in (i) wavelength-to-RGB
color representation for Peak Wavelength mask in panel d, (j) Average spectrum (blue
plot) for the entire dataset with boundaries used for TrueColor 32ch MIP in panel c. (k)
kernel used for panel e, average spectrum of the dataset (yellow plot), (l) kernel used for
panel f, average spectrum of the dataset (yellow plot), (m) kernel used for panel g,
average spectrum of the dataset (yellow plot), (n) kernel used for panel h, the average
spectrum of the dataset (yellow plot).

Figure 2.18 SEER of zebrafish volumes in Maximum Intensity Projection (MIP) and
Shadow Projection. The capability of SEER to improve the visualization of spectral
63
datasets is translatable to 3D visualizations with different visualization modalities. Here we
show a zebrafish embryo Tg(kdrl:eGFP); Gt(desmin-Citrine); Tg(ubiq:H2B-Cerulean)
labeling respectively vasculature, muscle, and nuclei. (a) MIP of an Angular map volume
with Mass Morph mode. (b) The same combination of map and mode is shown using
shadow projection. While the volume rendering approaches are different, the spatial
distinction between fluorescent labels is maintained. The Gradient Descent map in Max
Morph mode is here applied on the same dataset using (c) MIP and (d) shadow
projection. With the Gradient Descent map (c) MIP improves contrast for determining the
spatial distinction between fluorophores. (d) Shadow projection further enhances the
location of skin pigments (green).

Imaging and visualization of biological samples with multiple fluorescent labels are
hampered by the overlapping emission spectra of fluorophores and
autofluorescence molecules in the sample, complicating the visualization of the
sample. A triple labeled zebrafish embryo with Gt(desm-
Citrine)ct122a/+;Tg(kdrl:eGFP), H2B-Cerulean labeling respectively muscle,
vasculature, and nuclei, with contributions from pigments autofluorescence is
rendered with standard approaches and SEER in 1D and 3D (Figure 6). TrueColor
representation (Figure 6a, figure 17) provides limited information on the inner details
of the sample. Vasculature (eGFP) and nuclei (Cerulean) are highlighted with
shades of cyan whereas autofluorescence and muscle (Citrine) are in shades of
green (Figure 6a) making both pairs difficult to distinguish. The intrinsic richness of
colors in the sample is an ideal test for the gradient descent and radial maps.
The Angular map separates spectra based mainly on their central (peak)
wavelength, which corresponds to “phase” differences in the phasor plot. The
gradient descent map separates spectra with a bias on subtle spectral differences
closer to the center of the phasor plot. Here we applied the Mass Morph and Max
64
Morph modes to further enhance the distinction of spectra (Figure 6b-c). With the
Mass Morph mode, the muscle outline and contrast of the nuclei are improved by
increasing the spatial separation of the fluorophores and suppressing the presence
of autofluorescence from skin pigment cells (Figure 6e). With the Max Morph mode
(Figure 6c), pixels with spectra closer to skin autofluorescence are visibly separated
from muscle, nuclei, and vasculature.
The enhancements of SEER are also visible in volumetric visualizations. The
Angular and Gradient maps are applied to the triple labeled 4D (x, y, z, λ) dataset
and visualized as maximum intensity projections (Figure 6d-f). The spatial
localization of fluorophores is enhanced in the Mass Morphed Angular map, while
the Max Morphed Gradient Descent map provides a better separation of the
autofluorescence of skin pigment cells (Supplementary Video Movie 2). These
differences are also maintained in different visualization modalities (figure 18).
SEER helps to discern the difference between fluorophores even with multiple
contributions from bleed through between labels and from autofluorescence.
Particularly, morphed maps demonstrate a high sensitivity in the presence of subtle
spectral differences. The triple-labeled example (Figure 6) shows the advantage of
the morphed map, as it places the apex of the SEER map at the center of mass of
the phasor histogram and compensates for the different excitation efficiencies of the
fluorescent proteins at 458 nm.

65

Figure 2.19 Visualization of combinatorial expression on Zebrabow
34
samples. Maximum
Intensity Projection renderings of Tg(ubi:Zebrabow) muscle acquired live in multi-spectral
confocal mode with 458 nm excitation. (a) The elicited signal (e.g., white arrows) is difficult
to interpret in the TrueColor Image display (Zen Software, Zeiss, Germany). (b)
Discerning spectral differences are increasingly simpler with Gradient Descent map
scaled to intensities while compromising on the brightness of the image. (c) Gradient
Descent and (d) Gradient Ascent RGB Masks in scale mode show the color values
assigned to each pixel and greatly improve the visual separation of recombined CFP,
YFP, and RFP labels. Color bars represent the main wavelength associated with one
color in nanometers.

Figure 2.20 Visualization comparison for combinatorial expression with other RGB
standard approaches. Visualization of ubi:Zebrabow muscle (Figure 2.7) with different
66
standard approaches. Details for the visualization are reported in the methods section
2.5.2. The same slice is shown as an RGB mask which represents the color associated
with each pixel, independent from the intensity, or as a maximum intensity projection
(MIP) using: (a) SEER gradient descent map mask in scaled mode, (b) Average spectrum
(blue plot) for the entire dataset with boundaries used for TrueColor 32ch MIP in panel c.
(c) TrueColor 32 channels, (d) Peak wavelength mask, (e) Gaussian Default Kernel with
RGB centered respectively at 650nm, 510 nm, and 470 nm. (f) Gaussian Kernel at 10%
threshold, RGB values centered at 659 nm, 561 nm, and 463 nm. (g) Gaussian Kernel at
20% threshold, RGB values centered at 641 nm, 552 nm, and 463 nm. (h) Gaussian
kernel at 30% threshold, RGB values centered at 632 nm, 552 nm, and 472 nm. (i)
wavelength-to-RGB color representation for Peak Wavelength mask in panel d. A
representation of the RGB visualization parameters is reported in (j) kernel used for panel
e, average spectrum of the dataset (yellow plot), (k) kernel used for panel f, average
spectrum of the dataset (yellow plot), (l) kernel used for panel g, average spectrum of the
dataset (yellow plot), (m) kernel used for panel h, the average spectrum of the dataset
(yellow plot).

Spectral differences can be visualized in combinatorial approaches
Zebrabow34 is the result of a powerful genetic cell labeling technique based on the
stochastic and combinatorial expression of different relative amounts of a few
genetically encoded, spectrally distinct fluorescent proteins34,50,51. The Zebrabow
(Brainbow) strategy combines the three primary colors red, green, and blue, in
different ratios, to obtain a large range of colors in the visual palette, similar to
modern displays52. Unique colors arise from the combination of different ratios of
RFP, CFP, and YFP, achieved by a stochastic Cre-mediated recombination50.
This technique has been applied to multiple applications, from axon and lineage
tracing51–55 to cell tracking during development56,57, in which a specific label can
be used as a cellular identifier to track descendants of individual cells over time and
space. The challenge is acquiring and analyzing the subtle differences in hues
67
among these hundreds of colors. Multi-spectral imaging provides the added
dimension required for an improved acquisition; however, this modality is hampered
by both instrumental limitations and spectral noise. Furthermore, current image
analysis and visualization methods interpret the red, yellow, and cyan fluorescence
as an RGB additive combination and visualize it as a color picture, similar to the
human eyes’ perception of color. This approach is not well poised for distinguishing
similar, yet spectrally unique, recombination ratios due to our difficulty in reliably
identifying subtly different colors.
SEER overcomes this limitation by improving the analysis’ sensitivity using our
phasor-based interpretation of colors. Recombination of labels belongs to separate
areas of the phasor plot, simplifying the distinction of subtle differences. The
Standard Reference Maps and modes associate a color easily distinguishable by
eyes, enhancing the subtle spectral recombination. SEER simplifies the
determination of differences between cells for combinatorial strategies, opening a
novel window of analysis for Brainbow samples.
We imaged a Tg(ubi:Zebrabow) sample and visualized its multiple genetic
recombinations using SEER. The results (Figure 7, Figure 19) highlight the difficulty
of visualizing these datasets with standard approaches as well as how the
compressive maps simplify the distinction of both spectrally close and more
separated recombination.
68
2.4.4 Quantitative validation of SEER
Standard approaches for the visualization of hyperspectral datasets trade
computational expense for improved visualization. In this work, we show that the
phasor approach can define a new compromise between computational speed and
rendering performance. The wavelength encoding can be achieved by proper
conversion and representation by the spectral phasor plot of the Fourier transform
of real and imaginary components. The phasor representation offers an effortless
interpretation of spectral information. Originally developed for fluorescence lifetime
analysis22 and subsequently brought to spectral applications26,28,29, here the
phasor approach has been applied to enhance the visualization of multi- and hyper-
spectral imaging. Because of the refined spectral discrimination achieved by these
phasor-based tools, we call this approach Spectrally Enhanced Encoded
Representations (SEER).
SEER offers a computationally efficient and robust method that converts spectral (x,
y, λ) information into a visual representation, enhancing the differences between
labels. This approach makes more complete use of spectral information. Prior
analyses employed the principal components or specific spectral bands of the
wavelength dimension. Similarly, previous phasor analysis interpreted phasor using
selected regions of interest. Our work explores the phasor plot as a whole and
represents that complete information set as a color image while maintaining
efficiency and minimizing user interaction. The function can be achieved quickly and
efficiently even with large data sizes, circumventing the typical computational
69
expense of hyperspectral processing. Our tests show SEER can process a 3.7 GB
dataset with 1.26⋅108 spectra in 6.6 seconds and a 43.88 GB dataset with 1.47⋅109
spectra in 87.3 seconds, including denoising of data. Compared with the python
module, scikit-learn’s implementation of fast Independent Component Analysis
(fastICA), SEER provides up to a 67-fold speed increase (Figure 2.1) and lower
virtual memory usage.
Processing speed comparison between SEER and fastICA for the multispectral
fluorescent data shown in Figures 2.4-2.7 is presented in Supplementary Table 2.
SEER’s computation time ranged between 0.44 (for Figure 4) and 6.27 seconds (for
Figure 5) where the corresponding timing for fastICA was 3.45 and 256.86 seconds
respectively, with a speedup in the range of 7.9 to 41 folds (Figure 2.20), by the
trend shown in Figure 2.2. These results were obtained using Python, an
interpreted language. Implementation of SEER with a compiled language could
potentially increase speed by one order of magnitude. The spectral maps
presented here reduce the dimensionality of these large datasets and assign colors
to a final image, providing an overview of the data before a full-scale analysis.
A simulation comparison with other common visualization approaches such as
Gaussian kernel and peak wavelength selection (Figure 2.21, Methods) shows an
increased spectral separation accuracy (Methods) for SEER to associate distinct
colors to closely overlapping spectra under different noise conditions (Figure 2.22).
The spectral separation accuracy improvement was 1.4-2.6 fold for highly
overlapping spectra, 0nm-8.9nm spectra maxima distance, and 1.5-2.7 fold for
70
overlapping spectra with maxima separated by 17.8nm-35.6nm (Figure 23, 24,
Methods). Additional discussion is available in the Supplementary Peer Review file.
Quantification of RGB images by colorfulness, contrast, and sharpness shows that
SEER generally performs better than standard visualization approaches (Figure 25,
Methods). SEER’s average enhancement was 2%-19% for colorfulness, 11%-27%
for sharpness, and 2%-8% for contrast (Supplementary Table 3) for the datasets in
Figures 4-7. We then performed a measure of the Color Quality Enhancement
(CQE)58, a metric of the human visual perception of color image quality,
(Supplementary Table 4). The CQE score of SEER was higher than the standard,
with an improvement of 11%-26% for Figure 2.6, 7%-98% for Figure 2.12, 14%-25%
for Figure 2.13, and 12%-15% for Figure 2.14 ( Figure 2.25, Methods).
Flexibility is a further advantage of our method. The users can apply several
different Standard Reference Maps to determine which is more appropriate for their
data and enhance the most important image features. The modes provide a
supplementary enhancement by adapting the References to each dataset, in terms
of size and distribution of spectra in the dataset. Scaling maximizes contrast by
enclosing the phasor distribution; it maintains the linearity of the colormap. Max and
Center of Mass modes shift the apex of the distribution to a new center, specifically
the most frequent spectrum in the dataset or the weighted “color-frequency” Center
of mass of the entire dataset. These modes adapt and improve the specific
visualization properties for each map to the dataset currently being analyzed. As a
result, each map offers increased sensitivity to specific properties of the data,
amplifying, for example, minor spectral differences or focusing on major wavelength
71
components. The adaptivity of the SEER modes can prove advantageous for
visually correcting the effect of photobleaching in samples, by changing the apex of
the map dynamically with the change of intensities (Figure 2.26).
SEER can be applied to fluorescence, as performed here, or to standard reflectance
hyper- and multi-spectral imaging. These phasor remapping tools can be used for
applications in fluorescence lifetime or combined approaches of spectral and
lifetime imaging. With multispectral fluorescence, this approach is promising for
real-time imaging of multiple fluorophores, as it offers a tool for monitoring and
segmenting fluorophores during acquisition. Live imaging visualization is another
application for SEER. The Gradient Descent map, for example, in combination with
denoising strategies29 can minimize photobleaching and phototoxicity, by enabling
the use of lower excitation power. SEER overcomes the challenges in visualization
and analysis deriving from low signal-to-noise images, such as intrinsic signal
autofluorescence imaging. Among other complications, such image data can result
in a concentrated cluster proximal to the phasor center coordinates. The gradient
descent map overcomes this limitation and provides bright and distinguishable
colors that enhance subtle details within the dim image.
It is worth noticing that the method is generally indifferent to the dimension being
compressed. While in this work we explore the wavelength dimension, SEER can
be utilized, in principle, with any n-dimensional dataset where n is larger than two.
For instance, it can be used to compress and compare the dimension of lifetime,
space, or time for multiple datasets. Some limitations that should be considered are
that SEER pseudo-color representation sacrifices the “true color” of the image,
72
creating inconsistencies with the human eyes’ expectation of the original image, and
does not distinguish identical signals arising from different biophysical events
(Supplementary Note 2). SEER is currently intended as a pre-processing
visualization tool and, currently, is not utilized for quantitative analysis. Combining
SEER with novel color-image compatible segmentation algorithms59,60 might
expand the quantitative capabilities of the method.
New multi-dimensional, multi-modal instruments will more quickly generate much
larger datasets. SEER offers the capability of processing this explosion of data,
enabling the interest of the scientific community in multiplexed imaging.
2.5 SEER methods, simulations, and quantifications
2.5.1 Simulated hyperspectral test chart
To account for the Poisson noise and detector noise contributed by optical
microscopy, we generated a simulated hyperspectral test chart starting from real
imaging data with a size of x: 300 pixels, y: 300 pixels, and lambda: 32 channels.
S1, S2, and S3 spectra were acquired respectively from zebrafish embryos labeled
only with CFP, YFP, and RFP, where the spectrum in Figure 1a is identical to the
center cell of the test chart Figure 1d. In each cell, three spectra are represented
after shifting the maxima by d1 or d2 nanometers with respect to S2. Each cell has
its corresponding spectra of S1, S2, and S3 (figure 3).
73
2.5.2 Standard RGB visualizations
The TrueColor RGB image (Figure 3, 8, 21) is obtained through compression of the
hyperspectral cube into the RGB 3 channels color space by generating a Gaussian
radial basis function kernel61 𝐾 for each RGB channel. This kernel 𝐾 acts as a
similarity factor and is defined as:
𝐾
)
(𝑥
)
,𝑥
*
)= 𝑒
!|#
$
!#%|
&
&'
&
(eq. 2.1)
where 𝑥′ is the center wavelength of R or B or G. For example, when 𝑥′=650𝑛𝑚,
the associated RGB color space is (R:1, G:0, B:0). Both 𝑥 and 𝐾 are defined as 32
× 1 vector, representing, respectively, the 32-channel spectrum of one single pixel
and the normalized weight of each R, G and B channel. 𝑖 is the channel index of
both vectors. 𝐾
)
represents how similar channel 𝑖 is to each R/G/B channel, and 𝜎 is
the deviation parameter.
We compute RGB color space 𝑐 by a dot product of the weight vector 𝐾 and 𝜆 at
corresponding channel R/G/B:
𝑐 = ∑ 𝜆
)
× 𝐾
)

)+,-
)+.
(eq. 2.2)
where 𝜆 is a vector of the wavelengths captured by the spectral detector in an LSM
780 inverted confocal microscope with lambda module (Zeiss, Jena, Germany) and
𝜆
)
is the center wavelength of channel 𝑖. Gaussian kernel was set at 650nm, 510
nm, 470nm for RGB respectively as Default (Figure 3s, figure 8, figure 13e, figure
16e, figure 17e, figure 19j).
74
The same Gaussian kernel was also changed adaptively to the dataset to provide a
spectral contrast stretching on the visualization and focus the visualization on the
most utilized channels. The average spectrum for the entire dataset is calculated
and normalized. They intersect at 10% (figure 13f, 16f, 17f, 19f), 20% (figure 13g,
16g,17g, 19g) and 30% (figure 13h, 16h, 17h, 19h). of the intensity is obtained and
used as a center for the blue and red channels. The green channel is centered
halfway between red and blue. Representations of these adaptations are reported in
figure 21g,h,i.
The TrueColor 32 Channels image (Figure 1c, Figure 5a,c,e,g, Figure 6a,d,e,f,
figure 13c, 16c, 17c, 19c) was rendered as a 32 channels Maximum Intensity
Projection using Bitplane Imaris (Oxford Instruments, Abingdon, UK). Each channel
has a known wavelength center (32 bins, from 410.5 nm to 694.9 nm with 8.9 nm
bandwidth). Each wavelength was associated with a color according to classical
wavelength-to-RGB conversions62 as reported in figure 21f. The intensity for all
channels was contrast-adjusted (Imaris Display Adjustment Settings) based on the
channel with the largest information. A meaningful range for rendering was
identified as the top 90% in the intensity of the normalized average spectrum for the
dataset (figure 13b, 16j, 17j, 19b). Channels outside of this range were excluded
from rendering. Furthermore, for 1 photon excitation, channels associated with
wavelength lower than the laser excitation (for example channels 1 to 5 for laser
458nm) were excluded from rendering.
Peak Wavelength representation (figures 13d, 16d, 17d, 19d, 21 and 23)
reconstructs an RGB image utilizing, for each pixel, the color associated to the
75
wavelength at which maximum intensity is measured. Wavelength-to-RGB
conversion was performed using a python function
(http://bioimaging.usc.edu/software.html#HySP) adapted from Dan Bruton’s work62.
A graphical representation is reported in figure 21f.
2.5.3 Spectral separation accuracy calculation
We utilize the Simulated Hyperspectral Test Chart to produce different levels of
spectral overlap and signal to noise ratio (SNR). We utilize multiple RGB
visualization approaches for producing compressed RGB images (figure 21, figure
22). Each panel of the simulation is constructed by three different spectra,
organized as three concentric squares Q1, Q2, Q3 (figure 3). The maximal contrast
visualization is expected to have three well separated colors, in this case red, green
and blue. For quantifying this difference, we consider each (R, G, B) vector, with
colors normalized [0,1], in each pixel as a set of Euclidean coordinates (x, y, z) and
for each pixel calculate the Euclidean distance:
𝑙
.-
= E∑ F𝑝
/.
−𝑝
/-
I
)
-
0
)+1
(eq. 2.3)
where l12 is the color distance between square Q1 and Q2. pQ1 and pQ2 are the
(R, G, B) vectors in the pixels considered, 𝑖 is the color coordinate R, G or B. The
color distances l13 and l23 between squares Q1Q3 and Q2Q3 respectively, are
calculated similarly. The accuracy of spectral separation (figure 23) is calculated as:
𝑆𝑝.𝑠𝑒𝑝.𝑎𝑐𝑐 =
(3
(&
43
()
43
&)
)
3
*+, !-*++.
4 3
*+, !/01+
4 3
-*++. !/01+
(eq. 2.4)
76
where the denominator is the maximum color distance which can be achieved in
this simulation, where Q1 , Q2 and Q3 are respectively pure red, green and blue,
therefore:
𝑙
(78"9(77'
+ 𝑙
(78"&3:7
+ 𝑙
9(77'"&3:7
= 3√2 (eq. 2.5)
2.5.4 Compressive spectral algorithm and map reference design
Phasor calculations
For each pixel in an image, we acquire the sequence of intensities at different
wavelengths 𝐼(𝜆). Each spectrum 𝐼(𝜆) is discrete Fourier transformed into a
complex number 𝑔
;,=,>,#
+𝑖 𝑠
;,=,>,#
. Here 𝑖 is the imaginary unit, while (𝑥,𝑦,𝑧,𝑡)
denotes the spatio-temporal coordinates of a pixel in a 5D dataset.
The transforms used for real and imaginary components are:
𝑔
;,=,>,#
(𝑘)
|@+-
=
∑ B(C)∗EFG (
&234
5
) ∗∆C
4
5
4
6

∑ B(C)∗∆C
4
5
4
6
(eq. 2.5)

𝑠
;,=,>,#
(𝑘)
|@+-
=
∑ B(C)∗G)' (
&234
5
) ∗∆C
4
5
4
6
∑ B(C)∗∆C
4
5
4
6
(eq. 2.6)
Where 𝜆
!
and 𝜆
I
are the initial and final wavelengths respectively, 𝑁 is the number
of spectral channels, ∆𝜆 is the wavelength band width of a single channel. k is the
harmonic. In this work, we utilized harmonic 𝑘 =2. The effects of different harmonic
numbers on the SEER visualization are reported in Supplementary Note 1.
77

Standard map reference
Association of a color to each phasor coordinate (𝑔,𝑠) is performed in two steps.
First, the reference system is converted from Cartesian to polar coordinates (𝑟,𝜃).

(𝑟,𝜃)=WX𝑔
-
+𝑠
-
,
G
9
Y (eq. 2.7)

These polar coordinate values are then transformed to the Hue, Saturation, Value
(HSV) color model utilizing specific settings for each map, as listed below. Finally,
any color generated outside of the r=1 boundary is set to black.
Gradient Descent:
ℎ𝑢𝑒 = 𝜃
𝑠𝑎𝑡𝑢𝑟𝑎𝑡𝑖𝑜𝑛 =1
𝑣𝑎𝑙𝑢𝑒 =1−0.85∗𝑟
Gradient Ascent
ℎ𝑢𝑒 = 0
𝑠𝑎𝑡𝑢𝑟𝑎𝑡𝑖𝑜𝑛 =1
78
𝑣𝑎𝑙𝑢𝑒 =𝑟
Radius:
Each r value from 0 to 1 is associated to a level in the jet colormap from the
matplotlib package
Angle:
ℎ𝑢𝑒 = 0
𝑠𝑎𝑡𝑢𝑟𝑎𝑡𝑖𝑜𝑛 =1
𝑣𝑎𝑙𝑢𝑒 =1
Tensor map
Visualization of statistics on the phasor plot is performed by means of the
mathematical gradient. The gradient is obtained in a two-step process. First, we
compute the two-dimensional derivative of the phasor plot histogram counts by
utilizing an approximation of the second order accurate central difference. Each bin
𝐹(𝑔,𝑠) has a central difference:
JK
J9
,
JK
JG
,
JK
JG
, with respect to the differences in 𝑔
(horizontal) and 𝑠 (vertical) directions with unitary spacing ℎ. The approximation
becomes:
JK
JG
=
KLG4
(
&
M,9N"KLG"
(
&
M,9N
M
=
7(9:;,-):7(9,-)
&
"
7(9,-):7(9!;,-)
&
M
=
K(G4M,9)"K(G"M,9)
-M
(eq. 2.8)

79
And similarly:

JK
J9
=
K(G,94M)"K(G,9"M)
-M
(eq. 2.9)

Second, we calculate the square root of the sum of squared differences 𝐷(𝑠,𝑔)
as:
𝐷(𝑔,𝑠)= E(
JK
J9
)
-
+(
JK
JG
)
-
(eq. 2.10)
obtaining the magnitude of the derivative density counts. With this gradient
histogram, we then connect the phasor coordinates with same 𝐷(𝑠,𝑔) gradients with
one contour. All gradients are then normalized to (0,1). Finally, pixels in the
hyperspectral image corresponding to the same contour in phasor space will be
rendered the same color. In the reference map, red represents highly dense
gradients, usually at the center of a phasor cluster. Blue, instead, represents the
sparse gradient that appears at the edge circumference of the phasor distributions.
Scale mode
In this mode, the original square Standard Reference Maps are transformed to a
new boundary box adapted to each dataset’s spectral distribution.
The process of transformation follows these steps. We first determine the boundary
box (width 𝜔, height ℎ) based on the cluster appearance on the phasor plot. We
80
then determine the largest ellipsoid that fits in the boundary box. Finally, we warp
the unit circle of the original map to the calculated ellipsoid.
Using polar coordinates, we represent each point P of the standard reference map
with phasor coordinates (𝑔
)
,𝑠
)
) as:

𝑃(𝑔
)
,𝑠
)
)=𝑃(𝑟
)
∗𝑐𝑜𝑠 𝜃
)
,𝑟
)
∗𝑠𝑖𝑛 𝜃
)
) (eq. 2.11)

The ellipsoid has semi-major axes:

𝑎 =
O
-
(eq. 2.12)

and semi-minor axes

𝑏 =
M
-
(eq. 2.13)

Therefore, the ellipse equation becomes:

81
(
9
$
>
&
)
-
+ (
G
$
;
&
)
-
=𝑟𝑎𝑑
-
(eq. 2.14)

Where rad is a ratio used to scale each radius 𝑟
)
in the reference map to a
proportionally corresponding distance in the boundary box-adapted ellipse, which in
polar coordinates becomes:

𝑟𝑎𝑑
-
= 𝑟
)
-
∗((
EFG P
$

>
&
)
-
+ (
G)' P
$

;
&
)
-
) (eq. 2.15)

Each point P( 𝑔
)
,𝑠
)
) of the standard reference map is geometrically scaled to a new
coordinate (𝑔
F
,𝑠
F
) inside the ellipsoid using forward mapping, obtaining the
equation:

(𝑟
F
,𝜃
F
) =dE𝑔
)
-
+𝑠
)
-
/𝑟𝑎𝑑,
9
$
G
$
e (eq. 2.16)

This transform is applied to all Standard Reference Maps to generate the respective
scaled versions.
Morph mode
82
We linearly morph each point P(𝑔
)
,𝑠
)
) to a new point 𝑃′(𝑔
F
,𝑠
F
) by utilizing a shifted-
cone approach. Each standard map reference is first projected onto a 3D conical
surface centered on the phasor plot origin and with unitary height (figure 27a-c,
Supplementary Video Movie 4). Each point P on the standard map is given a 𝑧
value linearly, starting from the edge of the phasor universal circle. The original
standard map (figure 27c) can, thus, be interpreted as a top view of a right cone
with z=0 at the phasor unit circle and z=1 at the origin (figure 27a).

83
Figure 2.21 Morph mode algorithm pictorial abstraction. (a) A Radial map in standard
mode centered at the origin O can be abstracted as a (b) 3D Conic shape with height ℎ
and apex A. (c) Upon shifting the apex of the cone from 𝐴 to 𝐴′, the map reference center
translates from origin O to the projection 𝐴⏊′. During this shift, the edges of the cone base
are anchored on the phasor unit circle. (c-d) If we consider a plane cutting the oblique
cone horizontally, the resulting section is a circle with center 𝑂′ and radius 𝑟′. The
projection of this circle is centered on 𝑂⏊′ which lies on the line 𝑂𝐴⏊′, adjoining the fixed
center O and new apex projection 𝐴⏊
$
and has the same radius 𝑟′. As a result, (d) all of
the points in each of these projected circles are shifted along the vector 𝑂𝑂⏊′ on the
phasor plot.

We then shift the apex 𝐴 of the cone to the computed weighted average or maxima
of the original 2d histogram, producing an oblique cone (figure 2.21b) centered in
𝐴′.
In this oblique cone, any horizontal cutting plane is always a circle with center 𝑂′. Its
projection 𝑂⏊′ is on the line joining the origin 𝑂 and projection of the new center
𝐴⏊′(figure 2.21b-d). As a result, all of the points in each circle are shifted towards
the new center 𝐴⏊′ on the phasor plot. We first transform the coordinates (𝑔
)
,𝑠
)
) of
each point P to the morphed map coordinates (𝑠
F
,𝑔
F
), and then obtain the
corresponding (𝑟
F
,𝜃
F
) necessary for calculating Hue, Saturation, and Value.
In particular, a cutting plane with center 𝑂′ has a radius of 𝑟′ (figure 2.21). This
cross-section projects on a circle centered in 𝑂⏊′ with the same radius. Using
geometrical calculations, we obtain:

𝑂𝑂⏊′= 𝛼∗𝑂𝐴⏊′, (eq. 2.17)
84

were 𝛼 is a scale parameter. By taking the approximation,

∆𝑂
*
𝑂𝑂⏊
*
~ ∆𝐴
*
𝑂𝐴⏊
*
, (eq. 2.18)

we can obtain

𝑂𝑂′= 𝛼∗𝑂𝐴′. (eq. 2.19)

Furthermore, given a point 𝑁
*
on the circumference centered in 𝑂
*
, eq.14 also
implies that:

𝑂′𝑁
*
=(1−𝛼)∗𝑂𝑁⏊′, (eq. 2.20)

which is equivalent to

𝑟′=(1−𝛼)∗𝑅. (eq. 2.21)
85

where R is the radius of the phasor plot unit circle.
With this approach, provided a new center 𝐴⏊′ with a specific 𝛼, we obtain a
collection of scaled circles with their centers on the line 𝑂𝐴⏊′. In boundary cases,
when 𝛼 =0, the scaled circle is the origin, while 𝛼 =1 is the unit circle. Given any
cutting plane 𝑂′, the radius of this cross section always satisfies this identity:

𝑟′
-
=(𝑔
)
−𝛼∗𝑔
Q⏊
%)
-
+(𝑠
)
−𝛼∗𝑠
Q⏊
%)
-
=(1−𝛼)
-
∙𝑅
-
(eq. 2.22)

The coordinates of a point 𝑃′′(𝑔
F
,𝑠
F
) for a new morphed map centered in 𝐴⏊′ are:

(𝑔
F
,𝑠
F
)=(𝑔
)
−𝛼∗𝑔
Q⏊
%,𝑠
)
−𝛼∗𝑠
Q⏊
%) (eq. 2.23)

Finally, we compute

(𝑟
F
,𝜃
F
) =WX𝑔
F
-
+𝑠
F
-
,
G
?
9
?
Y (eq. 2.24)
86
and then assign colors based on the newly calculated Hue, Saturation, and Value to
generate the morph mode references.

2.5.5 Color image quality calculation
Colorfulness
Due to the inherent lack of “ground truth” in experimental fluorescence microscopy
images, we utilized an established model for calculating the color quality of an
image without a reference
63
. The colorfulness is one of three parameters, together
with sharpness and contrast, utilized by Panetta et al
58
to quantify the overall quality
of a color image. Two opponent color spaces are defined as:

𝛼 =𝑅−𝐺 (eq. 2.25)

𝛽 =0.5 (𝑅+𝐺)−𝐵 (eq. 2.26)

Where R, G, B are the red, green and blue channels respectively, α and β are red-
green and yellow-blue spaces. The colorfulness utilized here is defined as:

87
𝐶𝑜𝑙𝑜𝑟𝑓𝑢𝑙𝑛𝑒𝑠𝑠 =0.02 𝑙𝑜𝑔 W
S
@
&
|T
@
|
6.&
Y𝑙𝑜𝑔 d
S
B
&
UT
B
U
6.&
e (eq. 2.27)

With 𝜎
V
-
, 𝜎
W
-
, 𝜇
V
, 𝜇
W
respectively as the variances and mean values of the α and β
spaces58.
Sharpness
We utilize EME64, a Weber based measure of enhancement. EME is defined as
follows:

𝐸𝑀𝐸
GMX(Y
=
-
@
(
@
&
∑ ∑ 𝑙𝑜𝑔s
B
CD#,3,0
B
C$. ,3,0
t
@
&
3+.
@
(
3+.
(eq. 2.28)

Where 𝑘
.
, 𝑘
-
are the blocks used to divide the image and 𝐼
ZX;,@,3
and 𝐼
Z)',@,3
are the
maximum and minimum intensities in the blocks. EME has been shown to correlate
with a human observation of sharpness in color images58 when associated with a
weight 𝜆
E
for each color component.

𝑆ℎ𝑎𝑟𝑝𝑛𝑒𝑠𝑠 = ∑ 𝜆
E
𝐸𝑀𝐸
GMX(Y
,
E+.
(eq. 2.29)

88
Where the weights for different color components used in this article are 𝜆
1
=0.299,
𝜆
[
=0.587, 𝜆
0
=0.114 in accordance with NTSC standard and values reported in
literature58.
Contrast
We utilize Michelson-Law measure of enhancement AME65, an effective evaluation
tool for contrast in grayscale images, designed to provide larger metric value for
larger contrast images. AME is defined as:

𝐴𝑀𝐸
EF'#(XG#
=
.
@
(
@
&
∑ s∑ 𝑙𝑜𝑔s
B
CD#,3,0
4B
C$. ,3,0
B
CD#,3,0
"B
C$. ,3,0
t
@
&
3+.
t
"!.]

@
(
3+.
(eq. 2.30)

Where 𝑘
.
, 𝑘
-
are the blocks used to divide the image and 𝐼
ZX;,@,3
and 𝐼
Z)',@,3
are the
maximum and minimum intensities in the blocks. The value of contrast for color
images was then calculated as:

𝐶𝑜𝑛𝑡𝑟𝑎𝑠𝑡 = ∑ 𝜆
E
𝐴𝑀𝐸
EF'#(XG#
,
E+.
(eq. 2.31)

With the same weights 𝜆
E
utilized for sharpness.
Color Quality Enhancement
89
We utilize Color Quality Enhancement (CQE), a polling method to combine
colorfulness, sharpness and contrast into a value that has both strong correlation
and linear correspondence with human visual perception of quality in color
images58. CQE is calculated as:

𝐶𝑄𝐸 =𝑐
.
𝑐𝑜𝑙𝑜𝑟𝑓𝑢𝑙𝑛𝑒𝑠𝑠 +𝑐
-
𝑠ℎ𝑎𝑟𝑝𝑛𝑒𝑠𝑠+ 𝑐
,
𝑐𝑜𝑛𝑡𝑟𝑎𝑠𝑡 (eq. 2.32)

Where the linear combination coefficients for CQE measure were set to evaluate
contrast change according to values reported in literature58, 𝑐
.
=0.4358, 𝑐
-
=
0.1722 and 𝑐
,
=0.3920.

90
Chapter 3 Classification of multimodel 5-D
biomedical data with SpectraFLIM-LSTM
3 Classification of multimodal 5-D biomedical data
with SpectraFLIM-LSTM
3.1 SpectraFLIM enables multiplexing
3.1.1 Single-cell resolution SpectraFLIM autofluorescence imaging
SpectraFLIM is a multimodal optical microscopy imaging technique that extracts
both spectral and lifetime information from images. SpectraFLIM captures the full-
spectrum and fluorescence lifetime for each pixel in an image. This multimodal
optical microscopy approach combines and broadens the informational content
being imaged simultaneously with applications recently published for multiplexed
signals in live samples
25
.
However, the challenge of processing, analyzing, and interpreting fluorescence
imaging data increases as the number of imaging modalities increases. The focus
of this work is to explore the information content of 5-dimensional datasets (x, y, z,
wavelength, lifetime) and develop a classification algorithm for disease detection in
complex data. Our results show that information-rich SpectraFLIM data has the
91
potential to increase the capability of classifying diseased versus healthy tissues by
exploiting the connection between intrinsic autofluorescence and changes in
metabolism due to disease. Our approach integrates information from spectra and
lifetime with in-vivo autofluorescent-related changes of metabolites through various
deep learning network architectures.

Figure 3.1 Overview of SpectraFLIM data. (a-b) Spectrally resolved autofluorescence
intensity images of freshly isolated trachea explant tissues with detection wavelengths
from 405nm-455 nm, 455nm-505nm, 505 nm-555nm, and 555nm-605 nm. The color bar
represents photon counts. (c-d) Spectrally resolved lifetime images of the same samples
92
with the same detection wavelengths as (a, b). FLIM image in each spectral band is the
average lifetime of the pixel. The color bar indicates lifetime. Structural information does
not unravel the injury, while subtle lifetime differences exist between the two sets of
images. (e, f) Histogram plots of the two ROIs in the corresponding samples in lifetime
images (c, d).Lifetime distributions in the two ROIs suggest the metabolic activities are
changing in the two samples. (g) Average spectra of the two ROIs in each channel in the
corresponding samples in intensity images (a, b). The spectral distributions are highly
impacted by signal strength and we cannot classify these samples just by looking at the
distribution. The first, third, and fifth rows (Fig. 3a, 3c, 3e) come from a healthy sample
and the second, fourth and sixth rows (Fig. 3b, 3d, 3f) are from diphtheria+ tracheal
explant tissues. (h) A tile-scan image of a single-channel FLIM image from a healthy
sample. (i) A tile-scan image of a single-channel FLIM image from a diphtheria+ sample.
The tile scan images provide an overview of biological structures, but we cannot tell injury
from healthy tissue just by looking at these images. There is a need for a quantitative
analysis to differentiate injury.

3.1.2 Multi-dimensional SpectraFLIM data
SpectraFLIM produces high dimensional datasets where each voxel can have up to
6 dimensions: volume (x, y, z), time (t, seconds), wavelength( 𝜆, nanometers), and
lifetime (𝜏, nanoseconds) (Fig. 3.2). This 6-dimensional information is shaped as (x,
y, z, time [s], wavelength, lifetime [ns]). In this work, the time dimension is omitted
due to the end-point nature of the experiments, a 740 nm two-photon laser was
used as a light source, a spectrally resolved lifetime autofluorescence microscope
to acquire 4-dimensional images that extend in space (𝑥,𝑦), wavelength (𝜆), and
lifetime (𝜏). An anatomy of the SpectraFLIM data is shown in Figure 3.2. We extend
acquisition from single channel to collecting multi-channel intensity and lifetime
information simultaneously. The detected electromagnetic wavelength (405-605nm)
93
is divided into multiple spectral channels with a bandwidth of 50 nm (4 channels in
our configuration). In each spectral channel, we measure the lifetime information
that enables us to see the metabolic activities of cells in such spectrum range. In
addition to lifetime, the integral of photons captured in each spectral channel results
in an intensity image, shown in Figure 3.1c, e, which constitutes the multi-channel
fluorescence intensity dataset (𝑥,𝑦,𝜆). With spectrally resolved intensity images to
form a spectral cube in wavelength dimension, simultaneously same wavelength
bands resolved lifetime images are acquired to form a FLIM cube in lifetime
dimension, and we combine the information from two domains together to form this
4-dimensional SpectraFLIM data set. We then applied our workflow to classify
differential metabolic states of airway tissues.

Figure 3.2 Anatomy of 5-D multimodal SpectraFLIM data with dimensions of x, y, z, 𝜆 for
spectral dimension, and 𝜏 for lifetime dimension. (a) is a 5-D SpectraFLIM data that is a
volume at position (x, y, z). (b) is a spatial cross-section of the 5-D volume at position z
94
with dimensions (x, y, 𝜆, 𝜏). (c) If we zoom into one single pixel of the cross-section in the
4-D slice (x, y, 𝜆, 𝜏) at position (x, y), this 2-D plot shows this single pixel with information
in lifetime and wavelength (𝜆, 𝜏). (d,e) This pixel has a dimension of the spectrum and a
dimension of lifetime, where both lifetime and intensity are acquired at a narrow
wavelength band and the acquisition is repeated on multiple wavelength bands to span a
large wavelength range (405 – 605 nm). SpectraFLIM thus captures information from both
spectral and lifetime domains.
3.2 4D-imaging classification to separate metabolic states
3.2.1 Overview of current deep learning approaches
Convolutional neural networks (CNNs) are commonly used to address the
challenges of unsupervised learning for the classification of medical images in
diseases
26–31
. CNNs show great advantages for investigating biological events in
fluorescence microscopy imaging applications; from a single cell to fully formed
organisms at cellular resolution32–36. However, CNNs are generally applied to
grayscale (𝑥, 𝑦) or sometimes RGB (𝑥, 𝑦, 3) images and are not suitable for spectral
or lifetime cubes owing to the additional dimensions of wavelength and lifetime
respectively. 2-D CNNs commonly used for multi-channel image input must tradeoff
between processing spatial and spectral dimensions separately without considering
spatial-spectral as a whole. Fortunately, 3-D CNNs for hyperspectral data based on
spectral-spatial convolutions have been published37 but have high-complexity
computations due to the combination of spatial and spectral dimensions per
sampling point
34,38–42
.
Deep Learning methods have been applied individually to Fluorescence Spectral
and Lifetime microscopy datasets. Modified architectures have been implemented
for feature segmentation and classification in Fluorescence Spectral imaging
95
utilizing pure 1-D CNN for spectral
43
or a combination of both spatial and spectral
information
38
. Deep learning architectures have been applied to recovering and
reconstructing spatially resolved complex FLIM images
44,45
. 1-D CNN has been
developed for fast analysis of FLIM data to extract lifetime parameters from
fluorescence signals, is robust, and has good performance
46
. Similarly, with current
CNNs utilized in spectral imaging, this 1-D approach loses contextual spatial
information of the whole dataset. SpectraFLIM is only recently becoming a more
accessible type of data acquisition
25
, but the exploration of this multimodal data
through deep learning approaches is still limited.
Recurrent Neural Networks (RNN) have been acknowledged as effective solutions
for sequential data analysis. The RNN architecture uses recurrent connections
between neural activations at consecutive time steps where hidden layers and
memory cells learn the underlying dynamics of input sequences over time
47,48
.
Long Short-Term Memory (LSTM) is a commonly used RNN architecture that
avoids vanishing gradient problems
49
. LSTM has been adapted to multi-spectral
and hyperspectral reflectance data as the temporal variability of a sequential signal
is similar to the spectral variability of a pixel in a spectral image. The fundamental
advantage of Spectral LSTM architecture over standard CNNs is its ability to
sample spectral values of pixels in different channels sequentially to learn the
spectral features
50–52
. However, while LSTM treats the spectral feature extraction
as a sequence-learning problem, it also fails to consider the spatial information.
Convolutional LSTM (ConvLSTM) overcomes this limitation and extends the fully
connected LSTM to include convolutional structures, allowing the network to
96
capture not only the sequential dynamics but also the spatial correlations
53
.
ConvLSTM has been used in hyperspectral image classification to learn spectral-
spatial features
52
. Here we explore a combination of deep learning approaches and
multimodal measurements of live tissue autofluorescence to classify airway disease
injuries of an animal model. We developed a SpectraFLIM-LSTM model which
considers the entire multimodality of the 5-D dataset and compares it to different
deep learning models for dimensionally sectioned subsets of the 5D data set and
evaluated their performances.
3.2.2 Deep learning approaches for investigating 5-D SpectraFLIM data
We design different deep learning networks to explore the multiple dimensions of
5D spectral FLIM data (x, y, z, 𝜆, 𝜏 ) to investigate the performance of different
domains of information (intensity, spectral, lifetime) in classifying airway disease.

97
Figure 3.3 Workflow of SpectraFLIM classification using COPD mouse model with multiple
neural network strategies. (a) To understand the health status of airway epithelia, we
designed an experiment using mouse model for airway disease, where we generated a
genetic animal model in which airway epithelial cells are affected by diphtheria toxin,
inducing cell death. (b) The effect of this cell depletion in the mouse model was measured
utilizing a SpectraFLIM microscope with cellular resolution. We used this 2-Photon spectral
and lifetime autofluorescence microscope that is capable of acquiring 4-dimensional
images, that extends in space (x, y), wavelength(𝜆) and lifetime (𝜏). We applied this
instrument to detect differential metabolic states. (c) We designed different deep learning
networks to explore multiple dimensions of SpectraFLIM data (x, y, z, 𝜆, 𝜏) to investigate
the performance of classifiers using different domains of information (intensity, spectral,
lifetime) in classifying airway disease. We adapted a basic CNN to explore 3-D, 4-D
subsets of fluorescent data. We also developed a model called SpectraFLIM-LSTM, (a
ConvLSTM in combination with SpectraFLIM autofluorescence data) to produce an
accurate classification of airway disease. ConvLSTM, originally applied to sequential data,
is here used to process the fluorescence lifetime and spectral dimension with a
convolutional gated recurrent network, followed by spatial processing. (d) The input layer of
each network model connects to the rest of the convolutional layers. (e) We trained the
above models to predict the class label for disease classification.

In this work, we use a customized Convolutional Long Short-Term Memory
(ConvLSTM) network characterized by convolutional input transformations and
recurrent transformations. This network shows its capability to effectively harvest
spectral, lifetime, and spatial information of 5-D SpectraFLIM data by stacking the
ConvLSTM layer with 2-D convolutional layers. In addition, we compare its results
to a modular 2-D CNN for exploring dimensionally sectioned subsets of the data.
Experimental results on SpectraFLIM data suggest competitive performance in the
SpectraFLIM-LSTM approach.

98

Figure 3.4 Inner structures of 2-D convolution and ConvLSTM. (a) A scheme of
SpectraFLIM data set that includes a 4-ch FLIM stack and a 4-ch intensity stack. (b) 2-D
convolution takes a multi-channel image regardless of its spectral dimension, which does
not preserve the spectral correlations of the image. (c) ConvLSTM inputs the multi-
channel image sequentially and therefore preserves the correlations across channels. The
figure shows an input sequence {𝑥
%&!
, 𝑥
%&"
,…, 𝑥
%&'
} and a sequence of hidden states
{𝐻
%&!
, 𝐻
%&"
, …, 𝐻
%&'
}. At a given time point (in our case channel ch), the ConvLSTM layer
receives the input 𝑥
%&
and its previous output value 𝐻
%&(!
, then calculates the current
state 𝐻
%&
.

Figure 3.5 2-D CNN and ConvLSTM architectures. (a) An in-depth view of the input layer
for single-channel input, where the 2-d convolutional kernel is applied to the 2-d image
slice. (b) An in-depth view of the input layer for multi-channel input, where the 2-d
convolutional kernel is applied to the 4-channel input regardless of its spectral channel
correlations. (c) Network architecture for 2-D CNN input. (d) An in-depth view of the
99
ConvLSTM layer for SpectraFLIM input. (e) The network architecture that connects to the
ConvLSTM layer for SpectraFLIM data.

SpectraFLIM-LSTM
Sequence-based RNN has gained popularity for the analysis of time series
47,48
,
however, it has not been commonly used in spectrally resolved FLIM imaging. We
utilize an LSTM model with convolutional operations to characterize the sequential
property of the spectral information and spectrally resolved lifetime information of
pixel vectors in biomedical autofluorescent samples. The architecture exploits a
recurrent operation to understand correlation across channels in both intensity and
lifetime domains. To the best of our knowledge, this is the first use of the recurrent
neural network model to explore channel-to-channel variability both in the lifetime
domain and in a combination of spectra and FLIM. We apply ConvLSTM to a
convolutional neural network model to harvest all 5 dimensions of information for
the SpectraFLIM data input.
Multi-spectral image data can be treated as a set of ordered spectral sequences in
the intensity domain. Spectrally resolved lifetime image data follow the same
sequential order but in the lifetime domain; the order preserves spectral information.
SpectraFLIM data have information distributed in an ordered sequence across both
intensity and lifetime dimensions. The SpectraFLIM-LSTM model binds information
across spectral channels and addresses the interconnected properties while
analyzing spectrally resolved intensity and FLIM data. We combine the architecture
with 2-D convolutional blocks to extract additional spatial features. The purpose is
100
to explore 5-D SpectraFLIM data both spatially and sequentially accounting for both
multi-dimensional and -modal information. A voxel with multi-channel intensity and
lifetime values can be regarded as a data point with two independent sequences. In
our setup, ConvLSTM encodes spectral-spatial sequences of 5-D data and outputs
the same 3-D tensor shape as a 2d convolutional layer with 4-D inputs. Thus,
ConvLSTM stacked with a batch normalization layer, a pooling layer, and a dropout
layer can subsequently be adapted as a building block for more complex structures
(Figure 3.3). The adapted ConvLSTM block receives the spectrally sequenced
voxels and connects to 2-d convolutional layers. This ConvLSTM model
compresses the whole input sequence into 3-D tensors that preserve all the spatial
information, making it advantageous for classifying complex hyperdimensional
sequences. In the proposed ConvLSTM network, (Figure 3.3) the input of the
network is a multispectral image set 𝑥, where spectral channels are denoted as 𝑥
EM
;
each channel includes a FLIM image and an intensity image. The existing spectral
channel 𝑥
EM
is initially fed into the input layer, and the ConvLSTM layer receives 𝑥
EM

and calculates the hidden state information 𝐻
EM
. Subsequently, we can unfold the
network for 𝑐ℎ steps to get the output at channel 𝑐ℎ+1. The next channel 𝑥
EM4.

together with the state information of 𝑥
EM
are input to the ConvLSTM layer, and the
operation at spectral channel 𝑐ℎ+1 is computed by a convolutional kernel. Finally,
the ConvLSTM layer connects to the rest of the convolutional layers to predict the
class of the input.
101
2-D CNN
We adopt a basic CNN composed of convolutional layers, batch normalization,
pooling layers, and dropout layers (Figure 3.3) to explore 3-D, 4-D, and 5-D subsets
of fluorescent data. We implement different input strategies (Figure 3.2) to
accommodate various dimensionally sectioned inputs. The convolutional layer
extracts key features of input data through kernel filters. Following the convolutional
layer, a fully connected layer, with up to approximately 442k trainable parameters
connects to a dense layer with a sigmoid activation function to predict the probability
for each class.
3.2.3 Exploring subsections of SpectraFLIM data
We create dimensionally sectioned subsets of the 5-D SpectraFLIM fluorescent
data (Figure 3.1c) and pair them with deep learning networks (Figure 3.3) to
investigate how different information-rich subsets can classify airway disease. We
divide the data into dimensionally different sets containing: 3-D single-channel
intensity 𝐷(𝐼)
;,=,>
where 𝐷(𝐼) is the data with intensity 𝐼 at position (𝑥,𝑦,𝑧); 3-D
single-channel FLIM 𝐷(𝜏)
;,=,>
where 𝐷(𝜏) is the data with lifetime 𝜏 at (𝑥,𝑦,𝑧); 4-D
spectrally resolved intensity 𝐷(𝐼)
;,=,>,EM
where 𝐷(𝐼) is the data with intensity 𝐼 at
(𝑥,𝑦,𝑧,𝑐ℎ); 4-D spectrally resolved FLIM 𝐷(𝜏)
;,=,>,EM
where 𝐷(𝐼) is the data with
lifetime 𝜏 at (𝑥,𝑦,𝑧,𝑐ℎ), and 5-D SpectraFLIM 𝐷(𝐼)
;,=,>,EM
+𝐷(𝜏)
;,=,>,EM
that includes
the lifetime data 𝐷(𝜏) and intensity data 𝐷(𝐼) at (𝑥,𝑦,𝑧,𝑐ℎ).
102
Finally, the 5-D SpectraFLIM 𝐷(𝐼)
;,=,>,EM
𝐷(𝜏)
;,=,>,EM
includes the data in its entirety,
both spectral and lifetime information. We utilize SpectraFLIM-LSTM, a ConvLSTM
that accounts for spectral and lifetime information equally through the LSTM
portions while extracting spatial features through the convolutional operations. This
choice is driven by the limitation of applying typical CNNs to multi-spectral images.
CNN architecture is dedicated to encoding spatial channels starting from a single
spatial channel image. The typical 2-dimensional kernel of a convolutional filter is
neither well suited for hyperspectral stacks nor spectrally resolved FLIM stacks,
which include a third tensor dimension dedicated to the spectral channels.
3-D Single-channel intensity 𝐷(𝐼)
;,=,>|EM
and 3-D Single-channel FLIM 𝐷(𝜏)
;,=,>|EM

data are introduced in a dimensionally matched CNN model (Figure 3.2a, 3.2c) as
single z-section images. The process is repeated separately for different channels.
An example of this dimensionally sectioned subset is reported in Figure 3a-3d,
which includes intensity data 𝐷(𝐼)
;,=,>|EM+. ,-,, ,F( ^
in Figure 3a-3b and Figure 3c-3d
with lifetime data at channel 1, 2, 3, and 4. The input contains raw pixel values of
the image, in this case, a greyscale image of photon counts, or the averaged
fluorescence lifetime decay value scaled to 16 bits. The input layer in the ML model
used for data convolves the input data with a 2d filter(Fig. 4b, Supp Fig. 1a). We
tested the performance of the network when training with intensity images from four
spectral channels separately and compared the results with across FLIM channels.
4-D Spectra 𝐷(𝐼)
;,=,>,EM
and 4-D Spectrally resolved FLIM 𝐷(𝜏)
;,=,>,EM
datasets are
explored with the same 2-D CNN (Figure 3.2b, 3.2c). These subsets consist
103
respectively of intensity and average lifetime values of the image data along the 4
spectral channels. We explore these 4-D datasets using 2-D CNN (Fig. 3.3),
capturing spatial structures with 2d convolutional filters but not preserving the
spectral information with the purpose to investigate the classification performance of
spatial information across multiple channels for intensity and lifetime.
3.3 Improving classification by integrating spectral and FLIM
information into deep learning
Our results demonstrate the ability to classify airway tissue health based on
metabolic cellular autofluorescence utilizing spectral imaging in combination with
lifetime imaging with deep learning. We explored the informational content for
dimensional subsets of the complex 5-D SpectraFLIM data by designing and
comparing multiple different network architectures, processing different aspects of
spectral, lifetime, and spatial domains. Based on our results, 5-D SpectraFLIM
data, when explored in its multi-dimension and -domain entirety, captures the
highest information content, outperforming other dimensional subsets. These
results prove the potential of SpectraFLIM-LSTM in tissue classification and disease
diagnosis.
3.3.1 The impact of spectrally resolved spatial information
We evaluate the impact of spectrally resolved spatial information by changing the
number of channels on the classification of autofluorescent biomedical samples.
104
We utilize single-channel 𝐷(𝐼)
;,=,>|EM
and four-channel 𝐷(𝐼)
;,=,>,EM
intensity images
for training, without preserving, in either case, the spectral information. This
approach aims at evaluating the effect of increasing spatial information when
classifying this type of data.

Table 3.1 Results for all input strategies with data in original shape, comparing different
methods. Single-channel intensity input D(I)x,y,z|ch provides higher accuracy, than single-
channel lifetime input D(τ)x,y,z|ch, and this is because intensity images have stronger
signals and higher bit depth. Increasing the number of channels, FLIM accuracy improved
by 2.1%, while it doesn’t improve the performance for the intensity domain. This could be
FLIM images capturing fluorescent molecules in their distinct metabolic stages or different
fluorescent species in the specific wavelength range. SpectraFLIM has the best
performance over all other classification methods.

105

Table 3.2 Results for all input strategies with data after data binning. Binning yields higher
signals in lifetime and intensity. Comparing to Table 3.1, it appears that the accuracy for
single-channel FLIM data improved slightly (2.0%) from binning, while that of single-
channel intensity degraded (1.5% on average). Consistently, the spectrally resolved 4-
channel FLIM (D(τ)x,y,z,ch) had a 6.9% improvement from binning, while it shows no
significant difference for the intensity version D(I)x,y,z,ch. A larger improvement in FLIM
data is expected, as the precision in determining lifetime decay value is more sensitive to
noise and photon counts. After binning, signals are enhanced. So the FLIM classifiers
benefit from stronger signals. SpectraFLIM (D(I)x,y,z,ch D(τ)x,y,z,ch) accuracy improved
by 4.0% after binning, which provides the highest accuracy in our test.

106

Figure 3.6 Results for all the input strategies with data in original shape and data after
binning, comparing different methods. (a) Performances for all the classifiers with data in
original shape. Each classifier was evaluated using accuracy, sensitivity, specificity, and
AUC. All values are in percentage. (b) We then evaluated the effect of increased signal
quality by virtually increasing the number of signals through data binning. In this binning
process, the information from four adjacent voxels on the same optical plane was
summed into a single super voxel. The result of this operation is an increase of signal
quality for both spectra and lifetime, at the cost of reducing spatial resolution. (c) ROC
curves for all the classifiers with data in original shape. (d) ROC curves for all the
classifiers with binned data.

107
Single-channel intensity classification provides an average accuracy of 84.9% +/-
0.6% (Table 1) with minimal variation across the channels. By comparison, the
four-channel intensity, which increases linearly the spatial-information content,
provides an accuracy of 87.0%, approximately a 2.1% improvement over the single-
channel. This suggests the intensity information is more uniformly distributed
across channels and spatial information increase is not a major contributor to the
accuracy of classification for the intensity of autofluorescent data. Other
performance metrics such as sensitivity, specificity, and AUC provide the same
conclusion.
We perform a similar comparison for the fluorescence lifetime domain. Single
channel lifetime images 𝐷(𝜏)
;,=,>|EM
provide an average 77.4% accuracy (Table3.1
1) with a standard deviation of +/-1.9%. Increasing the number of lifetime channels
to four 𝐷(𝜏)
;,=,>,EM
increases classification accuracy by an absolute value ~8.4%;
this is confirmed by other performance metrics (Sensitivity, specificity, AUC). These
results suggest that the lifetime information is not uniformly distributed across
channels and that the increased spatial and channel lifetime information has a
pronounced effect on the performance of classification.

3.3.2 Comparing classification in intensity and lifetime domain
We compare the classification accuracy when utilizing intensity or lifetime, both in
single- and four-channel. Single-channel intensity data 𝐷(𝐼)
;,=,>|EM
provides on
108
average ~7.5% higher accuracy with a standard deviation >3 times smaller, in
comparison to single-channel lifetime input 𝐷(𝜏)
;,=,>|EM
. Spectrally intensity
𝐷(𝐼)
;,=,>,EM
provides 1.2% classification improvement over the corresponding
spectral resolving the lifetime information ( 𝐷(𝜏)
;,=,>,EM
), and the confidence interval
with 𝑝 <0.05 between these two classifiers are partially overlapped, suggesting that
the information content of the two-dimensional subsets, with respect to
classification, is similar. Increasing the number of channels improved by 2.1% the
performance for the intensity domain while the change was 4-fold larger for lifetime
classification. Intrinsic biomarkers are characterized by wide autofluorescent
spectra that result in reduced differences across intensity channels, producing
limited improvement over single-channel intensity. The same biomarkers are
characterized by more marked differences in average lifetime decay times for each
spectral channel, improving classification when considering four channels.
The cross-information between spectral channels in the lifetime domain enables the
capability of harvesting information on subtle metabolic changes in complex cellular
activities. Classifiers with single-channel FLIM improved by 2.0% by binning while
with SpectraFLIM input improved (accuracy) by 4.0% by binning, suggesting other
informational combinatorial factors have contributed to improvement during the
synthetic signal augmentation. Adding more dimensions improves classification
more significantly while signals are sufficient, this is possibly related to the
additional spectral information provided. Binning in this spectrally resolved data set
provides enhancement of spectral connections between lifetime channels.
109

3.3.3 Classification across spectral, lifetime, and spatial dimensions
Spectrally resolved FLIM in combination with intensity in spectral channels
𝐷(𝐼)
;,=,>,EM
𝐷(𝜏)
;,=,>,EM
using ConvLSTM gates presents the best classification
performance amongst all the different strategies. SpectraFLIM shows absolute
improvements up to 5.2% for accuracy, 3.6% for sensitivity, 7.2% for specificity, and
6.2% for Area Under the Curve (AUC) compared with 4-ch FLIM, and 4.0% for
accuracy, 3.8% for sensitivity, 4.1% for specificity and 3.3% for AUC compared with
4-ch Intensity. A full performance report is shown in Fig. 5a, 5c. We observe a
more significant improvement in comparison to 4-ch FLIM input than to 4-ch
intensity input.
In conclusion, SpectraFLIM-LSTM provides the highest performance across the
dimensional subsets of 5-D SpectraFLIM data, and the network architectures
tested, for the metrics of accuracy, sensitivity, specificity, and AUC. The results
here reported (Fig. 3.4) show a trend of improved performance with the increasing
number of informational dimensions. The maximum accuracy is reached when
considering autofluorescence spatial, spectral, and lifetime domains with higher
signal quality.
3.3.4 Effects of signal quality on classification
We evaluate the effect of increased signal quality on the classification metrics by
virtually increasing the number of signals through data binning. In this process, the
110
information from four adjacent voxels on the same optical plane is summed into a
single voxel. The result of this operation is an increase in signal quality for both
spectra and lifetime, at the expense of halving the spatial resolution. We adapt all
the dimensionality subset network architectures to account for this spatial-resolution
change and compute the performance metrics (Table 3.2).
We compare the performance of the binned higher signals (Table 3.2) with that of
the original lower signals (Table 3.1). The accuracy for single-channel FLIM data
(𝐷(𝜏)
;,=,>|EM
) improved by 2.0% from binning, while that of single-channel
intensity 𝐷(𝐼)
;,=,>|EM
data decreased 1.5% on average. Consistently, the spectrally
resolved dimensional subsets of 4-channel FLIM ( 𝐷(𝜏)
;,=,>,EM
) had a 6.9%
improvement from binning, while the confidence intervals show no significant
difference for the intensity version 𝐷(𝐼)
;,=,>,EM
. A larger improvement in FLIM data is
expected, as the precision in determining lifetime decay value is more sensitive to
noise and photon counts. Finally, SpectraFLIM data (𝐷(𝐼)
;,=,>,EM
𝐷(𝜏)
;,=,>,EM
)
accuracy improved 4.0% after binning, providing the highest accuracy in our test.
Signal quality in the SpectraFLIM data has an important role in the information
content of its dimensional subsets. The single-channel intensity (𝐷(𝐼)
;,=,>|EM
) subset
has a 7.5% higher accuracy than the corresponding FLIM subset in the original
shaped data (Table 3.1). This difference in accuracy can be explained by the
dynamic range between intensity and lifetime data. The photon count range is 16
bits, limited by the amount of signal integration during acquisition. The lifetime has
a resolution of 0.001 ns per gray level, with an instrument limitation of the lifetime
111
range of 0ns-12.5ns, which will produce a dynamic range of fewer than 14 bits. In
our data, the measured lifetime range was (0.3 ns, 11.4 ns). This decreased
dynamic range for a lifetime is compensated by more marked lifetime differences
across channels. For example, NADH, one of the large contributors of
autofluorescence at this excitation wavelength in live tissues, is known for having
considerably different lifetimes in its free and bound states4,5,15. This lifetime
variation across channels can be seen in the accuracy variation across channels in
Table 1. By comparison, the autofluorescent spectra, known to be broader in
nature3 produce more uniform accuracies across single-channel intensity (Table
3.1). Accordingly, the 4-channel FLIM classification accuracy shows no significant
difference from the one of 4-channel intensity, suggesting the diversity in lifetimes
across channels aids this type of classification.
To better characterize the effects of signal quality we explored a binned version of
the data (Table 3.2) that synthetically increased the accuracy of spectra and lifetime
in each voxel. In this scenario, the single-channel intensity (𝐷(𝐼)
;,=,>|EM
) accuracy is
on average 4.0% better than the single-channel input in FLIM, although the intensity
data classification shows no significant change from binning on average. Data
containing lifetime domain information benefit significantly from binning, and this
strategy is commonly utilized as preprocessing in FLIM analysis
44
.
Interestingly, after binning, the 4-channel FLIM 𝐷(𝜏)
;,=,>,EM
accuracy improved to
92.7%, and AUC improved to 94.8%, outperforming the 4-channel intensity
𝐷(𝐼)
;,=,>,EM
. We believe multiple factors contributed to this difference in
112
performance: increased lifetime signal quality reported variation of lifetime values
across spectral channels for autofluorescent metabolites and decreased spatial
resolution. Our data (Table 3.1, 3.2) suggests that spatial resolution is a major
contributor to CNN-based classification of autofluorescent intensities. The accuracy
of single-channel intensity 𝐷(𝐼)
;,=,>|EM
decreased 1.5% on average after binning
(Table 3.2, Figure 3.5), while the 4-channel intensity 𝐷(𝐼)
;,=,>,EM
decreased by
0.2%. In comparison the single-channel (𝐷(𝜏)
;,=,>|EM
) and 4-channel lifetimes
𝐷(𝜏)
;,=,>,EM
increased respectively 2.0% and 6.9%, suggesting the spatial resolution
decrease caused by binning did not outweigh the improvements on fluorescence
lifetime accuracy.
3.4 Methods
3.4.1 Image acquisition
A 740 nm two-photon laser was used as the light source for SpectraFLIM imaging
and Leica SP8 Falcon was employed for FLIM data acquisition. The spectral bands
ranging from 405 nm to 605 nm with a bandwidth of 50 nm. The spatial resolution is
512 × 512 pixels with an average of 8 frames in acquisition.
3.4.2 Data standardization
Images with a size of 512 × 512 pixels were collected with an average of 8
accumulated frames at a time to achieve a decent amount of photon counts and
then split into 4 sub-images with the size of 256 × 256. Lifetime values were
113
changed to 0 where the pixel has photon counts less than 20 in intensity (16-bit as
maximum). We normalized intensity and lifetime values per image slice. Pixel
values in each image slice were scaled into the range [0, 1] with unit-based
normalization (Eq. 1).
𝑋
*
=
_"_
C$ .
_
CD#
"_
C$ .
(Eq. 3.1)

3.4.3 Data Binning
Fluorescent images suffer from low signal-to-noise ratio (SNR) in general. Binning
the data may reduce the impact of noise on the image at a cost of a reduced
resolution. An array of 4 pixels is combined into a single pixel in a 2×2 kernel by
summing the pixel values across the entire image slice. Image size is reduced to
128 × 128 after binning.
3.4.4 2-D CNN Model
Our CNN design is composed of convolutional layers, max pooling, and dropout
layers as shown in Figure 3.3. All CNN architectures incorporate five 2-D
convolutional layers with size 3 by 3 along with max-pooling layers with kernel
size=2 and dropout layers. Then a fully connected layer with size=128 is
concatenated to reduce feature dimensions towards convergence of model training.
The convolutional layer extracts key features of input data through filters,
114
𝑞
;,=
())
= 𝜎 (∑ ∑ ∑ 𝑤
C,T
())
𝑞
;4C"`,=4T"`,EM
()". )
a".
T+!
a".
C+!
b".
EM+!
+𝑏
())
) (Eq. 3.2)
Where 𝑞
;,=
())
and 𝑞
;4C"`,=4T"`,EM
()". )
are the output and input data at spatial position
(𝑥,𝑦) in the 𝑖-th layer, respectively, and 𝑏
())
is the bias. 𝑤
C,T
())
denotes a filter of size
(𝐻×𝐻×𝐶), and 𝐾 =𝐻/2. The output of each filtering operation is computed
through an activation function 𝜎, which is rectified linear unit (ReLU) 𝜎(𝑧)=
𝑚𝑎𝑥 (0,𝑧). This architecture can be extended to three-dimensional problems while
𝑐ℎ >1 in accordance with multi-channel input data. In this study, the number of
channels 𝐶 is set to 1, 4 with respect to the number of spectral channels.
This approach can be directly extended to the use of FLIM domain by treating the
FLIM channels as intensity channels. We also stack all FLIM channels into the
CNN’s first layer’s input channel to compare the performance between intensity
inputs and FLIM inputs.
3.4.5 ConvLSTM Model
We use a convolutional LSTM that performs joint spectral and FLIM processing in
its units, where each channel is processed sequentially. We refer to this method
SpectraFLIM LSTM. As the input gets more complex from adding more channels in
lifetime and/or intensity domain, we adjust the model architecture by increasing filter
numbers with regards.
115
The LSTM model has a sequence of input {𝑥
.
,… 𝑥
EM
}, cell outputs {……𝑐
EM
}, hidden
states {…,..., ℎ
EM
}, input gate 𝑖
EM
, forget gate 𝑓
EM
, output gate 𝑜
EM
. Memory cell and
output can be computed by:

𝑖
EM
= 𝜎(𝑊
;)
𝑥
EM
+𝑊
M)
ℎ
EM".
+𝑊
E)
𝑐
EM".
+𝑏
)
) (Eq. 3.3)
𝑓
EM
= 𝜎F𝑊
;c
𝑥
EM
+𝑊
Mc
ℎ
EM".
+𝑊
Ec
𝑐
EM".
+𝑏
c
I (Eq. 3.4)
𝑐
EM
=𝑖
EM
∘ 𝑡𝑎𝑛ℎ(𝑊
;E
𝑥
EM
+𝑊
ME
ℎ
EM".
+𝑏
E
)+𝑓
EM
∘𝑐
EM".
(Eq. 3.5)
𝑜
EM
= 𝜎(𝑊
;F
𝑥
EM
+𝑊
MF
ℎ
EM".
+𝑊
EF
∘𝑐
EM
+𝑏
F
) (Eq. 3.6)
ℎ
EM
= 𝑜
EM
∘𝑡𝑎𝑛ℎ(𝑐
EM
) (Eq. 3.7)

where 𝑏
)
, 𝑏
c
, 𝑏
E
and 𝑏
F
are bias terms, 𝜎 is nonlinear activation function, and ∘
denotes the Hadamard Product. 𝑊 is the weight matrix with subscripts that denote
the gates, for instance, 𝑊
;F
is the weight matrix between the input gate and output
gate, and 𝑊
M)
is the hidden-input gate matrix, etc.
The ConvLSTM model is an extension of LSTM that has convolutional structures for
spatio-spectral sequence encoding. All the inputs {𝑋
.
,…𝑋
EM
}, cell outputs {…...𝐶
EM
},
hidden states {…,..., 𝐻
EM
}, input gate 𝑖
EM
, forget gate 𝑓
EM
, and output gate 𝑜
EM
are 3D
tensors whose last two dimensions are spatial dimensions (𝑥 and 𝑦 in width and
length). The computation of ConvLSTM is as below:
116

𝑖
EM
= 𝜎(𝑊
;)
∗𝑋
EM
+𝑊
M)
∗𝐻
EM".
+𝑊
E)
∘𝐶
EM".
+𝑏
)
) (Eq. 3.8)
𝑓
EM
= 𝜎F𝑊
;c
∗𝑋
EM
+𝑊
Mc
∗𝐻
EM".
+𝑊
Ec
∘𝐶
EM".
+𝑏
c
I (Eq. 3.9)
𝑐
EM
=𝑖
EM
∘ 𝑡𝑎𝑛ℎ(𝑊
;b
∗𝑋
EM
+𝑊
Mb
∗𝐻
EM".
+𝑏
b
)+𝑓
EM
∘𝐶
EM".
(Eq. 3.10)
𝑜
EM
= 𝜎(𝑊
;F
∗𝑋
EM
+𝑊
MF
∗𝐻
EM".
+𝑊
EF
∘𝐶
EM
+𝑏
F
) (Eq. 3.11)
ℎ
EM
= 𝑜
EM
∘𝑡𝑎𝑛ℎ(𝐶
EM
) (Eq. 3.12)

Where same as above, ∘ denotes the Hadamard product, and ∗ denotes the
convolution operator.
We train all the models with a cross-entropy loss and Adam optimizer. Weight
decay is used as L2 regularization. We perform class balancing by reweighting the
loss function with the inverse class frequency 𝑤
)
=
d
d
$
where 𝐷 is the total number of
samples and 𝐷
)
is the number of samples in class 𝑖. Hyperparameters such as
learning rate and feature map size are optimized for the individual models based on
validation performance. An early stopping criterion is utilized for the training
process.
117
3.4.6 Models for binned data
For the image input with 256 by 256, we apply 5 convolutional layers in the CNN
model and 1 ConvLSTM layer connected to 4 convolutional layers in the ConvLSTM
model. Each convolutional layer is followed by max pooling and dropout layers. For
reduced image size, we reduce the total number of pooling layers by 1. We stack
the last 2 convolutional layers directly on top of each other instead of staking a
pooling layer on top of each convolutional layer.
3.4.7 Performance Metrics
We consider accuracy, sensitivity, specificity, and Area under the ROC Curve
(AUC) as our performance metrics due to class imbalance.
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =
e(:7 fFG)#)g74e(:7 I79X#)g7
fFG)#)g74I79X#)g7
(Eq. 3.13)
𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 =
e(:7 fFG)#)g7
e(:7 fFG)#)g74KX3G7 I79X#)g7
(Eq. 3.14)
𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 =𝑇𝑁𝑅 =
e(:7 I79X#)g7
e(:7 I79X#)g74KX3G7 fFG)#)g7
(Eq. 3.15)
𝑇𝑃𝑅 =
e(:7 fFG)#)g7
e(:7 I79X#)g74KX3G7 fFG)#)g7
=1−𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 (Eq. 3.16)
𝐹𝑃𝑅 =
KX3G7 fFG)#)g7
e(:7 fFG)#)g74KX3G7 I79X#)g7
(Eq. 3.17)
The Receiver Operator Characteristic (ROC) curve evaluates binary classification
problems. It is a probability curve that plots TPR against FPR at different
118
classification threshold values. The Area under the ROC Curve (AUC) measures
the ability of a classifier to distinguish between classes across all possible
classification thresholds. The higher the AUC, the better the performance of the
model at distinguishing between positive and negative classes.
We derived 95% confidence intervals using DeLong’s method
58,59
, an approach for
comparing AUCs that allows the derivation of the asymptotic distribution of AUC
and the statistical test. In this work, we used an image-weighted average of AUC to
measure overall performance.
60

119
Chapter 4 The future of multi-dimensional
fluorescence microscopy data analysis
4 The future of multi-dimensional fluorescence
microscopy data analysis
Cells and proteins are deeply intertwined across tissues at multiple levels, leaving
lots of unanswered questions about the interactions between components.
Fluorescence microscopy enables scientists to monitor the temporal dynamics and
spatial structures of these biological processes in living organisms. However,
visualizing and understanding the dynamics in living organisms adds another layer
of complexity to this problem than fixed tissues. Scientists need careful planning
and strategies to balance the tradeoffs between imaging speed, photobleaching and
phototoxicity, spatial resolution, penetration depth, and signal intensity.
Furthermore, 3-D volumetric imaging alone does not satisfy the need of observing
the underlying mechanisms of complex biological processes. For example, in
Chapter 1.2 where we discussed hyperspectral imaging, scientists are limited by the
number of fluorophores due to overlapping spectra and bleed-through artifacts.
Biological dynamics and interactions at metabolic levels are difficult to observe from
a chromatic image with information in spectral dimension. There exists a fourth,
fifth, and even sixth dimension in living organisms that we want to capture with an
advanced instrument. We believe there is a need of collecting information from a
higher dimension and following that comes with a need of visualizing and
interpreting complex data.
120
However, there is a gap between fluorescence data acquisition and image analysis
of multimodal multi-dimensional data. In this work, we aimed to extend the
multiplexing capabilities of spectral and lifetime fluorescence imaging and extract
information from complex biological information to fill this gap. We addressed
challenges in visualization and classification from acquisition to analysis in our
previous chapters. In Chapter 2, we focus on the rapid, pre-analysis visualization of
the hyperspectral dataset that can be useful in guiding the users during data
acquisition. In this context, the main goal is to visualize the information contained in
the spectral dimension by means of a compressed representation in a standard
RGB display. In Chapter 3, we developed a multimodal classification algorithm to
not only distinguish cellular metabolisms but identify spatial and spectral features
that mark how they are unique from each other.
In the trending development of live imaging microscopy, the dimensionality of
acquisition is continuously increasing as we acquire information from another
domain and/or combine imaging modalities. It is important to extend multiplexing
capabilities with SEER and SpectraFLIM while exploring the potential improvements
in image analysis with complex biological data. SEER provides an enhanced
visualization that can distinguish fluorophore species with overlapping spectra.
SpectraFLIM integrates information from multiple channels and multiple domains
and therefore achieves significant improvements in classifying subtle metabolic
differences. These algorithms extend the multiplexing capabilities of lifetime and
hyperspectral fluorescence imaging and have shown scientists the potential to
unravel the complexity of life sciences.
121
In this chapter, we discuss considerations we could take and potential steps we
could do from acquisition to preprocessing to achieve a higher performance using
these algorithms. We then discuss where we could apply these algorithms to solve
real-world problems in biological sciences and clinical environments. Last but not
least, we discuss the trend in the field of multi-dimensional fluorescence data
analysis, and algorithms to develop as well as how to optimize them to extract
complex information from high dimensions in a low signal-to-noise ratio regime.
4.1 Enhancing current techniques for multiplexing fluorescence
data
While applying SEER or SpectraFLIM algorithms to fluorescent data, there are a
few additional strategies we could take into consideration to enhance the
performance. Low signal strength remains one of the biggest challenges in
fluorescent image data when transferring image processing strategies from general
RGB images. In this section, we will discuss how we could apply denoising
strategies to SEER and data binning to SpectraFLIM to enhance signal strength and
therefore result in improved performance. Data hungry is another issue in
supervised learning that hampers the efficacy of classification algorithms. We will
discuss challenges and possible approaches to augment fluorescent data for the
SpectraFLIM classifier.

122
4.1.1 Improve visualization with denoising
Fluorescence data suffer from low signal-to-noise ratios. Current analysis of
multi-dimensional fluorescent signals benefits from SEER. SEER accomplishes
visualization of large data through the Phasor approach combined with spatially
lossless spectral denoising of data. In the microscopy world where acquisition is
in low signal-to-noise regimes, such denoising processes benefit visualization
significantly. Adding denoising prior to SEER enhances the color contrast and
removes blur in noisy data. Figure 4.1 shows the spectral denoising effect when
applying medium filters prior to SEER visualization.

Figure 4.1 Spectral denoising effect on Angular and Radial maps visualization of standard
overlapping spectra (Simulated Hyperspectral Test Chart II (SHTC II)). Phasor spectral
denoising affects the quality of data along the spectral dimension, without changing
intensities. Here the second harmonic is utilized for calculations. Noisy data appears as a
spreading cluster on the phasor, here shown overlaid with the (a) Angular map and (b)
Radial map, with the overlaid visualization exhibiting salt and pepper noise. (c, d) When
123
denoising is applied on the phasor, the cluster spread is reduced, providing greater
smoothing and less noise in the simulated chart. (e, f) Increasing the number of denoising
filters results in a clearer distinction between the three spectrally different areas in each
block of the simulation. (a, c, e) In Max Morph Mode, each denoising filter introduces a
shift of the apex of the map, changing the reference center of the color palette (b, d, f) In
Scale Mode, the less scattered phasor cluster makes maximum use of the reference
maps, enhancing the contrast (d, f) of the rendered SHTC.

There exists a vast number of denoising strategies that can be applied to
fluorescent data. Non-local mean (NLM) denoising has its advantages over “local
mean” filters in that it preserves edges and textures in intensity. In a side project,
the author adapted NLM denoising filters to sequential data and developed an
algorithm for denoising hyperspectral fluorescent images called lambda-NLM. Our
results showed that this approach enhances signals prior to image analysis and has
the potential to improve analysis performance.

4.1.2 Improve classification by increasing signal strength
Signal quality in the SpectraFLIM data has an important role in the information
content of its dimensional subsets. In our results shown in Chapter 3, data
containing lifetime domain information benefit significantly from binning, and this
strategy is commonly utilized as preprocessing in FLIM analysis [44]. Such
significant improvement in FLIM data is expected, as the precision in determining
lifetime decay value is more sensitive to noise and photon counts.
124
There are various ways from acquisition to preprocessing to enhance signal
strength in lifetime data and binning is one of these. Acquiring multiple scanning
frames to accumulate sufficient photon counts is another strategy to achieve a good
signal quality. Denoising, as a computational approach prior to analysis, is also an
alternative to enhance signal strength.

4.1.3 Data augmentation to expand data set size for classification
The lack of labeled data in machine learning is still the biggest challenge while
machine learning technologies evolve. In supervised learning, despite great deep
learning models, insufficient data introduces bias to training. Supervised learning
models are data-hungry, and their performance is heavily influenced by the size of
the training data available. A model without enough data is not suited for real-world
problems as overfitting and underfitting occur. Data augmentation is an alternative
to address this challenge. However, fluorescence data, different from general RGB
data, capture complex information such as spectra and lifetime. Generating pseudo-
image requires careful planning in biological sciences.
As shown in Figure 1.4, the spectral wavelength of a voxel of a fluorophore does not
follow the shape of a perfect spectrum, let alone a voxel that is composed of
multiple fluorescent species. It is not realistic to simulate spectra by adding Poisson
noise to reference spectra. Generating an image that is a cell or tissue of a complex
biological system with great biological variability is even more challenging.
125
One simple way to generate synthetic data is using geometrical transformations in
preprocessing, where images are reshaped, geometrically rotated, or resized. More
recently, data augmentation studies have begun to switch to deep learning and
develop generative models to create artificial data. In 2018, Luo et al. observed that
EEG signals could be generated by Conditional Wasserstein Generative Adversarial
Networks (GANs), which were then introduced to the training set in a learning
model. It is a promising approach to apply to fluorescence image data and
potentially improve classification performance.

4.1.4 Expand image analysis to volumetric or time-lapse with SpectraFLIM
In our work, we sequentially analyzed the 3-D volumetric data (x, y, z) slice-by-slice
in the z dimension, providing a consistent approach for processing volumetric data.
Current microscopy image data is getting larger and larger as scientists capture z-
stack volumetric images or time-lapse images. However, video processing and
volumetric image processing are computationally expensive and time-consuming.
Moreover, the experimental complexity of collecting this type of data, together with
the very broad dynamic range of autofluorescent intensities and the intrinsic
variability of the dimensions of the data acquired, results in different volumes
acquired. This sample variability affects the consistency of training for different
network architectures.
126
This topic alone draws lots of attention and researchers have come up with
enormous deep learning models for multimodal multi-dimensional biological image
data. A solution that addresses pain points in large-scale image data classification
will make a profound contribution to current microscopy data analysis. Starting from
what we have developed in the SpectraFLIM-LSTM model, an ad-hoc network could
further improve on harvesting information from multimodal volumetric data by
adding an extra dimension of convolution or by projecting the volume into a
contiguous flattened set of slices.

4.1.5 Outlier detection with large in-class variance and imbalanced
classification
Current technologies, for this work, were tailored to the biological systems and
disease model we described. However, real-world problems are always not in favor
of the algorithms developed with abundant labeled training examples. Datasets are
insufficient and could be biased, where many clinical conditions are sometimes
infrequent for classification, therefore it could be difficult for the model to achieve
high effectiveness in real-world scenarios. In this section, we could target an
extreme but commonly seen scenario in disease classification, which is to detect
anomalies with data points that have lower incidence.
127
Large in-class variance hampers the classification accuracy
In our work, we applied a modified deep-learning model called ConvLSTM to
classify complex biological data into healthy or injured samples. The injured class is
from mice with diphtheria toxin exposure for 48h. Mice in the injured class show
different levels of severity due to individual differences, health conditions, and
experimental errors.
Taking advantage of the growing data set can generalize the model and make the
prediction more accurate. In our experiments, though we trained the network only
with data from mice with 48h toxin exposure, we also acquired mice data with 24h
toxin exposure. Including image data from mice with 24h toxin exposure could
expand the data set size and potentially boost the training performance. However, if
we include these image data, the severity of the injury varies greatly, which causes
a high in-class variance across samples. Large in-class variance makes it difficult
for the model to converge in training, as a consequence, the model fails to predict
class labels. There is a balance between expanding the data set size and
maintaining a decent in-class variance.
Class imbalance hampers classification accuracy
Machine learning models suffer class imbalance problems when class distributions
are highly imbalanced in training data. Class imbalance hampers the model
accuracy at a great scale. The machine learning classifier tends to be more biased
towards the majority class. In this context, classification algorithms tend to have low
prediction accuracy for the infrequent class.
128
In the real world, many lung diseases are individually infrequent for pre-condition
classification with supervised learning. Class imbalance occurs not only in the
disease model and experimental data we acquired for this project, but often in a
great number and various types of clinical settings, for example, high throughput
screening in drug discovery, anomaly detection in early-stage diseases. Failing to
recognize these infrequent data points in disease detection causes a high true
negative and low true positive rate, which could cause misdiagnosis and lose
valuable time for patients to be promptly treated.
Outlier detection of unseen conditions
Although these unhealthy conditions are individually infrequent, they can be
collectively common and therefore are clinically significant in aggregate. These
infrequent “outliers” need a sensitive and accurate classifier to separate them from
healthy samples, which we call “inliers”.
Expanding this project from a binary classification to adapting this classifier to a
series of real-world applications in disease detection, we could develop a one-class
classification to recognize unhealthy samples, which is of great use in lots of clinical
settings. We believe this new project will be interesting and beneficial for scientists
struggling with data biases. One-class classification is to fit a minimum hypersphere
that encloses the training inlier data in a transformed feature space. This
hypersphere geometrically is a free-shape surface in a high-dimensional feature
space that wraps around all the inlier data. The distance between a test sample and
the centroid of the hypersphere is used as a score to predict its category.
129
4.2 Potential applications of these techniques
SEER and SpectraFLIM allow us to multiplex high-dimensional fluorescent data and
unravel the complexity of life sciences. Our aim is to extract useful information with
these tools and apply them to real-world problems in clinical and biological
sciences. This section is an overview of the future applications I envision for the
technologies I have developed and presented in this thesis.
4.2.1 SEER for fast and enhanced visualization
SEER has shown its advantages in computational speed and great color quality in
noisy fluorescent data. We believe that SEER has a potential for fast and enhanced
visualization in biological research.
Speed improvement
For SEER visualization, our tests show SEER can process a 3.7 GB dataset with
1.26 × 108 spectra in 6.6 s. The processing time for a 43.88 GB dataset with
1.47 × 109 spectra, including the denoising process is 87.3 s. SEER can go up to 67
times faster, compared to regression-based algorithms, such as ICA. This can be
applied to large-scale screening.
Enhanced visualization in low signal-to-noise regimes
We have also shown SEER provides an enhanced visualization in low signal-to-
noise settings. SEER is more robust to noisy environments and allows faster
130
scanning, consequently allowing larger areas for acquisition and/or higher
resolution.
Highlighting subtle spectral differences
Moreover, SEER is sensitive to spectral differences. We can use this to identify
areas of interest for the acquisition, as often the developmental biology
experimentalist needs to “compromise” resolution for speed. Scientific discovery is
often aided by the ability to distinguish new events within noisy or spectrally similar
areas. On many occasions, microscopists need to compromise on the total area of
the image because of the level of expression or the size of the area with respect to
the resolution desired. Subtle spectral differences might not be obvious in research
settings and also clinical settings. SEER can be used to localize injuries and large-
scale screening and could have the potential of finding early-stage diseases.
Below shows some examples of how this improved visualization can improve the
experimental pipeline for a scientist:
1. identifying the independent spectral components for linear unmixing or HySP
analysis: For fluorescent samples, spectra can have subtle variations on a
sample-to-sample basis which depend on multiple factors (pH, expression level,
sensitivity of the instrument, excitation wavelength). A “canned” spectrum
acquired from literature or a different instrument can yield sub-par results.
SEER visualization can allow for the identification of pixels containing good
spectra based on uniformity of color and anatomical position.
131
2. identifying areas of interest for the acquisition, as often the developmental
biology experimentalist needs to “compromise” resolution for speed. Scientific
discovery is often aided by the ability to distinguish new events within noisy or
spectrally similar areas. On many occasions, the microscopist needs to
compromise on the total area of the image because of the level of expression or
the size of the area with respect to the resolution desired. This more sensitive
visualization is more robust to noisy environments and allows faster scanning,
consequently allowing larger areas for acquisition and/or higher resolution.
3. understanding the patterns and location of labels, with respect to intrinsic, often
undesired, signals Autofluorescence is present in the majority of biological
samples. Being able to distinguish the location of a signal of interest from the
surrounding spectrally similar pixels can aid experiments where fluorescent
labels emit in the autofluorescence range or where multiple labels spectrally
overlap.
4. understanding the different intrinsic signals and their relationship with
metabolism as we show in figure 2.4 and Chapter 2, there is information in
intrinsic signals. This information relates to the metabolism of tissues. While
standard visualization approaches show a uniform color (see below) or require
different imaging strategies (FLIM), SEER can capture these subtle differences
using spectra. A better understanding of intrinsic signals and metabolisms of the
sample can improve the understanding of developmental processes.
132
5. there are color-based segmentation algorithms available through Fiji (Fiji is Just
ImageJ) (Plugins -> Segmentation -> Color Clustering) While it is beyond the
scope of this manuscript, there is an extensive library of color-based algorithms
that perform analysis of data. A large portion of these algorithms are aimed at
brightfield pathology slides images but with SEER, in novel projects, these
algorithms could be applied to perform completely new experiments or to
reprocess observations on already spectrally acquired data.
The application in Figure 2.4 (which is an uncleared mouse tissue), on
autofluorescence, is one clear example of an application of this visualization that
can lead to improved biological findings, otherwise impossible with “standard color
scales”. Visually no useful information can be extracted in this “standard color
scale”, despite the anatomical knowledge of the basal and apical layer.

4.2.2 Near real-time visualization for sequential data
SEER was originally developed for hyperspectral data that extends the
electromagnetic wavelength to a full dimension in acquisition. The fundamental
hypothesis is to separate spectral sequences by distinguishing them in Fourier
space. SEER can be applied to sequential analysis in various types of data.
Integrate SEER into hyperspectral imaging instruments
We could Implement SEER to achieve near real-time visualization in a lot of
settings. We could Integrate SEER into hyperspectral imaging instruments. These
133
instruments could be both cameras we use in daily life, they could also be clinical
instruments such as bronchoscopes and endoscopes.
Extend SEER for sequential analysis with various data types
SEER fundamentally is to transform sequential wavelengths and assign colors from
there. This transform is not limited to the spectral dimension. We can extend the
SEER approach to visualize sequential information from another domain. For
example, echocardiograms image the heart using sound waves. The echo
echocardiogram used to produce images is an acoustic sequence over time.
Instead of the spectral dimension, we can transform the time dimension and detect
abnormalities by highlighting these regions with distinct colors using SEER maps.
Electroencephalography (EEG) records electrical signals to measure brain activities
over time and Electrocardiogram (ECG) records the electrical signals from the
heart. SEER could also be applied to physiological data analysis and detect
abnormal pulses in electrical signals.

4.2.3 SpectraFLIM for multi-dimensional and multimodal classification
We apply ConvLSTM to a convolutional neural network model to harvest all 5
dimensions of information for the SpectraFLIM data input, and this classifier exacts
information from both lifetime domains and spectral domains. We utilize an LSTM
model with convolutional operations to characterize the sequential property of the
spectral information and spectrally resolved lifetime information in autofluorescent
134
samples. The architecture uses a recurrent operation to understand correlation
across channels in both intensity and lifetime domains. To the best of our
knowledge, this is the first use of the recurrent neural network model to explore
channel-to-channel variability both in the lifetime domain and in a combination of
spectra and FLIM.
We could expand this to lots of biological data and medical data settings. This
classification algorithm has great potential for tissue classification and subsequently
disease diagnosis. SpectraFLIM could simplify diagnostics by reducing the time and
cost of accessing biomedical information within tissues, compared to performing
biopsies or complex pathological procedures in the lab, it provides a shorter path to
label-free in vivo diagnostics.
In our work, we focused on designing a scalable pipeline that could work from single
cell to tissue. We currently applied our pipeline to a commonly diffused plague,
which is airway disease. However, SpectraFLIM has a much broader potential to be
translated to a variety of medical image data in clinics, such as pathology image
classification, drug validation and discovery.

135
References

1. Jahr, W., Schmid, B., Schmied, C., Fahrbach, F. O. & Huisken, J. Hyperspectral light
sheet microscopy. Nat. Commun. 6, (2015).

2. Levenson, R. M. & Mansfield, J. R. Multispectral imaging in biology and medicine: Slices
of life. Cytometry Part A 69, 748–758 (2006).

3. Garini, Y., Young, I. & McNamara, G. Spectral imaging: principles and applications.
Cytom. Part A 747, 735–747 (2006).

4. Dickinson, M. E., Simbuerger, E., Zimmermann, B., Waters, C. W. & Fraser, S. E.
Multiphoton excitation spectra in biological samples. J Biomed Opt 8, 329–338 (2003).

5. Sinclair, M. B., Haaland, D. M., Timlin, J. A. & Jones, H. D. T. Hyperspectral confocal
microscope. Appl. Opt. 45, 6283 (2006).

6. Valm, A. M. et al. Applying systems-level spectral imaging and analysis to reveal the
organelle interactome. Nature 546, 162–167 (2017).

7. Cranfill, P.J. et al. Quantitative assessment of fluorescent proteins. Nat. Methods 13,
557–562 (2016).

8. Hiraoka, Y., Shimi, T. & Haraguchi, T. Multispectral Imaging Fluorescence Microscopy for
Living Cells. Cell Struct. Funct. 27, 367–374 (2002).

9. Dickinson, M. E., Bearman, G., Tille, S., Lansford, R. & Fraser, S. E. Multi-spectral
imaging and linear unmixing add a whole new dimension to laser scanning fluorescence
microscopy. Biotechniques 31, 1272–1278 (2001).

10. Jacobson, N. P. & Gupta, M. R. Design goals and solutions for display of hyperspectral
images. in Proceedings - International Conference on Image Processing, ICIP 2, 622–
625 (2005).

11. Hotelling, H. Analysis of a complex of statistical variables into principal components. J.
Educ. Psychol. 24, 417–441 (1933).

12. Jolliffe, I. T. Principal Component Analysis. J. Am. Stat. Assoc. 98, 487 (2002).

13. Abdi, H. & Williams, L. J. Principal component analysis. Wiley Interdisciplinary Reviews:
Computational Statistics 2, 433–459 (2010).

14. Tyo, J. S., Konsolakis, A., Diersen, D. I. & Olsen, R. C. Principal-components-based
display strategy for spectral imagery. IEEE Trans. Geosci. Remote Sens. 41, 708–718
(2003).

15. Wilson, T. A. Perceptual-based image fusion for hyperspectral data. IEEE Trans. Geosci.
Remote Sens. 35, 1007–1017 (1997).

136
16. Long, Y., Li, H. C., Celik, T., Longbotham, N. & Emery, W. J. Pairwise-distance-analysis-
driven dimensionality reduction model with double mappings for hyperspectral image
visualization. Remote Sens. 7, 7785–7808 (2015).

17. Kotwal, K. & Chaudhuri, S. A Bayesian approach to visualization-oriented hyperspectral
image fusion. Inf. Fusion 14, 349–360 (2013).

18. Kotwal, K. & Chaudhuri, S. Visualization of Hyperspectral Images Using Bilateral
Filtering. IEEE Trans. Geosci. Remote Sens. 48, 2308–2316 (2010).

19. Zhao, W. & Du, S. Spectral-Spatial Feature Extraction for Hyperspectral Image
Classification: A Dimension Reduction and Deep Learning Approach. IEEE Trans.
Geosci. Remote Sens. 54, 4544–4554 (2016).

20. Zhang, Y., De Backer, S. & Scheunders, P. Noise-resistant wavelet-based Bayesian
fusion of multispectral and hyperspectral images. IEEE Trans. Geosci. Remote Sens. 47,
3834–3843 (2009).

21. A, R. SVD Based Image Processing Applications: State of The Art, Contributions and
Research Challenges. Int. J. Adv. Comput. Sci. Appl. 3, 26–34 (2012).

22. Redford, G. I. & Clegg, R. M. Polar plot representation for frequency-domain analysis of
fluorescence lifetimes. J. Fluoresc. 15, 805–815 (2005).

23. Digman, M. A., Caiolfa, V. R., Zamai, M. & Gratton, E. The phasor approach to
fluorescence lifetime imaging analysis. Biophys. J. 94, (2008).

24. Vergeldt, F. J. et al. Multi-component quantitative magnetic resonance imaging by phasor
representation. Sci. Rep. 7, (2017).

25. Lanzanò, L. et al. Encoding and decoding spatio-temporal information for super-
resolution microscopy. Nat. Commun. 6, (2015).

26. Fereidouni, F., Bader, A. N. & Gerritsen, H. C. Spectral phasor analysis allows rapid and
reliable unmixing of fluorescence microscopy spectral images. Opt. Express 20, 12729
(2012).

27. Cutrale, F., Salih, A. & Gratton, E. Spectral phasor approach for fingerprinting of photo-
activatable fluorescent proteins Dronpa, Kaede and KikGR. Methods Appl. Fluoresc. 1,
(2013).

28. Andrews, L. M., Jones, M. R., Digman, M. a & Gratton, E. Spectral phasor analysis of
Pyronin Y labeled RNA microenvironments in living cells. Biomed Opt Express 4, 171–
177 (2013).

29. Cutrale, F. et al. Hyperspectral phasor analysis enables multiplexed 5D in vivo imaging.
Nat. Methods 14, 149–152 (2017).

30. Radaelli, F. et al. μmAPPS: A novel phasor approach to second harmonic analysis for in
vitro-in vivo investigation of collagen microstructure. Sci. Rep. 7, (2017).

137
31. Scipioni, L., Gratton, E., Diaspro, A. & Lanzanò, L. Phasor Analysis of Local ICS Detects
Heterogeneity in Size and Number of Intracellular Vesicles. Biophys. J. (2016).
doi:10.1016/j.bpj.2016.06.029

32. Sarmento, M. J. et al. Exploiting the tunability of stimulated emission depletion
microscopy for super-resolution imaging of nuclear structures. Nat. Commun. (2018).
doi:10.1038/s41467-018-05963-2

33. Scipioni, L., Di Bona, M., Vicidomini, G., Diaspro, A. & Lanzanò, L. Local raster image
correlation spectroscopy generates high-resolution intracellular diffusion maps. Commun.
Biol. (2018). doi:10.1038/s42003-017-0010-6

34. Pan, Y. A. et al. Zebrabow: multispectral cell labeling for cell tracing and lineage analysis
in zebrafish. Development 140, 2835–2846 (2013).

35. Ranjit, S., Malacrida, L., Jameson, D. M. & Gratton, E. Fit-free analysis of fluorescence
lifetime imaging data using the phasor approach. Nat. Protoc. 13, 1979–2004 (2018).

36. Zipfel, W. R. et al. Live tissue intrinsic emission microscopy using multiphoton-excited
native fluorescence and second harmonic generation. Proc. Natl. Acad. Sci. 100, 7075–
7080 (2003).

37. Rock, J. R., Randell, S. H. & Hogan, B. L. M. Airway basal stem cells: a perspective on
their roles in epithelial homeostasis and remodeling. Dis. Model. Mech. 3, 545–556
(2010).

38. Rock, J. R. et al. Basal cells as stem cells of the mouse trachea and human airway
epithelium. Proc. Natl. Acad. Sci. (2009). doi:10.1073/pnas.0906850106

39. Bird, D. K. et al. Metabolic mapping of MCF10A human breast cells via multiphoton
fluorescence lifetime imaging of the coenzyme NADH. Cancer Res. 65, 8766–8773
(2005).

40. Lakowicz, J. R., Szmacinski, H., Nowaczyk, K. & Johnson, M. L. Fluorescence lifetime
imaging of free and protein-bound NADH. Proc. Natl. Acad. Sci. (1992).
doi:10.1073/pnas.89.4.1271

41. Skala, M. C. et al. In vivo multiphoton microscopy of NADH and FAD redox states,
fluorescence lifetimes, and cellular morphology in precancerous epithelia. Proc. Natl.
Acad. Sci. (2007). doi:10.1073/pnas.0708425104

42. Sharick, J. T. et al. Protein-bound NAD(P)H Lifetime is Sensitive to Multiple Fates of
Glucose Carbon. Sci. Rep. (2018). doi:10.1038/s41598-018-23691-x

43. Stringari, C. et al. Phasor approach to fluorescence lifetime microscopy distinguishes
different metabolic states of germ cells in a live tissue. Proc. Natl. Acad. Sci. 108, 13582–
13587 (2011).

44. Stringari, C. et al. Multicolor two-photon imaging of endogenous fluorophores in living
tissues by wavelength mixing. Sci. Rep. (2017). doi:10.1038/s41598-017-03359-8

138
45. Sun, Y. et al. Endoscopic fluorescence lifetime imaging for in vivo intraoperative
diagnosis of oral carcinoma. in Microscopy and Microanalysis (2013).
doi:10.1017/S1431927613001530

46. Ghukasyan, V. V. & Kao, F. J. Monitoring cellular metabolism with fluorescence lifetime of
reduced nicotinamide adenine dinucleotide. J. Phys. Chem. C (2009).
doi:10.1021/jp810931u

47. Walsh, A. J. et al. Quantitative optical imaging of primary tumor organoid metabolism
predicts drug response in breast cancer. Cancer Res. (2014). doi:10.1158/0008-
5472.CAN-14-0663

48. Conklin, M. W., Provenzano, P. P., Eliceiri, K. W., Sullivan, R. & Keely, P. J.
Fluorescence lifetime imaging of endogenous fluorophores in histopathology sections
reveals differences between normal and tumor epithelium in carcinoma in situ of the
breast. Cell Biochem. Biophys. (2009). doi:10.1007/s12013-009-9046-7

49. Browne, A. W. et al. Structural and functional characterization of human stem-cell-derived
retinal organoids by live imaging. Investig. Ophthalmol. Vis. Sci. (2017).
doi:10.1167/iovs.16-20796

50. Livet, J. et al. Transgenic strategies for combinatorial expression of fluorescent proteins
in the nervous system. Nature 450, 56–62 (2007).

51. Weissman, T. A. & Pan, Y. A. Brainbow: New resources and emerging biological
applications for multicolor genetic labeling and analysis. Genetics 199, 293–306 (2015).

52. Pan, Y. A., Livet, J., Sanes, J. R., Lichtman, J. W. & Schier, A. F. Multicolor brainbow
imaging in Zebrafish. Cold Spring Harb. Protoc. 6, (2011).

53. Raj, B. et al. Simultaneous single-cell profiling of lineages and cell types in the vertebrate
brain. Nat. Biotechnol. 36, 442–450 (2018).

54. Mahou, P. et al. Multicolor two-photon tissue imaging by wavelength mixing. Nat.
Methods 9, 815–818 (2012).

55. Loulier, K. et al. Multiplex Cell and Lineage Tracking with Combinatorial Labels. Neuron
81, 505–520 (2014).

56. North, T. E. & Goessling, W. Haematopoietic stem cells show their true colours. Nature
Cell Biology 19, 10–12 (2017).

57. Chen, C. H. et al. Multicolor Cell Barcoding Technology for Long-Term Surveillance of
Epithelial Regeneration in Zebrafish. Dev. Cell 36, 668–680 (2016).

58. Panetta, K., Gao, C. & Agaian, S. No reference color image contrast and quality
measures. IEEE Trans. Consum. Electron. (2013). doi:10.1109/TCE.2013.6626251

59. Hall, M. et al. The WEKA Data Mining Software: An Update; SIGKDD Explor. (2009).
doi:10.1080/14733285.2016.1271943

139
60. Sommer, C., Straehle, C., Kothe, U. & Hamprecht, F. A. Ilastik: Interactive learning and
segmentation toolkit. in Proceedings - International Symposium on Biomedical Imaging
(2011). doi:10.1109/ISBI.2011.5872394

61. Vert, J., Tsuda, K. & Schölkopf, B. A primer on kernel methods. Kernel Methods Comput.
Biol. 35–70 (2004). doi:10.1017/CBO9781107415324.004

62. Bruton, D. {RGB} Values for visible wavelengths. (1996). Available at:
http://www.physics.sfasu.edu/astro/color/spectra.html.

63. Hasler, D. & Suesstrunk, S. E. Measuring colorfulness in natural images. in Human
Vision and Electronic Imaging VIII (2003). doi:10.1117/12.477378

64. Sos S. Agaian and Karen P. Lentz and Artyom M. Grigoryan. A New Measure of Image
Enhancement. IASTED Int. Conf. Signal Process. \& Commun. (2000).

65. Agaian, S. S., Silver, B. & Panetta, K. A. Transform coefficient histogram-based image
enhancement algorithms using contrast entropy. IEEE Trans. Image Process. (2007).
doi:10.1109/TIP.2006.888338

66. Westerfield, M. The Zebrafish Book. A Guide for the Laboratory Use of Zebrafish (Danio
rerio), 4th Edition. book (2000).

67. Trinh, L. A. et al. A versatile gene trap to visualize and interrogate the function of the
vertebrate proteome. Genes Dev. 25, 2306–2320 (2011).

68. Jin, S.-W., Beis, D., Mitchell, T., Chen, J.-N. & Stainier, D. Y. R. Cellular and molecular
analyses of vascular tube and lumen formation in zebrafish. Development 132, 5199–209
(2005).

69. Megason, S. G. In toto imaging of embryogenesis with confocal time-lapse microscopy.
Methods Mol. Biol. 546, 317–332 (2009).

70. Huss, D. et al. A transgenic quail model that enables dynamic imaging of amniote
embryogenesis. Development 142, 2850–2859 (2015).

71. Holst, J., Vignali, K. M., Burton, A. R. & Vignali, D. A. A. Rapid analysis of T-cell selection
in vivo using T cell-receptor retrogenic mice. Nat. Methods 3, 191–197 (2006).

72. Kwan, K. M. et al. The Tol2kit: A multisite gateway-based construction Kit for Tol2
transposon transgenesis constructs. Dev. Dyn. 236, 3088–3099 (2007).

73. Kawakami, K. et al. A transposon-mediated gene trap approach identifies
developmentally regulated genes in zebrafish. Dev. Cell 7, 133–144 (2004).

74. Urasaki, A., Morvan, G. & Kawakami, K. Functional dissection of the Tol2 transposable
element identified the minimal cis-sequence and a highly repetitive sequence in the
subterminal region essential for transposition. Genetics 174, 639–649 (2006).

75. White, R. M. et al. Transparent Adult Zebrafish as a Tool for In Vivo Transplantation
Analysis. Cell Stem Cell 2, 183–189 (2008).
140
76. Arnesano, C., Santoro, Y. & Gratton, E. Digital parallel frequency-domain spectroscopy
for tissue imaging. J. Biomed. Opt. 17, 0960141 (2012).

Abstract (if available)

Abstract Imaging the metabolism of intact cells and tissues would offer an important means to assay health and injury. Fluorescence multi-spectral (hyperspectral) imaging (fHSI) has emerged as a great tool to capture spatio-temporal dynamics across scales for molecules, cells, and tissues with multiple fluorescent labels. The resulting data are dense in information and high in dimensionality, and often require lengthy analyses to interpret the complex biological information for our eyes to understand. We developed Spectrally Encoded Enhanced Representations (SEER) for improved and computationally efficient color visualization of hyperspectral images with overlapping spectra. This visualization technique enhances subtle spectral differences in multi-labeled fluorescent and autofluorescent images, providing a fast, intuitive, and mathematical way to interpret hyperspectral images during collection, preprocessing, and analysis.
We furthermore combined fluorescence multi-spectral imaging with fluorescent lifetime imaging (SpectraFLIM) to assay the metabolism as well as the spatial and temporal dynamics of living samples. We demonstrated that SpectraFLIM offers improved insights into complex biology at cellular resolution. Our SpectraFLIM workflow can image metabolism and distinguish injured tissue from healthy tissue. To better analyze SpectraFLIM data, we employed deep learning methods using a 2-dimensional Convolutional Neural Network (2-D CNN) with inputs from spectral and/or lifetime domain(s). For comparison, we developed a convolutional long short-term memory (ConvLSTM) network to classify complex SpectraFLIM data and find that it extracts spatial features while preserving spectral correlations across all channels. We find that either deep learning approach improves classification accuracy, which should make imaging-based diagnostic tools more robust and easier to apply in diverse settings. Our methods in this thesis combine analytical tools with novel microscopy to understand complex biological processes in living organisms.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Hyperspectral phasor for multiplexed fluorescence microscopy and autofluorescence-based pathologic diagnosis

PDF

Multiplexing live 5d imaging with multispectral fluorescence: Advanced unmixing through simulation and machine learning

PDF

Machine learning and image processing of fluorescence lifetime imaging microscopy enables tracking and analysis of subcellular metabolism

PDF

Analytical tools for complex multidimensional biological imaging data

PDF

Democratizing optical biopsy with diffuse optical imaging

PDF

Hyperspectral and multispectral optical bioluminescence and fluorescence tomography in small animal imaging

PDF

Metabolic profiling of single hematopoietic stem cells for developing novel ex vivo culture strategies

PDF

Development of a quantitative proteomic molecular imaging platform using multiplexed nanoscale Raman scattering contrast agents

PDF

Time-resolved laser induced fluorescence spectroscopy for detection of brain tumors

PDF

Computational methods for fluorescence molecular tomography

PDF

Learning from limited and imperfect data for brain image analysis and other biomedical applications

PDF

New theory and methods for accelerated MRI reconstruction

PDF

Neural network integration of multiscale and multimodal cell imaging using semantic parts

PDF

The role of the environment around the chromophore in controlling the photophysics of fluorescent proteins

PDF

Learning logical abstractions from sequential data

PDF

Neighborhood and graph constructions using non-negative kernel regression (NNK)

PDF

Practice-inspired trust models and mechanisms for differential privacy

PDF

Improving arterial spin labeling in clinical application with deep learning

PDF

Brain connectivity in epilepsy

PDF

Visual representation learning with structural prior

Asset Metadata

Creator Shi, Wen (author)

Core Title Extending multiplexing capabilities with lifetime and hyperspectral fluorescence imaging

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Biomedical Engineering

Degree Conferral Date 2023-05

Publication Date 03/06/2023

Defense Date 06/01/2022

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag fluorescence microscope,hyperspectral imaging,image processing,lifetime imaging,machine learning,OAI-PMH Harvest

Format theses (aat)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Fraser, Scott (committee chair), Cutrale, Francesco (committee member), Leahy, Richard (committee member), Zavaleta, Cristina (committee member)

Creator Email jadesnotfake@gmail.com,shiwen0728@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-oUC112764253

Unique identifier UC112764253

Identifier etd-ShiWen-11491.pdf (filename)

Legacy Identifier etd-ShiWen-11491

Document Type Dissertation

Format theses (aat)

Rights Shi, Wen

Internet Media Type application/pdf

Type texts

Source 20230313-usctheses-batch-1009 (batch), University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the author, as the original true and official version of the work, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright. The original signature page accompanying the original submission of the work to the USC Libraries is retained by the USC Libraries and a copy of it may be obtained by authorized requesters contacting the repository e-mail address given.

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA

Repository Email cisadmin@lib.usc.edu