Trusting a map in her own head
As told by Doris Tsao

My parents grew up in China. My father is a mathematician who studied Marx, AI, and functional analysis at night after chopping trees all day in Manchuria during the Cultural Revolution. My mother was a computer programmer. Both my parents were raised as outcasts, my father by his grandmother and my mother by her aunt. They decided to marry after three weeks of courtship, during which my mother caught fish from a river and cooked them for my father, and my father taught my mother calculus.

I was born in Changzhou and emigrated to the U.S. with my parents in 1980. I grew up in College Park, Maryland, where both my parents were graduate students. My parents were too poor to afford childcare while they were both busy studying. One of my fondest memories is of summers in which every morning, they would drop me off at the library, and I would spend the whole day by myself reading. My favorite genre was biographies. There was nothing I loved more than to disappear into the life of Clara Barton or Kit Carson or Mozart. These summers were the happiest times. At school, I was extremely shy and teased a lot. I was sometimes mistaken for a boy because my hair was so short.

Many scientists recall having a curiosity about the world starting from a very young age. I cannot boast of any such natural proclivity for science. I don’t think I was curious about anything until one morning in 6th grade. I remember waking up and suddenly wondering whether space is infinite or not. If space were infinite, this seemed incredible and fundamentally different from anything I had ever experienced or thought about in life up to that point. On the other hand, I could not imagine space not being infinite: even if the universe were one giant room surrounded by an endless wall, the wall would still occupy space. I remember experiencing a peculiar mixture of terror and excitement thinking about this. This was the very first time a question about the world actually bothered me. And it turns out, this remains the question that governs me deep inside.

When I was in high school, my father went on a trip to California and took a tour of Caltech with a friend. He came back and told me about a monastery for science, located under the mountains amidst flowers and orange trees, where all the students looked very skinny and super smart, like little monkeys.

I was intrigued. I went to a presentation about Caltech by a visiting admissions officer, who showed slides of students taking tests under olive trees, swimming in the Pacific, or huddled in a dorm room working on a problem set. I decided: this is where I want to go to college! I dreamed every day about being accepted to Caltech. After I got my acceptance letter, I began to worry that I would fall behind in the first year, since I had heard about how hard the course load is. So, I went to the library and started reading the Feynman Lectures. This was another world, where one could see beneath the surface of things and ask why, why, why, why? And the results of one’s mental deliberations could actually be tested by experiments and reveal completely unexpected yet real phenomena, like magnetism as a consequence of the invariance of the speed of light. I was enthralled. At the same time, I tried to read Kant’s Critique of Pure Reason, and even though I didn’t understand everything, I found the idea that our spatial perception is due to our mental representation, a revelation.

Caltech
I started as a freshman at Caltech in the fall of 1993. The two blocks between S. Wilson Ave. and S. Hill Ave became my paradise, where I walked in euphoric circles after the last exam of each semester, and where I made friends for the first time in my life.

As undergrads, we were each given a “South Master” key that we could use to open almost any door on campus. On rainy days we would trek to our classes through underground steam tunnels. The intellectual life for undergrads at Caltech similarly grew from principles of freedom, empowerment, and highest faith. A running joke held that the postdocs were smarter than the faculty, the grad students smarter than the postdocs, and the undergrads smarter than the grad students. This was certainly how we undergrads felt. After conquering the rigors of the Caltech core curriculum together, we believed we could tackle any problem under the Sun. I am grateful to Caltech for giving me this sense of confidence (this is why years later, as a faculty member, I strongly opposed the proposal to remove quantum mechanics from the core curriculum, a battle that I unfortunately lost).

"There was no doubt in my mind that consciousness was the most profound and exciting scientific mystery of all, and I wanted to spend my life studying it."

I majored in math and biology, though neither was where my heart lay. I found math beautiful (I spent one summer studying set theory and learning the proof of the Banach Tarski Paradox), but I knew it was not what I wanted to spend my life doing. I knew with even more certainty that biology was not for me: I was generally bored by my biology classes and got mediocre grades in them.

I count two pivotal intellectual events during my time in college. One was reading “The Origin of Consciousness in the Breakdown of the Bicameral Mind” by Julian Jaynes. I was in ecstasy reading this book, whose account of the birth of consciousness remains as close to a Theory of Everything as I have ever come across. There was no doubt in my mind that consciousness was the most profound and exciting scientific mystery of all, and I wanted to spend my life studying it.

The second life-changing event was a visit from my father and little brother. The three of us went on a camping trip to Catalina Island. My father brought along a paper he had written, which he asked me to proofread for English mistakes. I had been doing this for him since high school; it usually involved correcting subject-verb agreement issues. But now, I could actually understand his paper, and I found the central idea astonishingly beautiful. The paper tackled the famous “correspondence” problem: how can one compute the 3D structure of the environment if one has 2D images of it from different observation points?

My father’s solution was to posit a set of “dynamic receptive fields” that span a space dual to the image space: a mirror reflection of the world. In this dual space, the receptive fields can warp to compensate for the warping of the image that occurs due to change of observation point; and they can do this in a precise way by following the gradient of an energy function, to thereby encode 3D structure.

The theory was inspired by Bruno Olshausen’s idea of “shifter circuits” (which Olshausen had developed as a graduate student at Caltech). I found the idea so beautiful because it showed how the geometry of the world could be precisely reflected in a mirror space in the brain, with every change in the world inducing an opposite change in the neural space to maintain equilibrium. Suddenly, all my deepest yearnings—to understand the infinity of space, to understand consciousness, to make my father proud of me—crystallized into a concrete goal: to discover the dynamic receptive fields in visual cortex responsible for driving the brain’s geometric engine.

"All of us desire to find a mentor who sees us as we wish to be, rather than as we are".

Harvard
In the fall of 1996, I started graduate school at Harvard, in the Department of Neurobiology. I joined the lab of Margaret “Marge” Livingstone, who at the time was mapping receptive fields in primary visual cortex of awake monkeys. The visual system of the macaque monkey is remarkably similar to that of the human visual system, thus providing the best experimentally tractable model to study human vision.

I first met Marge during grad school interviews. She was wearing a shiny silver dress (overturning all my pre-conceptions about the proper attire of a successful female scientist). In the middle of recording from a primary visual cortex (V1) cell, she told me to wait while she isolated her cell. Watching her it was obvious that she was completely in love with what she was doing. During a lull in the monkey's fixation, she asked me what I was interested in. Painfully shy, I muttered something about wanting to understand neural circuits in detail. Marge replied that if she wanted to understand neural circuits, she would go to Larry Katz's lab, and she then described Katz's glutamate uncaging experiments to me. It's been said that everything that will go wrong in a relationship is actually evident on the first date. During this first interaction with Marge, I already sensed her kindness and non-judgmental attitude, her crystal-clear way of thinking, and her readiness to get to the point.

All of us desire to find a mentor who sees us as we wish to be, rather than as we are. I felt that Marge did this naturally. This is what I loved most about her as a mentor. After I joined her lab, during my first three years, I was essentially completely unproductive. I wanted to prove that stereopsis (the perception of depth from fine differences between the images in the two eyes) can induce receptive field dynamics in primary visual cortex, inspired by my father's models. Although I knew nothing about how the brain actually worked, Marge allowed me to spend these three years chasing down my fantasies with complete freedom—which astonishes me now in hindsight.

She never made me feel stupid or ignorant just because I was. Instead, she created an environment in her lab where we each felt we were tackling the most important problem in the universe, and Marge had utter faith in our abilities to discover something new that she hadn't thought of. Essentially, she encouraged us to dream, and she treated us as children rather than as employees.

My siblingin the Livingstone Lab was Bevil Conway, who was deeply interested in color perception. Bevil and I shared many conversations and adventures including an exuberant attempt to perform intracellular recording, after reading a paper by Otto Creutzfeld, that was abandoned after half an hour of broken pipettes.

"This was a quest worth devoting a life to."

The two of us had the immense fortune to grow up as scientists under the tutelage not only of Marge, but also of David Hubel, Marge’s close collaborator. As a first-year graduate student, reading the Ferrier Lecture by Hubel and Wiesel describing the beautiful, precise sequence of transformations occurring in the early stages of vision, I knew I had made the right life choice to study monkey visual cortex. How can electrical activity in a thin sheet of cells create the percept of objects in space? This was a quest worth devoting a life to.

If there were a Mount Rushmore of Neuroscience, David Hubel and Torsten Wiesel’s visages would certainly be greeting the birds each morning. Yet, David remains in my memory the person who made us Scarlatti CDs, gave us nicknames (Bevil was “chucklehead” and I was “Dodo”), rode the subway all the way from Newton to Longwood on a Saturday to a look up a number in a monkey atlas for me, and expressed such love for and faith in us that I can only hope to be worthy one day.

Face patches and face cells
One year at the Society for Neuroscience Conference, around my third year in the lab, Marge told me she had just bumped into a colleague named Roger Tootell, who was looking for someone to help start a monkey functional magnetic resonance imaging (fMRI) project in his lab at Massachusetts General Hospital (MGH). At the time, I was recording from neurons in V1 to study their role in depth perception, and not finding anything interesting. For example, when I presented a 3D surface vividly jutting out in depth to the monkey, V1 cells remained silent if the surface lacked texture. It seemed clear that the key computations related to 3D vision occurred beyond V1. Thus, the opportunity to do monkey fMRI excited me—the approach could potentially provide a large-scale picture of where key computations related to 3D vision occurred in the brain and take me out of my rut.

"It seemed utterly strange that the brain would contain an area responding only to faces".

I embarked on an adventure to learn monkey fMRI. Roger introduced me to his collaborators Guy Orban and Wim Vanduffel, and Guy and Wim invited me to visit Leuven, where Wim had successfully set up monkey fMRI. Wim performed the surgeries on the first two fMRI animals at MGH. Soon, I was scanning monkeys every day, putting red-green glasses on them and showing them all kinds of 3D stimuli. I discovered strong activation to depth-rich stimuli in a small set of areas in the posterior parietal lobe, providing a promising target for single-cell recordings to study 3D vision beyond V1. But little did I know, I was about to go on a 20-year detour.

At Massachusetts General Hospital, I heard a lot of buzz about a scientist at MIT named Nancy Kanwisher. In 1997, she had published a paper reporting discovery of an area in the human temporal lobe that appeared to be specialized for processing faces. I remember my initial reaction after reading this paper: bewilderment. It seemed utterly strange that the brain would contain an area responding only to faces. From introspection, faces didn’t seem so different from other visual forms.

I decided to scan my monkeys to see if they also had face areas. If they did, there was potentially a huge payoff—we could record in the face areas and study in detail the electrical activity of single cells within them. While I wasn’t particularly interested in face perception, it seemed fun to venture into a new area of vision (object recognition) and a new part of the brain (the temporal lobe). And the cost of doing the experiment was minimal. Unlike single-unit electrophysiology, where one had to painstakingly prepare electrodes, clear away granulation tissue, and maintain a sterile chamber in order to record from a few neurons a day, performing fMRI scans on monkeys was like watching TV: one simply placed the animal inside the scanner, clicked “Scan,” and within a few hours, a map of activations across the brain rolled out.

When I showed monkeys pictures of faces and other objects, I found a circumscribed region in the macaque temporal lobe that responded much more strongly to faces—just as Nancy Kanwisher had found in humans. Indeed, improved scanning techniques subsequently revealed six discrete patches of face-selective cortex in inferotemporal cortex in each hemisphere. This finding was quite surprising. In the early 1970s, Charlie Gross had reported the discovery of face-selective cells in monkeys, but these cells appeared to be randomly scattered throughout inferotemporal cortex rather than concentrated within any specific area. I was invited to give a job talk at Caltech about these fMRI findings. But I was not offered the position: people were skeptical that a few blurry blobs of fMRI activity measuring blood flow could reveal anything profound about the brain.

Around this time, a postdoc in Nancy Kanwisher’s lab named Winrich Freiwald emailed me, asking if he could come watch one of my monkey fMRI experiments. Soon, he joined the project, working with me on the next big step: to target an electrode to one of the face patches.

"We realized there was a pattern: all the cells were face selective!"

I will never forget our first recording. We inserted an electrode into inferotemporal cortex and showed the monkey pictures of faces and objects. The first neuron we recorded turned out to be selective for faces. We advanced slightly and recorded another neuron. It was also selective for faces. Around the fourth or fifth face-selective neuron, we realized there was a pattern: all the cells were face selective! Moreover, each cell responded to many different faces and didn’t seem particularly selectivity for identity, violating my naïve expectations of what cells in a face area should do (much later, the rationale would become clear when we discovered the code for faces used by these cells). Around 1 am, we were ready to pull out the electrode. Out of curiosity, I looked up what brain area was located above the face patch in the atlas. Seeing it was auditory cortex, I shook my keys as the tip passed through. The cells responded in precise rhythm, “Cha Cha Cha.” Thus ended our first day recording from a face patch. I remember a feeling of such elation afterwards--a newfound sense that the brain might be understandable. I called my father after I got home, around 3 am, to tell him that I thought I got the Caltech job.

Scouting a new land
Day after day, we stuck our electrode in the same three grid holes and encountered face cell after face cell. It was clear that we had found a treasure trove. The ability to access such an extraordinary concentration of face cells made the problem of understanding how the brain represents a complex object suddenly tractable: How does a cell determine that an image contains a face? What features are individual cells selective for? How can cells work together to generate something as ineffable as a facial identity? I was sure the answer was lurking near the tip of our electrode. The cells were absolutely machine-like in their responses. It didn’t matter if the monkey had already seen our set of 96 screening stimuli hundreds of times. Each time a face came, the cell would blast away, and even the precise pattern of selectivity to different individuals was reproducible across repeated presentations.

Just three weeks after our first recording in a face patch, Winrich had to go back to Germany to take on a new independent position. Thereafter, I continued the recordings on my own in Marge’s lab. I showed everything I could think of: monkey faces, human faces, cartoon faces, faces with different head orientations, upside down faces, scrambled faces, blurry faces, occluded faces, face parts, objects that look like faces (I felt slightly silly, a Photoshop-wielding Imposter in the Land of Face Cells).

"This first paper was rejected by Nature but accepted by Science."

Three conclusions were clear after just a few months of recording: 1) the face patch really did consist almost entirely of face cells; 2) the face cells showed ramp-shaped tuning to specific face features, with many cells preferring large irises; 3) the cells showed delayed responses to degraded faces (upside down, occluded, blurry). I wanted to put all these findings into one big first paper, but Marge wisely suggested focusing the first paper on the central finding of an extraordinary concentration of face cells within a face patch. This first paper was rejected by Nature (one reviewer asked, “Why is it particularly interesting to find a cluster of face cells?”) but accepted by Science.

Around this time, I had to decide what to do next with my life. The traditional route was of course to do a postdoc in a new lab. But what I really wanted to do was continue to explore the face patches. I applied for, and was fortunate to receive, a Sofia Kovalevskaya Fellowship from the Alexander von Humboldt Foundation, providing me with funds to set up my own little lab in Bremen, Germany.

Bremen
I moved to Bremen, living in an apartment on H.H. Meier Allee with two roommates I found online. At the University of Bremen, I had a tiny lab constructed from a portable classroom. My lab was connected to Andreas Kreiter’s Institute for Brain Research and located down the hall from Winrich’s lab. Whenever anyone entered the room, the monkey’s eye position signal would shake slightly. I had a sign on the door, “Bitte sprechen Sie Deutsch,” that everyone ignored.

I was joined by Nicole Schweers and Sebastian Moeller. Nicole sold eggs at the Bremen Market before joining the lab as technician, yet she soon learned to prepare for surgeries, scan and train monkeys, and troubleshoot all manner of problems better than anyone. Sebastian came to neuroscience from a background in architecture and was an extraordinarily skilled and methodical experimentalist. We were later joined by Piercesare Grimaldi, a daydreamer with a background in neural development.

Together, we tackled two obvious fundamental questions: 1) What is the anatomical organization of the face patches? What are their inputs and outputs? (Eventually, we would routinely refer to the six patches as “the face patch system,” but at this point in our journey, we didn’t even know whether they were connected to each other.) 2) Do the six different face patches differ from each other functionally, and if so, how? The answers to both questions turned out to be strikingly clear, leading to two papers in Science that I consider foundational for our understanding of inferotemporal cortex: they not only clarified the hierarchical organization of the face patch system, but opened a new path for understanding the structure and function of all inferotemporal cortex (a path we would later exploit).

During my time in Bremen, I started taking violin lessons again. While I had never particularly liked the violin as a child, now I fell fully in love. Something about this period of my life and living in Northern Germany made me particularly susceptible to Brahms. Outside the lab, I lived in the dark harmonies-reaching-towards-light of his music.

Around my second year in Bremen, I returned to Caltech to interview again. This time, I could prove the value of finding a few blurry blobs of face-selective fMRI activity: this provided the address to the brain’s carpenter’s shop for faces, so we could visit the shop repeatedly to scrutinize every step and reverse engineer the entire assembly line. I was offered the job.

Return to Pasadena
In January 2009, I started as an Assistant Professor at Caltech. Laying on the carpet of my new faculty apartment, warmed by rays of Southern California sunshine, I felt giddy with happiness.

Nicole, Sebastian, and Piercesare moved with me from Bremen to Caltech. We were soon joined by Simon Kornblith, a brilliant Caltech undergraduate, and Shay Ohayon, an amazing graduate student who completely transformed our experimental infrastructure, replacing my jerry-rigged programs with much more efficient and streamlined software. Winrich also joined my lab, while he waited for his new lab at Rockefeller to be renovated.

At Caltech, we continued the quest to understand the face patch system. Shay tackled the problem of how face cells initially detect an object as a face. Piercesare injected tracers into different face patches to map their detailed anatomical connectivity. Sebastian explored how electrical microstimulation of face patches affects the monkey’s perception. It was a time of great productivity.

"In 2013, I was invited by Larry Swanson to give a Presidential Lecture at the Society for Neuroscience Conference."

In 2013, I was invited by Larry Swanson to give a Presidential Lecture at the Society for Neuroscience Conference. This was an incredible honor, to be able to share with the entire neuroscience community our discoveries about the face patch system. My brother Albert (also a neuroscientist, and much smarter and more experimentally talented than I am) must have listened to my talk a hundred times as he helped me to prepare. I sneaked in a few slides at the end of my talk about my dad’s models. In my mind, detours notwithstanding, understanding 3D vision remained my central quest.

Partly due to publicity from this lecture, afterwards our lab grew significantly. We were joined by postdocs Le “Steven” Chang, Pinglei Bao, Liang She, Lu Liu, Joe Wekselblatt, Francisco Luongo, and graduate student Janis Hesse. Francisco, Lu, and Joe delved into a new project to understand visual segmentation in mice and tree shrews.

Meanwhile, Steven, Pinglei, and Liang each taught me new ways of thinking about face patches, inferotemporal cortex, and the neural code for objects. Steven tackled the fundamental question of how face cells encode facial identity, discovering a simple linear code that he could use to reconstruct in detail the face shown to a monkey from activity of just ~200 neurons. Pinglei explored inferotemporal cortex outside face patches, leveraging approaches we had developed for studying face patches, and uncovered a new general principle: mapping object space. Liang examined how face memories are coded across the face patch system and discovered that at long latency, the cells use a unique code for representing familiar faces.

In 2015, I received the remarkable gift of being named an HHMI Investigator. This funding has allowed me to embark on the projects I believe are most exciting and important, regardless of whether they are at a stage ready to be defended in front of an NIH grant panel.

A voyage just begun
A number of years ago (well before the current ChatGPT-driven craze), I encountered the idea of “analysis-by-synthesis,” also known as “predictive coding.” The core idea was remarkably resonant with my father’s concept of a dynamic receptive field that had startled me with its beauty back in college: a neural representation mechanism based on the principle of dynamic homeostasis. In analysis-by-synthesis, a hierarchical perceptual system iteratively attempts to reconstruct sensory input, and successful representation is indicated by convergence.

I realized that the perfect playground to explore this profound idea actually lay right in front of us: the face patch system. How do different face patches interact during challenging perceptual situations, when the incoming stimulus is noisy/ambiguous and the visual system needs time to compute what is present? Does the system harbor machinery to regenerate sensory input, as predicted by the analysis-by-synthesis framework? Four graduate students in the lab, Janis Hesse (now a postdoc), Frank Lanfranchi, Yuelin Shi, and Varun Wadia set out, each in different and highly creative ways, to investigate this possibility.

"I believe that the importance of the face patch system extends far beyond the realm of object recognition."

Meanwhile, returning to the source of my bewilderment as a graduate student after reading Nancy Kanwisher’s 1997 paper: how does the brain so seamlessly integrate the representation of faces within the face patch system into a larger representation of the environment? Which areas and data structures does the brain use to represent the larger movie that makes up our visual experience of the world? Three postdocs, Krithika Mohan, Shi Chen, and Nate Dolensek, together with graduate student Jialiang Lu, are currently digging deep into this question.

Reflecting more broadly on why it is particularly interesting to find a cluster of face cells: I believe that the importance of the face patch system extends far beyond the realm of object recognition. The miracle of the brain is how it enables us to survive and thrive in a real 3D world of objects and other agents. By crystallizing neural representation of one of the most meaningful objects that we interact with into a small set of cortical regions, the face patch system makes it tractable to understand how embodied behavior is orchestrated.

Finally, returning to my core: Last year, I was happy to publish a paper with my father in PNAS, “A topological solution to object segmentation and tracking.” This paper, which we worked on over many years, presents an overarching theory of 3D vison. It synthesizes old ideas that my dad first developed when I was in high school together with new ideas about globally-invariant surface representation as well as computational experiments I carried out during the pandemic. I am deeply proud of this paper and believe it will become known in time.

Together with two first-year graduate students Dasheng Bi and Ruichang Sun, I am now back where I wanted to be 20 years ago: in posterior parietal cortex, in the part of the brain that generates our perception of 3D space.

With my husband Greg
In 2021, I moved together with my husband and two young boys from Pasadena to UC Berkeley, where my husband is an assistant professor of engineering. His dream is to develop intelligent self-assembling systems for applications in engineering and medicine (mirroring my dream to understand the most complex self-assembled system in existence). Greg is my best friend, who understands me, loves me, and expects from me to a degree that can only be called soulmate. We live in a house atop Panoramic Hill, overlooking the beautiful Berkeley campus, together with Tomi and Levi (who are just finishing first grade) as well as my father (who just turned 80).

I have been lucky in life to always be able to follow my deepest dreams. This was made possible by mentors (Marge, David, my father) whose guiding principle was love and faith. I can only hope to pass this on to the remarkable young scientists whom I have now have the privilege to mentor, because without a doubt the best is yet to come. As Rilke said, in a line that is a personal motto for me, “Welchen Sinn hätten wir, wenn der, nach dem wir verlangen, schon gewesen wäre?” [What meaning would we have if He whom we are longing for has already existed?]