Mice appear to have a specialized system for detecting and at least initially processing instinctually important smells such as those that denote predators. The finding raises a question about whether their response to those smells is hardwired.

A new study finds that mice have a distinct neural subsystem that links the nose to the brain and is associated with instinctually important smells such as those emitted by predators. That insight, published online this week in Proceedings of the National Academy of Sciences, prompts the question whether mice and other mammals have specially hardwired neural circuitry to trigger instinctive behavior in response to certain smells.

In the series of experiments and observations described in the paper, the authors found that nerve cells in the nose that express members of the gene family of trace amine-associated receptors (TAAR) have several key biological differences from the much more common and diverse neurons that express members of the olfactory receptor gene family. Those other nerve cells detect a much broader range of smells, said corresponding author Gilad Barnea, the Robert and Nancy Carney Assistant Professor of Neuroscience at Brown University.

The differences between TAAR neurons and olfactory receptor neurons led Barnea and his co-authors to conclude that they form an independent subsystem for certain smells.

"Our observations suggest that the TAAR-expressing sensory neurons constitute a distinct olfactory subsystem that extracts specific environmental cues that then elicit innate responses," Barnea said.

Different circuits and genes

The newly found differences include the way TAAR neurons are wired to structures called glomeruli in the olfactory bulb, which is the brain area that initially processes smells. The glomeruli relay smell signals deeper into the brain where the perception of smell occurs and behavioral responses are initiated.

Graduate students Mark A. Johnson and Lulu Tsai at the Barnea laboratory visualized TAAR neurons in the nose and their projections to the olfactory bulb by generating specific antibodies that could detect TAAR proteins in these neurons. They found that while neurons expressing a given olfactory receptor connect to only two glomeruli, neurons that express a given TAAR connect to four to six glomeruli. They also found that while olfactory receptor neurons connect to glomeruli throughout the olfactory bulb, TAAR glomeruli tend to be clustered within an area at the top of the bulb. That area was implicated in instinctual behaviors by previous research.

In another experiment they found that the TAAR axons leading to glomeruli at the top of the bulb express a cell adhesion molecule that is associated with olfactory receptor axons leading only to glomeruli at the bottom part of the olfactory bulb.

What convinced Barnea and his colleagues that TAAR neurons constitute an olfactory subsystem were differences in how these neurons choose the receptor gene that they express. They used mice in which one TAAR gene is swapped with a marker that turns the cell blue. When the same strategy was used in olfactory receptor-expressing neurons, the blue marker was switched off and another olfactory receptor gene was selected. By contrast, TAAR-expressing neurons did not turn off the blue marker. They selected another TAAR gene, but continued to be stained blue.

This observation led Barnea and longtime collaborator Stavros Lomvardas of the University of California-San Francisco to identify several other differences between TAAR neurons and olfactory receptor neurons in how the neurons carry out gene expression.

"The logic of gene choice in TAAR neurons is different," Barnea said. "Their patterns of projection [to the bulb] are different, and the mechanisms that control their projections are different. Altogether these observations suggest that it's a different subsystem."

The smell of fear?

TAARs were discovered in the nose in 2006 by Stephen Liberles of Harvard Medical School, one of Barnea's collaborators and co-authors of this paper, when Liberles was a postdoctoral fellow in the lab of Nobel laureate Linda Buck. Liberles' team also found that some TAARs detected odors that are avoided by mice, such as a chemical found in the urine of carnivores like lions and tigers.

The discovery that TAAR neurons operate separately from olfactory receptor neurons, Barnea said, will lead to research on whether the glomeruli to which TAAR neurons connect selectively send signals to primal behavior centers in the brain, such as the amygdala that governs the fear response. That would complete a circuit that allows for specialized processing of alarming or repulsive odors to produce an instinctual fear-related behavior.

"We hypothesize that the neurons expressing TAARs may be some of the olfactory neurons that project to this area and elicit these behaviors," Barnea said. "We are building a circumstantial case. Now we know what it looks like in the first relay, in the glomeruli. Our study provides the tools to look for the next step."

Hypothetically, having two different systems could reflect the different meanings that various smells can have for an organism. Most of the odors that a mouse encounters in life -- the smell of cheddar, for example -- become relevant only with experience and learning. These odors are detected by olfactory receptors and they are processed by parts of the brain that analyze and learn new information.

But when it comes to the scent of a predator, a mouse has very little room for mistakes. There may be many chances to consider cheese, but only one to recognize a threat. A hotline between the nose and an instinct to flee might promote survival.

As for human beings, Barnea said, there is more debate than certainty about whether smells trigger instinctual behaviors, but he notes that humans do have several intact TAAR genes.

The paper's lead authors are Johnson and Tsai. In addition to Barnea, Liberles, and Lomvardas, other authors include Dheeraj Roy of Harvard, David Valenzuela of Brown, and Colleen Mosley and Angeliki Magklara of UCSF.

The National Institutes of Health supported the research.