Population geneticist Montgomery Slatkin, graduate student Fernando Racimo and post-doctoral student Flora Jay were part of an international team of anthropologists and geneticists who generated a high-quality sequence of the Neanderthal genome and compared it with the genomes of modern humans and a recently recognized group of early humans called Denisovans.

The comparison shows that Neanderthals and Denisovans are very closely related, and that their common ancestor split off from the ancestors of modern humans about 400,000 years ago. Neanderthals and Denisovans split about 300,000 years ago.

Though Denisovans and Neanderthals eventually died out, they left behind bits of their genetic heritage because they occasionally interbred with modern humans. The research team estimates that between 1.5 and 2.1 percent of the genomes of modern non-Africans can be traced to Neanthertals.

Denisovans also left genetic traces in modern humans, though only in some Oceanic and Asian populations. The genomes of Australian aborigines, New Guineans and some Pacific Islanders are about 6 percent Denisovan genes, according to earlier studies. The new analysis finds that the genomes of Han Chinese and other mainland Asian populations, as well as of native Americans, contain about 0.2 percent Denisovan genes.

The genome comparisons also show that Denisovans interbred with a mysterious fourth group of early humans also living in Eurasia at the time. That group had split from the others more than a million years ago, and may have been the group of human ancestors known as Homo erectus, which fossils show was living in Europe and Asia a million or more years ago.

"The paper really shows that the history of humans and hominins during this period was very complicated," said Slatkin, a UC Berkeley professor of integrative biology. "There was lot of interbreeding that we know about and probably other interbreeding we haven't yet discovered."

The genome analysis will be published in the Dec. 19 issue of the journal Nature. Slatkin, Racimo and Jay are members of a large team led by former UC Berkeley post-doc Svante Pääbo, who is now at the Max Planck Institute for Evolutionary Anthropology in Leipzig, Germany.

In another analysis, Jay discovered that the Neanderthal woman whose toe bone provided the DNA was highly inbred. The woman's genome indicates that she was the daughter of a very closely related mother and father who either were half-siblings who shared the same mother, an uncle and niece or aunt and nephew, a grandparent and grandchild, or double first-cousins (the offspring of two siblings who married siblings).

Further analyses suggest that the population sizes of Neanderthals and Denisovans were small and that inbreeding may have been more common in Neanderthal groups than in modern populations.

As part of the new study, Racimo was able to identify at least 87 specific genes in modern humans that are significantly different from related genes in Neanderthals and Denisovans, and that may hold clues to the behavioral differences distinguishing us from early human populations that died out.

"There is no gene we can point to and say, 'This accounts for language or some other unique feature of modern humans,'" Slatkin said. "But from this list of genes, we will learn something about the changes that occurred on the human lineage, though those changes will probably be very subtle."

According to Pääbo, the list of genes "is a catalog of genetic features that sets all modern humans apart from all other organisms, living or extinct. I believe that in it hide some of the things that made the enormous expansion of human populations and human culture and technology in the last 100,000 years possible."

The Pääbo group last year produced a high-quality Denisovan genome based on DNA from a pinky finger bone discovered in 2008 in Denisova Cave in the Altai Mountains of Southern Siberia. That bone is from a young woman who lived about 40,000 years ago. The Neanderthal toe bone was found in the same cave in 2010, though in a deeper layer of sediment that is thought to be about 10,000-20,000 years older. The cave also contains modern human artifacts, meaning that at least three groups of early humans occupied the cave at different times. The Pääbo group developed new techniques to extract DNA from these old bones.

Slatkin noted that no one is sure how long the various now-extinct groups lasted, but that there is evidence that Neanderthals and modern humans coexisted in Europe and Asia for at least 30,000 years. Interbreeding was infrequent, though how infrequent is unclear given the genomic information available today.

"We don't know if interbreeding took place once, where a group of Neanderthals got mixed in with modern humans, and it didn't happen again, or whether groups lived side by side, and there was interbreeding over a prolonged period," he said.