A team of Australian scientists claimed to have developed a genetic test that predicts a person’s risk of developing autism spectrum disorder (ASD) with 72 percent accuracy. Writing in Molecular Psychiatry, the team led by Stan Skafidis and Carlos Pantelis from the University of Melbourne said that their panel of 237 genetic markers could “correctly classify ASD from non-ASD individuals” and “may provide a tool for screening at birth or during infancy to provide an index of at-risk status.”

But a new study, led by Benjamin Neale from Massachusetts General Hospital, suggests that those claims were overblown. Neale’s team replicated the Australian group’s research in a larger sample, and found that the proposed panel of markers did not accurately predict ASDs.

“The claims in the original manuscript were quite bold. If they were true, it really would have been quite a major advance for the field, with serious ramifications for patients and other risk populations,” said Neale. “I think it’s important to ensure that this kind of work is of the highest quality.”

“This is a convincing refutation that calls into question the original results on specific technical grounds, rather than simply a non-replication that leaves a puzzling discrepancy between the two studies,” said Leonid Kruglyak, a geneticist from the University of California, Los Angeles, who was not involved in either study.

In 2012, Skafidis’s team compared the genes of 732 European people with ASD from the Autism Genetic Resource Exchange database, with those of 123 neurotypical people from a different cohort. They searched for single nucleotide polymorphisms (SNPs) that were linked to ASD, especially those in genes with roles in relevant cellular pathways.

They eventually settled on 237 SNPs in 146 genes, which they used to create a classifier for predicting ASD risk. When they tested the classifier on 243 cases and 42 controls from the same databases, it correctly predicted ASD with an accuracy of 85.6 percent.

The team then tested the classifier on an independent group of people—525 with ASD taken from the Simons Foundation Autism Research Initiative and 2,620 controls from the Wellcome Trust Birth Cohort. It identified the ASD cases with an accuracy of 71.7 percent.

But to other geneticists, these results seemed too good to be true. They implied that this small set of SNPs can explain around 11 percent of the variation in ASD risk—an unprecedented figure for any psychiatric condition. If the set truly had such strong effects, genome-wide association studies (GWAS) should have identified those SNPs by now—and they had not. It will likely take a sample of hundreds of thousands of people to find SNPs with such predictive power, as has been the case for other traits like height. “The magnitude of the study you need is dramatically larger than what was presented,” said Neale.

Neale wrote to Skafidis’s team asking for the full list of 237 SNPs, but did not receive it. (Skafidis told The Scientist that they offered the code that they used to generate their results, which should have been even better.) As such, they focused on the 30 most important SNPs, which were detailed in the published paper.

By comparing 5,417 cases and 5,417 controls from the Psychiatric Genomics Consortium, Neale’s team found that none of the 30 SNPs were significantly associated with ASD risk. The researchers also combined the SNPs into a classifier, using methods detailed in the original paper, and tested it on 4,623 cases and 4,623 controls from the same group. Again, the set failed to predict ASDs any better than chance. Finally, they also showed that the cellular pathways which the Australian team identified are not significantly associated with ASDs. The team’s results were published as a letter to the editor on 22 October, also in Molecular Psychiatry.

Several factors could explain the differences between the two studies. The Melbourne team initially tested the accuracy of their risk classifier on the same group of people whom they used to identify their SNP set. This is bad practice. “To appropriately assess the accuracy of a classifier, the sample which is used to develop it must be fully distinct from the sample on which it is tested,” said Kruglyak.

The Australian researchers also drew their cases and controls from separate populations with subtly different ethnic compositions. The SNPs they identified could have reflected reflect random ancestral differences between the two groups, rather than meaningful differences in ASD risk. Daniel Geschwind from the University of California, Los Angeles, made the same argument in a letter regarding Skafidis’s paper, which was published in the same journal this April.

Kruglyak added a third possible explanation: “batch effects, in which cases and controls are genotyped at different times and on different technology platforms,” he proposed. This problem also plagued a similar recently-retracted paper, which identified a panel of SNPs that could supposedly predict longevity.

But Skafidis said that Neale’s team may have come to different conclusions because the group did not use the full set of SNPs, nor the code that was provided. His team has submitted a response to the new study, which is in revision with Molecular Psychiatry (and does list the full set of 237 SNPs).

Meanwhile, Neale emphasized that other research into the genetics of autism are yielding stronger results. Several studies have identified loss-of-function mutations, and differences in the number of copies of certain genes, that are linked to ASD risk. Promising GWAS results have been presented at conferences and are making their way into published papers. “Autism genetics shouldn’t be tarnished by science that hasn’t been robustly proven,” he said. “There are successes beginning to emerge, and that’s really exciting and important.”