Decentralising science may lead to more reliable results

Research results on drug-gene interactions are much less likely to be replicated if they are performed by hierarchical communities or close-knit groups of frequent collaborators who use similar methods, instead of independent groups of scientists using different methods, suggests a paper published last week in eLife.

The findings may help improve the reliability of scientific results by helping to identify possible factors that contribute to the publication of unreliable, misleading or false results about potential drug-gene interactions.

“The way science is often produced may inadvertently contribute to unreliable results,” says senior author James Evans, Professor of Sociology at the University of Chicago, and External Professor at the Santa Fe Institute, US. “For example, a large group of scientists who frequently collaborate, use similar methods, share equipment, and frequently cite similar works are prone to producing the same, self-confirming results. Although such a group may produce repeated published experiments, our results demonstrate that their findings are not independent. Independent labs perform experiments in different ways with different expectations and are less prone to peer pressure than a densely connected networks of scientists.”

To better understand how such factors may contribute to unreliable results in studies of interactions between drugs and genetics, Evans and his colleagues compared the results of 3,363 published studies on 51,292 drug-gene interactions in the Comparative Toxicogenomics Database with results from the LINCS L1000 program, which used robots to test thousands of drug-gene interactions.

They found that drug-gene interactions identified by multiple studies were verified by the LINCS L1000 results 45% of the time, while results from single studies could only be verified 19% of the time.

They also looked at a subset of gene-drug interactions that were investigated in more than one study. They found interconnected groups of authors using similar methods were more likely to confirm each other’s results than scientists with no apparent connections. The results of these interconnected groups were less likely to be replicated by LINCS L1000 than results from independent groups.

“Even if a drug-gene interaction claim garners support from many papers, if it is studied exclusively by a centralised scientific community, the claim has a predicted probability of replication that is similar to that for a claim reported in a single paper,” says lead author Valentin Danchev, PhD, a postdoctoral scholar previously in the Department of Sociology at the University of Chicago, and now at the Meta-Research Innovation Center at Stanford (METRICS), Stanford University, US.

He adds that the way science is currently organised encourages frequent collaborations between interconnected groups of researchers. These interconnected groups and the ‘star’ scientists that lead them not only publish results that are less likely to be replicated but may also gain disproportionate influence, potentially discouraging independent groups from researching the same drug-gene interactions or publishing findings that disagree. “Our findings highlight the importance of introducing science policies that promote decentralised and non-repeated collaboration as a path to independent replications that are robust across diverse teams, methods and settings,” Danchev concludes.

###

References

The paper ‘Meta-Research: Centralized scientific communities are less likely to generate replicable results’ can be freely accessed online at https://doi.org/10.7554/eLife.43094.

It was produced by lead author Valentin Danchev (now at Meta-Research Innovation Center at Stanford, US; previously at University of Chicago, US) and co-senior authors Andrey Rzhetsky and James Evans (both University of Chicago).

This work has been published as part of eLife‘s Meta-Research Collection, the full contents of which can be freely accessed online at https://elifesciences.org/collections/8d233d47/meta-research-a-collection-of-articles.

Contents, including text, figures and data, are free to reuse under a CC BY 4.0 license.

Media contact

Emily Packer, Senior Press Officer

eLife

[email protected]
01223 855373

About eLife

eLife is a non-profit organisation inspired by research funders and led by scientists. Our mission is to help scientists accelerate discovery by operating a platform for research communication that encourages and recognises the most responsible behaviours in science. We publish important research in all areas of the life and biomedical sciences, which is selected and evaluated by working scientists and made freely available online without delay. eLife also invests in innovation through open-source tool development to accelerate research communication and discovery. Our work is guided by the communities we serve. eLife is supported by the Howard Hughes Medical Institute, the Max Planck Society, the Wellcome Trust and the Knut and Alice Wallenberg Foundation. Learn more at https://elifesciences.org/about.