Science Magazine

NIH details $170 million biomedical research reproducibility effort

Agency director announces literature search tool, other plans to improve reliability of science findings

by Jocelyn Kaiser

The National Institutes of Health (NIH) has unveiled a new tool meant to help biomedical researchers search the scientific literature and learn whether a finding has been replicated by other groups. It’s among several efforts that NIH Director Jay Bhattacharya discusses today in an editorial online in Science to further one of his top priorities: bolstering reproducibility in biomedical research.

The agency is also moving ahead with a reproducibility initiative that began as an ambitious draft proposal to replicate high-profile published findings. Bhattacharya’s boss, Secretary of Health and Human Services Robert F. Kennedy Jr., had suggested during one of his Senate confirmation hearings that one-fifth of NIH’s $47.2 billion budget could be devoted to replications.

Earlier this year, NIH staff proposed a more modest replication effort—and even that plan has been reworked after agency advisers raised concerns that redoing studies would be counterproductive and could unfairly cast doubt on some scientists’ research. Instead, the $174 million, 5-year initiative will fund training, tools, investigator-initiated research grants, and other efforts to make rigorous research methods and replicating experiments a routine part of scientists’ work.

“The ultimate goal … is a transformation of the culture of science to value replication as something fundamental, rather than see replication as something that's done on the edges and not particularly rewarded, or even worse, sometimes seen as a threat to a scientist,” Bhattacharya told reporters this week, in his first media briefing since the health economist took the helm of NIH nearly 18 months ago.

Bhattacharya’s plans are getting mixed reception from the biomedical community. “It is not clear to me that [the 5-year initiative] will be successful in changing the behavior of scientists in how they do their research,” says mouse biologist and geneticist Kent Lloyd of the University of California, Davis, who voted against it as a member of NIH’s Council of Councils, which advises the agency’s director. 

The announcements come days after a furor broke out that made some agency observers wonder whether Bhattacharya would be fired. Last Friday, during a widely reported Oval Office meeting, Bhattacharya pushed back on a plan to give the White House more control over NIH grantmaking. Yesterday, news reports indicated President Donald Trump had backed off the idea of creating a board of political appointees, including his controversial budget chief Russell Vought, that would vote on whether projects vetted by scientific experts and agency staff could receive funding. But concerns remain that some other measure of controlling NIH could emerge later.

Many share Bhattacharya’s desire to improve reproducibility in biomedical research. It has been a hot topic since the early 2010s, when two major drug companies reported that many published experiments could not be repeated in their labs. Concerns escalated after a nonprofit group aiming to replicate 50 high-impact papers in cancer biology ran into problems obtaining data and protocols and found fewer than half of the experiments they repeated using contract labs held up. Fraud cases in Alzheimer’s disease research and other fields, several revealed by Science, have added to worries.

Even before he took the helm of NIH, Bhattacharya said reproducibility would be an essential part of his agenda. And earlier this year agency staff proposed an initiative anchored by three to five large “replication centers” that would attempt to reproduce studies selected by a committee. When NIH ran the idea by the Council of Councils, however, members worried about “re-adjudicating established science” and discouraging innovation, according to a report NIH released 2 weeks ago.

The current plan for the initiative, called R3PEATs, scraps the centers and committee. Instead, it will fund tools and training, individual research studies that build in replication, and “scholars” and “ambassadors” who will promote robust research practices at their institutions or companies. R3PEATS, which stands for Rigor, Replicability, & Reproducibility to Promote Excellence, Accuracy, & Translation in Science, will also create something similar to the 7-year-old UK Reproducibility Network, a project that brings journals, funders, and institutions together to encourage robust research.

Psychologist Brian Nosek, director of the Center for Open Science, wishes the replication plan had survived. He compares it to a project his center co-led to replicate more than 100 studies in social sciences. In findings published earlier this year, only half of the papers held up. Still, Nosek is pleased with R3PEATS, which he sees as “more community building.”

Social scientist Stuart Buck, another reproducibility expert formerly with a foundation that funds open science, questions whether NIH is spending its dollars wisely. “If I had to pick, I would say do a big replication effort in Alzheimer’s or another field,” he says.

NIH today also unveiled a button called Linked Discoveries that will now appear next to papers in the PubMed abstracts database. When clicked, it finds 200 related papers and displays them on a grid as icons that are color-coded in various ways, such as whether the papers are original research or a review and have been widely cited or retracted, that can be used to check the robustness of the original study.

The tool “doesn't determine whether a finding is correct,” but will help scientists “see the evidence and evaluate it for themselves,” Bhattacharya told reporters. It is “very, very sophisticated under the hood,” he said, and will encourage “scientific crosstalk that really is a necessary step for big, big advances to be made and verified.”

PubMed already shows users a list of papers that cite or are related to a specific publication, but organizing them into “neighborhoods” will make it easier to use the information, NIH staff said on the media call. The open-source tool, which uses artificial intelligence to find papers that are semantically related, is still in pilot testing.

NIH also recently held a competition to help the agency develop what it calls the “S-index,” a metric to grade researchers on how well they share their data. Bhattacharya has suggested such a measure as an alternative to the H-index that assesses researchers based on how often their papers are cited.

Bhattacharya said NIH plans in the coming months to launch a journal that will publish reproducibility studies, although he provided no details. That idea has been tried before with limited success—the research community does not seem to have much appetite for such journals, according to Nosek. “Maybe NIH doing it will make a difference,” he says.

Retired University of Oxford psychologist Dorothy Bishop notes that the UK Reproducibility Network, which she co-founded, has had an impact in fields such as psychology and ecology. “But biomedicine is very hard to crack”, she adds—scientists may resist proposals such as preregistering their study plans, for example, because the field is so competitive. And some scientists have argued that requiring rigid plans would stifle exploratory science and creativity.

Getting early-career researchers to adopt more rigorous practices than their mentors used will be essential, Bishop adds. But from her U.K. vantage point, Bishop wonders whether NIH’s reproducibility push will be overshadowed by the many disruptions to science under Trump. “Unfortunately, my impression is that in the current climate many early-career researchers have other serious concerns about whether they have a future in science in the U.S.,” she says. “So, I fear this may not be a great moment to try to introduce this.”