Is AI Flooding Medical Journals With ‘Meaningless’ Research?

As artificial intelligence (AI) grows more sophisticated, its use in medical research is becoming both more prominent and harder to detect, leading to what some describe as a flood of mediocre science being published in journals.

Even just 6 months ago, AI hallucinations like fabricated references were a big concern. But it’s now relatively easy to prompt AI to double-check these things and churn out research with no obvious errors, experts said.

At the same time, it’s become harder for journals to identify AI’s fingerprints.

“We are at a point now that we cannot distinguish fake from real,” said Elisabeth Bik, PhD, a science integrity consultant known for detecting fraud in research.

Ivan Oransky, MD, co-founder of Retraction Watch, explained that it’s become relatively easy to enter a dataset into ChatGPT or Claude and have it generate a study with findings of little consequence. Systematic reviews are particularly easy to create, experts said.

Essentially, AI slop has just been replaced by “kind of meaningless” science, Oransky said.

While it “may be deceptive if you’re not disclosing the AI use, it’s not fraudulent, and it’s not really slop,” he added.

Since generative AI came to market, the number of studies using public datasets such as CDC’s WONDER or the National Health and Nutrition Examination Survey has skyrocketed, according to Matt Spick, PhD, a research integrity expert at the University of Surrey in England.

Indeed, most biomedical publications have tells of LLM-assisted writing, according to a study cited by Spick.

At one publisher, things got so bad that leaders had to implement a new strategy for dealing with the influx of AI-generated studies.

Last year, the open-access publisher Frontiers became the first to require “new experimental validation or additional data from authors’ own institutions for all manuscripts comprising solely of bioinformatics analyses, computational studies of public data such as NHANES, or findings from Mendelian Randomization studies,” Elena Vicario, PhD, director of research integrity at Frontiers, told MedPage Today.

In the past year, the research integrity team has rejected “more than 12,000 submissions based on simple queries of public datasets, including over 3,000 NHANES-based submissions,” she added.

Other journals have reviewed their policies to ensure the flood of mediocre studies doesn’t overwhelm the peer review system.

Springer Nature uses “a combination of screening tools, AI-enabled approaches and expert human oversight” to identify and assess unethical or fraudulent content, according to director of research integrity Chris Graf.

A spokesperson for NEJM Group said it is “actively piloting AI detection tools in multiple areas and looking to improve disclosure efforts from our authors.”

Tools for detecting use of AI in manuscripts include Pangram, which analyzes blocks of text to estimate if it is AI or human-written, while programs like Imagetwin and Proofig can detect duplicated images and plagiarism.

But whether these tools can keep up with AI’s constant improvement remains a question. Bik noted that image fraud detection tools are better at detecting fraud in pre-AI research. The skills she and other research fraud sleuths have are “based on the fraud of yesterday,” she said.

Gideon Meyerowitz-Katz, PhD, of the University of Sydney, said generative AI has made outright fraud a lot easier to do as well — and that’s a whole other issue that journals and publishers have to deal with.

It has also aided authors’ responses to fraud accusations, he noted. In the past, when Meyerowitz-Katz flagged a potentially fraudulent study to a journal, he rarely received a response from the authors.

“Now, every time I complain about a paper to a journal, I will receive a 5-page, in-depth rebuttal to my points, which Pangram says is almost entirely AI generated,” he told MedPage Today.

As long as the “publish-or-perish” incentive lives on in academic medicine, journals are likely to continue to be bombarded with mediocre AI-created work, experts said.

“If medical schools and universities don’t want the literature to be overwhelmed with garbage,” Oransky said, “then they should probably stop giving every applicant … an incentive to publish a lot of crap.”