AI Slop Risks Future Discovery

With apologies to The Simpsons news anchor Kent Brockman: I, for one, welcome our new AI research overlords. Or maybe not so much. Catherine Aiken, Steph Batalis, and Greg Tananbaum, in “Charting a Course for AI in Science” (Issues, Spring 2026), provide a valuable critique of how artificial intelligence is rapidly reshaping the production of science. They describe how institutional incentives pressure researchers to adopt AI tools without adequate understanding, training, or oversight. At the same time, AI is undoubtedly bringing real innovation in research and development.

As the authors imply, this duality can be seen playing out in scholarly publishing. Commercial publishers have integrated AI into their business models, capitalizing on the bundling of scientific content for model training while deploying AI systems for integrity checks. Yet publishers remain highly vulnerable to AI-slop. In one recent case, for example, the journal Frontiers published articles with absurd, AI-generated images and hallucinated citations, demonstrating that the industry is in a rat-race against the technology while looking to profit from it.

I applaud the authors for pointing out the connection between AI and the perverse incentives of the publish-or-perish culture. I was surprised, though, that while they discussed the traditional routes for sharing research through journal publication, they overlooked the growing movement—and associated AI challenges—of sharing research through preprints. By facilitating the rapid, open sharing of discoveries, preprint repositories reduce the dependence on commercial gatekeepers and help move the needle in the publish-perish culture.

However, AI is threatening the preprint ecosystem on two fronts. The servers face severe strain from the demands of the AI ingestion machine, eager to gobble up openly licensed content for model training and context updating. And, similar to the publishers but perhaps on a larger scale, AI slop is flooding the field. The scale of this challenge is reflected in the recent decision by the open-access service arXiv to institute a strict one-year ban for authors of manuscripts containing unchecked AI-generated content, such as hallucinated citations or leftover chatbot instructions.

These challenges in scholarly publishing highlight a broader problem in the scientific enterprise: a possible future where innovation implodes. We do not yet know whether researcher reliance on AI to synthesize literature, draft manuscripts, propose hypotheses, analyze and generate data, and conduct reviews will erode the distinct human capacity for creative thought. We do know, however, that as the share of AI-generated content grows and eventually overtakes human-generated content in the scholarly record, it poses an increased risk to future discovery. Frontier models will continue to ingest the growing body of AI-generated scientific outputs. The resulting feedback loop is likely to degenerate into model collapse, where AI systems degrade in quality and lose the ability to provide novel or accurate insights due to being trained on (recursively input) synthetic information.

We’re unlikely to escape the AI research overlords in the long term. In the short term, the scientific community must anchor its principles and practices in open science. Teaching researchers how to responsibly share data, code openly, and write comprehensive AI-use disclosures should be a prerequisite in the training that the authors advocate for.

Christopher Steven Marcum

Catherine Aiken, Steph Battles, and Greg Tananbaum’s essay got me thinking about a new “pilot test” being conducted by the Centers for Medicare & Medicaid Services. Intended to evaluate the Wasteful and Inappropriate Service Reduction (WISeR) Model, the test will run for six years in six states, including Washington, where I live. The stated goal of WISeR is to reduce fraud, waste, abuse, and medically unnecessary care by using artificial intelligence and other advanced technologies to screen selected Medicare procedures before treatment is authorized.

AI is increasingly being introduced not only into health care and scientific research, but also into such diverse fields as finance and education. In many cases, AI is not merely analyzing data after the fact. It is becoming an active participant in the systems being studied, as it is in the WISeR test. This development raises an important methodological question: How to conduct valid scientific evaluation when the intervention itself changes the behavior being measured?

Economists, ecologists, sociologists, organizational theorists, and systems scientists have long studied reflexive systems. What may be new is the emergence of reflexive systems in which adaptation occurs in response to AI-driven processes whose internal logic is only partially discernible to participants, researchers, and, in some cases, even the developers of the systems themselves.

Traditional experimental design assumes that researchers can control an intervention while observing outcomes in a relatively stable environment. AI challenges that assumption. Once participants understand that an AI system is influencing decisions, they begin adapting their behavior. Physicians alter documentation practices. Organizations redesign workflows. Consultants develop optimization strategies. Software vendors modify products. Increasingly, participants employ AI tools of their own to anticipate, respond to, and influence AI-driven processes. As a result, the environment under study becomes a dynamic sociotechnical system characterized by continuous adaptation. The intervention affects participant behavior, participant behavior changes the data being generated, and those data may subsequently influence future versions of the intervention.

This creates several methodological challenges. First, causal inference becomes more difficult. Observed outcomes may reflect adaptation to the AI system rather than the effectiveness of the AI system itself. Second, data reliability may deteriorate. Changes in documentation practices, coding behavior, reporting conventions, and AI-assisted content generation may alter the meaning and consistency of the data over time. Third, adaptive feedback loops may emerge. Participants learn how to respond to the system, while the system increasingly operates on data generated by those responses. We must then ask: Are we evaluating the performance of the AI, or the behavior of a system adapting to the presence of AI?

The answer may have profound implications for the future of scientific research itself. And yet as the authors point out, “Researchers and scientists are largely navigating these changes on their own, with current incentives pushing them to adopt AI through hurried learning of new and insufficiently understood tools, methods, and models.”

This is certainly the case with the WISeR pilot test.

Richard V. Badalamente