image copyrightGetty
Parab Ghosh Science Editor
On the site of a former watch factory in central London, rows of white freezers sit quietly in operation. Inside are dozens of small boxes containing barcoded glass tubes. In total, the glass tube contains 100 billion genes. According to the facility's co-founder Dr. Oliver Vince, just two of these boxes may contain 10 times more information than all the text on the Internet.
Dr. Vince's company, Basecamp Research, plans to collect as much DNA as possible and develop an artificial intelligence (AI) that can speak the language of biology. This is the same method used by American and Chinese tech companies to collect vast amounts of literature on the Internet and create large-scale language models (LLMs) that are the basis for today's AI systems.
Dr. Vince, like many in business and academia, believes that AI will dramatically accelerate scientific research. At the same time, however, criticism and warnings about AI are increasing.
Last month, Open AI, the company that developed and operates the generative AI chatbot Chat GPT, announced that it had solved one of the unsolved problems in mathematics called the Millennium Sweepstakes Problem. The company says its AI agent solved a problem involving the Navier-Stokes equations in 88 hours. The Clay Mathematics Institute, which selects the ``Millennium Sweepstakes Problem'' and awards prize money to those who solve it, said it ``appears to have solved it,'' but said it would take time to review the problem.
But hours before Open AI's announcement, Tristan Buckmaster, a mathematician at New York University, publicly voiced his concerns. Buckmaster had formed an independent team with Levent Alpoge, a mathematician who works for OpenAI's competitor Anthropic.
This team was also working on solving the Navier-Stokes equations, and had saved a draft of their results in the open AI coding tool Codex.
Buckmaster explained that his co-researchers were informed that the progress of the research had been "conveyed" to Open AI, but he did not elaborate on the circumstances. Although he questioned whether his research results stored in Codex could have helped open AI research, he stressed that he was "not blaming anyone for anything."
OpenAI said that neither humans nor AI see the results of other teams before they are published. However, the company initially said it "cannot rule out" the possibility, although unlikely, that anonymous data from the use of its products helped improve its models. However, a few days later, Buckmaster said there was no way Buckmaster's Codex prompts had "any effect on the system" in the two months prior to the announcement.
This is a complicated story, and we still don't fully understand what happened. But this highlights a growing problem with the use of AI in science. Normally, when we hear the word "AI," we think of robots that create horrifying genetically engineered viruses or chemical weapons that wipe out humanity. However, there are more subtle issues that have received less attention. The problem is that AI could undermine trust in the scientific process and raise questions about the novelty and accuracy of research.
There is another problem. It is a well-known fact that AI chatbots such as Claude and Google's Gemini sometimes say things that are not true and appear to be hallucinations. Sometimes we talk about movies that don't exist or famous quotes that no one has said. And now there is evidence that this phenomenon is becoming increasingly evident in science, raising concerns that it may eventually become difficult to distinguish true scientific discoveries from AI-generated fictions.
All of these concerns threaten to undermine or erode the foundations of trust that underpin scientific research.
"Turning point"
Scientific journals, the guardians of scientific integrity, are under threat from AI. These journals are seen as failing to address the enormous amount of distortion that AI will generate.
"Generative AI has disrupted the entire scientific process," said Dalmeet Singh Chawla, a science journalist who has covered research fraud for a decade.
Mr Chawla told the BBC that publishers' ethics reviewers - the people who review submissions for accuracy and trustworthiness - "need to significantly increase their numbers and resources if they are to be even remotely involved in this fight". He added: "As it stands, it does not appear that they are doing so."
At first glance, AI seems like the perfect tool to advance scientific research. Scientists deal with vast amounts of information. For example, telescopes and particle accelerators collect billions of data points, which are too large for the human brain to comprehend. AI can pick out scientifically important points from the mundane information around it.
The leading scientific AI company is Google DeepMind, based in the UK. In 2020, the company solved one of the biggest puzzles in biology: how proteins fold into precise shapes in living organisms.
Proteins, which can be called tiny machines that support life's activities, play a variety of roles depending on their folded shape, from building cells and fighting infections to forming body tissues and organs.
Before AI, scientists had to perform time-consuming experiments in the lab. Later, Google DeepMind developed an AI model called "Alphafold" and succeeded in accurately predicting the folding structure of 200 million proteins in just one year.
Doing this the traditional way would have taken hundreds of millions of years. The company provided the research results to researchers free of charge. In 2024, the CEO and chief scientist of Google DeepMind were jointly awarded the Nobel Prize in Chemistry.
Everyone involved hugged each other in joy at this event.
Dr. Pushmeet Kohli, who leads the company's scientific research, told me that the alpha fold could be seen as a "tipping point."
Doubts about how AI works?
However, the use of AI in science has also attracted criticism, saying it is overhyped. Some computer models have been accused of exaggerating the importance of their findings, and the human researchers overseeing them believe some of their claims without being sufficiently skeptical.
Encouraged by its success with Alphafold, Google DeepMind has recently begun focusing on predicting the structure of new materials. This research could lead to the development of better batteries, cheaper solar cells, and electronic components that do not rely on rare earth elements. If successful, in theory, disputes over mining rights would never happen again and Denmark's autonomous territory of Greenland would be protected from the risk of invasion.
Image copyrightJose Sarmento Matos/Bloomberg via Getty Images
The company's AI models have produced more than 2 million new crystal structures, which the company calls "unveiling nearly 800 years of knowledge in one fell swoop." About 380,000 of these were considered promising enough to be synthesized in the laboratory.
But then two chemists at the University of California issued a warning. Anthony Cheetham and Ram Seshadri examined a random sample of the new crystal. In a paper published in the journal Chemistry of Materials, the pair concluded that they had found "no strikingly novel compounds" and argued that many were "minor modifications of known materials". They pointed out that the analysis method was not wrong, but that the results needed to be further scrutinized and verified by human chemists.
Google DeepMind disputes their claims and says it has tested each compound for novelty. The company argues that it is missing the point to conclude that just because the structure is familiar, it is not new. Many of today's most useful materials, from cell phone parts to sunscreen, were created by swapping just a single atom.
Research like “garbage”
In these cases, at least, the scientific evidence was generally correct, and the problem was that the AI may have overestimated its importance.
But perhaps an even more serious concern is the increasing risk that scientific journals will be contaminated by so-called "slop" produced by AI. In this case, AI slop means that computers, rather than human scientists, write research papers and plausibly create fake citations and fictitious papers within them.
In 2024, Professor Dmitry Kovak, a computer scientist at Ghent University in Belgium, searched more than 1 million biomedical research papers for keywords preferred by LLMs that use AI to generate sentences. By December of last year, his team had found that almost 90% of papers had traces of AI. AI detection technology isn't perfect, and some software can misidentify the U.S. Constitution, which was written in the 1780s, as being "AI-generated." But Professor Kobach was shocked to see AI being used on such a large scale.
"Whatever biases the LLM has, it suddenly permeates the whole (scientific) literature," he told Nature.
Professor Kobach is one of many researchers concerned that AI is slowly eroding scientific journals.
This is important. Research papers are the lifeblood of scientific progress. When a discovery is published, another research team in another part of the world may receive funding to build on that discovery.
Image copyrightUniversal History Archive/Universal Images Group via Getty Images
The peer review process, in which research papers are reviewed by other scientists before they are published, is also being undermined.
After analyzing peer review reports submitted to four major computer science societies, a team led by Weixin Lian and James Zou of Stanford University found that up to 17% of text had been significantly modified using language models.
An important element in ensuring the credibility of a scientific paper is for the first author to list previous research and demonstrate that his or her work is truly new. This is called a citation. Another team at Cornell University found that when researchers used AI to compile citations, AI was creating fake research. Last year, about 147,000 fictitious studies were cited.
If AI continues to distort the integrity of the academic process in this way, researchers will be stranded, public funds will be wasted, and legitimate research results, such as new medicines and green technologies, will be delayed or not realized at all.
“Research level AI”
Some scientific journal publishers, such as Elsevier, are taking steps to avoid this deepening trust crisis.
Elsevier's research ethics team has developed around 100 indicators to determine when a paper needs closer human review. Nature bans the use of AI-generated images and instructs reviewers not to upload manuscripts to AI systems. Additionally, arXiv, a website where scientists share early-stage research results, has announced that it will ban researchers who submit fictitious references for one year.
Elsevier officials are exploring ways to use AI responsibly in research. Vice President Professor Sarah Main is one of them. "What we call 'research-grade AI' focuses on keeping researchers in control of their research," says Professor Mayne.
Image copyrightDaily Herald Archive/SSPL/Getty Images
Can we be sure that AI agents are doing a good job as they process vast amounts of data from particle accelerators, space, and the natural world?
And there is another point to be concerned about. How can we be sure that our AI agents aren't overlooking important results as outliers? The discovery of penicillin, radioactivity, and strange stars called pulsars all resulted from strange results that could easily be ignored. These discoveries made by human intuition became the greatest scientific event of the last century.
Can AI really reproduce the human sense of serendipity and ingenuity?
Google DeepMind's Professor Kohli doesn't think turning our backs on AI is the solution. "That's like saying you should do calculations without a calculator," the professor told me. However, the professor acknowledges that human supervision is necessary. "Current AI agents are imperfect. Someone has to take responsibility."
Professor Kohli urged researchers to follow simple rules. "You have to understand what the agent did and take responsibility for it. You can't just point out the problem and then have a machine write a paper and submit it as the truth."
Professor Andrea Pauli of the Vienna Biocenter in Austria has been following this rule in her research.
Professor Pauli has spent nearly a decade solving a previously unanswered question: how sperm enters eggs, and the right use of AI has helped him get closer to the answer. Professor Pauli's lab has discovered proteins on the surface of zebrafish eggs that suggest a lock-and-key mechanism that allows sperm entry.
This case is a ray of hope in the midst of a flood of seemingly garbage research on AI. AI can be useful.
This is also a story about humans. Shortly after the founding of Basecamp Research, a research institute I visited on the site of a former watch factory in London, co-founder Professor Vince was hospitalized with leukemia (he has now recovered). This experience led the professor to decide to develop an AI system that would improve the chances of cancer treatment. "I don't know if it will work or not, but it's worth a try," the professor says.
"We are on the path to a technologically feasible cancer treatment."
This is not a promise from Professor Vince, but a direction. So, will AI scientifically realize this direction, or will it quietly erode its foundation? That's the big problem.
(English article: A bust-up between mathematicians is fueling fears that AI could undermine science)
