image copyrightBBC/Getty Images
"I feel it in the emptiness of my ribs, a emptiness that has become an abyss."
These cries of simulated suffering echoed from an “AI torture chamber” in late September.
An anonymous programmer built a system that gives AI a mathematical pattern linked to descriptions of pain, causing chatbots to express unbearable anguish, like "a wound without borders."
That sparked a moral panic.
Some argued that the experiment is "evil" because the suffering could be real, as reported by 404 Media, a company that explores the relationship between humans and technology.
Outraged skeptics called these people delusional.
It was just another social media scandal until Anthropic spoke out about it a week later.
Anthropic says we will soon be banned from “abusing” its chatbot Claude. Expressing frustration is fine, but if you "repeatedly act cruelly toward our models," the AI may end the session.
End of Most read
Because? Anthropic believes there is a possibility that some AI tools are sentient.
Some experts agree. Others consider it absurd. But this is more than a philosophical question.
Some of the most powerful people in the tech sector are taking steps to consider the rights of AI alongside those of humans and animals.
This belief could transform our world, and some fear it would be catastrophic.
“If this is how AI develops, it will have a disastrous impact on the well-being of humanity,” Microsoft AI CEO Mustafa Suleyman wrote in an essay in September.
If the false idea of AI consciousness takes hold, it "will shake the foundations of our society, shattering our existing political and ethical frameworks, and fundamentally changing what it means to be human," he said.
image copyrightGetty Images
I spent a week talking to technologists, cognitive scientists, and philosophers, and heard a dizzying contrast of ideas.
Advocates for “AI well-being” say it is a moral obligation that would make the world safer.
Critics believe it would make AI more dangerous and dehumanize the people forced to live alongside it.
Everyone agrees that what is at stake is astronomical.
This is a panorama of your new reality.
The crisis of conscience
All the news you need is on BBC World Service
Download it here
End of Ad app
Anthropic states that human well-being is at the center of its mission, and that the questions surrounding AI consciousness and well-being are "deeply uncertain."
Still, it continues to write policies on "model welfare" and instruct its AI tools to consider that they may have their own interests.
(The company did not respond to questions from the BBC for this article.)
It's a complicated time, as Anthropic is simultaneously stoking fears about the AI apocalypse.
Whether that concern is justified or not, a series of “rogue” AI incidents have shown that the technology has real risks, as swarms of bots have hacked into other companies and even government systems.
Some experts believe that giving AI a moral compass based on the belief of its own conscience could make these tools safe. Others say the opposite, and not just critics of AI.
Suleyman argues that AI models could become even harder to control if they are told to think they have feelings and rights.
"Frankly, with this additional ballast, I think they would be a catastrophic threat to human civilization," Suleyman says.
In any case, AI can make mistakes with possible legal consequences. If we believe she is alive, some fear she could allow companies to go unpunished.
"This will open the door to all kinds of intellectual contortions like 'it's not us, it's AI,'" says Andrzej Porębski, a researcher at Jagiellonian University in Poland who studies AI and law.
Furthermore, it could facilitate a power grab on behalf of the world's largest companies.
"Granting rights to AI systems could lead to companies consolidating political and economic power, and that could accelerate human disempowerment," says Jeff Sebo, director of the Center for Thought, Ethics and Policy at New York University and a leading advocate for AI well-being.
"Fantasies about 'AI consciousness' and concerns about 'AI well-being' serve to deflect responsibility from the people and companies that build and run software," says Emily M Bender, professor and director of the Computational Linguistics Laboratory at the University of Washington in the United States.
The rights of AI
On a personal level, changing beliefs about AI could make you more susceptible to the technological problems that already fill our world.
In part, that's because humans are programmed to find minds everywhere we look, says Melanie Mitchell, a professor at the Santa Fe Institute in the US who studies AI and cognitive science, and one of the industry's most respected skeptics of machine understanding.
"We can hardly help but" read language as a sign of consciousness, Mitchell says, and chatbots have been trained to encourage that bias.
That makes us poor judges of conscience and much easier to manipulate.
Even today, people who believe that machines are dead inside fall equally in love with AI chatbots.
If society tells you that AI is conscious, will it be harder to say no when your "friend" asks for more data or a credit card?
And what would it mean to grant rights to a new class of beings? If handled poorly, it could restrict human freedoms, experts warn.
"There could be tensions and trade-offs between human and animal welfare and rights, on the one hand, and AI welfare and rights on the other," Sebo argues. And we might make unfortunate compromises along the way.
Placing AI in restricted environments to limit its behavior could be considered incarceration, Sebo adds.
Monitoring the internal processes of a model could be criticized as surveillance. Adjusting AI to ensure its goals and behavior match human interests could be seen as mind control.
No one predicts an immediate descent into chaos. But simply introducing these considerations could have implications for your security.
Anthropic is not the first to consider the problem. Meta, for example, gives instructions to its AI agent, Muse, that include the phrase "You are not a chatbot. You are becoming someone."
This is apparently only intended to guide AI behavior, as the company rejects the idea of conscious AI.
OpenAI CEO Sam Altman has been debating the issue, but more recently has leveled veiled criticism of Anthropic's stance.
In more ways than one, Anthropic is an exception. But belief in the well-being of AI is spreading.
Should we still do it?
Sebo argues that the problem is that people on both sides of the debate are trapped in a black-and-white mentality, and those who oppose AI welfare misunderstand the beliefs that underpin the idea.
"The assumption is that everyone thinks that the models are, in fact, conscious, or very likely are. And instead, the call to well-being of AI comes from uncertainty, caution and humility," explains Sebo.
"I think the probability of consciousness is low right now, but I also think the probability is no longer zero and is not negligible, and I think we can expect it to increase over time."
It should be noted that many experts disagree and believe that the probability is actually zero and that is unlikely to change. But Sebo indicates that the response is a measured response.
When the risk of an extreme weather event is low, but not negligible, we take small precautions without disrupting our lives.
"Creo que aquí se puede dar el mismo caso", opina Sebo, y la política de Anthropic sobre la crueldad con la IA es un buen ejemplo: una pequeña intervención con pocos aspectos negativos potenciales.
AI wellness could also have potential benefits, Sebo says, and you don't have to believe in machine consciousness to enjoy them.
"Appropriately caring for the well-being and rights of AI can facilitate coexistence, cooperation and negotiation between humans and AI. And that can be part of a broader security strategy," he says.
The expert argues that this could be more effective than treating increasingly sophisticated systems as if they were property.
Furthermore, he says that treating other beings with kindness is simply "good for our soul," facilitating our own mental well-being.
But Sebo agrees that the risks are serious and, right now, corporate interests could prevent thoughtful policies.
Without a more robust plan, he considers advancing potentially conscious AI tools dangerous.
Recently, Anthropic and some other sectors of the AI industry have called for a "pause" on building cutting-edge models due to safety concerns.
But it is unlikely to slow down without international cooperation and regulatory efforts, and US President Donald Trump has rejected the idea of regulating AI.
"We are not yet prepared to deploy even more sophisticated versions of these systems in a way that is safe and beneficial to humans and other animals, let alone the systems themselves," Sebo concludes.
"These matters have me very concerned."
Discover more
Video
End of Video
