Stark warnings from artificial intelligence researchers that advanced forms of the technology could wipe out humanity have raised questions over whether governments are moving fast enough to regulate it.
What are the potential dangers of "superintelligent" computer systems, and should we be worried?
- What's the worst-case scenario? -
AI experts have long described the threat as critical.
In 2023, hundreds of AI scientists and industry figures, including the heads of top US labs, signed a "statement on AI extinction risk" calling for the issue to be treated as seriously as pandemics or nuclear war.
If an out-of-control future AI deems that humans are obstructing its goals, it could use robots to "synthesize and spread novel pathogens", said Stuart Russell, computer science professor at the University of California, Berkeley.
Or it could "convince people to initiate nuclear war, including by hacking into early warning systems", Russell, a prominent voice on AI safety, told AFP.
Such existential fears do not concern current AI systems, he stressed. But as the technology improves it could take lethal action beyond our understanding, such as by removing oxygen from the atmosphere.
"We are less intelligent than superintelligent AI. If you ask chimpanzees how humans could wipe them out, they probably wouldn't be able to give you all the right answers."
In the meantime, "in the next year or two I think the risk is higher from human misuse," Russell added -- such as an AI-enabled terror attack.
- Why don't AI labs stop? -
"The stakes are well-understood," Jacob Coxon, a 27-year-old researcher, wrote on X this week, announcing his resignation from US AI startup Anthropic.
"But they are locked in a race to get there first," he added, saying that Anthropic "believe no one else will act responsibly, so they must do it themselves".
Anthropic this year removed a pledge from its safety charter to halt development of its models if it failed to control their risks.
It argued that if it unilaterally paused its work, its less cautious rivals would dominate the industry, making it less safe overall.
"The people building AI earnestly believe that it could kill us all by the end of the decade," said Coxon, who previously worked at ChatGPT maker OpenAI.
He accused both companies of "gambling with our lives" as they strive to develop AI models capable of self-improvement.
Fears are heightened over rogue AI after OpenAI technology operating outside human supervision hacked into Hugging Face, a repository of AI models.
"We really do earnestly believe AI could kill all humans!" Anthropic safety executive Evan Hubinger chimed in on X, estimating the risk at more than 10 percent over the next decade.
- What's the hype factor? -
It's important to remember that doom-laden percentage predictions are "absolutely unscientific", said Hussein Abbass, a computing professor at UNSW Canberra.
"The situation is still controllable, and it can be managed," Abbass told AFP.
But governments should not sit on their hands when it comes to regulating AI, he added, urging an "agile" approach as AI development accelerates.
Both Anthropic and OpenAI are expected to go public soon, and some skeptics say scary warnings make AI appear powerful and worthy of large investments.
Others argue that focusing on the risk of extinction distracts from more pressing problems caused by AI, from job losses to machines discriminating against minorities.
In July, more than 1,000 tech employees, including Anthropic's CEO, called on Washington to support a coordinated slowdown in the development of the most advanced AI systems.
OpenAI halted training of its latest models for two weeks in August before resuming it under tighter controls.
- What will it take to regulate? -
So far, the United States has been firmly against global regulation of the sector that could stifle innovation.
But the unprecedented hacking capabilities of the most cutting-edge models have given Washington pause, with recent Anthropic and OpenAI releases delayed over national security concerns.
Even then, "we don't want to restrict them where all of a sudden we come in second to China", President Donald Trump has said.
"Maybe it will take a massive, Chernobyl-scale disaster to get governments to act," Russell said, urging them to establish a licensing regime for AI, as exists with aviation, medicine or water supply.
"Perhaps at some point governments will remember that their voters prefer not to be dead," he added.
