The head of one of the largest artificial intelligence start-ups in the world has warned against the development of AI superintelligence — machines that outstrip human brain capability.

In the letter released last week, Sam Altman, chief executive of Open AI, called AI superintelligence the greatest threat to the continued existence of humanity.

Dario Amodei, CEO of another major AI firm Anthropic, said: “I think there’s a 25 per cent chance that things go really, really badly”.

More than 29,000 people signed the letter in the span of 48 hours.

It states: “We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably and strong public buy-in.”

Harry, Duke of Sussex was also a signatory. In a personal statement in the note, he wrote: “The future of AI should serve humanity, not replace it. I believe the true test of progress will be not how fast we move, but how wisely we steer. There is no second chance.”

Stuart Russell, professor of computer science, Berkeley and the director of the Centre for Human-Compatible Artificial Intelligence said they did not want a ban in the usual sense.

He wrote: “It is simply a proposal to require adequate safety measures for a technology that, according to its developers, has a significant chance to cause human extinction. Is that too much to ask?”

Superintelligence is a theoretical software-based artificial intelligence system with an intellectual scope beyond human intelligence.

At the moment, AI intelligence is equivalent to a really smart adult with an IQ score of 110 to 120, but lacks humanity’s broad understanding, consciousness and reasoning.

According to a recent article in Psychology Today, pet dogs understand human intent better than AI.

In August, OpenAI’s competitor, Mark Zuckerberg, CEO of Meta, said AI superintelligence was “now in sight”.

Not everyone agrees. Brent Smolinski, IBM vice-president and global head of technology and data strategy called superintelligence hype.

“It is totally exaggerated,” he said in an article on IBM’s website. “I don’t think we’re even in the right zip code for getting to superintelligence.”