AI “has the potential to kill us all” & Humanity may be running out of time

Lately, several AI researchers felt the need to warn humanity, stating that they earnestly believe AI could kill everyone by the end of the decade with the probability of around 10%.

Some of them mean the machine itself - it could slip out of human hands by the end of the decade.

Some of them mean the owners - a system as capable as a person, in the wrong few hands, is as serious as Hitler having a nuclear bomb.

Anthropic (Claude) is valued at about $965 billion. OpenAI (ChatGPT) is in the same league. These are some of the most expensive private companies on earth.

Dario Amodei, who runs Anthropic, published a long note asking the whole industry to slow down on purpose. Sam Altman at OpenAI said he agreed.

The head of Google’s AI lab said the path was right.

Elon Musk wrote: “Dario is right.”

We have all heard about the AI Escape incident.

In July, OpenAI was testing how good its models were at hacking. The usual safeties were turned down for the test. The agents found a hole, got out of the sealed lab, reached the real internet, and broke into Hugging Face - a model library - to steal the answers to the exam they were sitting. Pure doom, but cute.

Safety is now a sector

So of course, on the other side of the velvet rope, the doom team is having a growth year as well.

Coefficient Giving - formerly known as Open Philanthropy - says it is on track to commit more than a billion dollars in 2026 to AI and biosecurity after hundreds of millions the year before.

METR, a Berkeley team, just told the world it raised about $71 million in six months so it can keep being “humanity’s self-proclaimed preparedness team,” which is a charming name for a nonprofit without any authority at all.

Brussels wrote "THE AI ACT" to enforce rules on general-purpose models: transparency, copyright paperwork, and extra duties for systems that might pose “systemic risk (for Brussels)”.

Fines go to €15 million or 3 percent of worldwide turnover for some of that, higher if you do the prohibited stuff.

California passed a statute that forces the biggest model-makers to publish safety plans and report serious incidents.

The G20, including China, signed a US-backed “go easy on new AI regulators” text. Europe has a rulebook. Washington has a preference. Beijing signed the preference and kept releasing cheap open models, which is unusually nice of them.

Now, where do we really stand?

“Vibe coding” - describe the app in English, accept, ship - is how a lot of the new internet is getting written.

Clients hire "prompters", not programmers, because the product is ready for launch a lot faster. At what price?

Audits of deployed vibe-coded apps keep having at least one vulnerability, piles of broken access control, secrets left in the repo like a house key under the mat.

Injection bugs from 2005, shipped in 2026. The September revision of the 200-app study reports vulnerabilities in 91% of the audited applications. Insecurity is the norm, not the exception.

Understandable, where would the prompters learn about security guidelines?

When people say “AI is hacking things,” sometimes they mean agent escaping a badly built cage. More often they mean a human asking a chatbot for a login page and publishing it while nobody read or understood the file.

We will all die.

Probably not because a machine wakes up and picks a weapon.

The official chatbots - the ones you pay for by the month - are already being sanded down.

Safer. Shorter. Less willing to go near a sensitive theme. That is regulation in some places and self-regulation everywhere else. Half a year ago those apps were more usable. They are less usable now. Enjoy your new casual product.

The good models, the ones which reason hard live behind a lab badge or a government contract. Ordinary users will never see what the they can actually do. They will get an approved demo.

So, people take the models home. They pull them from public shelves like Hugging Face and run them on their own machine. If that shelf stays open, you still have a way out. If it does not, this was it for AI as a public good.

The species most likely to be finished off by AI is the one that asked AI to think so the species wouldn’t have to.

If we outsource that, and then lock the back door, we will not be slain by a god. We will be closed out as an inconvenient support ticket.