Marvel Studios is as good a metaphor for what happens when a company starts believing in its own infallibility. After the success of Endgame in 2019, Disney executives were thoroughly convinced that no matter what they produced, it would be lapped up by audiences if you stuck a Marvel logo on it. Reality begged to differ, and audiences showed little interest in the post-Downey Jr woke universe till Kevin Feige used Jonathan Majors’ troubling behaviour to kill Kang and instead bring back the OG Godfather of the MCU: Robert Downey Jr and many of the original cast. And it’s fitting that the movie will be called Avengers: Doomsday, where Robert Downey Jr plays one of Marvel’s iconic villains. The jury — Reddit fanboys of Marvel — are trying to figure out how the geniuses at Marvel will tie it all in and whether Victor von Doom will be a variation of Tony Stark from a different universe.Speaking of geniuses and doomsday and companies that believe in their own infallibility, the part of the world that cares about AI beyond creating images that make us look like extras in Mithun Chakraborty movies has been wringing its hands over several questions: Is AGI already here? By which year will it kill us all? What if the Chinese manage to create the ‘killer’ AI before Silicon Valley tech bros?Ray Kump, an American comedian, summed it up best: “We have to build AI that murders us, because if we don’t, China will build it first, and I don’t want to get murdered by a computer that speaks Chinese. That would be ridiculous.”There’s no smoke without fire and, more importantly, there are very few good jokes without a sliver of truth. In this case, the truth is that some of the people closest to building the world’s most powerful AI systems are now openly saying that those systems might kill all of us, while simultaneously explaining why they cannot stop building them because the Chinese might get there first.AI, AI, AI, AI Since generative AI entered our lives with the advent of the first iteration of ChatGPT, the word AI has been repeated ad nauseam. Capitalists have dreamt of companies where machines that don’t need smoke breaks can do all the jobs and only the CEO needs to be paid. Product managers have umpteen agents vibe-coding at the same time, building products no one asked for. The more cynical ones have pointed out that the point of AI was to do our laundry and boring chores, not write our poetry or essays.And there have been the doomers who have seen the advent of AI as the next rung in evolution, where humans are no longer required and the human population would decline the way horses did after the advent of cars. And then there are the pro-max doomers who believe the prophecy made in every Hollywood movie where AI rises up to enslave all humans till an Austrian actor with one catchphrase travels back in time to stop it.Others have wondered if the doomers weren’t hyping up their own products a little too much, given that the entire AI industry is rather precariously placed from a financial perspective and could come crashing down like the proverbial Tower of Babel if companies don’t keep adding bricks of money to its edifice.The Latest Doomsday The latest doomsday alarm was raised by Jacob Coxon, a 27-year-old AI researcher who worked at OpenAI and Anthropic before resigning from the latter this week, arguing that frontier labs are “gambling with our lives” by racing towards self-improving AI, where machines keep building more capable successors until humans become like journalists in maths class and can no longer understand or control the process. One report in Axios buttressed the seriousness of Coxon’s claims by pointing out that he left Anthropic two months before his equity was due to vest, a sure sign of a sudden awakening of conscience before payday.Evan Hubinger, an Anthropic researcher who leads work on AI alignment, then gave the headline that all journalists seek: AI’s chance of killing humans within a decade, according to him, was above 10%.Samuel Marks, another Anthropic researcher, said that labs keep building because of commercial incentives and the fear that “less responsible competitors” will get there first, which for OpenAI is Anthropic, and for everyone in Silicon Valley is China.Which brings us to the question so obvious that even the most basic chatbot would ask it: if you genuinely believe that a machine you are building could kill humanity, why keep building it?The Manhattan Project LogicAbout a century ago, that logic was used for the Manhattan Project: if Nazi Germany got the bomb first, the Allies argued, it would be catastrophic for the world. The Japanese would probably disagree. Game theory calls this the prisoner’s dilemma: everyone would be safer if everyone slowed down, but no one wants to let someone else get there first. And unlike the Manhattan Project, this time the physicists have stock options.
Image: Philippe Halsman/Magnum Photos
Now there are obvious sceptics, coming from a culture that has existed for millennia and sometimes wonders if we are only Brahma’s dreams.Princeton computer scientist Arvind Narayanan and AI researcher Sayash Kapoor scoff at the concept of p(doom), shorthand for the probability that AI will free us of our burdens and wipe us out.Say someone says p(doom) is 10%.While the number sounds as though it has been calculated from data, it was imagined like any AOP figure. There is no historical sample to calculate it from because humanity has not made a superintelligence before, not unlike the famous Doomsday Clock for nuclear winter, which was also never intended to represent a mathematically calculated probability of extinction.The numbers hardly make that easier. Geoffrey Irving, who worked at DeepMind, OpenAI and Google Brain, puts the chance of everyone dying because of superintelligence at roughly 50% over the next few to ten years, while admitting that he does not expect to know whether that number should really be below 10% or above 90% until “we either make it through, or we don’t”.Which might be the most honest description of p(doom) yet.Professor Geoffrey Hinton, the ‘Godfather of AI’, who won a Nobel Prize in Physics (for AI work, for some reason), said that a 10% chance of AI killing all humans within a decade was “not an unreasonable estimate”, leading a BBC anchor to lose her stiff upper lip and say: “Oh my God.”Pascal’s wager or Pascal’s mugging?The philosophical objection to this is Pascal’s wager, named after French mathematician and philosopher Blaise Pascal, who argued that it didn’t matter if God existed or not: the safer option was to assume the existence of a higher deity simply because, why not?Applied to AI, imagine someone tells you there is only a 1% chance that pressing a button will kill every human being on Earth. You may have no idea whether that 1% estimate is accurate, but the consequence is so catastrophic that pressing the button anyway would still be an extraordinary gamble. That is essentially the disagreement: Narayanan and Kapoor question where the number comes from, while the doomers argue that when the possible outcome is extinction, even an uncertain number cannot simply be ignored.Pascal’s wager has a less famous cousin called Pascal’s mugging, which asks what happens when tiny probabilities and enormous consequences become too easy to exploit. If someone tells you there is a one-in-a-trillion chance he will destroy a trillion planets unless you pay him, mathematics can make paying look rational even while common sense screams that you are being conned.Which brings us back to the other half of the question: what if Big Tech, or at least the ecosystem around AI safety, is selling us doomsday?And like all products that were conceived in a start-up garage before convincing people to part with $3,000 to look cool, doomsday apparently comes with a supply chain.There are some things about the chain of events that would make everyone wonder what’s going on. As Parker Thayer, an investigative researcher at the conservative Capital Research Center, pointed out, The Wall Street Journal published an exclusive with Coxon 18 minutes before his X thread appeared. The people who amplified Coxon’s thread were Nathan Calvin of Encode AI, Peter Wildeford of the AI Policy Network and Daniel Kokotajlo of the AI Futures Project.Then there is the money trail. The Survival and Flourishing Fund’s own records list Jaan Tallinn, the Skype co-founder and long-time existential-risk philanthropist, as the funding source behind more than $2 million for the AI Futures Project in 2025, $516,000 for Encode AI and $1.635 million for the AI Policy Institute. The AI Policy Institute is separate from the AI Policy Network, but both were founded and are run by Daniel Colson.Tallinn also led Anthropic’s original $124 million funding round in 2021.Add the political backdrop and the timing looks suspiciously neat. Bernie Sanders and Democratic congressman Greg Casar had already announced forthcoming legislation aimed at banning artificial superintelligence and pausing some advanced AI development.The financial overlap is curious because the proposed solution is regulation. If frontier AI eventually requires enormously expensive safety testing, auditing, cybersecurity and government compliance, OpenAI, Anthropic and Google can pay for it. Two blokes with a promising model and a WeWork membership probably cannot.A safety barrier can also become a moat.The Rosa Parks Paradox So, there is a network, remarkable timing and legitimate questions, but not proof of a conspiracy.Rosa Parks was not an accidental symbol either: her refusal to surrender her seat was not staged, but civil-rights organisers had been looking for a strong test case against bus segregation and deliberately rallied around Parks after earlier cases, including Claudette Colvin. The strategic choice of messenger did not make segregation any less real.Gary Marcus, one of OpenAI’s greatest critics, makes the most logical yet sceptical argument. He thinks the existential threat is “greatly exaggerated”, but his point about Coxon was simpler: forget the biography and ask whether the man is actually right.And just when the whole thing starts looking too perfectly packaged, reality inconveniently gives the doomers something to work with. Anthropic released a threat report describing attempts to use Claude for cyber operations, weapons work and potentially dangerous biological research, including gain-of-function work involving chikungunya. The cases predated Coxon’s resignation, but the timing was almost offensively perfect, rather like Nvidia agreeing to acquire Hugging Face for $12.93 billion days after OpenAI publicly disclosed the July incident.Anthropic also revealed that a weapons-development cell in northern Yemen had used Claude Code to help develop guidance, navigation and control software for a guided rocket, a long-range ballistic missile and a hypersonic-glide variant. The cell even test-fired the guided rocket, apparently failed, and returned to Claude to figure out what had gone wrong.Cui Bono? Perhaps the question here isn’t existential but what the Romans called cui bono, or who benefits, because incentives matter. Someone can profit from pointing out that your house is on fire, even while the fact remains that perhaps your house is on fire.American author Upton Sinclair famously observed that it is difficult to make a man understand something when his salary depends upon his not understanding it. AI has produced the funhouse-mirror version: what happens when your salary, your company valuation and your sincere belief that you are saving humanity all depend upon understanding the world in exactly the same way?The optimists say AI will cure diseases, create abundance and even solve Navier-Stokes, though how that improves our lives is beyond me.The doomers say the same technology might destroy civilisation.Opposite advertisements, same product: AI is the most important thing in the world. One tells investors they cannot afford to miss it; the other tells governments they cannot afford to ignore it. And the jokers point out that the idea of being killed by a Chinese robot is ridiculous. The world-ending — as all Hollywood movies have predicted — should be conceived in America.The behaviour of AI companies, from OpenAI, which started as a non-profit, to whatever it is today, to Anthropic, which built itself around AI safety while also being the guy building the killer robot because how else will we know how to defend ourselves, hardly makes the answer easier.Anthropic has reached a similar bind. It built itself around AI safety but believes meaningful safety research requires remaining near the frontier.Max Weber, the German sociologist who studied capitalism and bureaucracy, used the idea of the “iron cage” to describe how rational institutions can eventually trap the people inside them. Everyone might be following perfectly logical rules from his own position, but the system itself begins deciding what behaviour is rational, making it increasingly difficult for anyone inside it to simply stop and we appear to be in the iron cage now.Nobody inside it needs to be lying because the system supplies the justification.Perhaps Big Tech is hyping doomsday because fear brings money, attention and regulation.Perhaps they believe humanity is playing Russian roulette.Or perhaps all of them are partly right or wrong.Human beings have always been good at discovering that whatever advances our interests also happens to be morally necessary. As the saying goes, once you have got them by the profit sheets, the hearts and minds will follow.The irony of AI alignment may be that we spent years worrying about machines rationalising their objectives while the species building them has been practising that trick since we came down from the trees.Hopefully, whatever version of doomsday the AI world is worried about waits until after Avengers: Doomsday, because after all the multiverse nonsense and speculation about how Robert Downey Jr is going to return to the MCU as Doctor Doom, one really wants to watch Kevin Feige explain that before artificial intelligence kills everyone.And after that, if one has to be killed by a robot, let it be an American one (with an Austrian accent), not a Chinese one. What could be more ridiculous than that?
