Why AI Labs Keep Building Despite Extinction Fears
Top researchers are resigning in protest over existential risks, yet development continues at full speed due to competitive pressures and a belief that benefits outweigh dangers.
Several prominent engineers have recently quit major AI companies, citing a genuine fear that their creations could lead to human extinction. Jacob Coxon, who left Anthropic, described the industry as gambling with public safety, while colleague Evan Hubinger stated plainly that they believe AI could kill all humans. These departures are part of a growing trend of insiders warning that the technology is moving faster than safety controls.
Despite these stark warnings, the labs continue to accelerate their work. The contradiction lies in the reasoning behind their decisions: a mix of perceived economic benefit, the belief that safety can only be tested through active development, and the intense pressure of a global competition where no single player wants to fall behind.
Weighing Existential Risk Against Benefit
Leaders at OpenAI, Anthropic, and Google DeepMind have publicly agreed that AI extinction risk is comparable to nuclear war or pandemics. Dario Amodei of Anthropic has estimated the chance of catastrophic failure between 10% and 25%. Yet, these same executives promise that AI will eliminate poverty and cure disease. The decision to proceed rests on the assumption that these massive societal gains justify the potential for total loss, a calculation that critics argue should involve broader democratic input rather than just corporate leadership.
The Necessity of Active Testing
A second justification is the idea that you cannot make a dangerous system safe without building it. Proponents argue that static analysis is insufficient; you must deploy the model, observe its failures, and iterate. This approach, known as iterative deployment, is akin to studying a cliff edge only by standing close to it. While this allows for rapid learning and correction, it also means that each new release carries the risk of unforeseen consequences before safeguards can be fully implemented.
A Global Race Without Brakes
The most potent driver is competition. With rivals in the United States and China all seeking dominance, companies assume that if they slow down, others will not. This creates a dynamic where each firm believes it is the responsible one, even as they collectively push the boundaries further. According to reporting from GN technics/ai (en-US), this race is fueled by the fear that a mutual agreement to pause would be secretly broken by competitors. Consequently, the industry behaves like an arms race, where speed and capability are prioritized over collective safety.
