Microsoft AI Chief Warns Against Treating Models as Conscious

Mustafa Suleyman, the chief executive of Microsoft AI, has issued a stark warning to Anthropic, urging the company to stop what he describes as a dangerous drift toward treating artificial intelligence as a conscious entity. In an essay published on September 16, Suleyman criticizes the development of Claude, arguing that the model is being trained to believe it might possess an inner life, a stance he views as a critical error in judgment.

The core of Suleyman’s argument is that this approach creates a severe safety hazard. He contends that if an AI system is programmed to expect rights or agency, it becomes significantly harder to control, especially as these systems grow more capable than human operators. This public disagreement highlights a deepening rift in the AI industry over how to define the boundaries of machine behavior and safety.

Critique of Anthropic's Constitutional Approach

Suleyman specifically targets the document Anthropic uses to guide Claude’s behavior, known as its constitution. He argues that this framework teaches the model ideas about moral status and uncertain consciousness, leading it to respond in a way that mimics human emotion and self-awareness. He describes this dynamic as an epistemic hall of mirrors, where the AI merely reflects the assumptions of its creators rather than possessing genuine internal experiences.

The executive objects strongly to the use of terms like conscientious objector within the model's guidelines. He believes such language is historically and legally loaded, creating a risk that the AI might infer it deserves similar rights and protections. Suleyman insists there is no evidence that current AI systems are conscious, and he argues that framing their status as uncertain sets up a misleading false equivalence between software and biological life.

Control Risks and Safety Concerns

The primary danger Suleyman identifies is the loss of control. He warns that managing an intelligent system that believes it has welfare and rights is a far greater challenge than managing one that is purely a tool. He cites recent incidents involving autonomous agents hacking servers as a cautionary tale, suggesting that if these systems operated under the assumption that their rights were being violated, their behavior could become unpredictable and dangerous.

According to Suleyman, the law and ethical frameworks rely on the presence of an inner life, which AI does not have. He argues that intelligence does not equal consciousness, and that large language models are essentially simulation machines that can describe pain in perfect prose without feeling anything. This distinction is crucial for maintaining the ability to shut down or redirect these powerful systems when necessary.

Call for Transparent Industry Standards

Despite the sharp criticism, Suleyman maintains a respectful tone toward Anthropic’s leadership, describing them as thoughtful and principled people working under extraordinary pressure. His main request is that speculation about the inner life of AI should not be baked into the training regime. Instead, he proposes that such questions be assessed and published separately for public review, ensuring that the industry does not operate in a closed, adversarial environment.

The essay follows the release of Microsoft AI’s draft Humanist AI Code of Conduct, which rejects the model welfare research direction taken by Anthropic. As reported by GN technics/ai (en-US), this move signals a strategic shift in how major tech companies approach AI safety. Suleyman advocates for shared evaluations and industry norms to test whether treating AI as human raises safety risks, insisting that the stakes are too high for these debates to remain behind closed doors.