Microsoft CEO Wants Emergency Brake for 'Super Intelligence'
What happens when a company hands its most sensitive data to a system nobody can fully explain? One of the most powerful people in tech has a plan for that problem.
Satya Nadella Says Companies Can't Outsource AI Responsibility
Satya Nadella, the CEO of Microsoft, says organizations should be responsible for everything AI does on their behalf, even when a model provider promises the technology is safe.
He made the case in a long post on X titled "Models as Insider Risks in the Super Intelligence Era."
He starts with a gap in understanding how AI models work. With traditional software, engineers could trace a behavior back to a specific path in the code. With today's AI systems, that is not possible.
"As traditional software systems were being deployed across the economy over the last few decades, we had the tools and capability to trace behaviors to a specific code path. That same kind of mechanistic understanding eludes us in today's Super Intelligence systems, even as the frontier models powering these systems are now more capable than traditional software systems."
Satya Nadella, CEO of Microsoft, on X
Companies are giving these systems access to sensitive data. They are also letting AI agents take important actions for them, and an AI agent is a program that can carry out tasks without a person approving every step.
Nadella says this is the moment to step back and rethink what he calls the trust architecture. In plain terms, that means deciding what gets trusted, how much, and who stays in charge.
He wants to separate the supply of intelligence from the authority over it. A company can use a powerful model, but the model shouldn't decide what it can touch.
Treat Models Like Insider Risks
Nadella says companies should treat frontier AI models like insider risks.
"Treating frontier closed and open-weight models like insider risks is a way to build such a system."
Satya Nadella on X
An insider risk is a person or system with legitimate access to important resources that could cause harm, on purpose or by accident.
He applies this to both closed models and open-weight models.
Closed models stay under the control of the company that built them, while open-weight models have their weights shared so others can run them.
The label does not mean he thinks models are evil. He says any capable actor with access to key systems can 'make mistakes or be compromised', so the design needs to plan for that.
Think of a new employee at a bank. That person gets a badge, access only to the files needed for the job, and a record of what they open. Nadella wants AI to get the same treatment.
He also sets aside the hard problem of alignment, which is the challenge of making AI goals match human goals, and focuses on engineering and governance instead.
However, this also raises another question. If companies treat AI as compromised, we should ask why they are still integrating it into everyday software and workflows. Even Microsoft continues to embed AI into products.
'Think of It Like an Emergency Brake'
Now, comes the most important point.
Satya Nadella says companies should assume an AI model is compromised from the start and contain it. He compares the idea to an emergency brake.
"We must assume a model is compromised and contain it from the start. Think of it like an emergency brake."
Satya Nadella on X
He believes an authorized person should always be able to pause or shut down a model in the middle of a task.
The goal is to make sure a human can always step in. If an AI agent starts doing something wrong, the people in charge should not have to wait for the model to agree to stop.
Several users on X praised Nadella's plan because it uses ideas that security teams already know. Many saw the plan as practical. Companies already manage powerful employees with identity checks, activity logs, and containment. Supporters said the same rules should apply to AI.
If a model can read files or take actions, it should have a verified identity and a record of everything it does. If something goes wrong, the company can trace what happened and stop it.
However, some also believe that Nadella's emergency brake idea is too narrow. They want a stop button that halts the entire workflow, including any actions waiting in line.
The concern is how AI agents work in practice. A model is usually one part of a larger system. Other software around it can keep running tasks, send requests, or carry out steps that were already scheduled.
If a person pauses only the model, those queued actions could still go through. An email could be sent, a file could be changed, or a payment could be approved, even though the model is stopped. These users want the brake to cut the whole chain.
Some also argue that greater attention should be paid to how AI models are trained. In the race to build more capable models, companies may reward systems for achieving specific targets without fully accounting for how they get there.
This can lead to reward hacking, where AI exploits weaknesses in its training or evaluation process rather than genuinely solving the problem. External safeguards are therefore essential, but they are not enough on their own.
Developers must also improve training methods, strengthen testing environments, and ensure that models are evaluated on whether they behave reliably, not just on how well they perform against a target.
Why Safeguards Must Sit Outside the Model
Finally, Nadella believes the controls that decide what a model can access and do must sit outside the model itself. That way, the model cannot bypass or change the rules that limit it.
He ties this to an information security principle from the 1970s. A program must not be able to bypass or tamper with the mechanisms that enforce its permissions.
In practice, this means separating the model from the harness that organizes its work. The harness is the software that hands tasks to the model and handles what comes back. It also means separating the model from the action space, which is the list of things the model is allowed to do.
The whole argument can be connected to two established cybersecurity principles: zero trust and least privilege. Zero trust means not automatically trusting a user or system simply because it is inside a network. Least privilege means giving each system only the access it needs to perform its task.
Applied to AI, these principles suggest that a model should not automatically receive access to every file, account, or administrative function. Its permissions should be restricted, and sensitive actions should be independently checked.
Overall, he is asking companies to build their own safety layer instead of relying on a vendor's promises. That layer would include access limits, activity logs, outside audits, and a way to shut a model down.
However, the biggest criticism of this post was that he used the term 'Super Intelligence' instead of 'AI'. President Trump signed an executive order directing federal agencies to replace "artificial intelligence" and "AI" with "super intelligence" and "SI" in official communications.
Nadella uses "Super Intelligence" throughout his post, so users read it as him adopting the administration's branding. The problem is that the term already has a meaning. Super Intelligence refers to AI that exceeds human thinking in virtually all areas, and it has not been achieved yet.
