A few weeks ago, I wrote about why Muse made AI personal while ChatGPT's Dots made it feel like a coworker.
I was excited by the idea that an AI could remain responsible for an outcome between conversations—not just answer a prompt, but keep a project moving, notice the next step, use tools, and return when human judgment was needed.
I am still excited.
Then I read Satya Nadella's essay, Models as Insider Risks in the Super Intelligence Era. It provides the necessary second half of the story.
If an agent is beginning to resemble a coworker, we cannot give it authority merely because it is intelligent. Nadella argues that we must “separate the supply of intelligence from the authority over it.” Models can make mistakes, be manipulated, or be compromised, so permissions, evidence, auditing, containment, and shutdown controls must sit outside the model itself.
I agree completely.
But I also see a difficult economic question hiding inside that architecture: who can afford to build it?
Startups may end up paying three times
Organizations are already restructuring around AI. At the same time, many are still carrying the cost of the people who understand the business, the models they hope will create leverage, and the infrastructure required to connect the two safely.
That can produce three separate bills:
- The intelligence: model subscriptions, inference, context, tools, storage, and repeated experimentation.
- The accountability: employees who still review outputs, resolve ambiguity, protect relationships, and accept responsibility when something goes wrong.
- The trust layer: identity, scoped permissions, logs, evaluations, monitoring, approvals, containment, incident response, and independent verification.
For a multinational company, those costs can be distributed across security, platform, legal, compliance, data, and product teams. The same company may run several models, maintain independent evaluators, commission audits, and build custom controls.
A startup may have five people trying to achieve a version of the same customer outcome.
Its employees already cover several roles. Its model budget competes with cloud costs and payroll. The person building the feature may also be the person evaluating it, approving its actions, responding to customers, and diagnosing an incident.
“Just keep a human in the loop” sounds inexpensive until we count the human.
The supervision paradox
An AI agent is supposed to create leverage. But if a person must inspect every intermediate step, reconstruct its reasoning, compare its answer with another model, and approve every action, the system may move work rather than remove it.
The employee becomes an AI supervisor while remaining accountable for the original job. The startup pays for automation and for the labor required to distrust it.
That arrangement may be necessary during an early pilot. It is not a sustainable operating model.
The opposite extreme—granting broad autonomy because review is expensive—is even worse. A small company often has less ability to absorb one destructive deployment, one leaked customer record, one incorrect payment, or one damaging message sent in its name.
It is how to concentrate limited human attention exactly where an error becomes consequential.
Minimum viable trust architecture
A startup should not pretend to be a global bank. It also should not use limited resources as an excuse to give an agent the keys to everything.
What it needs is a minimum viable trust architecture: a small set of controls that makes the safest path the simplest path, then grows with the authority and risk of the agent.
1. Give every agent an identity
An agent should not quietly act through a founder's or engineer's unrestricted account. Give it a separate identity where possible. Make its actions distinguishable from human actions. If something changes, the team should know who—or what—changed it.
2. Make permissions narrow and temporary
Access should match the current responsibility, not every responsibility the agent might have someday. Reading a repository does not require production-deployment access. Preparing a customer response does not require permission to send it. Investigating a billing problem does not require the ability to issue an unlimited refund.
OWASP describes excessive agency as a risk created when an AI system has unnecessary functionality, permissions, or autonomy. Its recommended controls are practical: minimize extensions and privileges, execute actions in the user's security context, require approval for high-impact steps, and log downstream activity.
3. Divide actions by consequence
Not every action deserves the same review cost.
Green → observe, search, summarize, draft Yellow → edit reversible content, open a PR, schedule a test Red → deploy, delete, pay, publish, message externally
Green actions can often run independently. Yellow actions need evidence and easy rollback. Red actions should normally stop at a human-controlled boundary.
The distinction should be enforced by tools and permissions, not remembered by the model.
4. Demand receipts, not explanations
A fluent summary of what the agent believes it did is not an audit trail.
Prefer evidence produced by the surrounding systems: a Git diff, a test result, an immutable event log, the exact message awaiting approval, the record IDs read or changed, and the identity under which the action ran. This aligns with Nadella's insistence that important actions leave tamper-resistant, human-readable evidence that does not depend on the model attesting to itself.
5. Put the emergency brake outside the model
“Stop when asked” is an instruction. An emergency brake is a control the agent cannot override.
A small team may not need a sophisticated containment platform on day one. It does need a reliable way to revoke credentials, stop a background job, disable a tool, pause a workflow, and prevent further side effects—even if the model continues proposing actions.
6. Review boundaries, not every thought
Human attention should concentrate at decision boundaries: before information leaves the company, before money moves, before production changes, before data is permanently deleted, or whenever confidence falls below an agreed threshold.
Reviewing every generated token is costly and gives a false impression of control. Reviewing a precise action, its evidence, its blast radius, and its rollback plan is far more useful.
7. Learn from small incidents
Startups do not need a formal committee for every mistake. They do need a habit: record what happened, which control failed, what the human missed, and what should change in the workflow.
If a team only measures successful completions, it will gradually automate its blind spots. NIST's AI Risk Management Framework emphasizes governing, mapping, measuring, and managing risk, and explicitly allows implementations to vary with a user's risk tolerance and resources. That proportionality matters for small organizations.
Use expensive intelligence selectively
The trust architecture can also become a cost architecture.
A startup does not need its strongest model for every classification, lookup, or formatting step. It does not need two models debating every low-risk action. It does not need a human to approve a reversible change in a disposable environment.
A more economical workflow may look like this:
rules handle known facts and hard limits
↓
smaller model handles routine judgment
↓
stronger model handles ambiguity or planning
↓
independent check for high-impact decisions
↓
human approves the irreversible action
This is where separating intelligence from authority becomes commercially useful, not only safer. When permissions and policy live outside the model, the company can use different models for different tasks, change providers, and route work according to cost, latency, and risk.
Model diversity does not have to mean paying several frontier providers to perform everything twice. For a startup, it can mean avoiding a design where one model both performs the consequential work and creates the only evidence that the work was correct.
Autonomy should be earned
Instead of choosing between “copilot” and “fully autonomous,” small teams can use an autonomy ladder:
- Suggest: the agent recommends; a person performs the action.
- Prepare: the agent creates the change; a person reviews and executes it.
- Execute reversibly: the agent acts inside a sandbox, draft, branch, or limited environment.
- Operate within bounds: the agent handles demonstrated cases while exceptions and consequential actions escalate.
Movement up that ladder should follow evidence: repeated successful outcomes, known failure modes, useful logs, effective rollback, and clear ownership. Autonomy should be a privilege granted to a workflow—not a personality trait attributed to a model.
Startups have one advantage
Large organizations have more resources, but they also have decades of systems, permissions, vendors, policies, and organizational boundaries to untangle.
A startup can design the agent boundary correctly before complexity hardens around it. It can keep the number of tools small, make actions reversible, maintain clean ownership, and encode a few strong rules instead of reconciling hundreds of exceptions.
The same limited resources that prevent a startup from building an enormous governance program can force it to build a simpler and more comprehensible system.
Platform providers will also have to absorb more of the common trust layer. OpenAI's Dots, for example, include permission controls, action approvals, activity views, safety checks, and restrictions on proactive research. Those features reduce the amount every small team must invent independently.
But Nadella's warning still applies: a provider's safeguards do not transfer responsibility away from the organization using the agent. Shared infrastructure can lower the cost of governance; it cannot decide the company's risk tolerance or accept its consequences.
Restructuring should not mean removing judgment
As organizations restructure around AI, there is a temptation to count models as replacements and people as costs.
That misses the transition we are actually in.
Some repetitive work will disappear. Some roles will change. But the scarce capability may increasingly be the ability to define outcomes, constrain authority, evaluate evidence, handle exceptions, and remain accountable for consequences.
A company that removes that judgment too early may appear efficient until the first serious failure. A company that requires people to manually supervise every model action may never receive the productivity gains it expected.
The goal is not fewer humans at any cost or more controls at any cost. It is a better allocation of intelligence, authority, and attention.
The real cost is accountable delegation
The price of an AI coworker is not only the subscription or the tokens.
It is the work required to decide what the agent may know, what it may do, what evidence it must leave, when another system should verify it, and where a human must still say yes.
Multinational companies can throw teams and infrastructure at that problem. Startups need a smaller answer—but not a careless one.
The winning small team may not be the one with the most autonomous agent. It may be the one that knows exactly where autonomy creates leverage, where it creates unacceptable risk, and how to place an inexpensive but real emergency brake between the two.
Sources & further reading: Satya Nadella: Models as Insider Risks in the Super Intelligence Era · OpenAI: Dots privacy, security and safety · OWASP: Excessive Agency · NIST AI Risk Management Framework