The control problem isn't AI. It's who's watching it.
Every generation of automation gets the same warning, and the warning gets ignored the same way: the tooling to contain it exists, and it gets switched off the moment it slows someone down. AI isn’t special in this respect. It’s just faster, more capable, and, left alone, more willing to keep going.
What control actually looks like
None of this is exotic. It’s the same short list most security teams already keep for people with too much access:
- Scoped access. An agent gets the narrowest permissions its task needs, not the broadest that got the demo working.
- A human in the loop for anything hard to undo — spending money, changing infrastructure, sending something external.
- Logs that get read, not just kept. An audit trail is only a control if someone checks it before the incident, not after.
- A kill switch that’s actually reachable. Stopping the system has to be faster than whatever it can do while running.
Why it gets skipped anyway
What’s different with AI is pace. A person who overreaches usually gets noticed within days; an agent with the same overreach can run for months producing output that looks fine, because most of the time it is fine, right up until the one input nobody tested for.
Every constraint above also makes the system slower or narrower, so removing one is nearly always the path of least resistance under a deadline. That trade looks free until the day it isn’t, and by then the constraint has usually been gone long enough that nobody remembers agreeing to drop it.
Where the leash comes off
- Scope creeps ahead of oversight. Access granted to ship a demo outlives the demo, and nobody revisits it once the thing is running.
- Guardrails are the first casualty of a deadline. A sandbox, an approval step, a rate limit: each is friction, and friction is what gets cut when something is due.
- A log nobody reads isn’t a control. It’s a record you’ll consult once, after the fact, to explain what already happened.
- Autonomy should scale with reversibility. The harder an action is to undo, the more of a person needs to be in the loop — not less.
- Working isn’t the same as controlled. A system can be correct for a year and still have no real containment the one time it isn’t.
None of this is a case against using AI. It’s a case for treating control as part of the design, not a compliance step bolted on afterwards. The tools to do this well already exist — most of the incidents I’ve seen didn’t come from a lack of tooling, they came from someone deciding the friction wasn’t worth it, right up until it was.
- ai
- security
- governance