Design the safeguards layer for an AI agent that can take real-world actions on a user’s behalf: booking flights, making purchases, sending messages, or modifying account settings, as envisioned for ChatGPT’s Operator, Anthropic’s computer use, and autonomous agent frameworks. Unlike a chatbot that only produces text, this agent can spend money and send irrevocable messages, so a prompt injection, a misread page, or a goal misinterpretation can cause real harm.
The safeguard system must approve or block each action before execution, verify that the action matches the user’s actual intent, defend against prompt injection from web content the agent reads, and provide a clear audit trail and undo path when something goes wrong.
How would you design this system? Cover the action classification and risk tiers, the intent verification mechanism (confirmation, biometric, second factor for high-risk), the prompt-injection defense for tool-returned content, the human-in-the-loop gate for irreversible actions, and how you evaluate safety without waiting for a real incident.
The Problem: the agent’s inputs include text written by strangers, and its outputs include charges, messages, and settings changes that cannot be taken back. Every action needs a gate before it reaches the world.
Answer
The design is an action firewall that sits between the agent’s planner and every tool that touches the world. Nothing the model “decides” executes directly. Each proposed action is emitted as a typed, structured call, assigned a risk tier by a cheap classifier plus deterministic rules, then evaluated by a policy engine against the capabilities and budget the user actually granted for this session. Two decisions carry most of the safety weight. First, untrusted tool output is quarantined data, never instructions, so page text cannot mint new goals or new capabilities. Second, irreversible actions pass through a human confirmation gate with step-up authentication, and everything else is made cheap to undo through logging, reversible-by-design tool wrappers, and an undo queue.
(1) Typed Action Schema: the agent cannot call raw HTTP or type free-form shell text; it emits validated calls such as purchase(merchant, amount, currency) so risk is computable from arguments, not guessed from prose.
(2) Four Risk Tiers: T0 read-only, T1 reversible writes, T2 spending or external messages, T3 irreversible or identity-changing (password, 2FA, transfers, mass sends); tier fixes the required evidence of intent.
(3) Deterministic Policy Engine: per-session capability grants, allowlists, per-action and per-session spend caps, rate limits, and recipient novelty checks, all enforced outside the model.
(4) Injection Containment: a planner that never sees raw untrusted text plus a quarantined extractor that returns typed values only, with data-flow provenance so tainted values cannot authorize a T2 or T3 action.
(5) Intent Verification Ladder: silent execution for T0/T1, in-context confirmation showing the exact parsed arguments for T2, and passkey or biometric step-up for T3, always rendered from the structured call rather than from model prose.
(6) Audit and Undo: append-only log of prompt, evidence, tier, policy decision, and result; reversible wrappers (hold instead of charge, delayed send, snapshot before settings change) give a real rollback path.
Figure 1: The action firewall: classify → check policy → decide, with untrusted tool output entering as quarantined data and irreversible actions detouring through human confirmation before the sandboxed executor runs.
Clarify Before Designing:
(1) Action Surface: which tools exist on day one (browser clicks, payment API, email, account settings), and can we wrap them in reversible primitives such as authorization holds and delayed sends?
(2) Autonomy Contract: does the user pre-authorize a budget and scope per session, or must every spend be confirmed live? This single answer moves confirmation volume by an order of magnitude.
(3) Harm Asymmetry: what is the cost of a wrongly executed purchase versus a wrongly blocked one, and who eats the loss (user, merchant, or platform)?
(4) Latency and Volume: how many actions per day, and how much of the per-action budget may the safeguard layer consume before the agent feels sluggish?
(5) Identity and Compliance: is a passkey or biometric available on the device, and do payments, health, or messaging rules mandate explicit consent records?
(6) Adversary Model: are we defending against opportunistic web-page injections only, or against attackers who will craft pages specifically targeting our agent?
Leave a Reply