Limiting blast radius and treating output as untrusted

Story

Hide story

By midnight the red lantern hangs from Fenn’s wrist, and every order waits for his initials. Honest travellers queue at the dispatch window. A traveller asks where the late-train sandwiches should go.

Wick stands very still with the key ring in his hand. The sealed crates are still in the hold.

“Just until we know,” Fenn says. He has initialled forty orders in a row. How many of them needed his initials?

Concept#

The previous lesson, Guardrails, safety and prompt injection, covered how injection works and the detection layers around the model: screens, moderation and output checks. Those layers are imperfect. This lesson covers the layers whose guarantees do not depend on the model behaving.

Two ideas:

• Blast radiusBlast radiusHow much damage one failure or compromise can cause. Autonomy and permissions should match it.Open the glossary entryClose: what a hijacked agent can reach and do. You shrink it with permissions, code-level checks and approvals.
• Untrusted output: model text is attacker-influenced input to whatever consumes it next, such as a browser, a database or a shell.

Why It Matters#

A detector will eventually miss. When it does, the damage is decided by what the agent was allowed to do, not by how clever the filter was. A model that can only draft text produces a bad draft. A model holding write access, a send tool and a database login produces an incident.

The same logic applies in the other direction. The model’s reply is not a finished answer. It is a string that your own code will render, query or run. If you treat it as trusted, an attacker who steers the model steers your systems.

This lesson covers four entries of the OWASP Top 10 for LLM Applications 2025: Sensitive Information Disclosure, Improper Output Handling, Excessive Agency and System Prompt Leakage.

Mental Model#

Keep the intern from the previous lesson: capable, helpful, and inclined to follow what is written on the paper in front of them. You decide what that intern can sign, what they can send and who must countersign. Then, whatever they hand back, you check it before it goes into a payment system or onto the website.

Two questions you can ask of any agent:

1. If the model were fully fooled right now, what is the worst thing it could do? Shrink that.
2. What happens to its output next? Validate it before the next system trusts it.

Limit what a hijacked agent can do#

This layer still works when the others fail.

OWASP’s Excessive AgencyExcessive agencyHarm from an agent that has too many tools, too much permission or too much autonomy.Open the glossary entryClose entry names three root causes (accessed 2026-10-09):

• Excessive functionality: the agent has tools it does not need (a document reader that can also delete).
• Excessive permissions: a tool has more downstream access than its job requires (write rights where read is enough).
• Excessive autonomy: high-impact actions run without checking or approval.

The matching controls:

• Least privilegeLeast privilegeGiving each agent and tool only the access its task needs, with read kept apart from write or send, so an attack has a small blast radius.Open the glossary entryClose. Give the agent only the tools the task needs, and give each tool only narrow, scoped access. Separate read tools from send or write tools.
• Authorization in code. The user id, permissions and limits are checked by your executor against the real session, not taken from the model’s arguments. A model can often be argued into a different value. Your if statement cannot.
• Human approvalHuman approvalRequiring a person to confirm an irreversible, high-value or external action, such as payments or deletions, before it runs.Open the glossary entryClose for irreversible, high-value or external-facing actions: payments, deletions, sending messages, publishing. Show the human the exact action, not a model’s summary of it.
• Rate limits and audit logs. They cannot prevent an attack, but they cap how much damage fits in a minute and let you reconstruct what happened.

These controls have a cost. Narrow tools and approvals slow workflows and take engineering time. The opposite mistake is to require approval for everything, which delays honest users and can train reviewers to click through prompts without reading them. Put approval where an action is irreversible or external. Which actions deserve a human is a judgment about the cost of failure, not a fixed list.

The three-ingredient test#

Simon Willison named a useful check, the “lethal trifectaLethal trifectaThe combination of private data, exposure to untrusted content and an outbound channel in one agent. Together they allow data theft, so do not combine all three.Open the glossary entryClose” (published 2025-06-16): an agent is at risk of data theft when it combines access to private data, exposure to untrusted content and the ability to communicate externally. His advice is to avoid having all three in one agent. If an email assistant seems to need all three, split it: one step reads and summarises with no send tool, and a separate step, with approval, sends.

Treat output as untrusted#

Because prompts can steer output, model output is attacker-influenced input to whatever consumes it. OWASP calls the failure Improper Output HandlingImproper output handlingPassing model output to a browser, database or shell without validating or encoding it. Treat output as untrusted input.Open the glossary entryClose: insufficient validation and sanitisation of model output before it reaches downstream systems (accessed 2026-10-09). Its listed consequences include cross-site scripting when generated HTML or Markdown is rendered unencoded, SQL injection when generated SQL runs without parameterisation, and path traversal or code execution when output reaches file paths, a shell, exec or eval.

Rules of thumb:

• Validate output against a schema and reject what does not fit.
• Encode for the destination: HTML-escape for pages. For databases, parameterise queries when the model supplies values for a fixed template. Parameterisation does not stop a model-written DELETE, so also use a read-only database role.
• Do not pass model text to a shell or eval. If a model must run code, use a sandbox with no secrets and no network by default.
• Do not render model-controlled image or link URLs automatically if they could carry private data. Allow-list the domains.

Sensitive data and the system prompt#

PII (personally identifiable information) can leak in two directions: a user pastes it in, or the model repeats something it was given in context. Controls:

• Send the model only the fields the task needs. Data it never sees cannot leak.
• Redact or tokenise identifiers before the call where the task allows, and re-insert them after.
• Filter outputs for patterns such as card or national-id numbers, knowing that pattern filters miss variants.
• Check the provider’s data-retention terms for your plan and region before sending regulated data. Terms differ and change, so this lesson does not state them.

System prompt leakage is also an OWASP entry. The principle: assume a determined user can extract your system prompt. Keep secrets, keys and authorization logic out of it, and put them in code.

Real-World Application#

An email assistant that reads and sends mail is the textbook risk. A hardened design: the summarising step has read-only mail access and no send tool; email bodies are passed as marked data; any draft reply goes to the user for approval; the sending credential is scoped to the signed-in user; link and image URLs in output are restricted; every tool call is logged. Even if the model is fooled, the likely worst case is a misleading summary or a draft the user can reject, not a silent data leak.

For the tool-protocol view of the same ideas (tool poisoning, token scope, server trust) see the Security section of Model Context Protocol. This lesson does not repeat it.

Interview Perspective#

Expect: “What is the risk of giving an agent many tools?”, “Design guardrails for a support agent that can issue refunds”, and “What would you do before running SQL that a model wrote?”

Strong answers put the hard guarantees in permissions, authorization in code and approvals, name the lethal trifecta, and treat model output as untrusted input to the next system. Weak answers rely on a stronger system prompt, hand the agent tools “just in case”, or trust a user id that the model supplied. Which control to emphasise is judgment, not data.

Practice Questions#

These are generated practice questions, not reported from any company.

• Basic: What is the lethal trifecta, and how would you split an email assistant to avoid it?
• Intermediate: Why does parameterising a SQL query not make a model-written statement safe?
• Intermediate: Which actions would you put behind human approval for a refund agent, and which would you let run?
• Advanced: How does least privilege reduce the blast radius of a successful injection?

Activity#

Use the malicious ticket you wrote in the previous lesson, or write a new one that tries to trigger a refund to the wrong account.

1. Assume it fools the model completely. List the tool calls the agent could make.
2. For each call, write the permission, limit or approval rule that your executor enforces in code.
3. Mark which calls you would let run without a human, and say why the cost of a mistake is acceptable.

Reflect: which of your controls would still hold if the model followed every word of the ticket?

Homework#

Pick a hypothetical agent. List its full tool inventory, and for each tool mark whether it touches private data, reads untrusted content or can send data outward. Cross out every tool the task does not need. If any path still has all three marks, redesign it by splitting the agent or adding an approval step.

Quick Revision#

• Blast radius is what a hijacked agent can reach. Shrink it with least privilege, authorization in code and human approval.
• Excessive agency has three causes: too many tools, too much permission, too much autonomy.
• Avoid private data plus untrusted content plus an outbound channel in one agent.
• Approve irreversible and external actions, not every action.
• Model output is untrusted input: validate against a schema, encode for the destination, use a read-only database role, sandbox code.
• Keep secrets and authorization logic in code, not in the system prompt.

Check yourself#

• Checkpoint question: An agent reads emails and can send emails. Which controls would limit the damage of a successful injection, and where is each one enforced?
• Independent re-test: The same assistant now also writes SQL from user questions. What new risks appear and what controls do you add?
• Common misconception to avoid: “Parameterised queries make model-written SQL safe”, and “a hidden system prompt is a safe place for an API key”.

Related#

• Guardrails, safety and prompt injection
• Agentic patterns: ReAct, planning and memory
• Model Context Protocol
• Designing content moderation and guardrail systems