AI agent security: a practical threat model

Start with the threat model, not the model

For each agent, write down three things: what data it can read, what untrusted content can reach its context, and what actions it can take – especially actions that send data somewhere. The danger is the combination. An agent that can read private data, ingest untrusted content, and communicate externally can be turned into an exfiltration tool by a single crafted message. This combination is sometimes called the lethal trifecta, and removing any one leg breaks the attack.

That framing is more useful than any checklist, because it tells you where to cut. A support assistant that reads customer tickets (untrusted) and account data (private) should not also be able to send email to arbitrary addresses or fetch arbitrary URLs.

Prompt injection: why prevention is mostly containment

Direct injection is the user typing “ignore your instructions”. Indirect injection is worse: instructions embedded in content the agent processes – a hidden line in a web page, white text in a PDF, a comment in a pull request, a field in a CRM record, the output of another tool.

There’s no complete fix at the model layer today. Classifiers and careful system prompts reduce the rate; they don’t bring it to zero, and attackers iterate faster than you retrain. Prompt injection prevention in practice means designing so that a successful injection can’t do much:

• Treat every tool result as untrusted input, including results from your own systems if users can write to them.
• Separate planning from untrusted data where you can. A privileged step decides what to do; a quarantined step processes untrusted content and can only return constrained, structured output (an enum, a number, a validated schema) – never free text that flows back into the planner as instructions.
• Constrain outputs that cause effects. If a tool takes a recipient, restrict it to an allow-list; if it takes a URL, restrict the domain.
• Block silent exfiltration channels. Don’t auto-render markdown images or links from model output pointing at arbitrary domains – a URL with data in its query string is a classic leak. Apply a content security policy to anywhere model output is displayed.
• Detect and log suspected injections for review, but don’t rely on detection as the control.

Tool permission scoping

Every tool is a capability you’re handing to something that can be manipulated. Scope it like you’d scope an API key for an untrusted integration.

• Least privilege per agent, not per platform. A triage agent gets read and label; it doesn’t get delete.
• Act as the user, never as a superuser. Every tool call should run through the same authorization checks as your UI, with the identity of the human who started the session.
• Split read and write tools, and split destructive operations from benign ones. archive_card and delete_project shouldn’t share a tool with a mode flag.
• Validate arguments server-side against a strict schema. The model’s JSON is user input.
• Bound the blast radius: rate limits, maximum records per call, spending caps, and tenant isolation enforced in the data layer.
• Short-lived, narrowly scoped credentials for anything the agent touches downstream.

The confused deputy problem

A confused deputy is a component with more authority than its caller that gets tricked into using that authority on the caller’s behalf. Agents are confused deputies by default: they hold credentials, and they take direction from whatever text lands in their context.

In MCP servers this shows up concretely. A server that proxies to a third-party API with its own static client credentials, and lets any MCP client obtain consent through it, can let one client ride on consent another user granted. The MCP spec addresses the most common versions: servers must validate that tokens were issued for them (audience binding), must not pass client tokens through to upstream APIs, and proxy servers that use a static client ID must obtain user consent for each dynamically registered client. If you’re building one, read the security best-practices section of the spec, not just the transport docs.

MCP server security checklist

MCP servers are where agents meet your systems, so most of the controls land here. For background on the protocol itself, see what is an MCP server.

• OAuth 2.1 with PKCE for remote servers; tokens audience-bound to your server and validated on every request.
• No token passthrough. Downstream calls use the server’s own credentials, scoped to the acting user.
• State handles (the IDs a server mints to carry state between calls, now that MCP has no session IDs) are random and never treated as authentication; bind them to the authenticated user.
• For local stdio servers: don’t run with more filesystem or shell access than the job needs, and never bind a local HTTP server to all interfaces.
• Tool annotations (read-only, destructive) set honestly – and on the client side, never trusted from servers you don’t control.
• Tool descriptions reviewed like code. A malicious or compromised third-party server can put instructions in its descriptions (tool poisoning) or change them after you’ve approved it.
• Pin and review third-party servers the way you’d review a dependency; don’t let users install arbitrary servers into an agent that holds production credentials.
• Input size limits and output truncation to stop context flooding.

Human approval gates

Approval is the control that still works when everything else has been bypassed, so put it where consequences are irreversible: payments and refunds, emails to external recipients, deleting data, merging code, changing permissions, anything touching production infrastructure.

Make approvals meaningful. Show the exact action and arguments, not the model’s summary of them – a summary can be manipulated too. Avoid approval fatigue by not asking about safe, reversible reads; if people click “approve” forty times an hour, the gate has stopped working. For high-volume flows, batch actions into a reviewable plan, or let the agent prepare a draft that a human sends.

Audit logs and monitoring

• Log every tool call with: acting user, agent or client identity, tool name, arguments, result summary, timestamp and the conversation or run ID.
• Keep the model inputs and outputs for a retention period you can justify under your privacy policy – you’ll need them to investigate an incident.
• Alert on anomalies: unusual tool sequences, spikes in data read volume, calls to rarely used destructive tools, repeated authorization failures.
• Make logs tamper-evident and outside the agent’s own reach.
• Run red-team tests – injected instructions in documents, tickets and tool outputs – as part of your eval suite, and rerun them on every prompt or model change.

Where to start

If you’re adding an agent to an existing product, the fastest risk reduction is usually: remove one leg of the trifecta, route every tool through your existing authorization layer, and put an approval gate on the irreversible actions. Then add guardrails and logging. I build agents and custom MCP servers with these controls in from the first sprint, because retrofitting permissions onto an agent that already has broad access is much harder than starting narrow.

Related

• Custom MCP server development for your product
• What is an MCP server?
• LLM guardrails for real product features
• AI agent development for work that actually ships
• Agentic workflows: patterns that hold up in production
• A code audit that tells you what you’ve actually got