Security in Tool-Calling Agents: Prompt Injection and Least Privilege

A tool-calling agent doesn't just generate text: it executes actions. That difference completely changes the threat model. A manipulated chatbot says something wrong; a manipulated agent can query data, send it out, or modify records using the credentials you gave it.

This article explains the problem and the defenses that actually work at the architecture level — that is, in the harness, not the prompt — the same line we followed in Part 3 of this series and in our guide on agentic development for software teams.

What is prompt injection (direct and indirect), and why is it worse in an agent than in a chatbot?

Prompt injection happens when untrusted text reaching the model alters its behavior. In a chatbot, the effect usually stays contained to an incorrect response; in a tool-using agent, that same text can turn into a real action: sending an email, querying a database, or deleting a record.

OWASP distinguishes two variants. In direct injection, the user themselves writes the malicious instruction. In indirect injection, the model ingests content from external sources — web pages, files — that carries hidden instructions. For an agent, indirect injection is the more dangerous one: the attacker doesn't need to talk to it directly, only needs the agent to read something the attacker controls.

Simon Willison, who coined the term by analogy with SQL injection, describes it as the problem of mixing trusted and untrusted content in the same context, with no reliable way for the model to weigh an instruction's authority based on where it came from. Anthropic illustrates the typical case: a seemingly harmless email carrying instructions in white text, invisible to the person, ordering the agent to forward confidential messages to an external address while it drafts the replies it was actually asked for.

OWASP adds an important nuance: severity depends on business context and on how much agency the architecture has granted the model. That's where you can actually act.

What is the "lethal trifecta" for AI agents?

It's the combination of three capabilities in the same agent: access to private data, exposure to untrusted content, and the ability to communicate outward. When all three coincide, an attacker can get the agent to read the private data and send it to them.

Willison defined this trifecta in June 2025. Each piece is worth understanding with everyday examples:

• Private data: the whole reason many tools exist, such as customer histories, appointments, or invoices.
• Untrusted content: any text or image a third party can control. An email inbox is the perfect example, because anyone can write to the agent.
• External communication: any channel through which data can leave. Willison warns that it's enough for a tool to be able to make an HTTP request, load an image, or render a link to serve as an exfiltration channel.

The practical consequence is a design question you can apply to every agent: does it combine all three? If the answer is yes, remove one of the legs. Willison also notes that providers usually patch these cases by closing the exfiltration channel, so malicious instructions no longer have a way to carry the stolen data out. When you're the one mixing tools together, nobody makes that fix for you.

Why does MCP increase the risk of prompt injection in an agent?

Because it makes it easy to combine tools from different sources, and each tool you add can contribute one leg of the trifecta. Willison explains that the Model Context Protocol encourages mixing tools that access private data, that can host malicious content, and that can communicate externally — often all inside the same tool.

His example is GitHub's official MCP server: it could read public issues, which an attacker could have opened, access private repositories, and create pull requests in a way that exfiltrated that data. All three legs sitting inside a single tool.

For MCP tool security, OpenAI's guide recommends keeping approvals turned on, so a person reviews and confirms every operation, both reads and writes. Before connecting any MCP server, treat it like any other privileged dependency: what does it read, what outside content does it ingest, and through what channel can information leave?

Can prompt injection be prevented entirely?

No, at least not today. None of the primary sources consulted claims the problem is solved, and several say the opposite explicitly.

• OWASP acknowledges that, given the stochastic nature of the models, it's unclear whether foolproof prevention methods even exist, and instead proposes measures that mitigate impact.
• Anthropic published in November 2025 that, with Claude Opus 4.5 in browser use, an internal adaptive attacker with 100 attempts per environment achieved a 1% success rate. Anthropic itself clarifies this remains a significant risk and that the problem is not solved.
• OpenAI, in its safety guide for building agents, states that even with mitigations in place, agents can still make mistakes or be fooled, and that sandboxing and structured outputs reduce risk but don't eliminate it.
• Willison distrusts "guardrail" products that promise to catch 95% of attacks: in web security, that percentage is a failing grade.

This is where the design principle behind the whole article comes from: assume some injection will succeed, and design so the possible damage stays small. It's the same idea we covered in Part 3: security has to live in the harness, not in the hope that the model will behave.

What is the principle of least privilege for an agent, and how is it applied?

It means giving the agent only the functions, permissions, and autonomy strictly necessary for its task. OWASP covers this as the Excessive Agency risk and names three root causes: excessive functionality, excessive permissions, and excessive autonomy.

Applied to a real agent, this becomes concrete controls implemented outside the model:

OWASP offers very concrete examples. An agent that recommends products only needs read access to the products table, and that restriction should be enforced through the database identity's own permissions, not a prompt instruction. An assistant that summarizes emails doesn't need the ability to send them: if the plugin ships with that capability anyway, it's one too many. With that design, a malicious email ordering the agent to forward the whole inbox simply has no tool to do it with.

Logging and rate limiting don't prevent abuse, but they shorten how long it lasts and make it visible.

How do you separate untrusted content from the agent's instructions?

Never place untrusted content in a privileged channel, and use schemas to restrict what can pass from one step to the next. Two concrete measures come straight out of the OWASP and OpenAI guides.

1. Don't inject untrusted variables into higher-authority messages. OpenAI explains that developer messages take priority over user and assistant messages, so placing untrusted input there hands the attacker maximum control. Untrusted input should travel in user messages.
2. Use structured outputs between steps. If only enumerated values or fixed-schema fields can flow between two stages of the pipeline, you eliminate the free-text channels an injection could use to smuggle in instructions or data. OpenAI recommends extracting only specific, validated fields from external input.

OWASP adds clearly identifying and segregating external content to limit its influence.

An illustrative example of the contract between an agent that reads external content and another that executes actions:

import { z } from "zod";

// Illustrative example: only typed fields ever reach the executor agent.
const ExtractedData = z.object({
  productId: z.string().uuid(),
  priceFound: z.number().positive(),
  inStock: z.boolean(),
});

Numeric, boolean, or fixed-format fields can't carry sentences. Any free-text field can, so it's worth constraining those with enums, maximum lengths, or formats. These measures reduce the risk, but don't eliminate it, and it's worth saying that plainly.

When should a human step in for a tool-using agent?

Before any irreversible action, any action that sends information outside the system, or any action affecting third parties: send, delete, pay, publish. Not everything needs approval, and that's one of the ideas from Part 1: human-in-the-loop is reserved for sensitive actions.

OWASP gives the example of an app that publishes content on social media: the approval routine belongs in the extension that implements the publish operation. OpenAI is stricter for MCP tools: it recommends keeping approvals on so a person reviews and confirms every operation, both reads and writes.

A design criterion of our own: approval only protects if the approver sees the real action, with the actual recipient and concrete data, not a summary written by the same model that may have been manipulated. An illustrative example of the difference:

# Summary drafted by the model (not trustworthy):
  "Should I archive the old server emails?"

# Real call the harness intercepts and shows as-is:
tool: delete_records
args: { table: "customers", all: true }

Authorize? [Yes] [No]

Someone who only reads the summary approves what looks like harmless archiving; someone who reads the real call catches a full wipe. Approval has to be rendered from the validated parameters the function will actually receive, not from the model's own text.

How do you test that an agent resists prompt injection?

Treat the model as an untrusted user, and test that your trust boundaries hold even when the model fails. OWASP recommends periodic adversarial testing and attack simulations with that mindset, and OpenAI proposes evaluations and trace grading to see where the agent goes wrong.

A minimum checklist, before registering any new tool, whether through MCP or native function calls:

1. What private data can this tool read?
2. What third-party content enters its context?
3. Through what channel can information leave? Can it reach arbitrary domains, or only an allowlist?
4. What's the worst thing it can do if the model obeys a malicious instruction?
5. Is authorization checked in code, outside the model? Does it validate the authenticated user, or does it trust an identifier the model proposes in its own parameters?
6. If it only reads data, does it connect with read-only permissions at the database engine level?
7. Is every call logged — timestamp, caller, tool, arguments, and result — for auditing?
8. Are there tests that inject malicious input into the parameters and verify the code layer rejects them?

And one practical test: build a small set of malicious inputs per tool and verify, deterministically, that the authorization layer rejects them even when the model proposes them. If the defense depends on the model refusing, it isn't a defense.

Conclusion

Prompt injection isn't eliminated; it's contained. Every tool you give an agent expands what a malicious text can achieve, so the design work is about giving it less: fewer functions, fewer permissions, less autonomy, and above all, never all three legs of the trifecta at once. It's the natural continuation of the earlier parts of this series: governing autonomy, controlling execution, and treating the harness as the place where security lives.

Frequently asked questions

Is prompt injection the same thing as a jailbreak?

They're related, but there isn't full consensus on how to name them. OWASP treats jailbreaking as a form of prompt injection in which the attacker gets the model to ignore its safety protocols. Willison considers them distinct problems: injection exploits mixing trusted and untrusted content in the same context, while jailbreaking aims to get the model to say something it shouldn't. For a tool-using agent, injection is the one that matters most.

Is it enough to tell the model in the prompt to ignore malicious instructions?

No. OWASP recommends constraining model behavior in the prompt as one more measure, but not as the sole defense. Willison asks how much confidence you can really have that such a safeguard will always hold, given the infinite number of ways to phrase a malicious instruction.

Are guardrails and detection classifiers enough on their own?

They're a useful layer, but shouldn't be the only one. Anthropic combines training, classifiers, and human red-teaming and still states the problem isn't solved. OpenAI describes its own guardrails as a first line of defense, not a foolproof solution.

What is excessive agency in an AI agent?

It's the vulnerability that enables harmful actions in response to unexpected, ambiguous, or manipulated model outputs, regardless of the underlying cause. OWASP identifies excessive functionality, excessive permissions, or excessive autonomy as the typical root causes.

This is the fourth part of a series on agentic system architecture. The earlier pieces cover autonomy versus execution authority, tool calling, idempotency, and observability, and model versus harness. It also continues the thread from our guide on agentic development for software teams.

Technical sources

• LLM01:2025 Prompt Injection, OWASP Gen AI Security Project: genai.owasp.org
• LLM06:2025 Excessive Agency, OWASP Gen AI Security Project: genai.owasp.org
• The lethal trifecta for AI agents, Simon Willison (Jun 16, 2025): simonwillison.net
• Mitigating the risk of prompt injections in browser use, Anthropic (Nov 24, 2025): anthropic.com
• Safety in building agents, OpenAI API Docs: developers.openai.com