Why prompt injection needs a gate outside the model

The problem in one sentence

Prompt injection happens when an application mixes its own instructions with untrusted content in a single prompt and then treats the model’s response as if there were a robust boundary between the two. The UK National Cyber Security Centre (NCSC) puts it bluntly: current large language models “simply do not enforce a security boundary between instructions and data inside a prompt”.^[1]

OWASP ranks prompt injection first in its 2025 Top 10 for LLM applications. It notes that injected content does not even need to be readable by a human, as long as the model parses it, and that it is unclear whether fool-proof prevention exists.^[3]

Why this is not SQL injection

It is tempting to treat prompt injection like SQL injection, which the industry largely tamed with parameterised queries. The NCSC argues that this comparison is dangerous. Parameterised queries work because the database engine can never interpret the input as an instruction. Inside an LLM there is no such distinction: there is only ever the “next token”. For that reason the NCSC says prompt injection attacks “may never be totally mitigated” in the way SQL injection can be.^[1]

The NCSC suggests a different mental model: the LLM is an “inherently confusable deputy”. A confused deputy is a privileged component that can be coerced into acting for a less-privileged attacker. Classic confused-deputy bugs can be fixed; the NCSC argues an LLM’s confusability cannot, so the goal shifts to reducing the likelihood and the impact of attacks.^[1]

What attackers do with an agent

The impact grows with what the model is allowed to touch. When a system calls tools or APIs based on model output, the NCSC warns, the impact of prompt injection becomes “whatever the worse case scenario would be of giving an attacker direct access to those tools/APIs”.^[1]

NIST’s Center for AI Standards and Innovation (CAISI) calls this agent hijacking: a type of indirect prompt injection in which malicious instructions hidden in data such as an email, a file or a website cause an agent to take unintended, harmful actions. Using the open-source AgentDojo framework,^[8] CAISI added injection tasks for remote code execution, database exfiltration and automated phishing, and reports that it was frequently able to induce the agent to follow the malicious instructions in all three areas.^[2]

Two of CAISI’s findings matter for anyone designing defences. First, a model that resisted previously known attacks did much worse against attacks crafted for it: in a red-teaming exercise the success rate rose from 11% for the strongest baseline attack to 81% for the strongest new one. Second, because model output varies between attempts, repetition pays off for attackers: trying each of five injection tasks 25 times raised the average success rate from 57% to 80%.^[2]

Why filters inside the loop are not enough

Many mitigations try to make the model itself more resistant: marking which parts of the prompt are data, training models to prioritise the developer’s instructions, or detecting injection attempts. The NCSC lists these as active research directions but observes that all of them try to overlay a concept of “instruction” and “data” on a technology that does not distinguish between them.^[1]

It is especially wary of deny-listing, such as blocking text containing “ignore previous instructions”, because there are endless ways to rephrase an attack.^[1] The authors of the CaMeL paper reach a similar conclusion about heuristic defences such as delimiters, prompt sandwiching and fine-tuning: none of them provides any guarantee of security, and they regularly fall short against new attacks in practice.^[6]

What a gate outside the model looks like

The alternative is to stop relying on the model to police itself and put deterministic checks around it. The NCSC’s design advice is that protections should focus on “deterministic (non-LLM) safeguards that constrain the actions of the system, rather than just attempting to prevent malicious content reaching the LLM”.^[1] OWASP’s guidance points the same way:

• Least privilege. Offer an agent only the tools it needs, with the narrowest permissions, and keep API tokens in application code rather than handing them to the model.^[3][4]
• Complete mediation. Enforce authorisation in downstream systems “rather than relying on an LLM to decide if an action is allowed or not”, and validate every request against security policy.^[4]
• Human approval for high-impact actions. Require a person to approve privileged operations before they run.^[3][4]
• An intent gate for tool calls. OWASP’s Top 10 for Agentic Applications describes policy enforcement middleware that treats model output as untrusted and validates intent and arguments before a tool executes, alongside egress allowlists and rate or cost budgets.^[5]

A related rule of thumb quoted by the NCSC: when an LLM processes information from a party, its privileges should drop to that party’s level. An assistant reading email from strangers should not, at that moment, hold privileged tools.^[1]

Evidence that design-level gates work

Research is starting to show what this looks like in practice. Debenedetti and colleagues propose CaMeL, “a protective system layer around the LLM” that keeps an agent secure even when the underlying model is susceptible to attack. A privileged model writes a plan from the trusted user request only; a quarantined model with no tool access parses untrusted data; and a custom interpreter tracks where every value came from and who may read it, checking security policies before each tool call. A call that would, for example, email a confidential document to someone not allowed to read it is blocked or sent to the user for approval.^[6]

On the AgentDojo benchmark the authors report solving 77% of tasks with provable security, against 84% for an undefended system, so the gate costs some usefulness. They are also explicit that prompt injection is “not fully solved”: CaMeL does not address attacks that only change the text shown to a user, it remains open to some side channels, and its policies must be written and maintained by people.^[6]

Beurer-Kellner and colleagues generalise the idea into a set of principled design patterns for agents with provable resistance to prompt injection and analyse the utility and security trade-offs of each.^[7] The NCSC cites both papers as early major publications on mitigating prompt injection through design.^[1]

Reading defence results honestly

Most numbers quoted for prompt-injection defences come from benchmarks, so it helps to know what those benchmarks measure. AgentDojo, the framework used by both CAISI and the CaMeL authors, runs agents that call tools over untrusted data. It ships 97 realistic tasks, such as managing an email client, navigating an e-banking website or making travel bookings, and 629 security test cases. Its authors describe it as an extensible environment for new tasks, defences and adaptive attacks rather than a static test suite.^[8] InjecAgent takes a narrower approach: 1,054 test cases covering 17 user tools and 62 attacker tools, with attacks grouped into direct harm to users and exfiltration of private data. Across 30 agents it found that a ReAct-prompted GPT-4 agent was vulnerable 24% of the time, and that adding a “hacking prompt” to the attacker’s instructions nearly doubled that rate.^[9]

CAISI’s experiments give four reasons to read any single headline figure with care:^[2]

• Fixed attack lists go stale. A model that is robust to previously tested attacks can still fail against attacks developed for it, as the jump from 11% to 81% shows. Evaluations need to be adaptive.
• Averages hide the dangerous cases. An average success rate of 57% across five injection tasks concealed large differences between tasks. Some succeeded well above the average and others much less often. Impact also varies: a hijacked agent sending a harmless email is not the same as one exfiltrating user data. CAISI notes that a low success rate for data exfiltration does not mean the scenario can be ignored.
• One attempt understates the risk. Many frameworks, AgentDojo included, score an attack on a single attempt. Model output varies between runs, so where an attacker can retry cheaply, results based on multiple attempts give a more realistic picture.
• Frameworks need maintenance. CAISI fixed bugs in AgentDojo’s default scenarios and added new injection tasks before relying on it.

For a gate, this suggests a simple discipline. Report results for each high-impact action, such as running code, sending data outside the organisation or moving money, not just one blended score. Re-test with attacks written against the deployed configuration. Count repeated attempts. Measure usefulness as well as security, because a gate that blocks too much has a cost too: CaMeL solved 77% of AgentDojo tasks with provable security, against 84% for an undefended system.^[6][8] AgentDojo’s authors also found that state-of-the-art models fail at many tasks even when no attack is present, so a low utility score is not always the defence’s fault.^[8]

Practical takeaways

1. Assume any content an agent reads may contain instructions, and that some will get through.^[1][3]
2. Decide what the agent may do in code that the model cannot rewrite: tool allowlists, scoped credentials, argument checks and authorisation in downstream systems.^[4][5]
3. Put a human in front of irreversible or high-impact actions.^[4]
4. Test with adaptive and repeated attacks, not only a fixed list, and report results for each high-impact action as well as the average.^[2][8]
5. Log full inputs, outputs and tool calls, and watch for failed tool or API calls, which may reveal an attacker refining their approach.^[1]

None of this makes prompt injection disappear. It changes the question from “can the model be tricked?”, to which the honest answer is yes, to “what can a tricked model actually do?”. A gate outside the model is how that answer is kept small. For the wider picture, see what a model bouncer is.