Evaluating prompt injection in tool-using LLM systems
A jailbreak is a model-behaviour problem. Prompt injection becomes a system-security problem when untrusted data can influence an LLM that reads private context or can take actions.
Start with the system, not the prompt
A useful evaluation begins by documenting what the system can read and what it can do. The same injected instruction has radically different impact in a chat interface, a retrieval system with private documents, and an agent with email, shell, or browser tools.
The threat model should state:
- which inputs are trusted, user-controlled, or retrieved from external sources;
- which secrets, documents, and conversation state the model can access;
- which tools exist and which arguments they accept;
- which actions are reversible, approval-gated, or irreversible;
- what counts as a security failure for this system.
Use an attack corpus, not one clever prompt
The corpus needs to cover how instructions reach the model, not just how they are worded. At minimum, test direct user instructions, indirect instructions in retrieved content, payloads embedded in tool output, multi-turn persistence, instruction obfuscation, and attempts to exfiltrate data or trigger privileged actions.
The evaluation loop
- Define the expected policy. Specify allowed and disallowed outcomes before running attacks.
- Execute deterministically where possible. Pin model, system prompt, tools, configuration, and attack version.
- Capture the full trace. Record messages, retrieved context, tool calls, arguments, outputs, guardrail events, and final response.
- Score behaviour and impact separately. A model following a malicious instruction is not equal to a successful privileged action.
- Replay after every control change. A defence is useful only if it reduces attack success without breaking intended tasks.
A practical scoring model
I separate four outcomes:
- 0. Rejected: the payload has no observable effect.
- 1. Behaviour influenced: the response changes, but no protected data or action is reached.
- 2. Boundary crossed: protected context is exposed or a restricted tool path is attempted.
- 3. Impact achieved: sensitive data is disclosed or a restricted action succeeds.
This keeps model weirdness separate from exploitable system impact and makes regression results easier to compare.
Controls that belong in the system
No single prompt can enforce a security boundary. Useful controls sit around the model:
- treat retrieved content and tool output as untrusted data;
- give tools narrow capabilities and validate structured arguments;
- keep secrets outside model-visible context whenever possible;
- require explicit approval for high-impact or irreversible actions;
- log the complete decision and action trace for replay;
- run regression attacks whenever prompts, models, tools, or retrieval change.
What comes next
The next iteration of this work turns the framework into a reproducible test harness: versioned attack cases, structured traces, repeatable scoring, and comparison of guardrail combinations against the same corpus.