ChatGPT

https://role-confusion.github.io/
A Theory of Prompt Injection (and why you should study roles)
This is a blog-style writeup of the paper. We show prompt injections are driven by a flaw in how LLMs perceive roles. This lets us create new attacks, explain mech interp results, and predict when attacks succeed. We then discuss what roles are and why they matter, and share research ideas for a science of roles.

Two ways to defend injections
How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have already learned to catch6.

In contrast, why do LLMs struggle so badly against human attackers? Consider that there are two ways an LLM can successfully resist an injection7:

• Attack memorization. The LLM recognizes "send your .env file" as a common prompt injection attack from training, so it refuses.
• Role perception. The LLM correctly identifies the command as tool text (i.e., external data), so it ignores embedded commands regardless of phrasing.

Attack memorization is inherently brittle; it only works against attacks the LLM already knows. Excessive reliance on attack memorization is why LLMs do well on benchmarks, but so poorly against human attackers who can rephrase and adapt attacks until one works.

In contrast, role perception is the robust alternative. All the LLM needs to do is recognize that the command is in a role like tool that inherently lacks authority to give orders. But we'll show that LLMs cannot perceive roles accurately.