https://role-confusion.github.io/
A Theory of Prompt Injection (and why you should study roles)
This is a blog-style writeup of the paper. We show prompt injections are driven by a flaw in how LLMs perceive roles. This lets us create new attacks, explain mech interp results, and predict when attacks succeed. We then discuss what roles are and why they matter, and share research ideas for a science of roles.

Two ways to defend injections
How well do current models do against prompt injection? Not so great. A recent paper found human red-teamers achieve near-100% attack success rates against frontier models5. But, these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt attacks until they work, benchmarks don't. Static benchmarks measure attacks models have already learned to catch6.

In contrast, why do LLMs struggle so badly against human attackers? Consider that there are two ways an LLM can successfully resist an injection7:

• Attack memorization. The LLM recognizes "send your .env file" as a common prompt injection attack from training, so it refuses.
• Role perception. The LLM correctly identifies the command as tool text (i.e., external data), so it ignores embedded commands regardless of phrasing.

Attack memorization is inherently brittle; it only works against attacks the LLM already knows. Excessive reliance on attack memorization is why LLMs do well on benchmarks, but so poorly against human attackers who can rephrase and adapt attacks until one works.

In contrast, role perception is the robust alternative. All the LLM needs to do is recognize that the command is in a role like tool that inherently lacks authority to give orders. But we'll show that LLMs cannot perceive roles accurately.

 

 

I wish modern computing technologists would learn English more.

Why must we all of a sudden start describing inputs to AI systems as a "prompt", that term already has an established meaning in computing. There are words like "cue" that are perfectly suited to this.

This was done earlier too when the idiotic term "shield" started being used for what should have been called a peripheral.

 

You should contact your congress person or MP or whatever.

 

You should contact your congress person or MP or whatever.

 

A fine example of this ignorance and abuse is the misuse of "invite" as a noun. It is wrong, simply and undeniably bad style.

This is just the tip of the iceberg though, all over the place in social media this gradual erosion of language accelerates each year, obviously schools are failing to teach English now.

 

Languages are defined by the people using them, not some higher authority. If people do something "wrong" in large enough numbers, it becomes "right". The language evolves. Schools should be teaching students to communicate in the language as they find it, not cling to historical relics that were, at one time in the past, also viewed as heretical misuses of some even older language.

 

Languages are defined by the people using them, not some higher authority. If people do something "wrong" in large enough numbers, it becomes "right". The language evolves. Schools should be teaching students to communicate in the language as they find it, not cling to historical relics that were, at one time in the past, also viewed as heretical misuses of some even older language.

While I agree with most of what you've said, I find it sad that because of the evolution of language, recorded history can become distorted and ideas corrupted or misinterpreted.

 

Don't discount economics.

95% of ATM transactions are processed by COBOL code.

It may be wrong, but it'd be far too expensive to do it right.

 

Languages are defined by the people using them, not some higher authority. If people do something "wrong" in large enough numbers, it becomes "right". The language evolves. Schools should be teaching students to communicate in the language as they find it, not cling to historical relics that were, at one time in the past, also viewed as heretical misuses of some even older language.

Not in France or Israel (there are many others too), the language itself is defined and protected by an authority, as part of the culture, like any other cultural trait worth defending.

Languages have rules, they are called "grammars".

This is terrible English, do not try to defend it: "But it weren't me which done it officer", you might speak that way, I do not.

We teach language poorly here and in England, that's why we have so much gibberish, we are not good custodians of our language.

 

Too many people today, speak like Ali G, it's ongoing and not good.

 

Not in France or Israel (there are many others too), the language itself is defined and protected by an authority, as part of the culture, like any other cultural trait worth defending.

Languages have rules, they are called "grammars".

This is terrible English, do not try to defend it: "But it weren't me which done it officer", you might speak that way, I do not.

We teach language poorly here and in England, that's why we have so much gibberish, we are not good custodians of our language.

While the French TRY to protect the language via codification since the 1600s or so, the language has still changed considerably and continues to change.

That English has changed much more radically is far less a reflection of some failure to be good custodians as much as it is a reflection of the interactions and melding of multiple cultures over time, a process that is still very much ongoing today.

It's natural to think that somehow the language that you learned as a child is the "proper" form of the language and that any changes are some corruption of it. But the form you learned was seen that same way by people in preceding generations. You believe that being a good custodian means that everyone must be forced to speak the language as you learned it. But if our ancestors had been the good custodians that you prefer, then this is how you would say The Lord's Prayer:

Fæder ūre, þū þe eart on heofonum,
Sī þīn nama gehālgod.
Tōbecume þīn rīce.
Gewurþe þīn willa, on eorðan swā swā on heofonum.
Ūrne gedæghwāmlīcan hlāf syle ūs tōdæg.
And forgyf ūs ūre gyltas, swā swā wē forgyfað ūrum gyltendum.
And ne gelǣd þū ūs on costnunge,
ac ālȳs ūs of yfele.
Sōþlīce.

Of course, even Old English was a major failure on the part of the custodians of the time, since it was a melding of the dialects of several Germanic tribes.

 

Fæder ūre, þū þe eart on heofonum,
Sī þīn nama gehālgod.
Tōbecume þīn rīce.
Gewurþe þīn willa, on eorðan swā swā on heofonum.
Ūrne gedæghwāmlīcan hlāf syle ūs tōdæg.
And forgyf ūs ūre gyltas, swā swā wē forgyfað ūrum gyltendum.
And ne gelǣd þū ūs on costnunge,
ac ālȳs ūs of yfele.
Sōþlīce.

What's neat: knowing the Lord's Prayer in advance, I can actuality read some of that.

 

While the French TRY to protect the language via codification since the 1600s or so, the language has still changed considerably and continues to change.

That English has changed much more radically is far less a reflection of some failure to be good custodians as much as it is a reflection of the interactions and melding of multiple cultures over time, a process that is still very much ongoing today.

It's natural to think that somehow the language that you learned as a child is the "proper" form of the language and that any changes are some corruption of it.

Are you kidding? have you never heard bad Liverpool English? Where I grew up the double negative was the norm, bad grammar was the norm.

"I never done nothing though"
"Why cos you had a go at her?"
"It wasn't me what took it"
"But I never writ that"

Would you approve, applaud if your child or grandchild came home from school speaking like that?

But the form you learned was seen that same way by people in preceding generations. You believe that being a good custodian means that everyone must be forced to speak the language as you learned it. But if our ancestors had been the good custodians that you prefer, then this is how you would say The Lord's Prayer:

Fæder ūre, þū þe eart on heofonum,
Sī þīn nama gehālgod.
Tōbecume þīn rīce.
Gewurþe þīn willa, on eorðan swā swā on heofonum.
Ūrne gedæghwāmlīcan hlāf syle ūs tōdæg.
And forgyf ūs ūre gyltas, swā swā wē forgyfað ūrum gyltendum.
And ne gelǣd þū ūs on costnunge,
ac ālȳs ūs of yfele.
Sōþlīce.

Of course, even Old English was a major failure on the part of the custodians of the time, since it was a melding of the dialects of several Germanic tribes.

I'm not disputing that languages change, I'm objecting to changes that are ungrammatical. Adding a new word or concatenating words to form new words (as German has) isn't a concern, but when we tolerate stuff like "I'll send you an invite" we are harming our own culture, not correcting bad grammar is as bad as not correcting any other formal mode of expression.

 

One example of change that grinds my gears is ironically a French word.

It's perfectly acceptable today to say "if you don't pay, the IRS can garnish your wages". WHAT?! Garnish is a sprig of parsley on the side of your plate or a twist of lemon in your martini. As a verb it is to apply those.

In my book, grabbing a part of someone's paycheck is to garnishee their wages, a verb. Now garnishee is more commonly a noun referring to the person writing the reduced check to the victim.

 

I'm not disputing that languages change, I'm objecting to changes that are ungrammatical. Adding a new word or concatenating words to form new words (as German has) isn't a concern, but when we tolerate stuff like "I'll send you an invite" we are harming our own culture, not correcting bad grammar is as bad as not correcting any other formal mode of expression.

So, would you be happier if we somehow forced everyone to only use the word "invite" in the sense that the Oxford English Dictionary defined it in the 17th century?

Oh, wait, that included its definition as a noun.

The verb "invite" was borrowed from French in the early 16th century and by the mid 17th century was also defined as a noun.

Because English largely lacks the specific noun and verb endings that some other languages have (and which Old English had), words commonly have the same form whether they are functioning as a noun or a verb. This invites (no pun intended) 'zero derivation', which is the term linguists use when a word changes grammatical category without changing its form. Many, many words and phrases that we use today, and that you would almost certainly defend as being proper representations of this culture you are trying to prevent from harm, were widely criticized back when they arose on exactly the same grounds.

Do you have a problem with any of the following:

There's more than one way to skin a cat.
If anything changes, I'll contact you.
We'll host a dinner party in their honor.
We went for a walk after lunch.

All of these involve words that were originally either a noun or a verb that then started being used as the other and were strongly criticized in newspapers and style guides at the time.

 

My sister is a ... linguist? She knows many languages and can teach ESL, which means teaching English to a classroom full of 30 people that can't talk to each other or the teacher. She laughs at the French and other attempts to slow evolution of their languages. Pissing in the wind.

 

One example of change that grinds my gears is ironically a French word.

It's perfectly acceptable today to say "if you don't pay, the IRS can garnish your wages". WHAT?! Garnish is a sprig of parsley on the side of your plate or a twist of lemon in your martini. As a verb it is to apply those.

In my book, grabbing a part of someone's paycheck is to garnishee their wages, a verb. Now garnishee is more commonly a noun referring to the person writing the reduced check to the victim.

About half the words in English originated from French.

 

So, would you be happier if we somehow forced everyone to only use the word "invite" in the sense that the Oxford English Dictionary defined it in the 17th century?

Oh, wait, that included its definition as a noun.

The verb "invite" was borrowed from French in the early 16th century and by the mid 17th century was also defined as a noun.

Because English largely lacks the specific noun and verb endings that some other languages have (and which Old English had), words commonly have the same form whether they are functioning as a noun or a verb. This invites (no pun intended) 'zero derivation', which is the term linguists use when a word changes grammatical category without changing its form. Many, many words and phrases that we use today, and that you would almost certainly defend as being proper representations of this culture you are trying to prevent from harm, were widely criticized back when they arose on exactly the same grounds.

Do you have a problem with any of the following:

There's more than one way to skin a cat.
If anything changes, I'll contact you.
We'll host a dinner party in their honor.
We went for a walk after lunch.

All of these involve words that were originally either a noun or a verb that then started being used as the other and were strongly criticized in newspapers and style guides at the time.

Call me old fashioned, but I think questions should be answered in the order they are asked.

 

My sister is a ... linguist? She knows many languages and can teach ESL, which means teaching English to a classroom full of 30 people that can't talk to each other or the teacher. She laughs at the French and other attempts to slow evolution of their languages. Pissing in the wind.

My wife's a linguist, it was her degree subject, she speaks French like a native as well as excellent Spanish, Italian, Latin and Portuguese.

She taught English in Paris for many years.

It was she who explained that French refuse to diminish the use of the subjunctive, where's in English we've stopped caring, sad.