Bruce Schneier and Barath Raghavan explore why LLMs struggle with context and judgment and, consequently, are vulnerable to prompt injection attacks. These 'attacks' are cases where LLMs are tricked into doing things that typical guardrails are supposed to prevent them from doing. LLMs are too easily detached from context, and they are present as overly confident in their assertions.
In IEEE Spectrum, Schneier and Raghavan write: "The overconfidence and obsequiousness of LLMs are training choices. The lack of an interruption reflex is a deficiency in engineering. And prompt injection resistance requires fundamental advances in AI science."