Berkeley AI Researchers Propose StruQ and SecAlign to Defend LLMs Against Prompt Injection
Researchers from the Berkeley Artificial Intelligence Research (BAIR) blog have introduced two new defense mechanisms, StruQ and SecAlign, to combat prompt injection attacks in Large Language Models (LLMs). Prompt injection, identified by OWASP as the top threat to LLM applications, occurs when untrusted data contains malicious instructions that override trusted system prompts. The researchers attribute this vulnerability to the lack of separation between prompts and data, and the tendency of LLMs to follow any detected instruction. To address this, they propose a Secure Front-End that uses special tokens to delimit data, ensuring clear separation. StruQ employs structured instruction tuning to teach models to ignore injected commands, while SecAlign utilizes preference optimization to favor responses to intended instructions over malicious ones. These methods reportedly reduce the success rate of optimization-free attacks to nearly 0% and lower optimization-based attack success rates to under 15%, significantly outperforming previous state-of-the-art defenses without adding computational or labor costs.
Wire timeline
Berkeley AI Researchers Propose StruQ and SecAlign to Defend LLMs Against Prompt Injection
Researchers from the Berkeley Artificial Intelligence Research (BAIR) blog have introduced two new defense mechanisms, StruQ and SecAlign, to combat prompt injection attacks in Large Language Models (LLMs). Prompt injection, identified by OWASP as the top threat to LLM applications, occurs when untrusted data contains malicious instructions that override trusted system prompts. The researchers attribute this vulnerability to the lack of separation between prompts and data, and the tendency of LLMs to follow any detected instruction. To address this, they propose a Secure Front-End that uses special tokens to delimit data, ensuring clear separation. StruQ employs structured instruction tuning to teach models to ignore injected commands, while SecAlign utilizes preference optimization to favor responses to intended instructions over malicious ones. These methods reportedly reduce the success rate of optimization-free attacks to nearly 0% and lower optimization-based attack success rates to under 15%, significantly outperforming previous state-of-the-art defenses without adding computational or labor costs.
The Berkeley Artificial Intelligence Research Blog