CachePrune: Defending LLMs Against Indirect Prompt Injection via KV-Cache Editing
Researchers have introduced CachePrune, a novel defense mechanism designed to protect Large Language Models (LLMs) from indirect prompt injection attacks. These attacks exploit the inability of LLMs to distinguish between data and instructions within a prompt context, leading to unintended compliance with malicious inputs. CachePrune addresses this vulnerability by identifying and pruning specific neurons associated with instruction-following during the KV cache encoding phase. This process steers the model to interpret the context strictly as data rather than executable commands. The method employs a neural attribution mechanism guided by a preferential attribution loss, which is theoretically linked to the Direct Preference Optimization (DPO) objective. By leveraging observed triggering effects in instruction-following, the approach enhances attribution fidelity without altering prompt formatting or adding test-time overhead. Experimental results demonstrate that CachePrune significantly reduces attack success rates while maintaining the model's capacity to follow legitimate user instructions. This development represents a significant advancement in AI security, offering a robust solution to a critical vulnerability in modern language models without compromising performance or usability.
Wire timeline
CachePrune: Defending LLMs Against Indirect Prompt Injection via KV-Cache Editing
Researchers have introduced CachePrune, a novel defense mechanism designed to protect Large Language Models (LLMs) from indirect prompt injection attacks. These attacks exploit the inability of LLMs to distinguish between data and instructions within a prompt context, leading to unintended compliance with malicious inputs. CachePrune addresses this vulnerability by identifying and pruning specific neurons associated with instruction-following during the KV cache encoding phase. This process steers the model to interpret the context strictly as data rather than executable commands. The method employs a neural attribution mechanism guided by a preferential attribution loss, which is theoretically linked to the Direct Preference Optimization (DPO) objective. By leveraging observed triggering effects in instruction-following, the approach enhances attribution fidelity without altering prompt formatting or adding test-time overhead. Experimental results demonstrate that CachePrune significantly reduces attack success rates while maintaining the model's capacity to follow legitimate user instructions. This development represents a significant advancement in AI security, offering a robust solution to a critical vulnerability in modern language models without compromising performance or usability.
cs.AI updates on arXiv.org