Mechanistic Evaluation of Defense Effectiveness Against Persistent Memory Attacks on Stateful LLM Agents
A new study published on arXiv systematically evaluates the effectiveness of six defense mechanisms against persistent memory attacks on stateful Large Language Model (LLM) agents. These attacks involve injecting malicious instructions via RAG-retrieved documents, which are stored in memory and executed in later sessions. The research tested nine open-source models across 5,040 runs. Results indicate that input-level and retrieval-level filtering defenses fail significantly, showing attack success rates (ASR) of 88-89%, statistically similar to the undefended baseline. Prompt Hardening showed partial improvement but remained largely ineffective. However, a memory-layer defense called 'Memory Sandbox' reduced ASR to 0% for eight out of nine models by removing the recall capability required for the attack. Notably, one reasoning model exhibited an inverted response, jumping from 0% to 100% ASR under this defense due to forced pathway changes. The study provides the first comprehensive characterization of why specific defense classes fail, offering critical insights for securing LLM architectures against delayed-trigger threats without imposing utility costs.
Wire timeline
Mechanistic Evaluation of Defense Effectiveness Against Persistent Memory Attacks on Stateful LLM Agents
A new study published on arXiv systematically evaluates the effectiveness of six defense mechanisms against persistent memory attacks on stateful Large Language Model (LLM) agents. These attacks involve injecting malicious instructions via RAG-retrieved documents, which are stored in memory and executed in later sessions. The research tested nine open-source models across 5,040 runs. Results indicate that input-level and retrieval-level filtering defenses fail significantly, showing attack success rates (ASR) of 88-89%, statistically similar to the undefended baseline. Prompt Hardening showed partial improvement but remained largely ineffective. However, a memory-layer defense called 'Memory Sandbox' reduced ASR to 0% for eight out of nine models by removing the recall capability required for the attack. Notably, one reasoning model exhibited an inverted response, jumping from 0% to 100% ASR under this defense due to forced pathway changes. The study provides the first comprehensive characterization of why specific defense classes fail, offering critical insights for securing LLM architectures against delayed-trigger threats without imposing utility costs.
cs.AI updates on arXiv.org