Seed Hijacking of LLM Sampling and Quantum Random Number Defense
Researchers have identified a critical security vulnerability in Large Language Models (LLMs) related to their reliance on deterministic pseudorandom number generators (PRNGs) for autoregressive sampling. The study introduces 'SeedHijack,' a novel backdoor attack that manipulates PRNG outputs to force the selection of attacker-specified tokens without altering the model's underlying logits. In extensive benchmarks involving GPT-2 and various aligned models ranging from 1.5B to 7B parameters, the attack achieved near-perfect success rates, effectively bypassing all tested alignment methods including RLHF and SFT. To counter this threat, the authors propose a defense mechanism utilizing hardware-based Quantum Random Number Generators (QRNG). This solution neutralizes the attack within the evaluated threat model while introducing negligible performance overhead, specifically adding only 0.6% latency and 7.7 MB of memory usage. The findings highlight a previously overlooked supply-chain attack surface in AI systems and offer a practical, deployable solution to enhance the security of LLM sampling processes against sophisticated manipulation attempts.
Wire timeline
Seed Hijacking of LLM Sampling and Quantum Random Number Defense
Researchers have identified a critical security vulnerability in Large Language Models (LLMs) related to their reliance on deterministic pseudorandom number generators (PRNGs) for autoregressive sampling. The study introduces 'SeedHijack,' a novel backdoor attack that manipulates PRNG outputs to force the selection of attacker-specified tokens without altering the model's underlying logits. In extensive benchmarks involving GPT-2 and various aligned models ranging from 1.5B to 7B parameters, the attack achieved near-perfect success rates, effectively bypassing all tested alignment methods including RLHF and SFT. To counter this threat, the authors propose a defense mechanism utilizing hardware-based Quantum Random Number Generators (QRNG). This solution neutralizes the attack within the evaluated threat model while introducing negligible performance overhead, specifically adding only 0.6% latency and 7.7 MB of memory usage. The findings highlight a previously overlooked supply-chain attack surface in AI systems and offer a practical, deployable solution to enhance the security of LLM sampling processes against sophisticated manipulation attempts.
cs.AI updates on arXiv.org