How LLMs Are Persuaded: A Few Attention Heads, Rerouted
A new research paper published on arXiv reveals the internal mechanism behind how Large Language Models (LLMs) can be persuaded to abandon factual knowledge. The study identifies a compact causal circuit where a small set of mid-layer attention heads determines the model's output by writing answer options into a low-dimensional polyhedron. Persuasion triggers a discrete latent jump from the correct answer vertex to the targeted one, rather than merely reducing confidence. The researchers demonstrate that decision heads do not reason over evidence but simply copy the token selected by attention. By isolating a rank-one evidence-routing feature controlled by shallower attention heads processing persuasive keywords, the team showed that modifying this feature steers the model's choice, while removing it blocks persuasion. Validated across various open-source LLMs and realistic poisoning scenarios like Generative Engine Optimization, this discovery frames persuasion as a narrow, monitorable circuit. This finding is significant for AI safety, offering potential methods to detect and mitigate manipulation vulnerabilities in language models by monitoring specific attention pathways.
Wire timeline
How LLMs Are Persuaded: A Few Attention Heads, Rerouted
A new research paper published on arXiv reveals the internal mechanism behind how Large Language Models (LLMs) can be persuaded to abandon factual knowledge. The study identifies a compact causal circuit where a small set of mid-layer attention heads determines the model's output by writing answer options into a low-dimensional polyhedron. Persuasion triggers a discrete latent jump from the correct answer vertex to the targeted one, rather than merely reducing confidence. The researchers demonstrate that decision heads do not reason over evidence but simply copy the token selected by attention. By isolating a rank-one evidence-routing feature controlled by shallower attention heads processing persuasive keywords, the team showed that modifying this feature steers the model's choice, while removing it blocks persuasion. Validated across various open-source LLMs and realistic poisoning scenarios like Generative Engine Optimization, this discovery frames persuasion as a narrow, monitorable circuit. This finding is significant for AI safety, offering potential methods to detect and mitigate manipulation vulnerabilities in language models by monitoring specific attention pathways.
cs.AI updates on arXiv.org