In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
A new research paper titled 'In-Context Fixation' reveals a critical vulnerability in large language models during few-shot classification tasks. The study demonstrates that when demonstration labels are homogeneous, even if semantically valid, model accuracy collapses to 12% or lower across various architectures including Pythia, Llama, and Qwen. The researchers identify that models treat tokens in the label position as an exhaustive answer vocabulary, leading to 'set-level fixation' where probability mass shifts to demonstrated nonsense tokens rather than semantically correct answers. This finding contradicts latent-concept Bayesian accounts of in-context learning, suggesting instead that output is constrained by vocabulary retrieval from the provided examples. The effect persists in multi-token verbalizers and 4-way classifications. Mechanistic analysis using activation patching localizes this fixation to specific circuit layers, particularly layer 7 in Pythia-1B, confirming an 'encode-then-override' trajectory. These results highlight that models bind outputs to demonstrated token inventories regardless of semantic plausibility, offering new insights into the mechanical interpretability and limitations of current in-context learning mechanisms.
Wire timeline
In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classification
A new research paper titled 'In-Context Fixation' reveals a critical vulnerability in large language models during few-shot classification tasks. The study demonstrates that when demonstration labels are homogeneous, even if semantically valid, model accuracy collapses to 12% or lower across various architectures including Pythia, Llama, and Qwen. The researchers identify that models treat tokens in the label position as an exhaustive answer vocabulary, leading to 'set-level fixation' where probability mass shifts to demonstrated nonsense tokens rather than semantically correct answers. This finding contradicts latent-concept Bayesian accounts of in-context learning, suggesting instead that output is constrained by vocabulary retrieval from the provided examples. The effect persists in multi-token verbalizers and 4-way classifications. Mechanistic analysis using activation patching localizes this fixation to specific circuit layers, particularly layer 7 in Pythia-1B, confirming an 'encode-then-override' trajectory. These results highlight that models bind outputs to demonstrated token inventories regardless of semantic plausibility, offering new insights into the mechanical interpretability and limitations of current in-context learning mechanisms.
cs.AI updates on arXiv.org