Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs

A new research paper published on arXiv investigates the difficulties Large Language Models (LLMs) face in handling context switching during multi-turn conversations. The study highlights that users often refine requests or pivot to new topics, but LLMs frequently miss these shifts, carrying over irrelevant previous context and producing inaccurate responses. The authors constructed synthetic benchmarks based on real-world datasets to simulate various levels of context shift difficulty. They evaluated the zero-shot performance of ten different LLMs, including open-weight, closed-source, and reasoning models. Results indicate that only specific reasoning and strongly instructed models accurately detect topic pivots. In contrast, open-weight models struggle significantly, often retaining stale context despite explicit cues. Furthermore, all tested models exhibited position bias. The paper provides key insights for improving the long-term robustness of multi-turn capabilities in LLMs, emphasizing the need for better context management mechanisms to enhance user interaction accuracy and reliability in complex dialogue scenarios.