Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse

Researchers have identified the underlying mechanisms causing catastrophic collapse in large language models during sequential knowledge editing, particularly with parameter-modifying methods. The study reveals that a model's general abilities are closely linked to the dominant singular directions of pretrained weight matrices, which are highly sensitive to perturbations from repeated edits. To address this, the authors propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace. REVIVE operates by representing parameter updates in the spectral basis of original weights and filtering out components that interfere with protected regions. Extensive experiments across various models and benchmarks demonstrate that REVIVE significantly improves editing efficacy while maintaining general capabilities, even under extreme conditions involving up to 20,000 sequential edits. This work provides both theoretical insights into spectral analysis of neural network weights and a practical solution for robust long-horizon knowledge updating in AI systems.