HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing
Researchers have introduced HoReN, a novel method for editing large language models (LLMs) to update factual knowledge without costly retraining. Existing model editing techniques often suffer from knowledge disruption or performance degradation when handling sequential updates at scale. HoReN addresses these limitations by employing a codebook-based, parameter-preserving approach. It wraps a single MLP layer with a discrete key-value codebook, interpreting entries as both knowledge-memory keys and modern Hopfield stored patterns. The system projects keys and queries onto a unit hypersphere to ensure retrieval is based on angular similarity, eliminating magnitude-driven mismatches. Additionally, it refines queries using damped Hopfield attractor dynamics, allowing paraphrases to converge on correct patterns while leaving unrelated queries undisturbed. Evaluations across benchmarks like ZsRE, WikiBigEdit, and UnKE demonstrate consistent performance gains. Notably, HoReN maintains stable performance above 0.9 even after 50,000 sequential edits, significantly outperforming prior editors that typically degrade before reaching 10,000 edits. This advancement offers a scalable solution for lifelong learning in LLMs, ensuring updated behavior without compromising existing knowledge integrity.
Wire timeline
HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing
Researchers have introduced HoReN, a novel method for editing large language models (LLMs) to update factual knowledge without costly retraining. Existing model editing techniques often suffer from knowledge disruption or performance degradation when handling sequential updates at scale. HoReN addresses these limitations by employing a codebook-based, parameter-preserving approach. It wraps a single MLP layer with a discrete key-value codebook, interpreting entries as both knowledge-memory keys and modern Hopfield stored patterns. The system projects keys and queries onto a unit hypersphere to ensure retrieval is based on angular similarity, eliminating magnitude-driven mismatches. Additionally, it refines queries using damped Hopfield attractor dynamics, allowing paraphrases to converge on correct patterns while leaving unrelated queries undisturbed. Evaluations across benchmarks like ZsRE, WikiBigEdit, and UnKE demonstrate consistent performance gains. Notably, HoReN maintains stable performance above 0.9 even after 50,000 sequential edits, significantly outperforming prior editors that typically degrade before reaching 10,000 edits. This advancement offers a scalable solution for lifelong learning in LLMs, ensuring updated behavior without compromising existing knowledge integrity.
cs.AI updates on arXiv.org