MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
Researchers have introduced MemQ, a novel framework designed to enhance the episodic memory capabilities of Large Language Model (LLM) agents. Unlike current methods that evaluate memory retrieval in isolation, MemQ accounts for dependency chains by applying TD(lambda) eligibility traces to memory Q-values. This approach propagates credit backward through a provenance Directed Acyclic Graph (DAG), which records memory retrieval history during new memory creation. The system replaces temporal distance with structural proximity, where credit weight decays based on DAG depth. Formalized as an Exogenous-Context Markov Decision Process, MemQ decouples task streams from memory stores. Extensive evaluations across six diverse benchmarks, including OS interaction, code generation, and multimodal reasoning, demonstrate that MemQ achieves superior success rates in generalization and runtime learning. The framework shows significant performance gains, up to 5.7 percentage points, particularly in multi-step tasks involving deep provenance chains, while offering principled guidance for parameter selection in future AI agent development.
Wire timeline
MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
Researchers have introduced MemQ, a novel framework designed to enhance the episodic memory capabilities of Large Language Model (LLM) agents. Unlike current methods that evaluate memory retrieval in isolation, MemQ accounts for dependency chains by applying TD(lambda) eligibility traces to memory Q-values. This approach propagates credit backward through a provenance Directed Acyclic Graph (DAG), which records memory retrieval history during new memory creation. The system replaces temporal distance with structural proximity, where credit weight decays based on DAG depth. Formalized as an Exogenous-Context Markov Decision Process, MemQ decouples task streams from memory stores. Extensive evaluations across six diverse benchmarks, including OS interaction, code generation, and multimodal reasoning, demonstrate that MemQ achieves superior success rates in generalization and runtime learning. The framework shows significant performance gains, up to 5.7 percentage points, particularly in multi-step tasks involving deep provenance chains, while offering principled guidance for parameter selection in future AI agent development.
cs.AI updates on arXiv.org