Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents
Researchers have introduced Nautilus Compass, a novel black-box persona drift detection system and agent memory layer designed for production Large Language Model (LLM) coding agents. Addressing the issue where agents forget constraints or confabulate during long sessions, this tool operates entirely at the prompt-text layer using BGE-m3 embeddings and cosine similarity, avoiding the need for model weights required by white-box approaches. Unlike other public memory layers such as Mem0 or Letta, Nautilus Compass does not call an LLM at index time to extract facts, significantly reducing costs to approximately $3.50 for end-to-end reproduction. The system supports various interfaces, including a Claude Code plugin and REST API, and features a Merkle-chained audit log for security. Testing on real Claude Code session traces demonstrated an ROC AUC of 0.83 for drift detection. While its retrieval scores are lower than white-box leaders, it outperforms existing baselines on EverMemBench. The project is open-source under an MIT license, offering a cost-effective solution for maintaining agent consistency in closed API environments like GPT-4 and Claude.
Wire timeline
Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents
Researchers have introduced Nautilus Compass, a novel black-box persona drift detection system and agent memory layer designed for production Large Language Model (LLM) coding agents. Addressing the issue where agents forget constraints or confabulate during long sessions, this tool operates entirely at the prompt-text layer using BGE-m3 embeddings and cosine similarity, avoiding the need for model weights required by white-box approaches. Unlike other public memory layers such as Mem0 or Letta, Nautilus Compass does not call an LLM at index time to extract facts, significantly reducing costs to approximately $3.50 for end-to-end reproduction. The system supports various interfaces, including a Claude Code plugin and REST API, and features a Merkle-chained audit log for security. Testing on real Claude Code session traces demonstrated an ROC AUC of 0.83 for drift detection. While its retrieval scores are lower than white-box leaders, it outperforms existing baselines on EverMemBench. The project is open-source under an MIT license, offering a cost-effective solution for maintaining agent consistency in closed API environments like GPT-4 and Claude.
cs.AI updates on arXiv.org