Dropbox Optimizes Dash Relevance Judge Using DSPy Framework
Dropbox engineers detailed how they enhanced the relevance judging capability of Dropbox Dash, an AI-powered search tool, by implementing DSPy, an open-source optimization framework. Previously, the team relied on manual prompt engineering for their Large Language Model (LLM) judges, a process that was fragile, prone to regressions during model swaps, and difficult to scale. By adopting DSPy, they transformed this manual task into a measurable, automated optimization loop. The system evaluates judge performance by comparing LLM-generated relevance scores against human annotations using Normalized Mean Squared Error (NMSE) and monitoring structural reliability via JSON formatting success rates. This approach allowed Dropbox to systematically minimize disagreement with human judgments while ensuring output consistency. The implementation resulted in improved task performance, reduced operational costs by enabling the use of smaller models, and increased reliability in production environments. This technical advancement addresses common challenges in scaling LLM-as-a-judge systems, specifically regarding prompt brittleness and the need for robust evaluation metrics across diverse data candidates.
Wire timeline
Dropbox Optimizes Dash Relevance Judge Using DSPy Framework
Dropbox engineers detailed how they enhanced the relevance judging capability of Dropbox Dash, an AI-powered search tool, by implementing DSPy, an open-source optimization framework. Previously, the team relied on manual prompt engineering for their Large Language Model (LLM) judges, a process that was fragile, prone to regressions during model swaps, and difficult to scale. By adopting DSPy, they transformed this manual task into a measurable, automated optimization loop. The system evaluates judge performance by comparing LLM-generated relevance scores against human annotations using Normalized Mean Squared Error (NMSE) and monitoring structural reliability via JSON formatting success rates. This approach allowed Dropbox to systematically minimize disagreement with human judgments while ensuring output consistency. The implementation resulted in improved task performance, reduced operational costs by enabling the use of smaller models, and increased reliability in production environments. This technical advancement addresses common challenges in scaling LLM-as-a-judge systems, specifically regarding prompt brittleness and the need for robust evaluation metrics across diverse data candidates.
Dropbox Tech Blog