Datadog and CMU Introduce ARFBench for Time Series Question-Answering
Researchers from Carnegie Mellon University and Datadog have introduced ARFBench, a new benchmark designed to evaluate AI models on Time Series Question-Answering (TSQA) tasks. With system failures costing over a trillion dollars annually, rapid troubleshooting via observability metrics is critical for engineers. ARFBench utilizes real internal telemetry from 63 incidents at Datadog, comprising 750 question-answer pairs that test compositional reasoning across three difficulty tiers. The study reveals that leading Large Language Models (LLMs), Vision-Language Models (VLMs), and Time Series Foundation Models (TSFMs) currently struggle with these complex tasks. To address this, the team developed a hybrid TSFM-VLM model that achieves performance comparable to top frontier models. Furthermore, the research highlights significant complementarity between AI and human experts, as their error profiles differ markedly. By combining model predictions with expert insights, the study establishes a new superhuman frontier for incident response. This work underscores the need for advanced AI capabilities in Site Reliability Engineering (SRE) and provides a robust framework for future developments in anomaly detection and automated system troubleshooting using real-world production data.
Wire timeline
Datadog and CMU Introduce ARFBench for Time Series Question-Answering
Researchers from Carnegie Mellon University and Datadog have introduced ARFBench, a new benchmark designed to evaluate AI models on Time Series Question-Answering (TSQA) tasks. With system failures costing over a trillion dollars annually, rapid troubleshooting via observability metrics is critical for engineers. ARFBench utilizes real internal telemetry from 63 incidents at Datadog, comprising 750 question-answer pairs that test compositional reasoning across three difficulty tiers. The study reveals that leading Large Language Models (LLMs), Vision-Language Models (VLMs), and Time Series Foundation Models (TSFMs) currently struggle with these complex tasks. To address this, the team developed a hybrid TSFM-VLM model that achieves performance comparable to top frontier models. Furthermore, the research highlights significant complementarity between AI and human experts, as their error profiles differ markedly. By combining model predictions with expert insights, the study establishes a new superhuman frontier for incident response. This work underscores the need for advanced AI capabilities in Site Reliability Engineering (SRE) and provides a robust framework for future developments in anomaly detection and automated system troubleshooting using real-world production data.
Machine Learning Blog | ML@CMU | Carnegie Mellon University