Interactive Benchmarks: A New Paradigm for Evaluating AI Reasoning

Researchers have proposed 'Interactive Benchmarks,' a novel evaluation framework designed to address the limitations of existing AI reasoning assessments, such as benchmark saturation and subjective preference-based judgments. Published on arXiv, this study argues that true intelligence involves deciding what information to acquire and how to use it effectively. The proposed paradigm assesses models through budgeted multi-turn interactions in two primary settings: Interactive Proofs, where models solve Logic, UI2Html, and Mathematics tasks with objective feedback from a judge; and Interactive Games, which require strategic reasoning to maximize long-term utilities. The results indicate that interactive benchmarks offer a more robust assessment of model intelligence, highlighting significant areas for improvement in interactive scenarios. This approach aims to provide a unified and dynamic method for evaluating the reasoning capabilities of artificial intelligence systems, moving beyond static and potentially contaminated fixed benchmarks.