Efficient Ensemble Selection from Binary and Pairwise Feedback
This research paper addresses the challenge of efficiently selecting small, high-performing ensembles of AI systems, a process often hindered by costly model calls and evaluations. The authors frame the selection problem as a distributional variant of multiwinner voting, where tasks are drawn from an unknown domain distribution and a committee's value is determined by its best-performing member. The study analyzes two feedback types: binary (correct/incorrect outcomes) and pairwise (preference comparisons). For binary feedback, the team designs a failure-conditioned greedy algorithm that maintains standard performance guarantees while reducing query costs. In the pairwise setting, they demonstrate that while full-information optimization allows for a PTAS, it lacks an EPTAS under Gap-ETH. To address this, they propose a weighted ordinal coverage relaxation that supports a greedy oracle, converting results back into robust guarantees via auditing or minimax wrappers. Small-scale experiments with Large Language Models (LLMs) validate the predicted query savings and highlight the importance of complementarity in committee selection, offering a theoretical and practical framework for optimizing AI system deployments.
Wire timeline
Efficient Ensemble Selection from Binary and Pairwise Feedback
This research paper addresses the challenge of efficiently selecting small, high-performing ensembles of AI systems, a process often hindered by costly model calls and evaluations. The authors frame the selection problem as a distributional variant of multiwinner voting, where tasks are drawn from an unknown domain distribution and a committee's value is determined by its best-performing member. The study analyzes two feedback types: binary (correct/incorrect outcomes) and pairwise (preference comparisons). For binary feedback, the team designs a failure-conditioned greedy algorithm that maintains standard performance guarantees while reducing query costs. In the pairwise setting, they demonstrate that while full-information optimization allows for a PTAS, it lacks an EPTAS under Gap-ETH. To address this, they propose a weighted ordinal coverage relaxation that supports a greedy oracle, converting results back into robust guarantees via auditing or minimax wrappers. Small-scale experiments with Large Language Models (LLMs) validate the predicted query savings and highlight the importance of complementarity in committee selection, offering a theoretical and practical framework for optimizing AI system deployments.
cs.AI updates on arXiv.org