Stanford Researchers Introduce Domino for Detecting Systematic ML Errors

The Stanford Machine Learning Group has introduced Domino, a novel approach designed to discover systematic errors made by machine learning models. While many models achieve high overall accuracy, they often underperform on specific data subsets known as slices, which are groups of samples sharing common characteristics. Identifying these underperforming slices is critical for safety-critical applications, such as medical diagnostics, where hidden biases can lead to severe consequences. For instance, lung collapse detection models may fail on images lacking chest drains if trained incorrectly. Domino addresses the challenge of finding these hidden slices in unstructured data like images and videos by using cross-modal embeddings. The method automatically identifies semantically meaningful subgroups where model performance is significantly worse than average. Alongside Domino, the researchers present a framework for quantitatively evaluating slice discovery methods. This innovation aims to help practitioners make informed deployment decisions, debug models, and improve robustness through targeted dataset updates or optimization techniques, ultimately enhancing fairness and safety in AI systems.