Discovering the systematic errors made by machine learning models

Discovering systematic errors with cross-modal embeddings In this blog post, we introduce Domino, a new approach for discovering systematic errors made by machine learning models. We also discuss a framework for quantitatively evaluating methods like Domino. Links: 📄 Paper (ICLR 2022) 🌍 Longer Walkthrough 💻 GitHub 📘 Docs 📒 Google Colab Machine learning models that achieve high overall accuracy often make systematic errors on coherent slices of validation data. What is a slice? A slice is a set of data samples that share a common characteristic. As an example, in large image datasets, photos of vintage cars comprise a slice (i.e. all images in the slice share a common subject). The term slice has a number of synonyms that you might be more familiar with (e.g. subgroup, subpopulation, stratum). These terms are largely interchangeable, but we’ll stick with “slice” throughout this post. We say that a model underperforms on a slice if performance on the data samples in the slice is significantly worse than its overall performance. The search for underperforming slices is a critical, but often overlooked, part of model evaluation. When practitioners are aware of the slices on which …