Datenschutzerklärung|Data Privacy

K. Forster

The Paper "Learning to Validate the Predictions of Black Box Machine Learning Models on Unseen Data" was Accepted for Publication at the HILDA Workshop, Co-Located with the SIGMOD '19

Learning to Validate the Predictions of Black Box Machine Learning Models on Unseen Data, Sergey Redyuk, Sebastian Schelter, Felix Biessmann, Tammo Rukat and Volker Markl . 2019. In Proceedings of Human-in-the-Loop-Data-Analytics Workshop at the ACM SIGMOD (HILDA '19). ACM, New York, NY, USA, Article 4, 4 pages.

When end users apply a machine learning (ML) model on new unlabeled data, it is difficult for them to decide whether they can trust its predictions. Errors or shifts in the target data can lead to hard-to-detect drops in the predictive quality of the model. We therefore propose an approach to assist non-ML experts working with pretrained ML models. Our approach estimates the change in prediction performance of a model on unseen target data. It does not require explicit distributional assumptions on the dataset shift between the training and target data. Instead, a domain expert can declaratively specify typical cases of dataset shift that she expects to observe in real-world data. Based on this information, we learn a performance predictor for pretrained black box models, which can be combined with the model, and automatically warns end users in case of unexpected performance drops. We demonstrate the effectiveness of our approach on two models – logistic regression and a neural network, applied to several real-world atasets.

A preprint-version is available here.

If you want to learn more about SIGMOD/PODS 2019 and the HILDA workshop visit: and