Learning to Validate the Predictions of Black Box Classifiers on Unseen Data
Sebastian Schelter, Tammo Rukat, Felix Bießmann
Abstract
Machine Learning (ML) models are difficult to maintain in production settings. In particular, deviations of the unseen serving data (for which we want to compute predictions) from the source data (on which the model was trained) pose a central challenge, especially when model training and prediction are outsourced via cloud services. Errors or shifts in the serving data can affect the predictive quality of a model, but are hard to detect for engineers operating ML deployments.
Topics & Concepts
Black boxComputer scienceMachine learningCloud computingArtificial intelligenceData modelingTraining setQuality (philosophy)Big dataData miningDatabaseOperating systemEpistemologyPhilosophyAnomaly Detection Techniques and ApplicationsMachine Learning and Data ClassificationAdversarial Robustness in Machine Learning