FLEX: fixing flaky tests in machine learning projects by updating assertion bounds
Saikat Dutta, August Shi, Saša Misailovíc
Abstract
Many machine learning (ML) algorithms are inherently random – multiple executions using the same inputs may produce slightly different results each time. Randomness impacts how developers write tests that check for end-to-end quality of their implementations of these ML algorithms. In particular, selecting the proper thresholds for comparing obtained quality metrics with the reference results is a non-intuitive task, which may lead to flaky test executions.
Topics & Concepts
RandomnessAssertionComputer scienceFLEXTask (project management)ImplementationQuality (philosophy)Machine learningTest (biology)Artificial intelligenceSoftware engineeringProgramming languageEngineeringBiologyEpistemologyTelecommunicationsPhilosophyPaleontologyMathematicsStatisticsSystems engineeringSoftware Testing and Debugging TechniquesAdversarial Robustness in Machine LearningMachine Learning and Data Classification