Detecting flaky tests in probabilistic and machine learning applications
Saikat Dutta, August Shi, Rutvik Choudhary, Zhekun Zhang, Aryaman Jain, Saša Misailovíc
Abstract
Probabilistic programming systems and machine learning frameworks like Pyro, PyMC3, TensorFlow, and PyTorch provide scalable and efficient primitives for inference and training. However, such operations are non-deterministic. Hence, it is challenging for developers to write tests for applications that depend on such frameworks, often resulting in flaky tests – tests which fail non-deterministically when run on the same version of code.
Topics & Concepts
Computer scienceProbabilistic logicScalabilityInferenceArtificial intelligenceMachine learningCode (set theory)Programming languageTheoretical computer scienceComputer architectureComputer engineeringOperating systemSet (abstract data type)Software Testing and Debugging TechniquesSoftware Reliability and Analysis ResearchSoftware Engineering Research