A study on the lifecycle of flaky tests
Wing Lam, Kıvanç Muşlu, Hitesh Sajnani, Suresh Thummalapenta
Abstract
During regression testing, developers rely on the pass or fail outcomes of tests to check whether changes broke existing functionality. Thus, flaky tests, which nondeterministically pass or fail on the same code, are problematic because they provide misleading signals during regression testing. Although flaky tests are the focus of several existing studies, none of them study (1) the reoccurrence, runtimes, and time-before-fix of flaky tests, and (2) flaky tests in-depth on proprietary projects.
Topics & Concepts
Regression testingComputer scienceFocus (optics)Reliability engineeringTest (biology)Code (set theory)Regression analysisRegressionSoftware engineeringStatisticsProgramming languageEngineeringMachine learningSoftwareSoftware developmentMathematicsSoftware constructionPaleontologyBiologyOpticsPhysicsSet (abstract data type)Software Testing and Debugging TechniquesSoftware Engineering ResearchSoftware Reliability and Analysis Research