The RERS challenge: towards controllable and scalable benchmark synthesis
Falk Howar, Marc Jasper, Malte Mues, David Schmidt, Bernhard Steffen
Abstract
Abstract This paper (1) summarizes the history of the RERS challenge for the analysis and verification of reactive systems, its profile and intentions, its relation to other competitions, and, in particular, its evolution due to the feedback of participants, and (2) presents the most recent development concerning the synthesis of hard benchmark problems. In particular, the second part proposes a way to tailor benchmarks according to the depths to which programs have to be investigated in order to find all errors. This gives benchmark designers a method to challenge contributors that try to perform well by excessive guessing.
Topics & Concepts
Benchmark (surveying)Computer scienceScalabilityTheory of computationRelation (database)Theoretical computer scienceComputer engineeringIndustrial engineeringAlgorithmData miningDatabaseGeographyGeodesyEngineeringFormal Methods in VerificationSoftware Testing and Debugging TechniquesRadiation Effects in Electronics