Finding Label and Model Errors in Perception Data With Learned Observation Assertions
Daniel Kang, Nikos Aréchiga, Sudeep Pillai, Peter Bailis, Matei Zaharia
Abstract
ML is being deployed in complex, real-world scenarios where errors have impactful consequences. In these systems, thorough testing of the ML pipelines is critical. A key component in ML deployment pipelines is the curation of labeled training data. Common practice in the ML literature assumes that labels are the ground truth. However, in our experience in a large autonomous vehicle development center, we have found that vendors can often provide erroneous labels, which can lead to downstream safety risks in trained models.
Topics & Concepts
Computer scienceSoftware deploymentComponent (thermodynamics)Pipeline transportKey (lock)Ground truthPerceptionData modelingDownstream (manufacturing)Pipeline (software)Artificial intelligenceData scienceComputer securitySoftware engineeringEngineeringProgramming languagePsychologyOperations managementPhysicsEnvironmental engineeringThermodynamicsNeuroscienceSoftware Testing and Debugging TechniquesMachine Learning and AlgorithmsAdvanced Neural Network Applications