Keytone: Silent Data Corruptions at Scale
Harish V. Dixit
Abstract
Silent data corruptions (SDC) in hardware impact computational integrity for large-scale applications. Sources of corruptions include datapath dependencies, temperature variance, and age among other silicon factors. These errors do not leave any record or trace in system logs. As a result, silent errors stay undetected within workloads, and can propagate across the stack to the applications. Silent errors can result in data loss and can require months of debug engineering time. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across device generations. Based on this experience, we determine that reducing silent data corruption requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.