Khaos: Dynamically Optimizing Checkpointing for Dependable Distributed Stream Processing
Morgan K. Geldenhuys, B. Pfister, Dominik Scheinert, Lauritz Thamsen, Odej Kao
Abstract
Distributed Stream Processing systems are becoming an increasingly essential part of Big Data processing platforms as users grow ever more reliant on their ability to provide fast access to new results. As such, making timely decisions based on these results is dependent on a system's ability to tolerate failure. Typically, these systems achieve fault tolerance and the ability to recover automatically from partial failures by implementing checkpoint and rollback recovery. However, owing to the statistical probability of partial failures occurring in these distributed environments and the variability of workloads upon which jobs are expected to operate, static configurations will often not meet Quality of Service constraints with low overhead.