Datenschutzerklärung|Data Privacy

K. Forster

"On-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Engines", Paper Accepted for Publication at BTW 2019.

On-the-fly Reconfiguration of Query Plans for Stateful Stream Processing Engines, Adrian Bartnik, Bonaventura Del Monte, Tilmann Rabl, and Volker Markl . 2019. 18. Fachtagung für Datenbanksysteme für Business, Technologie und Web (BTW), Rostock, Germany.

Stream Processing Engines (SPEs) must tolerate the dynamic nature of unbouded data streams and provide means to quickly adapt to fluctuations in the data rate. Many major SPEs however provide very little functionality to adjust the execution of a potentially infinite streaming query at runtime. Each modification requires a complete query restart, which involves an expensive redistribution of the state of a query and may require external systems in order to guarantee correct processing semantics. This results in significant downtimes, which increase the operational cost of those SPEs. We present a modification protocol that enables modifying specific operators as well as the data flow of a running query while ensuring exactly-once processing semantics. We provide an implementation for Apache Flink, which enables stateful operator migration across machines, the introduction of new operators into a running query, and changes to a specific operator based on external triggers. Our results on two benchmarks show that migrating operators for queries with small state is as fast as using the savepoint mechanism of Flink. Migrating operators in the presence of large state even outperforms the savepoint mechanism by a factor of more than 2.3.

A preprint version of the paper is available.