Modern data services need to meet application developers’ demands in terms of scalability and resilience, and also support privacy regulations such as the EU’s GDPR. We outline the main systems challenges of supporting data privacy regulations in the context of large-scale data services, and advocate for causal snapshot consistency to ensure application-level and privacy-level consistency. We present Pods, an extension to the dataflow model that allows external services to access snapshotted operator state directly, with built-in support for addressing the outlined privacy challenges, and summarize open questions and research directions.
QC 20220616
Part of proceedings: ISBN 978-3-030-93662-4; 978-3-030-93663-1