Designing Data-Intensive Applications
By Martin Kleppmann
Shop now
Quite honestly,I’d take things a step further. I would argue that well-designed streaming systems actually provide a strict superset of batch functionality. Modulo perhaps an efficiency delta[1],there should be no need for batch systems as they exist today. And kudos to the Flink folks for taking this idea to heart and building a system that’s all-streaming-all-the-time under the covers,even in “batch” mode; I love it.
The corollary of all this is that broad maturation of streaming systems combined with robust frameworks for unbounded data processing will,in time,allow the relegation of the Lambda Architecture to the antiquity of big data history where it belongs. I believe the time has come to make this a reality. Because to do so,i.e. to beat batch at its own game,you really only need two things:
-
Correctness — This gets you parity with batch. At the core,correctness boils down to consistent storage. Streaming systems need a method for checkpointing persistent state over time (something Kreps has talked about in his Why local state is a fundamental primitive in stream processing post),and it must be well-designed enough to remain consistent in light of machine failures. When Spark Streaming first appeared in the public big data scene a few years ago,it was a beacon of consistency in an otherwise dark streaming world. Thankfully,things have improved somewhat since then,but it is remarkable how many streaming systems still try to get by without strong consistency; I seriously cannot believe that at-most-once processing is still a thing,but it is. To reiterate,because this point is important: strong consistency is required for exactly-once processing,which is required for correctness,which is a requirement for any system that’s going to have a chance at meeting or exceeding the capabilities of batch systems. Unless you just truly don’t care about your results,I implore you to shun any streaming system that doesn’t provide strongly consistent state. Batch systems don’t require you to verify ahead of time if they are capable of producing correct answers; don’t waste your time on streaming systems that can’t meet that same bar. If you’re curious to learn more about what it takes to get strong consistency in a streaming system,I recommend you check out the MillWheel and Spark Streaming papers. Both papers spend a significant amount of time discussing consistency. Given the amount of quality information on this topic in the literature and elsewhere,I won’t be covering it any further in these posts.
-
Tools for reasoning about time — This gets you beyond batch. Good tools for reasoning about time are essential for dealing with unbounded,unordered data of varying event-time skew. An increasing number of modern data sets exhibit these characteristics,and existing batch systems (as well as most streaming systems) lack the necessary tools to cope with the difficulties they impose. I will spend the remainder of this post,and the bulk of the next post,explaining and focusing on this point. To begin with,we’ll get a basic understanding of the important concept of time domains,after which we’ll take a deeper look at what I mean by unbounded,unordered data of varying event-time skew. We’ll then spend the rest of this post looking at common approaches to bounded and unbounded data processing,using both batch and streaming systems.
Event time vs. processing time
Learning Path
Real-Time Data Applications
This Learning Path provides an in-depth tour of technologies used in processing and analyzing real-time data.
Shop now
To speak cogently about unbounded data processing requires a clear understanding of the domains of time involved. Within any data processing system,there are typically two domains of time we care about:
-
Event time,which is the time at which events actually occurred.
-
Processing time,which is the time at which events are observed in the system.
（编辑：应用网_阳江站长网）
【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!
|
