流数据处理的博文

[1] One which I propose is not an inherent limitation of streaming systems,but simply a consequence of design choices made in most streaming systems thus far. The efficiency delta between batch and streaming is largely the result of the increased bundling and more efficient shuffle transports found in batch systems. Modern batch systems go to great lengths to implement sophisticated optimizations that allow for remarkable levels of throughput using surprisingly modest compute resources. There’s no reason the types of clever insights that make batch systems the efficiency heavyweights they are today couldn’t be incorporated into a system designed for unbounded data,providing users flexible choice between what we typically consider to be high-latency,higher-efficiency “batch” processing and low-latency,lower-efficiency “streaming” processing. This is effectively what we’ve done with Cloud Dataflow by providing both batch and streaming runners under the same unified model. In our case,we use separate runners because we happen to have two independently designed systems optimized for their specific use cases. Long-term,from an engineering perspective,I’d love to see us merge the two into a single system which incorporates the best parts of both,while still maintaining the flexibility of choosing an appropriate efficiency level. But that’s not what we have today. And honestly,thanks to the unified Dataflow Model,it’s not even strictly necessary; so it may well never happen. (Return)

（编辑：应用网_阳江站长网）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!