Conference Proceedings

Performance Analysis of Large-scale Distributed Stream Processing Systems on the Cloud

Minh Truong Tri, Aaron Harwood, Richard O Sinnott, Shiping Chen

PROCEEDINGS 2018 IEEE 11TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING (CLOUD) | IEEE | Published : 2018

Abstract

Real-time data processing is often a necessity as it can provide insights that have less value if discovered off-line or after the fact. However, large-scale stream processing systems are non-trivial to build and deploy. While there are many frameworks that allow users to create large-scale distributed systems, there remains many challenges in understanding the performance, cost of deployment and considerations and impact of potential (partial) outages on real-time systems performance. Our work considers the performance of Cloud-based stream processing systems in terms of back-pressure and expected utilization. The performance of an exemplar stream application is explored using different Clo..

View full abstract