Stream processing has emerged from the confluence of advances in data management, parallel and distributed computing, signal processing, statistics, data mining, and optimization theory.

Stream processing is an intuitive computing paradigm where data is consumed as it is generated, computation is performed at wire speed, and results are immediately produced, all within a continuous cycle. The rise of this computing paradigm was the result of the need to support a new class of applications. These analytic-centric applications are focused on extracting intelligence from large quantities of continuously generated data, to provide faster, online, and real-time results. These applications span multiple domains, including environment and infrastructure monitoring, manufacturing, finance, healthcare, telecommunications, physical and cyber security, and, finally, large-scale scientific and experimental research.

In this book, we have discussed the emergence of stream processing and the three pillars that sustain it: the programming paradigm, the software infrastructure, and the analytics, which together enable the development of large-scale high-performance SPAs.

In this chapter, we start with a quick recap of the book (Section 13.1), then look at the existing challenges and open problems in stream processing (Section 13.2), and end with a discussion on how this technology may evolve in the coming years (Section 13.3).

Book summary

In the two introductory chapters (Chapters 1 and 2) of the book, we traced the origins of stream processing as well as provided an overview of its technical fundamentals, and a description of the technological landscape in the area of continuous data processing.