How-to for real-time alerting, analytics and reporting at scale with Apache Kafka and Apache Ignite
Are you tasked to build a system or upgrade an existing architecture to a solution capable of handling unbound streams of data, do real-time alerting, storing always growing terabytes and petabytes of data and, finally, act on the data within milliseconds SLAs. Sounds like a challenging but rather a common task? Yes! Should you as an engineer or architect build it up off the ground? Unlikely!
What if we tell you that by integrating Apache Kafka with Apache Ignite you’ll solve all of the requirements faster and easier. A battle-tested recipe is simple - take Kafka Connect and have your data stream through Kafka pipelines, add a pinch of KSQL to act on the streams with SQL in real-time with zero delays, rinse and flush the preprocessed data in Ignite as in-memory databases and get further insights by analyzing your hot and cold datasets.
In this talk, Denis and Viktor will demonstrate how to implement the solution in practice, will explain architectural reasoning and the benefits of real-time integration, share common usage patterns. As always, we'll build a streaming data pipeline using nothing but our bare hands, Apache Ignite, Kafka Connect, and KSQL.
Before joining GridGain and becoming a part of Apache Ignite community, Denis worked for Oracle where he led the Java ME Embedded Porting Team -- helping bring Java to IoT.
Working in the field, Viktor Gamov developed comprehensive expertise in building enterprise application architectures using open source technologies. He enjoys helping different organizations design and develop low latency, scalable and highly available distributed systems.
Back in his consultancy days, he co-authored O’Reilly’s «Enterprise Web Development.»
He is a professional conference speaker on distributed systems, Java, and JavaScript topics, and is regular on events including JavaOne, Devoxx, OSCON, QCon, and others (http://lanyrd.com/gamussa). He blogs at http://gamov.io and produces the podcasts Razbor Poletov (in Russian) and co-hosts DevRelRad.io.
Follow Viktor on Twitter @gamussa, where he posts there about gym life, food, open source, and, of course, Kafka and Confluent!