Roman’s presentation titled “Apache Ignite as a Data Processing Hub” will cover the technical details of how CyberAgent, which develops apps around internet ads, game development, Internet TV, and other big data areas, is integrating Apache Ignite with Apache Kafka and Apache Flume into a high-performance data processing system. Roman was kind enough to answer a few questions to give us a high-level look at what he will cover in his talk.

IMC Summit:
The abstract for your talk states “Such processing can rarely be done with one tool or framework — a number of tools are often involved”. What tool would you say is the closest to being a complete solution, i.e. requires the fewest ancillary tools? Is that Apache Ignite?

Roman Shtykh:
It depends much on the requirements you set for your data processing flow — the degree of “completeness” of a particular tool reveals in each particular context, and it changes when the context changes. Having compute grid in addition to a rich data grid functionality, I believe Apache Ignite can cover a wide range of requirements. To increase this degree of completeness even greater and in many diverse contexts, and bring its power to existing data pipelines, Apache Ignite community has been working on a number of integrations with other platforms.

IMC Summit:
Are there any must-have tools that you would say should be part of any implementation, or is it very dependent on the nature of the project?

Roman Shtykh:
It is dependent on the nature of the project. In CyberAgent we are moving large amount of logs from our Web services, and services like Apache Flume and Kafka are essential. In my talk I will introduce how such log services can be integrated with Apache Ignite.

IMC Summit:
Do you see a consolidation of tools taking place so that future projects won’t require a patchwork of solutions? Or do you just see more and more tools becoming available?

Roman Shtykh:
With new needs new solutions will emerge to take that particular niche, but with interfaces to easily integrate to existing systems. Many platforms today have a large arsenal of functionalities for standard data processing, and already provide rich integration functionalities and out of the box solutions that enable developers to easily plug them into existing running environments. And that is also a crucial factor for a system to be quickly adopted and become widely used. Furthermore, we can witness the consolidation of projects to create new OSS integration standards, such as Apache Arrow.

Attend Roman’s talk along with many other deep dive presentations and keynote talks from thought leaders in the field of In-Memory Computing, and network with other developers and decision makers who are pushing the limits of performance and scalability of big data systems at the In-Memory Computing Summit in San Francisco May 23-24. Register today.

Platinum

Gold