Fast is a Must-Have, But Not Enough: Real-World Anti-Fraud Using an In-Memory Data Processing Engine

There has been so much emphasis on streaming and real-time analytics lately that the challenges of Online Transaction Processing (OLTP) at scale (which often looks like a streaming application) are often overlooked. Building a system that performs transactions on relentless streams of data, at rates of 100,000 to 1,000,000+ transactions a second, presents many hurdles. Concurrency, latency, consistency, and correctness are among the toughest challenges.

Being fast is not enough. Accuracy, consistency and ease of use are also key to the solution. Users often look for familiar concepts such as relational data model and SQL to lower the learning curve and keep existing tooling working.

In this talk, we will look at a real-world OLTP use case, anti-fraud (detection and prevention), and explore the challenges it poses, and possible solutions.

Anti-fraud is the process to identify and protect user from fraudulent online activities based on patterns. For example, processing credit card transactions at a rate of 10,000 times a second, while detecting fraud per transaction, as opposed to post-credit card transaction detection, is the challenge. Attendees will learn how an in-memory computing engine like VoltDB solves this real-world problem.

We will dive deep into how anti-fraud works, why high throughput and low latency is a must-have, but not enough, and how much accuracy and strong consistency guarantees matter. This talk will also cover the challenges posed by an in-memory database solution in this use case, such as high availability, bulk deletion and in-memory compaction, and recovery time (should service be interrupted).