Kafka Simple Consumer Failure Recovery

June 21st, 2016 A modern data platform requires a robust Complex Event Processing (CEP) system, a cornerstone of which is a distributed messaging system. Today, many people use Kafka to fill this latter role. Kafka can be complicated, particularly when it comes to understanding how to recover from a failure during consumption of a message. There are numerous options for such recovery; in this post we’ll just be looking at one of them. We’ll walk through a simple failure recovery mechanism, as well as a test harness that allows you to make sure this mechanism works as expected. First, we’ll create a test Kafka producer and consumer with failure recovery logic in Java. Then, we’ll discuss a bash script that starts up a local Kafka cluster using Docker Compose, sends…


Link to Full Article: Kafka Simple Consumer Failure Recovery