Distributed Systems

Event-driven architectures with Kafka have become a standard way of building modern microservices. At first, everything works smoothly - services communicate via events, state is rebuilt from event streams, and the system scales well. But as your data grows, you face an inevitable challenge: what happens when you need to access historical events that are no longer in Kafka? 1. The Problem: Finite Retention & The Need for Backfills In a perfect world, we would keep every event log in Kafka forever. In the real world, however, storing an ever-growing history on high-performance broker disks is prohibitively expensive. ...

Duplicate Messages are Inevitable Duplicate messages are an inherent aspect of message-based systems and can occur for various reasons. In the context of Kafka, it is essential to ensure that your application is able to handle these duplicates effectively. As a Kafka consumer, there are several scenarios that can lead to the consumption of duplicate messages: There can be an actual duplicate message in the kafka topic you are consuming from. The consumer is reading 2 different messages that should be treated as duplicates. You consume the same message more than once due to various error scenarios that can happen, either in your application, or in the communication with a Kafka broker. To ensure the idempotent processing and handle these scenarios, it’s important to have a proper strategy to detect and handle duplicate messages. ...