Posts

Duplicate Messages are Inevitable Understanding the Intricacies of exactly-once semantics in Kafka Achieving Idempotent Processing with Kafka Idempotent Consumer Pattern Ordering of Messages Retry Handling Idempotent Processing and External Side Effects Publishing Output Messages to Kafka and Maintaining Data Consistency The Simplest Solution Transactional Outbox Pattern Without Transactional Outbox How it compares to Synchronous REST APIs Final Thoughts Duplicate Messages are Inevitable Duplicate messages are an inherent aspect of message-based systems and can occur for various reasons....

Building backend systems today will likely involve building many small, independent services that communicate and coordinate with one another to form a distributed system. While there are many resources available discussing the pros and cons of microservices, the architecture, and when it is appropriate to use, I want to focus on the functional testing of microservices and how it differs from traditional approaches. In my experience, the “best testing practices” have evolved with the introduction of microservices, and traditional testing pyramids may not be the most effective or even potentially harmful in this context....

I’ve been spending a lot of time with AWS S3 recently building data pipelines and have encountered a surprisingly non-trivial challenge of unzipping files in an S3 bucket. A few minutes with Google and StackOverflow made it clear many others have faced the same issue. I’ll explain a few options to handle the unzipping as well as the end solution which has led me to build nejckorasa/s3-stream-unzip. To sum up:...