Software Architecture

Duplicate Messages are Inevitable Duplicate messages are an inherent aspect of message-based systems and can occur for various reasons. In the context of Kafka, it is essential to ensure that your application is able to handle these duplicates effectively. As a Kafka consumer, there are several scenarios that can lead to the consumption of duplicate messages: There can be an actual duplicate message in the kafka topic you are consuming from. The consumer is reading 2 different messages that should be treated as duplicates. You consume the same message more than once due to various error scenarios that can happen, either in your application, or in the communication with a Kafka broker. To ensure the idempotent processing and handle these scenarios, it’s important to have a proper strategy to detect and handle duplicate messages. ...

Building backend systems today will likely involve building many small, independent services that communicate and coordinate with one another to form a distributed system. While there are many resources available discussing the pros and cons of microservices, the architecture, and when it is appropriate to use, I want to focus on the functional testing of microservices and how it differs from traditional approaches. In my experience, the “best testing practices” have evolved with the introduction of microservices, and traditional testing pyramids may not be the most effective or even potentially harmful in this context. In my work on various projects and companies, including the development of new digital banks and the migration of older systems to microservices as they scale, I have often encountered disagreements about the most appropriate testing strategies for microservices. ...