Search
5 results for “message-queues”
Search results
Message Queues & Event-Driven Architecture
Why a visibility timeout, not a delete, is what actually protects a message from being processed twice, and why "at-least-once delivery" means your consumer has to handle duplicates even when everything is configured correctly.
What is a visibility timeout, and what problem does it solve that simply having multiple consumers poll a queue would create?
When a consumer receives a message, the queue doesn't delete it immediately, it starts a visibility timeout during which that message is hidden from other consumers, so a second consumer polling the same queue won't also pick up and process the same message concurrently. The consumer is expected to finish processing and explicitly delete the message before the timeout expires; if it does, the message never reappears. Without this mechanism, multiple consumers competing for messages would routinely double-process the same one, exactly the race condition visibility timeouts exist to prevent.
A consumer receives a message, starts processing, but crashes before deleting it. What happens to that message, and why is this actually the desired behavior?
Once the visibility timeout expires without the message being deleted, it automatically becomes visible again in the queue and can be picked up by the same or a different consumer for another processing attempt. This is deliberate, not a bug: the alternative (a crashed consumer's message vanishing permanently) would silently lose data, whereas reappearing after timeout guarantees eventual processing at the cost of a possible duplicate attempt, which is exactly the trade-off "at-least-once delivery" describes. Setting the visibility timeout too short causes premature, unnecessary reprocessing of messages still legitimately being worked on; too long delays legitimate retries after a real crash.
What does "at-least-once delivery" actually guarantee, and what does it not guarantee, and why does that mean a consumer must be idempotent?
At-least-once delivery guarantees a message will eventually be delivered and processed at least one time, it does not guarantee exactly one delivery, the same message can legitimately be delivered and processed more than once, for example if a consumer's deletion request is lost after it already finished processing, or a visibility timeout expires just as processing completes. Because duplicate delivery is a normal, expected outcome of this model rather than a rare edge case, a consumer has to be written so that processing the same message twice produces the same end result as processing it once (an idempotent operation), rather than assuming the queue itself will prevent duplicates.
Kafka vs RabbitMQ
How Kafka and RabbitMQ differ in messaging model, routing, replay, and throughput, and which fits event streaming versus task queues.