A short-lived lock for a long-running evaluation
I recently added a deduplication safeguard to a queue-driven system. The problem was simple to describe but harder to solve well; many obvious answers turned out to have some bad trade-offs. The setup The system is event-driven. Workers pull messages off an SQS FIFO queue, where each message is grouped by an entity ID. Processing a message may create an “incident” for the entity. The processing flow looks like this: ...