The key takeaway
Treat duplicate delivery as an expected condition. Give each business operation a stable identity, bound retries, and provide an explicit recovery path for failed work. A successful queue acknowledgement should mean the intended outcome is safely recorded, not simply that a worker started running.
Start with the business operation
Name the action before defining the message. “Generate the monthly report for account A” is a better unit of work than “run this function.” Record the account, reporting period and requested output so a later worker can reconstruct the intent without relying on a browser session.
Separate accepting a request from completing it. Return a job identifier and show a visible pending state. Users need to distinguish slow work from a failed submission before they decide to click again.
Make duplicate delivery safe
Assign an operation key at the business boundary. A repeated submission for the same intended action should find the existing operation rather than create another one. Enforce that rule in durable storage; a process-local check cannot coordinate competing workers.
For database-only work, record the result and operation state in one transaction where possible. External side effects need their own strategy: use a provider's idempotency facility when available, and preserve its operation identifier for reconciliation. A local “completed” flag alone cannot close the gap between a remote success and a crashed worker.
Bound retries and keep failures visible
Retry failures that may clear, such as a temporary network interruption. Invalid input needs correction instead of repeated execution. Set an attempt limit and increase the delay between attempts, with jitter to avoid workers retrying together.
- Track job age as well as queue depth.
- Preserve the last error without including secrets or sensitive payloads.
- Route exhausted work to a reviewable failure state.
- Alert on growing failure volume and unusually old pending work.
Build a recovery workflow
An operator should be able to inspect the operation, understand which effects occurred and choose a safe next step. Replaying a message blindly can repeat an external action, so reconciliation belongs before retry when the outcome is uncertain.
Test a worker crash immediately before and after each side effect. Include duplicate delivery and concurrent processing in those checks. Connect job identifiers to your application monitoring so support can trace a user's request through the worker.
Keep the first implementation small
Start with one job type, explicit states and a documented recovery procedure. Add concurrency only after you know which operations can run together safely. For the wider application boundaries, see the guide to monolith and microservice decisions.