Webhook Retries and Idempotency
Webhook senders retry on failure — which means your handler will receive the same event more than once. If your handler is not idempotent, that results in duplicate orders, duplicate charges, or duplicate emails. This guide explains how retries work and how to build handlers that are safe to call multiple times.
TL;DR
- •Webhook senders retry when they receive 5xx, 4xx (sometimes), or a timeout
- •Retries can fire hours or days after the original delivery
- •Idempotent means: processing the same event twice produces the same result as once
- •Use the delivery ID as a deduplication key — store it before processing
- •Return 200 immediately, do heavy processing in a background job
Why Webhook Reliability Is Hard
A webhook delivery is an HTTP request from someone else's server to yours. You can't control when it fires, how long the sender waits for a response, or how many times they will retry. Every major webhook provider retries on failure, but they do so with different strategies, different windows, and different trigger conditions. Your handler must be correct under all of them.
Retries are automatic
If your server returns a 5xx, times out, or is unreachable, the sender queues a retry. You have no control over when it fires. Your handler may be called 1, 3, or 10 times for the same event over the course of hours.
Partial processing
Your handler may process half an event before crashing — updating a record but not sending a confirmation email. When the retry fires, you need to complete the remaining work without duplicating the part that already succeeded.
Out-of-order delivery
Retries and new events can arrive interleaved. A payment.updated event may arrive before the payment.created retry that triggered it. Your handler cannot assume events arrive in the order they were sent.
How Webhook Retries Work
Most webhook senders use exponential backoff — each retry waits progressively longer than the last. The exact timing varies by service but the pattern is consistent. What triggers a retry also varies.
| Provider | Retry on | Retry window | Delivery ID header |
|---|---|---|---|
| Stripe | 4xx, 5xx, timeout | Up to 72 hours, ~8 attempts | Stripe-Signature (contains timestamp) |
| GitHub | 4xx, 5xx, timeout (10s) | Up to 3 days, exponential | X-GitHub-Delivery |
| Shopify | 5xx, timeout (5s) | Up to 48 hours, 19 attempts | X-Shopify-Webhook-Id |
| Twilio | 5xx | Up to 1 hour | X-Twilio-Idempotency-Token |
| PagerDuty | 5xx, timeout | Up to 4 hours | X-Webhook-Id |
Important: timeout is the most common retry trigger
Most senders have a response timeout of 5–30 seconds. If your handler does any significant work synchronously (database write, email, downstream API call), you will exceed the timeout under load. The sender sees a timeout, marks delivery as failed, and retries — even though your handler may have completed the work. This is the most common source of duplicate processing.
What Idempotency Means for Webhook Handlers
An operation is idempotent if applying it multiple times produces the same result as applying it once. In the context of webhook handlers, this means: processing the same event twice must not create duplicate side effects.
This sounds obvious but is easy to get wrong. The naive pattern is:
Not idempotent
A retry creates a second order and sends a second email.
Idempotent
A retry returns immediately after the delivery ID check. No duplicates.
Detecting and Deduplicating Events
The delivery ID is the key to deduplication. Most webhook senders include a unique delivery ID in a request header. Before processing an event, check whether you've already processed a delivery with that ID.
Extract the delivery ID
Read the delivery ID from the header before any processing. Different senders use different header names (X-GitHub-Delivery, X-Shopify-Webhook-Id, Stripe's event ID in the body). For Stripe, use event.id from the parsed body — it's stable across retries.
// Extract at the top of your handler, before any await const deliveryId = req.headers['x-github-delivery'] || body.id
Check your deduplication store
Look up the delivery ID in a fast store — a database table, Redis, or an in-memory cache. If found, return 200 immediately without processing. This must happen atomically: check-then-insert, not check-then-process-then-insert.
// Atomic upsert — returns false if ID already existed
const isNew = await db.processedDeliveries.insert({ id: deliveryId, processedAt: new Date() })Store the ID before processing
Insert the delivery ID into your deduplication store before doing any work — not after. If your processing fails midway, the retry will re-enter and attempt the work again. Store the status (pending / complete) and update it after successful processing.
// Order matters: insert ID first, then process
await db.deliveries.insert({ id: deliveryId, status: 'processing' })
await processEvent(event)
await db.deliveries.update({ id: deliveryId, status: 'done' })Set a TTL on your deduplication store
Delivery IDs don't need to be kept forever — only as long as the sender might retry. If Stripe retries for up to 72 hours, keep IDs for 96 hours. In Redis, set an expiry on the key. In a database, periodically delete rows older than your retention window.
// Redis: set delivery ID with TTL matching the sender's retry window
await redis.set(`delivery:${deliveryId}`, '1', 'EX', 345600) // 4 daysDesigning Idempotent Handlers
Deduplication at the handler entry point handles exact retries (same delivery ID). You also need to design the operations inside your handler to be safe when run multiple times — because not all duplicate deliveries carry the same delivery ID.
Use upsert instead of insert
When creating or updating records, use upsert semantics — insert if not exists, update if exists. This prevents duplicate records if the same logical event arrives through different delivery paths.
// SQL INSERT INTO orders (id, status) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET status = $2
Check state before acting
Before taking an action (sending an email, charging a card, provisioning a resource), check whether it has already been done. Store a flag or status on the relevant record.
// Check before sending
if (!order.confirmationSentAt) {
await email.send(order.id)
await db.orders.update({ confirmationSentAt: new Date() })
}Use natural idempotency keys
When calling external APIs that support idempotency keys (Stripe, many payment processors), use the event's delivery ID or entity ID as the idempotency key. The API will return the original result instead of performing the action twice.
// Stripe: pass idempotency key
await stripe.charges.create(params, {
idempotencyKey: `charge_${event.orderId}`,
})Use state machines
Model your entity's lifecycle as a state machine with explicit valid transitions. Attempting an invalid transition (e.g., shipping an already-shipped order) is a no-op or returns an error — not a duplicate action.
// Transition only if in expected state
if (order.status !== 'paid') return
await order.update({ status: 'fulfilling' })Return 200 immediately, process asynchronously
The single most effective change you can make for reliability: acknowledge the delivery with a 200 within 2–5 seconds, then process the event in a background job. This eliminates timeout-triggered retries entirely.
Receive webhook
Verify signature, extract delivery ID, enqueue job, return 200.
Background job
Check dedup store, process event, update state, call external APIs.
Dead letter queue
Failed jobs after N retries go to DLQ for manual inspection and replay.
Common Mistakes
Deduplicating after processing
Storing the delivery ID after your handler completes means a crash between processing and storing leaves no dedup record. The retry will process the event again. Store the delivery ID first, then process.
Using only the event type as a dedup key
Two different payment.succeeded events for different customers share the same event type. The dedup key must be the delivery ID — the unique identifier for this specific delivery, not the event type.
Ignoring retries from different delivery IDs
Some senders generate a new delivery ID on each retry attempt. Your dedup logic must also handle business-level deduplication — checking whether the underlying entity (payment, order) has already been processed, not just the delivery.
Making synchronous external calls in the handler
Calling Stripe, sending emails, or writing to S3 inside the webhook handler adds latency and failure modes. Any of these can cause a timeout, triggering retries. Enqueue the event and do external calls in a background worker.
Not handling partial processing
If your handler creates an order but crashes before sending the confirmation email, the retry will attempt both steps again. Design each step to be independently idempotent, not just the handler entry point.
Returning non-200 for business logic failures
If you can't process an event because the referenced entity doesn't exist (yet), returning a 404 tells the sender to retry — which may be what you want. But returning a 400 for a known, unrecoverable error tells the sender to stop retrying. Be deliberate about which status codes you return.