Webhook delivery is not the same as successful business processing. An external platform may send an order-created, payment-updated or account-changed event and receive a timeout even though the receiving application has already applied the action. An operator who clicks Replay merely because the sender shows failure can therefore create a second order, notification or entitlement change. Software teams need to investigate what the recipient actually did, whether replay remains appropriate and how a safe recovery will be verified.
This is particularly important for small software houses supporting several client integrations. The sender's event identifier, delivery attempt, the receiver's processing record and downstream business transaction may each be different objects. A single red status does not reveal which stage failed. The objective is not to clear the red dashboard at any cost; it is to reconcile the intended one-time business outcome while preserving event evidence, access boundaries and traceability.
Identify the exact source event
Start from the sender's stable event ID, event type, creation timestamp, intended recipient endpoint, client tenant and related business reference. A payment gateway may issue multiple events concerning the same payment, and an ecommerce service may generate several updates to one order. Do not assume the latest visible error corresponds to the earliest unprocessed action. Link each delivery attempt to the same logical event and identify its actual payload version.
Confirm which environment and organisation the record belongs to. Production and test systems may look similar, and the wrong tenant can receive a valid-looking but unrelated event. Preserve only non-sensitive identifiers in ordinary support notes; payment information, signatures, access tokens and customer payloads belong in appropriately restricted systems.
Separate event creation, transport and processing
A source can create an event successfully but fail while opening a connection. Alternatively it can transmit the request and receive a rejection, or the receiver can process it before its acknowledgement is lost. These are different states requiring different actions. Inspect the sender's delivery log and authorised receiver evidence together. An HTTP timeout alone cannot prove no work occurred.
Where the receiving service acknowledges requests before completing asynchronous processing, an accepted HTTP response may also fail to prove that the business action succeeded. Identify which step the application actually records as complete and how that state can be verified.
Determine whether the business action already happened
Search the authoritative destination by event or correlation ID and, where appropriate, the actual order, transaction or account reference. Did the receiver store the event? Did it commit a new record, change a status or send a message? Did a subsequent worker complete the effect? A database row named processed is not necessarily proof that every downstream consequence occurred. Verify the relevant business outcome rather than reading one generic counter.
Investigate duplicates already present before replaying. A prior automatic retry may have succeeded while the original attempt still looks failed in a support tool. If the system does not retain reliable correlation, escalate to engineering rather than creating another externally visible action to learn whether it works.
Check whether the receiver is genuinely idempotent
Idempotency means that repeating the same logical operation does not create an additional unintended effect. It must be implemented at the relevant business boundary, not assumed from a sender's retry facility. The receiver may deduplicate delivery IDs while still allowing its own background worker to issue duplicate invoices. Document which invariant is protected and for how long the deduplication history is retained.
Stripe documents idempotent request handling for its API, and webhook guidance emphasises handling potentially repeated events, but those mechanisms have defined scope. They are examples rather than proof that every custom integration is safe. If the destination ignores source event IDs, an ordinary retry can perform the same operation twice.
Consider delivery ordering and old events
Events may arrive out of order, and replaying one historic state can overwrite a newer legitimate change. A customer order could be cancelled after a payment event was originally sent, or a user could lose access after an earlier role grant. Confirm the current authoritative business state and whether the event is still valid to apply. Do not replay merely to make all source statuses green.
If several events depend on sequence, map their relationships and identify a safe recovery order. Where the sender does not guarantee ordering, design the receiver's state transitions to reject stale or inappropriate changes rather than silently accepting every delivered message.
Review the original payload safely
Use the supported event archive to inspect the exact event and relevant schema version. Do not reconstruct a missing payload from memory or change an event type to bypass validation. Some providers sign webhook payloads, and altering the body may affect verification. Preserve restricted access and redact unnecessary sensitive data in general discussion.
If the sender no longer retains the event, determine whether an approved reconciliation route can establish the current truth without fabricating a replacement. A manually created new event is not equivalent to retrying the original one and may require separate business authorisation.
Identify automatic retry and queue activity
Before triggering a manual resend, check whether the sender is still retrying, a delivery is queued or the receiver has an unprocessed copy. A manual attempt can overlap existing work. Record the status and planned stop condition. If retries are disabled under the client's supported configuration, explain why and obtain the relevant owner decision before changing policy.
When the receiving application uses background queues, distinguish request acceptance from worker completion. A new event should not be sent merely because the worker is delayed; resolve the actual processing backlog through the approved operational route.
Assess the effect of the event
Some events only refresh a read-only cache, while others send money, release inventory, provision access or contact customers. Classify the real consequence and the damage a duplicate might cause. The correct escalation for a payment event may involve finance and an integration specialist, while a duplicated harmless lookup could be handled through an approved standard runbook.
Do not expand the work from one identified webhook to an entire time range without separate evidence. A broad replay may affect unrelated customers whose events already completed normally.
Obtain bounded authorisation for recovery
Specify the event ID, destination, known receiver state, expected one-time effect and chosen replay mechanism. Identify the technical operator and, where business consequences are material, the relevant client approver. A software developer's ability to press Replay is not automatically authority to create another shipment or payment transaction. Record the decision with a reliable audit reference.
Use a narrowly scoped supported tool. Avoid direct database changes intended to trick a retry system into accepting a new event, because bypassing normal checks can conceal the original defect and leave inconsistent logs.
Execute once and observe the result
Submit the approved replay under named access and record the new delivery attempt against the original source event. A successful HTTP response is evidence about transport, not necessarily proof of correct processing. Check the receiving system's actual accepted record, relevant worker result and business outcome. If the replay times out again, inspect the state before attempting another submission.
Where an external supplier owns the receiver, collect its authoritative case or status information. Do not infer correctness from a sender's dashboard alone.
Validate that no duplicate effect occurred
Compare source and destination records after recovery, including cumulative financial amounts, shipment references or access assignments where relevant. Confirm the intended outcome exists once. If an unintended duplicate is discovered, stop further replay and follow the client's approved correction process. Do not silently delete transactions or logs in order to present a clean result.
For asynchronous systems, revisit the case after the normal processing interval. If a delayed original message later arrives, the receiver must still handle it safely. Record this subsequent check instead of claiming immediate completion while delayed jobs remain.
Improve instrumentation and future handling
Recurring webhook failures may point to unstable endpoints, missing source identifiers, unclear status reporting, broken signature verification or an inadequate deduplication design. Use actual traces to determine the cause. Improve the retry runbook with clear no-replay conditions and decision owners; support staff should know which events they may safely repeat and which require developer or finance review.
Protect data in diagnostics. Adding every full payload to a general ticket may speed one investigation while exposing customer information. Prefer stable references, structured safe logging and restricted evidence links.
Close on the restored business state
The incident should close when the target business state is correct, unintended duplicate effects have been excluded at the agreed scope and remaining systemic fixes have owners. The source event's status may require reconciliation, but clearing a retry alert is not itself the goal. A competent handover tells the next engineer which event was involved, what actually happened and why another replay should not be attempted without a fresh decision.
Sources: Stripe webhook documentation and Stripe idempotent requests. These first-party examples illustrate event delivery and repeat-action controls; the receiver's actual implementation and authorised business state determine safety.
Failed webhook replay: verify the state before resending
An HTTP timeout is not proof that a downstream business action failed. Reconciliation must precede replay.
| Control | Investigation | Safe acceptance evidence |
|---|---|---|
| Identity | Confirm event ID, type, tenant, target and timestamp. | Correct original event located without exposing secrets. |
| Delivery | Inspect attempts, response codes and provider retry policy. | Known outcome rather than an inferred failure. |
| Business state | Check orders, refunds, inventory or other downstream results. | No already-completed action would be duplicated. |
| Replay permission | Verify signature, idempotency and authorised scope. | Approved retry method with controlled concurrency. |
| Closure | Reconcile the replay result against source and receiver. | Stable accepted state and documented exception if unresolved. |
Source A: Stripe webhooks documentation, vendor-specific delivery and duplicate-handling practices. Source B: Microsoft Azure idempotent-consumer guidance, general application design. Actual provider guarantees and business transaction state must be confirmed before replay.
Frequently asked questions
Does a webhook timeout prove the receiver did nothing?
No. A receiver may have committed the effect before its acknowledgement was lost. Inspect the destination state first.
Can a support engineer replay every failed event?
Not automatically. Confirm current business validity, duplicates, replay mechanism and authorisation for the specific impact.
Does using an event ID guarantee idempotency?
No. The receiving system must enforce the relevant business invariant across the actual processing chain.
Should the event payload be manually reconstructed?
Prefer the preserved original through the supported provider route. A new fabricated payload can alter semantics and require separate approval.
What proves replay recovery is complete?
The correct downstream business result, absence of unapproved duplicates and clear ownership of any remaining software defect.