Webhook delivery reconciliation: separate receipt from business outcome

A model to inspect

Delivery is not outcome

01Verify envelope
02Record receipt
03Apply guarded projection
04Reconcile authority

Receipt order is not automatically business order.

Original conceptual diagram · not a live trace or measured result.

Give delivery its own identity

A webhook envelope needs a sender identity, authenticated signature or equivalent verification result, event identifier, event type, received time, raw-body retention policy, and delivery attempt metadata. Verify before parsing into an action path, and bind the verification key or configuration version used. A delivery ID answers “have we processed this envelope?” It does not answer whether the invoice, booking, or user state inside is the newest truth.

Use a durable receipt table keyed by the verified sender and event ID. Store enough to replay safe processing or investigate rejection, but do not turn raw payload retention into an unbounded shadow database. A trace ID can join the route, while the sender’s delivery identity anchors duplicate detection.

Order is a domain question

Event arrival order can differ from occurrence order. Define the domain’s ordering evidence: upstream object version, monotonic sequence, update timestamp with tie rule, or a fetch-from-authority step. If none exists, mark the result uncertain and request reconciliation rather than assuming a later arrival represents a later change. Keep delivery order in diagnostics, but do not use it as business order by accident.

Hypothetical example: a subscription.cancelled delivery arrives before a delayed subscription.updated event. If the upstream object has version 12 for cancellation and 11 for update, retain version 12. If there is no comparable version, fetch the current subscription from the authority before changing local access.

Make replay safe but meaningful

A valid replay should re-enter the same delivery processing path with the same idempotency key. It may repair a transient downstream failure, but it must not create another refund, email, or access grant. Record the processing state separately from the domain projection: received, verified, applied, deferred for reconciliation, rejected, or failed transiently. A prior receipt can be processed again if its prior handling failed; it need not be discarded blindly.

Counterexample: “event ID seen” is set before a database transaction commits, then the worker crashes. A later redelivery is ignored and the business change is lost. Commit receipt state and projection atomically where possible, or leave a recoverable pending state that a reconciler can inspect.

Reconcile downstream state

For consequential changes, compare the local projection with a declared authoritative object and record the comparison time, identity, and result. A reconciliation job should be bounded, authorized, and capable of explaining whether it repaired, deferred, or refused a difference. It should not automatically choose a winner when local policy and upstream state conflict. That conflict may require a service owner.

Treat a successful HTTP acknowledgement as “we accepted delivery,” not “the business outcome is complete.” Likewise, a timeout after applying a local change can cause redelivery. The user-facing conversation should report verified local state or a pending reconciliation, never guess from transport acknowledgement alone.

Test the delivery ledger

Test invalid signatures, same event redelivered, two senders reusing an ID, delayed older events, worker crash between receipt and projection, replay after a schema upgrade, and an upstream object that cannot be fetched. Assert sender scoping, one effective domain transition, neutral error language, and a visible reconciliation queue.

Acceptance checks: verified deliveries have immutable identity; business projections name the source version or reconciliation basis; duplicate receipt does not imply duplicate action; and unknown order remains explicitly unknown. This guide does not establish any provider’s retry policy or signature scheme—use that provider’s primary documentation for those details.

Preserve an investigation boundary

Keep receipt processing, business projection, and reconciliation logs linkable but access-controlled. A support role may need to see that an event was received and deferred without receiving the whole signed payload or another tenant’s object. Define retention separately for payload evidence and for the smaller state ledger.

Acceptance checks include an operator replay request, a denied cross-scope lookup, and a reconciliation result that cannot be applied twice. The goal is accountable recovery: the system can explain whether a delivery was verified, applied, or still uncertain without treating its logs as the authority for the customer’s current state. Reviewers must be able to identify the source authority, the comparison time, and why a repair was permitted.

Take it into the review

Delivery and domain ledger

Record	Identity	Meaning	Retry rule
delivery	sender + event ID	one envelope	safe replay
projection	object + version	local domain state	compare authority
reconciliation	object + time	difference review	bounded retry
action	operation ID	side effect	separate control

A starting artifact to adapt to your service—not a ready-made policy, compliance certificate or test result.

Primary reading

Sources and limits

These links support the architecture, policy, or product behavior discussed above. Vendor documentation describes vendor features; it is not independent proof of performance. Current details should be rechecked before a production decision.

What the sources establish

Stripe: Receive events in a webhook endpoint

Stripe support guidance says webhook signature verification needs the raw, unmodified request body and describes timestamp-tolerance failures.

Limits: It is provider-specific guidance and does not define business ordering or reconciliation policy.

Checked 2026-09-19 · Stripe · source publication date not established.

Open original source ↗

W3C Trace Context

Trace Context supports correlation across service boundaries.

Limits: It supplies no webhook authentication, replay, or ordering semantics.

Checked 2026-09-19 · W3C · source publication date 2021-11-23.

Open original source ↗

Procedures and worked examples are editorial synthesis. Preparation/review dates are not claimed historical publication dates.