The scenario

Imagine a SaaS product that needs to notify customer systems when a project changes state. Customer endpoints may be unavailable, respond slowly, or receive the same event more than once. This example would make those delivery conditions visible and recoverable instead of treating one HTTP request as a reliable integration.

Commit an event before trying to deliver it

The product transaction would write a versioned event to an outbox alongside the state change. A delivery worker would create a separate record for each subscribed endpoint. A slow customer API would not hold open the transaction that updates the product.

The event identity would remain stable across delivery attempts. Attempt identifiers and timestamps would change as needed, while the payload would describe the original event. Customers would receive a documented deduplication key and instructions for handling repeated delivery.

Sign a documented request format

Each endpoint would have a managed signing secret. The request contract would specify the exact bytes covered by the HMAC, the timestamp format, and how the receiver verifies a signature. Key rotation would have an overlap and retirement process rather than silently invalidating queued deliveries.

The delivery service would accept only permitted HTTPS destinations and enforce outbound network restrictions. Destination validation would consider resolved addresses and redirects, including the risk of reaching internal or metadata services. Signing authenticates the payload; it does not make an arbitrary destination safe to call.

Make retries and replay understandable

Each attempt would have a timeout and a classified outcome. Temporary failures could receive bounded retries with backoff, while permanent configuration errors would pause or fail delivery with an actionable reason. The system would not assume that a timeout means the customer performed no work.

A delivery log would expose timestamps, response status, and a redacted failure reason. Manual replay would reuse the event identity and create a recorded new attempt. Tests would include an endpoint that commits its work but drops the response, so consumers can verify their duplicate-handling path.

What to validate

  • A product rollback does not emit a completed-change event.
  • The same event retains its identity through retries and replay.
  • A modified or stale signed request is rejected by the receiver policy.
  • Unsafe destinations and unbounded retry loops are blocked.

The tradeoff

Webhooks shift some reliability work to the receiving system. A polling API or a managed message channel may suit some integrations better. The delivery contract should state its retry, ordering, and retention behaviour plainly.

Technical references

Official documentation supporting the tools and mechanisms discussed above. The proposed architecture is illustrative.