123eworld Knowledge Hub → Transactional SMS API → Page 221
Transactional SMS API Webhook Retry Handling: Backoff, Deduplication and Endpoint Recovery
Developer reference guide for transactional sms api webhook retry handling: backoff, deduplication and endpoint recovery, covering implementation, validation, security, reliability, testing and production practices.
Webhook retries
A webhook system should retry only when the failure is likely temporary. Network errors, timeouts and selected 5xx responses are commonly retryable; malformed endpoint configuration should not be retried forever.
Backoff
Use exponential backoff with jitter and a maximum retry window. This prevents a customer outage from generating synchronized traffic.
Duplicate delivery
Retries naturally create duplicate events. Every event should have a stable event ID so consumers can process the same event more than once safely.
Endpoint health
Track failure rate and latency per endpoint. A consistently failing endpoint can be paused or placed into a degraded state while preserving pending events.
Response semantics
Document which HTTP responses count as successful acknowledgment. The receiver should acknowledge durable acceptance, not necessarily business completion.
Dead letters
Events that exceed the retry policy should remain visible in a controlled dead-letter area. Customers or operations should be able to replay them safely.
Replay
Replay the original event identity and payload where possible. Creating a new event ID would defeat consumer deduplication.
Security
Retries must preserve signature semantics. The receiver should verify each delivery independently.
Testing
Test endpoint timeout, 500 response, 429 response, duplicate delivery, replay and secret rotation.
Observability
Monitor oldest pending webhook, retry count, endpoint success rate and delivery latency.
Customer guidance
Provide implementation examples that acknowledge quickly and process asynchronously.
Reference pattern
Generate event → sign → deliver → acknowledge or retry → deduplicate → process → replay if required.
Retry state
Webhook delivery should store event ID, endpoint, attempt number, next-attempt time, last response and final state. This makes retries durable and observable instead of relying on an in-memory timer.
Response classification
Treat timeouts and selected server errors as temporary conditions. Authentication failures, invalid endpoint configuration or permanent client errors should move toward a controlled failure state rather than repeated delivery.
Backoff and jitter
Use increasing delays with jitter. During a platform-wide incident, this prevents all customer endpoints from receiving synchronized retry bursts.
Endpoint isolation
Each endpoint should have independent retry and concurrency controls. A slow endpoint must not consume all webhook workers.
Replay
A replay operation should reuse the original event ID and should be audited. The customer consumer can therefore use the same idempotency logic.
Dead-letter recovery
Dead-letter events should remain available for inspection and controlled replay. The platform should distinguish customer configuration failures from transient infrastructure failures.
Security
Every retry must be signed according to the same verification contract. Never bypass signature verification during replay.
Production scenario
A customer's CRM is offline for 30 minutes. The SMS platform continues normal message processing, stores webhook events, retries with backoff and begins successful delivery when the CRM recovers. The CRM sees stable event IDs and can safely deduplicate any repeated attempts.
Testing
Simulate endpoint timeout, 429, 500, DNS failure, slow response, recovery and replay.
Final rule
Webhook retry handling should make customer outages survivable without turning them into platform-wide incidents.
Retry ordering
Webhook retries should preserve event order where ordering is contractually required, but consumers should still be designed to tolerate duplicates. A per-endpoint queue can provide stronger ordering without allowing one endpoint to block the entire platform.
Backpressure
If an endpoint remains slow, reduce concurrency rather than increasing parallel retries. Backpressure is healthier than creating a growing outbound connection pool.
Retry budget by endpoint
A single customer endpoint should have a bounded retry workload. This protects both the customer and the platform from a feedback loop during prolonged downtime.
Replay controls
Replay should require authorization and should be rate-limited. Bulk replay can otherwise overwhelm a recovered customer system.
Evidence
Store the last response code, latency and attempt time. This lets support distinguish a customer outage from a platform delivery problem.
Reference implementation
Event queue → endpoint policy → signed request → response classification → backoff → retry/dead letter → controlled replay.
Implementation and operations note
Webhook retry scheduling should survive process restarts. The next-attempt time belongs in durable state rather than only in a worker timer.
Implementation and operations note
If an endpoint becomes healthy again, recovery should ramp gradually rather than releasing the entire backlog at maximum concurrency.
Implementation and operations note
A webhook delivery record should preserve the original event timestamp separately from each delivery-attempt timestamp. This distinguishes event age from delivery delay.
Implementation and operations note
Customers should be able to identify the event that caused a business action even after several retries. Stable event identity is therefore more important than attempt identity.
Implementation and operations note
A final test should stop all webhook workers, restart them and verify that pending events continue from durable state without losing or duplicating logical events.
Reference architecture guidance
Webhook retry handling should be treated as an outbound queue with the same engineering discipline as SMS delivery. Each event needs durable identity, due time, attempt count and endpoint context. This makes the retry mechanism restartable and observable.
Reference architecture guidance
An endpoint that consistently returns errors should enter a degraded state rather than receiving unlimited attempts. The platform can continue accumulating events for that endpoint while protecting healthy endpoints. Once recovery is detected, delivery can ramp gradually.
Reference architecture guidance
Consumers should not assume that successful HTTP acknowledgment means their entire business workflow completed. The contract should mean that the event has been durably accepted by the consumer. Business processing can happen asynchronously after that acknowledgment.
Reference architecture guidance
A reference-grade webhook system therefore combines signed events, stable IDs, durable retries, endpoint isolation, dead-letter recovery and controlled replay. These features make integrations resilient during customer-side outages.
Advanced reference scenario
Webhook retry systems should also consider ordering requirements. Some customers may need events for one logical message to be processed in sequence, while others only need at-least-once delivery. If ordering is part of the contract, use a per-endpoint or per-message ordering mechanism rather than globally serializing all webhooks. This preserves scalability. Even with ordering, consumers should remain idempotent because a network failure can cause the same event to be retried. The retry queue should record the original event timestamp separately from each attempt. This allows support to distinguish a newly generated event from an old event that was delivered late. When an endpoint recovers after a long outage, the platform should gradually drain its backlog and monitor response latency. A sudden replay of millions of events can overload the customer's recovered system and cause another outage. Controlled recovery is therefore part of webhook reliability, not an optional operational feature.
Operational safeguard
Webhook retry systems benefit from endpoint-level circuit breaking. When a customer's endpoint repeatedly fails, the platform can pause new attempts temporarily while retaining events. A controlled probe can then determine whether the endpoint has recovered. This reduces unnecessary traffic during long outages.
Closing engineering principle
Finally, document the maximum retry window and what happens after it expires. Customers should know whether an event remains available for manual replay, enters a dead-letter state or is permanently discarded under the retention policy. Predictable lifecycle rules are essential when webhook events update financial, CRM or operational records.
Operational reference
Operationally, keep webhook delivery workers separate from the core message-submission workers. A large customer outage should consume only its allocated webhook capacity. This isolation prevents callback retries from competing with the resources needed to accept and process new SMS traffic.
Final operational note
The customer-facing documentation should explain retry timing without promising exact delivery timestamps. A webhook event may arrive later because the customer's endpoint was unavailable, so applications should use the event timestamp and message ID rather than assuming arrival time represents event time. This distinction is important for audit trails and business workflows.
Quick reference
Keep retry evidence permanently traceable to the original event.
Checklist item
Monitor retry age.
Continue through the 123eworld Knowledge Hub
Explore the 123eworld Knowledge Hub for practical SMS API, transactional messaging and developer architecture guides.
Visit 123eworld.com for messaging and digital communication services.