123eworld Knowledge Hub → SMS API → Page 298
SMS API Webhook Retry Strategy: Handling Failed Callbacks Safely
A practical developer reference designed to solve real implementation and production problems around sms api webhook retry strategy: handling failed callbacks safely.
Why webhook retries happen
A webhook sender may retry when the customer endpoint is unavailable, times out or returns an error. Retries are necessary for reliability but can create duplicate events if the receiver has already processed an earlier attempt.
Receiver acknowledgement
The receiver should acknowledge after the event is durably accepted, not after a lengthy downstream business operation. This sharply reduces timeout-driven duplicates.
Exponential backoff
Retries should normally use increasing delays with a maximum retry window. A fixed one-second retry can overload an already unhealthy customer endpoint.
Retry classification
Network failures and 5xx responses are usually candidates for retry. Many 4xx responses indicate a configuration or authentication problem and should follow a different policy.
Idempotent event processing
Persist event IDs or another stable idempotency key before executing non-repeatable business actions.
Dead letters
After the retry window ends, move the event to a durable dead-letter state and make it visible to authorized support or customer administrators.
Manual replay
A replay function should preserve the original event ID and create a separate delivery-attempt record. Never mutate the historical event.
Backpressure
A receiver that is overloaded should be able to accept and queue events without executing every business action synchronously.
Observability
Monitor webhook success rate, retry age, endpoint latency and dead-letter volume.
Testing
Simulate 500 errors, timeouts, connection resets, slow responses and duplicate delivery.
Reference flow
Send → timeout/error → backoff → retry → acknowledgement or dead letter → optional controlled replay.
Production checklist
Use bounded retries, idempotency, durable queues, clear ownership and replay auditing.
Security and privacy
Treat phone numbers, message content, credentials and delivery data as sensitive operational information. Avoid unnecessary logging and ensure tenant authorization is applied before data access.
Production reliability
Design for timeouts, duplicates, retries, provider failures and delayed events. A messaging platform is asynchronous infrastructure, so success-path testing alone is insufficient.
Developer-first principle
The public API should hide unnecessary telecom complexity while exposing enough structured information for developers to build correct integrations.
Related 123eworld guides
Explore the 123eworld SMS & WhatsApp Knowledge Hub for related developer, API, routing and production guides.
Retry schedule
A retry schedule should protect both sides of the connection. Use short initial delays for transient failures followed by progressively longer delays. Add jitter so thousands of events do not retry at exactly the same moment. Define a maximum delivery window rather than retrying forever. The schedule should be configurable but bounded by platform safety limits.
HTTP response policy
Treat network timeouts and many 5xx responses as retry candidates. A 400 caused by an invalid customer endpoint configuration normally requires customer action, not repeated traffic. Authentication failures should be surfaced clearly and may require immediate operator attention. Document the exact policy so customers can understand why an event is being retried or stopped.
Idempotent receiver
The receiver should store the event ID before executing a non-repeatable action. If the same event arrives again, the receiver can acknowledge it without performing the business action twice. This pattern should be demonstrated in SDK examples and documentation because it is one of the most important webhook implementation requirements.
Dead-letter workflow
When an event reaches the retry limit, preserve its complete delivery history and move it into a dead-letter state. Provide authorized users with the failure reason, last response, attempt count and last-attempt timestamp. A replay action should create a new delivery attempt against the same event. Do not edit the historical record because support may need it later.
Backpressure
A customer endpoint may be technically available but overloaded. A fast acknowledgement combined with an internal queue allows it to absorb bursts. The messaging platform should also respect endpoint response behaviour and avoid turning a slow receiver into an ever-growing retry storm.
Operational testing
Run controlled tests where the endpoint returns 500 for several attempts, times out after receiving the event, becomes unavailable for an hour and then recovers. Confirm that retries back off, duplicates are harmless and dead-lettered events can be replayed safely.
Retry budgets
A retry policy should have both an attempt budget and a time budget. Ten retries in ten seconds is not equivalent to ten retries over several hours. Use the message's event importance and customer expectations to determine an appropriate maximum delivery window. The platform should stop retrying once the event is no longer useful.
Jitter
Add randomized jitter to retry delays. If a provider outage causes ten thousand webhook failures at the same instant, deterministic backoff can cause all ten thousand endpoints to be hit simultaneously again. Jitter spreads the recovery load and protects both the platform and customer infrastructure.
Endpoint health
Maintain endpoint-level health metrics without permanently disabling an endpoint based on a short incident. A customer endpoint may recover quickly. Use consecutive failures, retry age and response patterns to decide whether to reduce delivery pressure or notify the customer.
Replay controls
Manual replay should require appropriate authorization and should show the original event, last failure and number of previous attempts. The replay action should be audited. Bulk replay should be carefully controlled because a large dead-letter queue can generate a sudden traffic spike.
Retry notifications
Customers should be able to see when webhook delivery has repeatedly failed. An endpoint-level alert can prevent a dead-letter queue from growing unnoticed. The notification should identify the affected webhook configuration and last failure category without exposing message content.
Avoiding retry storms
If thousands of events fail because one endpoint is down, retries should be distributed over time. A circuit-like delivery suppression policy can temporarily reduce attempts while preserving the events for later delivery. This protects the endpoint when it returns.
Replay after recovery
After an endpoint recovers, replay should be controlled by age and volume. Start with a small sample, confirm successful acknowledgement and then increase the replay rate. This prevents a recovered endpoint from being overwhelmed by its historical backlog.
Retry observability
Store every webhook attempt with timestamp, response code, latency and failure classification. Keep the original event ID constant across attempts. This lets support see that five delivery attempts were made for one event rather than mistakenly counting five events.
Endpoint suspension policy
If an endpoint fails continuously, the platform can temporarily suspend automatic attempts after the configured retry window. Suspension should be visible and reversible. The event data should remain available for replay after the customer corrects the endpoint.
Customer guidance
Tell customers that returning a 2xx response after durable acceptance is the recommended pattern. They should not wait for downstream CRM or database work before acknowledging. This single rule greatly improves webhook reliability.
Retry ownership
The webhook sender owns delivery retry, while the customer owns business retry after successful webhook acknowledgement. Keeping these responsibilities separate prevents the platform from attempting to repeat customer business actions and prevents customers from asking the platform to replay SMS messages when they only need an event.
SLA communication
Document the maximum automatic retry period and the expected acknowledgement time. Customers can then design their endpoint infrastructure and alerting around a known delivery contract rather than an undefined best-effort process.
Retry testing under scale
Run a test where many events fail simultaneously and then recover. Verify that backoff and jitter prevent a synchronized retry storm, that dead-letter records remain durable and that recovery does not overload customer endpoints.
Final reliability principle
A webhook retry system should make eventual delivery likely without making failure more dangerous. Bounded retries, idempotent receivers, durable events and controlled replay provide that balance.
Production implementation detail
The retry policy should also communicate endpoint ownership. If an endpoint repeatedly fails, the customer administrator should be able to identify the URL, last successful delivery, current retry state and corrective action. This prevents support teams from repeatedly replaying events to an endpoint that has not been repaired. After correction, a small controlled replay should verify recovery before the complete dead-letter backlog is released.