123eworld Knowledge Hub → SMS API → Page 337
SMS API Webhook Retry Design: Reliable Callback Delivery Without Duplicates
A practical, developer-focused reference designed to solve real integration and production messaging problems.
Why this topic matters
SMS API Webhook Retry Design: Reliable Callback Delivery Without Duplicates is an advanced developer reference for building a dependable messaging platform. The goal is to solve the real integration and production problems that appear after a simple SMS API call works: security boundaries, retries, scale, observability, failure recovery and long-term maintainability.
Why webhook retries exist
Networks fail, customer endpoints restart and temporary errors occur. A reliable webhook design therefore expects delivery attempts to be repeated. The receiving system must process duplicates safely while giving the sender a clear acknowledgement when an event has been durably accepted.
Acknowledge quickly
Do not keep the provider connection open while performing expensive database queries, downstream API calls or customer notifications. Authenticate, validate and persist the event, then acknowledge according to the provider contract.
Idempotent event processing
Use the provider event ID as an idempotency key where possible. If the same event arrives again, the receiver should recognize it and return a successful acknowledgement without applying the business transition a second time.
Retry classes
A temporary network error, a 5xx response and a permanent 4xx validation error should not necessarily have the same retry behaviour. Document which errors invite another attempt.
Backoff
Exponential backoff with jitter prevents thousands of receivers from retrying simultaneously after a shared outage. The receiving platform should also avoid creating an internal retry storm.
Durable inbox
A webhook inbox table or durable event stream can separate callback receipt from business processing. This is especially useful when provider callbacks arrive faster than downstream consumers can process them.
Ordering
Delivery events can arrive out of order. Consumers should use event timestamps, attempt identifiers and explicit state-transition rules rather than assuming network arrival order equals business order.
Dead letters
Events that repeatedly fail business processing should move to a controlled dead-letter workflow with the original event identity and failure reason preserved.
Replay
Authorized operators should be able to replay a failed event through the normal idempotent processor. Replay must not bypass tenant authorization or create a second logical notification.
Monitoring
Measure callback success, retry volume, processing latency, duplicate rate and dead-letter growth. These metrics reveal both receiver health and provider behaviour.
Customer endpoint webhooks
If the SMS platform forwards events to customer systems, apply the same principles: signed requests, event IDs, fast acknowledgement, retry policy and replay-safe processing.
Testing
Simulate duplicate delivery, delayed delivery, out-of-order events, customer endpoint downtime and provider retry bursts.
Developer takeaway
Webhook reliability is achieved through durable acceptance, idempotent processing and explicit retry semantics, not by assuming callbacks arrive exactly once.
Retry storm prevention
If a customer endpoint is down, blindly retrying every event at the same interval can amplify the outage. Use exponential backoff, jitter and bounded concurrency. The platform should maintain a per-destination or per-tenant retry policy where required so one failing customer does not consume all webhook workers.
Replay safety
A replay tool should use the same event processor as normal delivery. It should not directly call business functions because that bypasses idempotency, authorization and audit controls.
Security baseline
Treat recipient numbers, message content, credentials, provider evidence and customer configuration as sensitive. Use TLS, least privilege, tenant-scoped authorization and safe logging. Never put secrets into URLs, error messages or ordinary analytics fields. Security should be enforced at the service boundary and repeated at important downstream boundaries rather than assumed because the request passed through an API gateway.
Production troubleshooting method
Start with the request or logical message ID and follow the lifecycle through authentication, validation, durable acceptance, queue processing, provider interaction, delivery evidence and webhook handling. Compare the affected path with a known-good request. This method prevents teams from changing routing or retry settings before they know which layer actually failed.
Implementation checklist
Before production use, verify authentication, authorization, idempotency, rate limits, queue durability, provider routing, delivery reporting, monitoring, auditability, retention, backup and rollback. Test both successful and deliberately failed paths. A messaging feature is production-ready only when its failure behaviour is as well defined as its happy path.
Related 123eworld Knowledge Hub Guides
Visit the complete 123eworld Knowledge Hub for the wider SMS API, WhatsApp API, messaging and developer reference library.
Retry architecture
Webhook delivery should use a durable queue or equivalent mechanism so a temporary customer endpoint failure does not lose an event. Each delivery attempt should record destination, attempt number, response class and next retry time. Exponential backoff with jitter prevents synchronized retry storms. A maximum event age should also be defined; an event that remains undeliverable forever can otherwise consume storage and worker capacity. After the maximum age, move the event to a dead-letter workflow and preserve enough evidence for authorized replay.
Receiver guidance
Customer systems should acknowledge a valid event quickly and process it asynchronously when business work is expensive. They should store the event ID before applying the business transition where possible. If an event is received again, the stored identity allows the system to return success without repeating the action. Documentation should explicitly state that webhook delivery is at-least-once unless the platform has a stronger verified guarantee.
Failure matrix
Document behaviour for connection timeout, DNS failure, 2xx acknowledgement, 4xx rejection, 5xx response, malformed response and prolonged customer downtime. Each class should have a clear retry or terminal rule. This prevents different worker implementations from developing inconsistent behaviour.
Advanced implementation note
The receiving side should expose delivery attempts as an operational concept. A webhook event may have been successfully persisted even though its final customer endpoint delivery failed. Keeping these states separate lets the platform retry the customer callback without reprocessing the original provider event. It also allows support to answer whether the provider event was received, whether it was normalized correctly and whether the customer system acknowledged it. This separation is particularly valuable for banking, ecommerce and transactional applications where an incorrect duplicate callback can trigger a duplicate business action. Event identity, delivery identity and logical message identity should therefore be distinct but correlated.
Production architecture guidance
A mature webhook system should also distinguish provider retry policy from platform retry policy. The provider controls how it retries callbacks to the platform, while the platform controls how it retries callbacks to customers. These are separate delivery chains and should have separate metrics. If the provider sends the same receipt five times, the platform should record one logical event and several receipt attempts. If the platform then sends that event to a customer three times because the customer endpoint returned errors, those attempts should not create three delivery events in the customer's business system. Explicit identities at each layer make this possible. The platform can also expose webhook delivery history to authorized customers so they can see whether their endpoint acknowledged an event. This reduces support tickets and makes integration failures easier to diagnose. A replay mechanism should preserve the original event ID and indicate that the delivery was a replay rather than a new provider event.
Final engineering review
A final engineering review should verify the failure cases, not just the normal path. For each page's subject, test what happens when the dependency is unavailable, when a request is repeated, when data arrives late and when configuration changes during processing. Record the expected outcome and compare it with the actual result. This creates a practical acceptance record that can be reused during future releases. The platform should also expose safe operational identifiers so support can trace an issue without requesting secrets or unnecessary personal data. These controls make the implementation easier to operate and easier to trust as customer traffic grows.
Reference implementation note
The same discipline should apply when the provider changes its retry behaviour. Keep provider-specific retry semantics inside the adapter or delivery policy so customer applications continue to see one documented contract. If a provider retries aggressively, the platform should still deduplicate the resulting event stream. If a provider stops retrying after a short window, the platform may need its own reconciliation or recovery process. The important point is to make the final customer-facing behaviour independent of undocumented provider assumptions.