123eworld Knowledge Hub → Transactional SMS API → Page 250
Transactional SMS API Webhook Retry System: Delivery Attempts, Backoff, Dead Letters and Replay
A practical developer reference covering implementation, architecture, security, testing and production controls for transactional sms api webhook retry system: delivery attempts, backoff, dead letters and replay.
Webhook delivery as a queue
Customer webhook delivery should be asynchronous. The provider callback handler should normalize the event and place it into a durable delivery queue rather than waiting for the customer endpoint.
Attempt records
Record endpoint, event ID, attempt number, response class, latency and next retry time. This makes retry behaviour observable and prevents an opaque loop.
Backoff
Use bounded exponential backoff with jitter. Retry intervals should reflect the fact that customer endpoints may be temporarily overloaded.
HTTP classification
Successful 2xx responses normally acknowledge delivery. Selected 5xx responses and network failures are usually retryable, while many 4xx responses indicate configuration or authorization problems that require a different policy.
Endpoint isolation
Each customer endpoint should have its own concurrency and retry budget. One broken endpoint must not consume the entire webhook delivery pool.
Dead letters
Events that exhaust retries should become visible dead letters with safe replay capability. Dead-letter retention should be defined separately from ordinary event retention.
Replay safety
Replay the same event ID and preserve the original logical message relationship. Customers should be able to process the replay idempotently.
Secret rotation
Webhook retry can span a secret rotation. Store the appropriate signing-key version with the event or delivery configuration so a replay remains verifiable.
Backlog management
Monitor webhook queue age and endpoint-specific backlog. Pause or slow a failing endpoint while allowing healthy endpoints to continue.
Customer experience
Provide clear documentation about retries, event IDs and expected duplicate handling. Customers should never assume exactly-once HTTP delivery.
Testing
Test slow endpoints, connection failures, 400/401/404 responses, 429 throttling, 500 errors, duplicates and replay.
Reference flow
Normalized event → durable endpoint queue → signed delivery → response classification → retry or success → dead letter or replay.
Error and failure model
Document what happens when each dependency is slow, unavailable, invalid or ambiguous. A good messaging system does not merely catch exceptions; it turns them into predictable states. This includes deciding when work is retried, when it is held, when it becomes a dead letter and when a human or reconciliation process is required.
Developer usability
Every technical feature should be understandable through documentation, examples and stable identifiers. Developers should be able to diagnose an issue using a request ID or message ID rather than requiring direct database access. This is particularly important for asynchronous systems where the initial API call and final delivery outcome happen at different times.
Security and privacy
Apply authentication, authorization, tenant isolation and least privilege throughout the feature. Keep sensitive values out of logs and traces, and define retention for operational evidence. Security controls should be tested under concurrency and failure, not only on the happy path.
Observability
Measure both success and degradation. Include latency, error rates, queue age, retry counts, provider outcomes and resource saturation where relevant. Correlation identifiers should survive service boundaries so an operator can reconstruct the lifecycle of a message.
Production operations
Provide runbooks for common incidents, ownership for important alerts and safe administrative controls. Configuration changes should be auditable and reversible. A feature is not production-ready if engineers cannot explain how to diagnose and contain its failures.
Reference scenario
Consider a customer sending a large transactional batch while a provider is intermittently unavailable. The correct system authenticates the request, applies quotas, stores logical message identity, queues work, retries only transient failures, protects the provider from retry storms, records delivery events and exposes stable status to the customer. When the provider recovers, backlog drains gradually and reporting reconciles late events.
Final checklist
Before release, verify API contract, authorization, idempotency, queue behaviour, provider handling, observability, security, retention and recovery. Test both ordinary traffic and realistic failure sequences. Confirm that documentation describes what the client should do after timeout, throttling, duplicate submission and asynchronous delivery failure.
Retry ownership
The webhook delivery system should own delivery retries rather than making the provider callback handler responsible for them. This separates inbound provider processing from outbound customer availability.
Endpoint-specific policy
A customer endpoint that repeatedly returns 429 may need slower retry release than one that returns transient 503 responses. Policies can be normalized while retaining endpoint-specific telemetry.
Delivery attempt identity
Each delivery attempt should have its own identifier while retaining the same event ID. This allows support to distinguish one logical event from multiple network attempts.
Ordering guarantees
If a customer requires ordering for certain events, document the scope. Strict global ordering can reduce throughput; per-message or per-resource ordering is often more practical.
Replay authorization
Replay should require an authenticated operator or customer action with appropriate scope. It should not be possible to replay another tenant's event by guessing an ID.
Event retention
Retain failed events long enough for normal operational recovery, but do not keep sensitive event data indefinitely. Dead-letter retention should have a documented lifecycle.
Testing scenario
Create an endpoint that alternates between 500 and 200 responses. Verify that retries occur with backoff, successful acknowledgment stops further delivery and the event remains traceable throughout.
Webhook delivery fairness
Schedule webhook work so a high-volume customer cannot monopolize workers. Per-endpoint concurrency and tenant-level quotas allow the system to maintain predictable service across many customers.
Delivery state
Maintain a state such as pending, delivering, acknowledged, retry_scheduled or dead_lettered. State transitions should be atomic enough that worker restarts do not create uncontrolled duplicate attempts.
Replay semantics
A replay should preserve the original event ID and clearly indicate that it is a replayed delivery attempt. Customers can then deduplicate the business event while still observing the retry history.
Webhook observability
Track endpoint success rate, response latency, retry count, queue age and signature failures. A healthy global webhook metric can hide one customer endpoint that is failing continuously.
Operational runbook
The runbook should explain how to pause a failing endpoint, inspect a dead letter, rotate its secret and safely replay an event. These actions should be tenant-scoped and audited.
Retry storm control
A webhook outage can create thousands of pending deliveries. Use endpoint-specific concurrency, maximum retry age and global protection so recovery traffic does not overwhelm the customer's endpoint when it returns.
Customer visibility
Expose safe webhook delivery history so customers can see event ID, attempt count, last response and next retry time. This reduces support dependency and helps developers diagnose endpoint problems.
Security interaction
Webhook retries must remain signed and authorized after every attempt. A replay mechanism must not bypass normal signature generation or tenant checks simply because the event originated internally.
Webhook ordering
If events for one message can arrive out of order, the consumer should use event timestamps, sequence values or state precedence rather than assuming arrival order. A delivered event should not be overwritten by a later-arriving submitted event.
Endpoint lifecycle
When a customer disables an endpoint, queued deliveries should follow a defined policy. Some systems discard future attempts; others retain events for a configured period. The decision should be explicit and auditable.
Webhook capacity review
Measure peak endpoint concurrency, average and tail response time, retry volume and queue age. Use these measurements to size delivery workers and prevent a single customer from creating a platform-wide backlog.
Production depth and implementation guidance
Webhook systems should document the difference between event delivery and business processing. A 200 response from the customer endpoint means the endpoint acknowledged receipt; it does not necessarily mean the customer's internal order system has completed its work. This is why the platform should send stable event IDs and the customer should persist them before acknowledging when exactly-once business handling is important. The messaging platform should also keep its own delivery state separate from the customer's business state. This separation makes retries safe, supports replay and allows the customer to recover from an internal outage without asking the provider to recreate the original delivery event.
Continue through the 123eworld Knowledge Hub
Explore the 123eworld Knowledge Hub for practical SMS API, transactional messaging and developer architecture guides.
Visit 123eworld.com for messaging and digital communication services.