123eworld Knowledge Hub → Transactional SMS → Page 157
Transactional SMS API Queue Architecture: Durable Queues, Workers, Backpressure, Dead Letters and Recovery
An engineering guide to durable SMS processing queues, worker design, backpressure, dead-letter handling, queue recovery and predictable throughput.
Why queues are the heart of SMS processing
A transactional SMS API should normally separate customer request acceptance from provider submission. A durable queue absorbs traffic bursts, protects provider connections and allows workers to process messages at a controlled rate. Without a queue, a sudden campaign or provider slowdown can turn a temporary spike into API-wide instability.
Durable queue requirements
A production queue should survive worker crashes and broker restarts. A message should not disappear simply because a consumer process terminated after claiming it. Use durable storage, acknowledgements and visibility or lease semantics appropriate to the queue technology.
Message lifecycle
A useful lifecycle is accepted, queued, claimed, processing, submitted, awaiting receipt and completed or failed. Keep the queue state separate from the business message status. The queue describes work execution; the message status describes customer-visible progress.
Worker design
Workers should be stateless where practical and should safely restart. They need bounded concurrency, provider-specific limits and controlled shutdown behaviour. A worker should acknowledge queue work only after the required durable state has been recorded.
Backpressure
When providers slow down, workers should reduce concurrency instead of consuming unlimited messages. Queue depth and queue age become capacity signals. Backpressure protects databases and providers while making customer-visible delay explicit.
Priority queues
Not every message has identical urgency. Priority can be represented by separate queues or scheduling weights, but the number of priority classes should remain small. Priority must not allow one tenant to consume all worker capacity.
Dead-letter queues
Messages that repeatedly fail should eventually leave the normal retry path. A dead-letter queue preserves them for investigation without allowing a bad payload to consume worker capacity indefinitely. Replay must re-run validation, suppression and authorization rules.
Queue fairness
Use tenant-aware quotas or weighted scheduling when shared infrastructure serves many customers. A large customer should be able to use its purchased capacity without starving smaller tenants.
Scaling workers
Scale workers using queue age, depth, provider capacity and processing latency rather than queue length alone. Adding workers during a provider outage can increase failed requests and retries instead of improving throughput.
Recovery
After a broker or worker outage, recover in stages. Restore durable state first, validate queue connectivity, release a controlled amount of work and monitor provider and database load. Avoid restarting every worker at maximum concurrency simultaneously.
Testing
Test worker crashes, duplicate queue delivery, broker restart, provider slowdown, poison messages, dead-letter replay and tenant overload. Verify that logical message identity remains stable across every retry.
Production checklist
Define queue durability, acknowledgement semantics, visibility timeout, worker concurrency, backpressure, priority, fairness, dead-letter handling and recovery procedures.
Developer takeaway
A queue is not simply a buffer. It is the control plane that determines how an SMS platform behaves when demand, provider capacity or internal resources change.
Implementation architecture
Implement workers around a durable message state and explicit acknowledgement rules. A worker should claim work with a bounded lease, record processing progress and acknowledge only after durable state is safe. If the worker dies, the queue should make the item available again. Dead-letter processing must create a controlled replay path rather than bypassing ordinary validation, suppression and authorization.
Operational reference
Queue depth alone is not a sufficient health metric. A queue containing many messages may be healthy if it is draining quickly, while a small queue can be dangerous if workers have stopped. Queue age and drain rate should therefore be monitored together.
Operational reference
Poison messages should be identifiable without exposing sensitive content. Store a reason code, message ID and attempt count so operators can inspect the failure while keeping payload access restricted.
Operational reference
Worker deployment should use graceful draining. Stop claiming new work, finish safe operations, record state and then terminate. This avoids unnecessary redelivery during routine deployments.
Production checklist
Production check: verify worker crash recovery, duplicate queue delivery, dead-letter replay, provider slowdown, priority fairness and controlled shutdown.
Additional implementation guidance
A queue should preserve the distinction between accepted business state and work-execution state. If a worker is processing a message, the message record should already identify the logical operation, while the queue controls which worker is responsible for the next processing step.
Additional implementation guidance
Visibility timeouts must be long enough for normal processing but short enough to recover a crashed worker. If the timeout is too short, the same work can be claimed by multiple workers while the first worker is still active.
Additional implementation guidance
Worker concurrency should be configurable per provider route. A provider that permits high throughput can use more workers than a constrained route. This avoids forcing every provider into the same capacity model.
Additional implementation guidance
Queue metrics should include enqueue rate, dequeue rate, active workers, oldest item age and dead-letter volume. Together these show whether the queue is healthy, draining or accumulating work.
Additional implementation guidance
During recovery, release work gradually. A large backlog can tempt operators to start many workers, but doing so can overload the database or provider and create a second incident. Controlled ramp-up is usually safer.
Additional implementation guidance
Queue messages should contain references to durable business state rather than unnecessarily large payloads. This reduces queue size and makes it easier to change message processing logic without duplicating sensitive content.
Additional implementation guidance
Queue architecture should also support graceful deployment. Workers can stop accepting new work, finish safe processing, persist state and exit. This reduces duplicate processing during routine releases.
Reference architecture note
Queue capacity planning should account for provider throughput, database write capacity and callback processing. The queue is only as healthy as the slowest downstream component. If provider throughput is 10,000 messages per second but database status updates can sustain only 6,000, increasing workers above that point creates backlog or database pressure. Document the limiting component for each route and keep operational headroom. Queue workers should also expose graceful shutdown and health signals so orchestration systems do not mistake a busy worker for a failed worker.
Advanced production guidance
Queue priorities should be explicit business policy. If OTP or security notifications receive higher priority, the platform should reserve enough capacity for them rather than simply placing them at the front of an unbounded shared queue.
Advanced production guidance
Dead-letter queues need ownership. Every dead-letter reason should have a documented next action, such as correct configuration, contact the customer, wait for provider recovery or perform a controlled replay.
Advanced production guidance
Queue storage should be monitored for retention growth. A backlog that remains after the incident is resolved can consume disk and increase recovery time.
Advanced production guidance
Queue changes should be load-tested before production. Changing acknowledgement or visibility behaviour can alter duplicate processing even when the application code is unchanged.
Engineering reference
A queue should support controlled visibility into individual work items without exposing sensitive message content. Operators need message ID, tenant, age, attempt count and reason codes more often than the full SMS body.
Engineering reference
Queue partitioning can be useful for high-volume tenants or specialized workloads, but it adds operational overhead. Use it when measured isolation or capacity requirements justify the additional infrastructure.
Engineering reference
Worker autoscaling should include a cool-down period. Rapidly adding and removing workers can create connection churn and make provider load less predictable.
Final engineering guidance
A queue's operational contract should specify what happens to work that remains invisible or unacknowledged beyond its lease. Make the recovery behaviour explicit so workers do not accidentally create duplicate processing.
Final engineering guidance
Keep queue configuration versioned. Changes to visibility, priority or retry handling can change message behaviour even when application code remains unchanged.
Final acceptance test
The final acceptance test should restart workers while messages are actively processing and verify that every logical message remains traceable. The queue must recover work without creating a second customer operation.
Continue through the 123eworld Knowledge Hub
Explore the complete 123eworld Knowledge Hub for practical SMS API, transactional messaging, queue, security and developer architecture guides.
Visit 123eworld.com for messaging and digital communication services.