Transactional SMS API Queue Architecture: Durable Queues, Workers and Backpressure

Why queues are central

An SMS API should not make the customer-facing request wait for every provider operation. A durable queue separates API acceptance from downstream processing and allows the platform to absorb short bursts without turning every provider delay into an API timeout.

Durable message handoff

The queue should receive work only after the application has established a durable source of truth. An outbox pattern can bridge database state and queue publication when losing a message between those two operations would be unacceptable.

Producer-consumer model

API services produce message work while workers consume it. This separation allows each side to scale independently and makes provider-specific processing asynchronous.

Worker concurrency

Worker count should be controlled by downstream capacity rather than simply maximizing CPU utilization. Too many workers can trigger provider throttling, database contention and connection exhaustion.

Backpressure

Backpressure prevents the platform from accepting work faster than it can safely process. It can operate through API rate limits, queue admission controls, worker limits or provider-specific throttles.

Visibility timeout

Queue systems that use visibility or lease timeouts must set them longer than normal processing while allowing recovery from crashed workers. Extensions may be necessary for unusually slow provider calls.

Acknowledgement

A worker should acknowledge or remove work only after the required durable state transition has succeeded. Acknowledging too early can permanently lose a message.

Dead-letter queues

Repeatedly failing messages should eventually move to a dead-letter queue or quarantine state. Operators can then inspect the reason without allowing one poison message to consume worker capacity indefinitely.

Priority queues

Critical OTP and security notifications may require higher priority than bulk traffic. Priority should be implemented with fairness so a continuous stream of urgent work does not completely starve normal messages.

Queue ordering

Strict global FIFO ordering is expensive and often unnecessary. Define the business ordering requirement first. Some systems need ordering only per customer, transaction or campaign.

Queue monitoring

Monitor queue depth, oldest-message age, processing rate, retry count, worker utilization and dead-letter volume. Queue depth alone can remain stable while customer-visible latency becomes unacceptable.

Implementation checklist

Define durability, publication semantics, worker concurrency, backpressure, acknowledgement, retry and dead-letter behaviour, priority policy, ordering requirements and queue observability.

Queue durability

Choose a queue implementation whose durability matches the business consequence of losing an accepted message. For transactional banking or security notifications, an in-memory best-effort queue may be inappropriate because an infrastructure restart could lose work.

Persistence should be confirmed before the producer considers the handoff successful.

Outbox pattern

When a database transaction creates the message record and a separate operation publishes queue work, there is a small consistency gap. An outbox record written in the same database transaction can represent the pending publication. A publisher then reads the outbox and safely retries publication until the queue accepts it.

This pattern is especially useful when the database is the authoritative source of message identity.

Worker lease recovery

A worker can crash after claiming a message but before completing it. Lease-based queues allow the work to become visible again after a timeout.

The processing design should be idempotent because a lease can expire while the original worker is still running.

Poison-message isolation

A malformed provider response or unexpected data condition can cause the same message to fail repeatedly. Without isolation, workers may repeatedly consume the same bad item.

Dead-letter or quarantine handling should capture the failure category and enough context for an operator to decide whether reprocessing is safe.

Backpressure design

Backpressure should protect both the platform and the customer experience. If queues are already beyond the safe latency window for OTP traffic, accepting additional OTP work may create messages that expire before processing.

Admission control should therefore consider traffic class and message expiry, not only raw queue size.

Queue partitioning

High-volume tenants or traffic classes can be partitioned across queues or shards. Partitioning can reduce contention and improve isolation, but it also introduces routing and rebalancing complexity.

Choose partitions based on actual throughput and fairness requirements.

Queue testing

Test worker crashes, duplicate delivery, queue-node failure, delayed acknowledgement, full queues, dead-letter routing and recovery after an outage. Measure not only whether messages eventually process but whether duplicate provider submissions remain impossible.

Production scenario

During a festival sale, API traffic doubles for fifteen minutes. The durable queue absorbs the safe burst, workers remain below provider limits, OTP traffic uses reserved capacity and the queue drains after the peak. Operators monitor oldest-message age rather than assuming that a large queue is automatically healthy.

Queue/database consistency

When the database is authoritative, avoid designing a queue as a second independent source of truth. A queue item should identify the durable message and processing operation. If the same message is delivered twice by the queue, the worker can use the message state and provider-attempt records to decide whether another submission is safe.

Queue admission

Admission rules can reject or defer work when a tenant exceeds its quota or when a traffic class has no safe capacity. The response should clearly distinguish an API validation failure from temporary capacity protection so client applications know whether retrying later is appropriate.

Worker autoscaling

Autoscaling workers from queue depth alone can be dangerous. If the provider is throttling, adding workers may increase failed attempts without increasing useful throughput. A better signal combines queue age, processing rate and downstream capacity.

Operational runbook

The queue runbook should cover rising queue age, worker crashes, dead-letter growth, provider throttling, queue infrastructure failure and backlog recovery. Operators should know when to add capacity and when adding capacity would make the problem worse.

Developer rule

Treat queue delivery as at-least-once unless the technology and architecture explicitly guarantee otherwise. Make message processing idempotent so duplicate work is safe.

Queue security

Queue access should be restricted by service role. An API producer should not automatically be able to consume or delete every queue. Worker credentials should be limited to the queues they process, and administrative queue operations should be separately controlled and audited.

Queue metrics

Useful queue metrics include arrival rate, processing rate, oldest item age, retry rate, visibility timeout expirations, dead-letter rate and successful acknowledgement rate. These metrics should be segmented by traffic class and tenant where cardinality remains manageable.

Queue acceptance test

Stop workers while continuing safe API traffic, verify that durable work accumulates without loss, restart workers and confirm that the backlog drains without duplicate provider submissions. Repeat the test with one queue partition unavailable if the architecture supports partitioning.

Reference rule

A queue is a reliability boundary only when work survives infrastructure failure and duplicate processing is safe.

Design decision: FIFO or priority

Choose ordering according to the business requirement. If one customer's payment alerts must remain ordered, partition by customer or transaction while allowing independent customers to process concurrently. Global FIFO can reduce throughput unnecessarily and make one slow item block unrelated traffic.

Design decision: queue count

One queue is simpler, but separate queues can isolate traffic classes and failure domains. Too many queues create operational overhead and make capacity balancing difficult. Start with clear classes and partition only where the workload justifies it.

Design decision: synchronous fallback

Avoid turning queue failure into direct provider calls from the API as an emergency shortcut. That can bypass rate limits, idempotency and observability. A safer fallback is to return a temporary service error unless a fully tested alternate path exists.

Failure-mode walkthrough

Consider a worker crash after provider acceptance but before the queue acknowledgement. The queue may redeliver the work. The worker must consult the durable message and provider-attempt state before submitting again. If the provider supports idempotency, use the same provider identity; otherwise place the message into an uncertainty workflow rather than blindly sending it twice.

This example shows why queue correctness cannot be separated from message identity and provider semantics.

Architecture checklist

Confirm durable publication, at-least-once assumptions, idempotent workers, controlled concurrency, backpressure, priority isolation, dead-letter handling, queue security, operational metrics and recovery procedures. Then test the complete failure path rather than checking each component independently.

Operator decision tree

When queue age rises, first determine whether arrivals exceed service rate, workers are failing, provider capacity is constrained or a priority class is consuming reserved capacity. Add worker capacity only when downstream capacity exists; otherwise reduce admission or traffic safely.

Practical implementation

Keep queue configuration versioned and auditable. Record the queue, priority class and processing policy used for each message so operators can reconstruct why work was delayed or retried.