123eworld Knowledge Hub → SMS API → Page 303
SMS API Pagination: Designing Message Logs, Reports and Status APIs for Large Data
A practical developer reference designed to solve real implementation and production problems around sms api pagination: designing message logs, reports and status apis for large data.
Why pagination matters
Message logs can contain millions of records. Returning all results in one API response is slow, expensive and unreliable. Pagination lets applications retrieve data in controlled portions.
Offset pagination
Offset and limit are easy to understand but become less efficient at large offsets and can produce inconsistent results when records change during traversal.
Cursor pagination
Cursor-based pagination uses a stable position such as an encoded message ID and timestamp. It is often better for large, changing datasets.
Stable ordering
Pagination requires deterministic ordering. Use a unique tie-breaker when timestamps can be identical.
Filters
Allow filters such as date range, status, sender profile and application where appropriate. Validate combinations and protect expensive unbounded queries.
Page size
Set a sensible default and maximum page size. Let clients request smaller pages when bandwidth is limited.
Next cursor
Return a machine-readable cursor rather than requiring clients to calculate offsets themselves.
Reports and exports
Large exports should normally use asynchronous jobs rather than huge synchronous API responses.
Consistency
Document whether pagination provides a snapshot or reflects changes during traversal.
Security
Apply tenant authorization before pagination and filtering so one customer cannot infer another customer's data.
Testing
Test duplicate records, deleted records, new records inserted during traversal and large datasets.
Reference response
items + pagination metadata + next cursor + optional has_more indicator.
Security and privacy
Treat phone numbers, message content, credentials and delivery data as sensitive operational information. Avoid unnecessary logging and ensure tenant authorization is applied before data access.
Production reliability
Design for timeouts, duplicates, retries, provider failures and delayed events. A messaging platform is asynchronous infrastructure, so success-path testing alone is insufficient.
Developer-first principle
The public API should hide unnecessary telecom complexity while exposing enough structured information for developers to build correct integrations.
Related 123eworld guides
Explore the 123eworld SMS & WhatsApp Knowledge Hub for related developer, API, routing and production guides.
Cursor design
A cursor should be opaque to the client. It can encode the last timestamp and unique ID used by the server's ordering rule, but the client should treat it as an uninterpreted token. This allows the server to change internal storage without changing the API contract.
Stable ordering
If two messages have the same created timestamp, use a unique ID as a tie-breaker. Otherwise a record can appear twice or be skipped between pages. The ordering expression and cursor encoding must use the same fields.
Filtering before pagination
Authorization and filters must be applied before selecting the page. Never fetch a large unrestricted dataset and then filter it in application memory. This is both a performance and a data-isolation concern.
Changing data
Message logs continue to change while a client is paginating. Cursor pagination can reduce some inconsistencies, but it does not automatically create a full snapshot. Document whether newly inserted or updated records can appear during traversal.
Large exports
If a customer needs millions of records, provide an asynchronous export job. The API can return a job ID and later expose a download or transfer mechanism. This prevents long-running synchronous requests from consuming API workers.
Testing
Test page boundaries with identical timestamps, records inserted between requests, records deleted or archived during traversal and maximum page sizes. Verify that authorization is applied consistently to every page.
Snapshot versus live pagination
If a customer needs an exact historical report, an asynchronous export or snapshot query may be more appropriate than ordinary live pagination. Document whether records can change between pages. For financial reconciliation, deterministic export semantics are usually preferable.
Indexing strategy
Filters and ordering should match database indexes. A common pattern is an index beginning with tenant scope and then created timestamp plus unique message ID. Exact indexing depends on the storage engine, but the principle is universal: authorize and filter efficiently before sorting a large dataset.
Cursor expiry
Cursors can have an expiry period so the server does not need to preserve arbitrary historical query state forever. If a cursor expires, return a clear error and instruct the client to restart from the original filter.
API evolution
Avoid exposing database offsets or internal primary keys as pagination contracts. Opaque cursors allow the storage implementation to change without breaking customer integrations.
Report API design
A status endpoint can support cursor pagination for interactive applications, while a reporting API can provide aggregated counts for dashboards. Do not force a client to download millions of individual messages just to display daily totals.
Performance protection
Set maximum page sizes and maximum date ranges for expensive queries. For larger ranges, direct customers to asynchronous exports. This protects the API from accidental or malicious resource-intensive queries.
Privacy-aware pagination
Authorization filters must be applied before pagination and before count calculations. Even a total-count field can leak information about another tenant if the query scope is wrong.
Cursor security
Cursors should not expose internal database IDs, tenant identifiers or sensitive filter information in plain text. Use opaque encoding and, where appropriate, integrity protection. The server remains authoritative for authorization.
Count semantics
If an API returns total counts, document whether the count is exact, approximate or limited by a maximum. Exact counts over very large datasets can be expensive and should not silently degrade API performance.
Export lifecycle
Asynchronous exports should have job status, creation time, expiry and access control. Export files can contain sensitive message data and must not become publicly accessible simply because the API created them.
Pagination and caching
If a status list is polled frequently, support conditional requests or appropriate caching where the API semantics permit it. This reduces repeated database work. Do not cache data across tenants or authorization scopes.
Search versus pagination
Pagination is for controlled traversal; search filters are for narrowing the dataset. Combining both carefully prevents expensive unbounded queries. Document which filters are indexed and which may require asynchronous reporting.
Pagination contract
Document default page size, maximum page size, ordering, cursor lifetime, filter semantics and consistency model. These details are not implementation trivia; they determine whether an integration can safely synchronize large datasets.
Final pagination principle
Make large datasets predictable. Stable ordering, opaque cursors, authorization-aware filtering and asynchronous exports are the foundation of scalable reporting APIs.
Production implementation detail
For synchronization integrations, cursor pagination should be combined with a reliable checkpoint strategy. Store the cursor only after the application has successfully processed the page. If processing fails, retry the same cursor rather than advancing and risking data loss. Because duplicate records can still occur in some live-data models, downstream processing should remain idempotent. These rules make pagination suitable for CRM, ERP and reporting synchronization rather than only interactive screens.
Final developer checklist
A reporting API should avoid returning a huge exact total by default if counting the complete dataset is expensive. Where appropriate, provide a bounded or asynchronous count. For interactive dashboards, aggregate endpoints are usually better than repeatedly paginating through individual messages. This keeps the API responsive as the knowledge-base platform grows.
Closing reference note
For enterprise synchronization, document a recommended page-processing loop: request the first page, process items idempotently, commit application changes, then persist the next cursor. If processing fails, restart from the last successfully committed cursor. This gives developers a practical recovery pattern and prevents skipped records during long-running synchronization jobs.
Reference implementation reminder
For status APIs, prefer filters that map to indexed business fields such as message ID, client reference, date range and status. Avoid encouraging customers to request unbounded historical data through a single synchronous endpoint.
Release and maintenance note
A pagination endpoint should also document whether the last page is represented by an absent cursor, a null cursor or an explicit has_more flag. Pick one consistent convention across related APIs so developers do not have to implement different loops for every resource.
Consistency principle
For developers consuming multiple pages, consistent pagination conventions are more valuable than clever pagination. Keep cursor format, page-size parameter naming, ordering semantics and error behaviour aligned across messages, reports and status resources.