123eworld Knowledge Hub → SMS API → Page 353
SMS API Unicode and SMS Encoding: GSM-7, UCS-2 and Message Segmentation
A developer-focused reference designed to solve real SMS API architecture, integration, security, scalability and production problems.
Why this topic matters
SMS API Unicode and SMS Encoding: GSM-7, UCS-2 and Message Segmentation is a practical developer reference for teams building, integrating or operating an SMS API. The goal is to solve real implementation problems rather than provide a surface-level overview. The design choices below focus on reliability, security, scalability, cost control and accurate customer-facing behaviour.
Why encoding matters
SMS is not simply a stream of arbitrary Unicode text. Character encoding determines how many characters fit into a segment, and the resulting segment count affects cost, throughput and sometimes delivery behaviour.
GSM-7
Many common Latin characters can use GSM-7 encoding. Some characters require special handling or escape representations, so a naive character count is not always an accurate segment calculation.
UCS-2
Messages containing characters outside the supported GSM-7 set may require UCS-2 encoding. This reduces the number of characters available per segment and can increase segmentation.
Segment calculation
Calculate encoding and segment count after the final template variables are inserted. The same template can consume different segment counts depending on customer data.
Concatenated SMS
Long messages are divided into segments with concatenation metadata. Applications should understand that one logical message can create multiple physical SMS segments.
Unicode normalization
Different Unicode representations can look identical but have different byte or character behaviour. Normalize content where appropriate and test real-world language data.
Emoji
Emoji can push messages into Unicode encoding and increase segment count. Do not estimate cost using visible character count alone.
Provider differences
Providers may expose encoding behaviour differently. The platform should normalize segment calculations and document any provider-specific limitations.
Validation
Reject or warn about unsupported characters only when the business requirement calls for it. Avoid silently replacing customer content without a defined policy.
Cost impact
Segment count should feed billing and capacity calculations. A campaign of Unicode messages can consume substantially more physical SMS traffic than expected.
Testing matrix
Test GSM-7 text, extended characters, accented languages, Devanagari, Arabic, CJK, emoji and mixed content.
API design
Return useful metadata such as segment count where appropriate so developers can understand cost and payload behaviour.
Security
Encoding conversion should not bypass content validation or create logging problems. Keep sensitive content out of diagnostic telemetry.
Reference principle
Correct SMS encoding is both a technical and commercial concern because it affects what recipients receive and how many physical segments the platform submits.
Security and privacy baseline
Treat recipient numbers, message content, credentials, provider evidence and customer configuration as sensitive. Use TLS, tenant-scoped authorization, least privilege, safe logging and controlled access to reports. Never place API secrets in URLs or ordinary logs.
Troubleshooting workflow
Start with a logical message ID or correlation ID. Follow the lifecycle through validation, durable acceptance, queue processing, provider attempt, provider response, delivery evidence and webhook processing. Compare the affected path with a known-good baseline before changing routing or retry policy.
Production checklist
Before production use, verify authentication, authorization, idempotency, rate limits, queue durability, provider routing, delivery reporting, monitoring, backup and recovery, retention, auditability and rollback. Test both normal traffic and predictable failure scenarios.
Developer takeaway
A production messaging platform should make the right behaviour easy to implement and the wrong behaviour difficult to create. Clear contracts, durable state, explicit policy and observable processing are more valuable than isolated features.
Segment-aware billing
If billing is based on segments, calculate segments from the actual encoded message. Do not estimate cost from character count alone.
Language testing
Test representative messages in Indian and international scripts, including Devanagari, Arabic, Bengali, Tamil, Chinese and mixed-language content where relevant to the product.
Extended characters
Some characters can change encoding even when the message looks mostly Latin. Test punctuation, currency symbols, smart quotes and accented characters.
Normalization policy
Define whether the platform normalizes text before encoding. Any transformation should be documented because changing message content can have customer-visible consequences.
Provider compatibility
Different providers can have subtle differences in encoding and segmentation. The platform should normalize behaviour where possible and document exceptions.
API metadata
Returning segment count or encoding metadata can help developers estimate cost and troubleshoot unexpected billing.
Performance
Encoding calculations should be efficient because they may run for every message. Cache reusable template metadata but always evaluate final content where variables can change encoding.
Testing long messages
Test messages just below and above segment boundaries. Boundary tests are especially valuable because one extra character can change physical segment count.
Security
Do not put raw message text into performance logs merely to diagnose encoding. Use safe identifiers and encoding metadata.
Long-term principle
Encoding is part of the message contract. Correct calculations protect delivery, cost reporting and customer expectations.
Implementation pattern
Keep the public API stable while isolating provider-specific behaviour behind internal services or adapters. Persist the logical message before asynchronous work begins, attach a correlation identifier to every downstream operation and keep provider attempts separate from the customer-facing message. This pattern makes retries, reporting, billing and support easier to reason about. It also allows infrastructure changes to happen without forcing every customer application to understand internal implementation details. When a component fails, the remaining lifecycle evidence should still make it possible to determine whether the message was accepted, submitted, delivered or left uncertain.
Failure scenarios to test
Do not limit testing to successful requests. Include invalid input, authentication failure, provider timeout, provider throttling, queue delay, worker restart, database failure, duplicate request, delayed delivery receipt and webhook retry. For each scenario define the expected customer-facing state and the expected internal evidence. This is particularly important for messaging because a timeout does not necessarily mean the provider did not accept the SMS. Testing uncertain outcomes is one of the best ways to prevent duplicate messages and misleading status information.
Observability requirements
At minimum, monitor API latency, acceptance errors, queue age, worker throughput, provider response categories, delivery outcomes and webhook processing. Use message IDs and correlation IDs rather than sensitive phone numbers as primary troubleshooting keys. Dashboards should allow drill-down by tenant, provider, country and message class where appropriate. Metrics show the symptom, traces show the execution path and structured logs provide detailed evidence. Together they make production troubleshooting substantially faster than relying on one source of telemetry.
Security and privacy
Recipient numbers, message content, credentials and enterprise configuration should be treated as sensitive. Use TLS for transport, least-privilege service accounts, tenant-scoped authorization and secure secret storage. Avoid placing API keys, OTP values or complete message content into ordinary logs. Exports and reports should expire according to policy and remain tenant-scoped. Security should be tested during failure and migration scenarios because recovery tooling, background jobs and support utilities can accidentally bypass the controls used by the normal API path.
Production readiness
Before production rollout, verify authentication, authorization, idempotency, rate limits, queue durability, provider eligibility, delivery reporting, backup and recovery, monitoring, audit logging and rollback. Test the real message mix rather than only short ASCII examples. Confirm that support can trace a message without accessing secrets. Document known limits and define the traffic ramp. Production readiness is evidence that the system can behave correctly under normal load and predictable failure, not merely proof that a sample API request returned HTTP 200.
Reference checklist
A developer should be able to answer five questions before shipping an integration: what identifies the logical message, what state does the API guarantee, what happens if the provider times out, how is duplicate processing prevented, and how can the final outcome be investigated? If any answer depends on an undocumented assumption, the integration is not yet robust. Clear contracts, durable state, explicit retry policy and observable lifecycle events create a much stronger foundation than ad hoc provider calls scattered through business code.
Advanced implementation note
Developers should calculate encoding after template rendering and before final cost estimation. A message that contains ordinary English in its template may become Unicode after a customer name or dynamic value is inserted. Segment calculation should therefore operate on the final content. Boundary tests are especially important: add or remove one character around the segment threshold and verify that the platform reports the expected change. This makes billing and capacity estimates far more reliable for real customer traffic.