123eworld Knowledge Hub → Transactional SMS API → Page 286
SMS API Unicode and GSM-7 Encoding: Character Sets, Length Limits and Multipart Messages
A practical developer reference designed to solve real implementation and production problems around sms api unicode and gsm-7 encoding: character sets, length limits and multipart messages.
Why encoding is an engineering decision
SMS text is not simply a string sent to a provider. The characters in that string determine the encoding, the number of segments and sometimes the downstream route. A robust SMS API should inspect content before submission and make the encoding decision explicit.
GSM-7 basics
GSM-7 provides a compact character set commonly used for standard Latin SMS. Some characters occupy an extended table and require an escape mechanism, so a simple character count is not always enough.
Unicode SMS
Unicode allows languages and symbols outside the GSM-7 repertoire, including many Indian scripts. The trade-off is fewer characters per segment. A single unsupported character can change the encoding decision for the entire message.
Segment impact
When a message exceeds the single-segment limit, concatenation reduces the usable payload per segment because metadata is needed to reassemble the parts. Developers should calculate segments after encoding, not before.
Normalization
Unicode normalization, smart quotes, emoji and copied text can unexpectedly change encoding. Normalize or validate content according to the application's requirements before calculating cost.
API validation
The API should return useful metadata such as detected encoding and estimated segment count where appropriate. This helps developers understand why a message consumes multiple SMS units.
Templates
Template systems should validate real substituted content, not only the template text. A variable containing a non-GSM character can change the final encoding.
Testing
Use representative Latin, Marathi, Hindi, Arabic, accented and emoji-containing samples. Test boundaries around segment limits.
Cost and billing
Logical messages and billable SMS segments are different quantities. Reporting should make the distinction visible.
Security
Encoding validation should not become a way to bypass content controls. Validate the final decoded message and preserve safe logging practices.
Reference algorithm
Normalize input → detect encoding → calculate septets/code units → determine segments → validate provider limits → submit.
Production checklist
Test character boundaries, mixed scripts, extended GSM characters, empty values, substitutions and provider-specific encoding behaviour.
Developer implementation principle
Keep the customer-facing API simple. Application developers should submit a logical message and receive stable identifiers and status semantics. Encoding, segmentation, routing and provider-specific transport should be handled by the messaging platform unless an advanced integration explicitly requires lower-level control.
Production reliability
Test the failure path as seriously as the success path. Provider throttling, delayed callbacks, ambiguous timeouts, configuration changes and failover can expose bugs that ordinary send tests never find.
Security and privacy
Avoid placing phone numbers, message content, credentials or authentication values unnecessarily into logs and metrics. Use tenant-aware authorization and stable opaque identifiers for troubleshooting.
Reference architecture
A robust design normally follows: API authentication → validation → logical message creation → encoding/segment calculation → routing → durable queue → provider adapter → delivery evidence → normalized status → reporting/webhook.
Related 123eworld guides
Continue through the 123eworld SMS & WhatsApp Knowledge Hub for related developer architecture, integration and production guides.
How to detect GSM-7 safely
A reliable encoding detector should work against a complete character table rather than a hand-written list of common characters. The detector should first inspect the entire final message after template substitution. If every character is supported by the selected GSM-7 repertoire, the message can use that encoding. If even one character is outside the repertoire, the platform may need Unicode encoding. The exact rule should follow the provider and telecom route being used. Developers should not implement their own approximate character counter in each application because small differences can create inconsistent segment estimates. A server-side authoritative calculation is safer. For customer-facing tools, a preview calculator can provide an estimate, but the final send API should remain the source of truth.
Extended GSM characters
Some GSM characters require an escape representation and therefore consume more encoded space than a simple visual character count suggests. This is a classic source of unexpected segment changes. A message can appear comfortably below a developer's character limit and still cross the transport boundary because of extended-table characters. The knowledge-base guidance should therefore teach developers to calculate encoded length, not visible length. Test symbols such as brackets, braces, certain currency signs and other extended characters supported by the chosen GSM table. Do not assume that a character accepted by one provider will have identical treatment on every route.
Indian-language messaging
Hindi, Marathi, Bengali, Gujarati, Tamil, Telugu and other Indian-language content frequently requires Unicode. The platform should make Unicode handling automatic and should not force developers to understand telecom encoding rules for every message. However, developers should understand the commercial effect: Unicode messages can contain fewer characters per segment, so a message that looks short can become multiple physical SMS units. Template authors should preview final rendered content and segment count before approving large notification campaigns.
Emoji and copied content
Emoji are a frequent cause of unexpected Unicode selection. Copying text from word processors, websites or social applications can also introduce curly quotes, non-breaking spaces or invisible characters. A production SMS API should normalize only when doing so is safe and expected. Silent replacement of characters can alter the meaning of a regulated or transactional message. A better approach is to identify the encoding change, report the segment impact and allow the business application to decide whether the text should be modified.
Provider-specific behaviour
The internal encoding decision should be separated from provider submission. The platform determines the logical encoding and segment requirement, while the provider adapter translates it into the fields required by the selected transport. If one provider has a special encoding flag or limit, that detail belongs in the adapter. This prevents the public API from becoming a collection of provider-specific switches.
Testing strategy
Create automated tests for boundary lengths in GSM-7, extended GSM-7 and Unicode. Include template variables, line breaks, punctuation, emoji and mixed-language text. Then compare calculated segment counts with controlled provider submissions. Store regression cases whenever a production incident reveals a previously unrecognized character or encoding rule.
Deep architecture and production guidance
A robust encoding service should expose its decision as structured metadata. For example, an internal message object can contain encoding_profile, encoded_length, segment_count and calculator_version. The calculator version matters because encoding rules may evolve and historical reports should remain explainable. If the same message is recalculated later after a library update, a different result should not make the original billing record impossible to understand. Store the calculation inputs and version with the logical message where commercially or operationally necessary. For high-volume systems, perform the calculation once at message creation and reuse the result through queue processing rather than recalculating independently in every worker. However, the provider adapter should still validate route-specific limits before submission. This layered model gives developers a predictable result while protecting the platform from provider-specific constraints. It also makes support much easier: when a customer asks why a message consumed three segments, the operator can see the final text characteristics, encoding decision, segment calculation and route rather than guessing from the visible message alone. Never log the full content simply to debug encoding. Store safe diagnostics and use controlled access for message content when required.
Developer implementation note
For production, expose a preview or validation capability so developers can inspect encoding and segment impact before committing a large notification batch. Keep the preview calculation consistent with the send path by sharing the same library and versioned rules. Do not allow client applications to override the final encoding merely to reduce segment count unless the provider contract explicitly supports that choice. Automatic selection is safer for most integrations.
Practical troubleshooting and decision guide
Production troubleshooting should begin with the final rendered message. Ask what exact characters were submitted, what encoding was selected, how many segments were calculated and which provider route was used. Then compare those facts with the provider's submission evidence. If the platform and provider disagree, determine whether the discrepancy comes from a different encoding table, provider-specific segment rules or an implementation defect. Keep a regression test for every discovered discrepancy. For multilingual enterprise systems, maintain test fixtures for each language used in production. This prevents a future library upgrade from silently changing segment counts. Also test templates after localization because translation can change both encoding and length. An SMS platform intended as a developer reference should make this reasoning explicit: character count is an input, encoding is a decision, segment count is a derived transport property and billing is a commercial result.