Base64 and UTF-8: diagnose Chinese text, emoji and replacement characters

When Base64 appears to “break Chinese,” the problem is usually the boundary between characters and bytes. Base64 does not know that a sequence represents Chinese text or emoji. It preserves bytes; the application chooses how those bytes are produced and interpreted.

Follow the bytes

The character 中 is U+4E2D. In UTF-8 it uses the bytes E4 B8 AD, whose Base64 representation is 5Lit. A different character encoding produces different bytes and therefore a different Base64 value. Agree on UTF-8 at both ends before comparing encoded strings.

const bytes = new TextEncoder().encode('中');
console.log([...bytes]); // [228, 184, 173]
console.log([...bytes].map(b => b.toString(16)).join(' ')); // e4 b8 ad

Emoji illustrate another distinction: JavaScript '😊'.length is two UTF-16 code units, while its UTF-8 representation has four bytes. Neither number alone is a count of visible user-perceived characters in arbitrary text. Use byte length for file and transport calculations.

Make replacement visible

The replacement character � can mean that an earlier decoder already lost information. Once unknown bytes have been replaced in a string and that string is re-encoded, the original values cannot generally be reconstructed. Inspect the raw decoded byte array before making further conversions.

const decoder = new TextDecoder('utf-8', { fatal: true });
console.log(decoder.decode(Uint8Array.of(0xe4, 0xb8, 0xad))); // 中
try { decoder.decode(Uint8Array.of(0xff)); }
catch { console.log('Invalid UTF-8; preserve the original byte.'); }

Normalization is a separate operation

Two strings can look identical while containing different Unicode sequences. The accented letter é may be one precomposed character or an e followed by a combining accent. UTF-8 and Base64 correctly preserve that difference. If your application wants normalized text, choose a normalization form explicitly and apply it before encoding; never call a changed result a byte-preserving repair.

Also check invisible characters, line endings and a possible byte-order mark when results differ. These are real input bytes, not necessarily encoder bugs. For signed messages and checksums, even an invisible difference changes the result.

A reliable troubleshooting order is: preserve the source, inspect bytes, identify the expected character encoding, decode with explicit error handling, and only then consider normalization. MDN: btoa() explains why direct browser btoa calls are unsuitable for arbitrary Unicode text. Binary files should bypass text decoding entirely.