🧰Daily Toolbox
ENES
unicodeEngineering Guide

UTF-8 Encoding Explained: Variable-Length Byte Patterns, BOM Pitfalls, and Mojibake Fixes (2026)

AS
Alex Sun
Published on 2026-10-18Β·10 min readΒ·Daily Toolbox Engineering

UTF-8 Encoding Explained: Variable-Length Byte Patterns, BOM Pitfalls, and Mojibake Fixes

Every text file, web page, JSON API response, and database text column depends on character encoding β€” the mapping between abstract characters and bytes. UTF-8 won this war decisively: over 98% of web pages use it. Yet developers still get bitten by encoding bugs: garbled text (mojibake), invisible BOM characters breaking parsers, and emoji counting as two characters in JavaScript.

This article explains UTF-8's variable-length design at the bit level, with hand-verifiable examples.

Why Variable-Length?

Before UTF-8, the dominant encodings were fixed-width: ASCII used 1 byte per character (but only covered English), while UTF-16 and UTF-32 used 2 or 4 bytes for everything. The problem: English text in UTF-32 takes 4Γ— the space of ASCII for zero benefit.

UTF-8 (RFC 3629) uses 1 to 4 bytes per character, with a clever prefix scheme that makes the encoding self-synchronizing β€” you can always tell where a character starts, even mid-stream:

Code point range Byte 1 Byte 2 Byte 3 Byte 4
U+0000 – U+007F 0xxxxxxx β€” β€” β€”
U+0080 – U+07FF 110xxxxx 10xxxxxx β€” β€”
U+0800 – U+FFFF 1110xxxx 10xxxxxx 10xxxxxx β€”
U+10000 – U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The leading bits tell you the sequence length: 0 = 1 byte, 110 = 2 bytes, 1110 = 3 bytes, 11110 = 4 bytes. Continuation bytes always start with 10. This design means:

  • ASCII is valid UTF-8. Every ASCII byte (0xxxxxxx) decodes identically. This single property drove UTF-8's adoption β€” existing ASCII systems upgraded seamlessly.
  • No byte is ambiguous. A byte starting with 10 is always a continuation; you never mistake it for a character start.

Note: the original UTF-8 spec allowed up to 6 bytes, but RFC 3629 restricted it to 4 (matching Unicode's U+10FFFF limit). Five- and six-byte sequences are now invalid.

Worked Examples: Bits to Bytes

Example 1: 'A' (U+0041) β€” 1 byte

U+0041 = 65 decimal = 1000001 binary (7 bits). Fits in the 1-byte pattern 0xxxxxxx:

Code point:  1000001
UTF-8:       0 1000001  β†’  0x41

Hex: 41. Same as ASCII. βœ“

Example 2: 'Γ©' (U+00E9) β€” 2 bytes

U+00E9 = 233 decimal = 11101001 binary (8 bits). Needs the 2-byte pattern 110xxxxx 10xxxxxx (11 payload bits).

Pad to 11 bits: 00011101001. Split into 5 + 6:

Payload:     00011  101001
Byte 1:      110    00011    β†’  11000011  β†’  0xC3
Byte 2:      10     101001   β†’  10101001  β†’  0xA9

Hex: C3 A9. Verify in Python: 'Γ©'.encode('utf-8').hex() β†’ 'c3a9' βœ“

Example 3: 'δΈ­' (U+4E2D) β€” 3 bytes

U+4E2D = 0100111000101101 binary (16 bits). Uses the 3-byte pattern 1110xxxx 10xxxxxx 10xxxxxx (16 payload bits). Split into 4 + 6 + 6:

Payload:     0100  111000  101101
Byte 1:      1110  0100     β†’  11100100  β†’  0xE4
Byte 2:      10    111000   β†’  10111000  β†’  0xB8
Byte 3:      10    101101   β†’  10101101  β†’  0xAD

Hex: E4 B8 AD. Verify: 'δΈ­'.encode('utf-8').hex() β†’ 'e4b8ad' βœ“

Example 4: 'πŸ˜€' (U+1F600) β€” 4 bytes

U+1F600 = 00011111011000000000 binary (20 bits). Uses the 4-byte pattern 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (21 payload bits). Pad to 21 bits and split 3 + 6 + 6 + 6:

Payload:     000  011111  011000  000000
Byte 1:      11110 000     β†’  11110000  β†’  0xF0
Byte 2:      10    011111  β†’  10011111  β†’  0x9F
Byte 3:      10    011000  β†’  10011000  β†’  0x98
Byte 4:      10    000000  β†’  10000000  β†’  0x80

Hex: F0 9F 98 80. Verify: 'πŸ˜€'.encode('utf-8').hex() β†’ 'f09f9880' βœ“

The BOM Problem

The Byte Order Mark (U+FEFF) is an invisible zero-width character that some editors (notably Windows Notepad) prepend to UTF-8 files: bytes EF BB BF.

UTF-8 doesn't need a BOM (byte order is irrelevant β€” it's a byte sequence, not 16-bit units). But the BOM causes real bugs:

  • JSON parsers may choke: JSON.parse('\uFEFF{"a":1}') throws in strict parsers.
  • CSV headers get corrupted: the first column becomes "\uFEFFid" instead of "id", breaking database imports.
  • Shebang lines break: #!/usr/bin/env python preceded by a BOM isn't recognized as a shebang on Unix.
  • String comparison fails silently: "hello" !== "\uFEFFhello" but they look identical.

Detection and removal:

def strip_bom(text: str) -> str:
    """Remove UTF-8 BOM if present."""
    return text.lstrip('\ufeff')

# Or when reading files:
with open('data.csv', encoding='utf-8-sig') as f:
    content = f.read()  # utf-8-sig automatically strips BOM

// JavaScript
const clean = text.replace(/^\uFEFF/, '');

Python's utf-8-sig codec handles this transparently β€” use it when reading files from untrusted sources (user uploads, Windows-generated exports).

Mojibake: When Encodings Collide

Mojibake (ζ–‡ε­—εŒ–γ‘, "character corruption") happens when bytes in one encoding are decoded as another. The classic case: UTF-8 bytes read as Latin-1 (ISO-8859-1).

Take 'Γ©' (UTF-8: C3 A9). If a Latin-1 decoder reads those two bytes:

Byte C3 β†’ Latin-1 character 'Γƒ' (U+00C3)
Byte A9 β†’ Latin-1 character 'Β©' (U+00A9)

Result: Γ© becomes é. If you've ever seen é or Ò€ℒ on a web page, that's UTF-8 misread as Latin-1 or Windows-1252.

Common causes and fixes:

Symptom Cause Fix
é, Ò€ℒ UTF-8 decoded as Latin-1/Win-1252 Set correct charset in Content-Type header and <meta charset>
???? or οΏ½ Decoder hit invalid bytes, replaced with U+FFFD Ensure source is actually UTF-8; check for truncated multi-byte sequences
Chinese shows as À¸­ UTF-8 decoded as Latin-1 Same as above β€” declare UTF-8 everywhere

The nuclear option for already-corrupted data (when you know the misdecoding path):

# Fix double-encoded UTF-8 (UTF-8 bytes misread as Latin-1, then re-encoded as UTF-8)
def fix_mojibake(text: str) -> str:
    try:
        return text.encode('latin-1').decode('utf-8')
    except (UnicodeEncodeError, UnicodeDecodeError):
        return text  # Not fixable this way

fix_mojibake('é')  # β†’ 'Γ©'

JavaScript's String Length Trap

JavaScript strings are UTF-16. Characters outside the Basic Multilingual Plane (emoji, rare CJK, historic scripts) are stored as surrogate pairs β€” two 16-bit code units:

'πŸ˜€'.length        // 2, not 1!
'πŸ˜€'[0]            // '\uD83D' (lone surrogate β€” garbage if used alone)
[...'πŸ˜€'].length   // 1 (spread iterates code points, not code units)

'A'.length         // 1
'Γ©'.length         // 1 (U+00E9 is in BMP)
'δΈ­'.length        // 1 (U+4E2D is in BMP)
'πŸ˜€'.length        // 2 (U+1F600 needs surrogate pair)

This breaks naive string truncation, character counting, and cursor positioning. The fix: use Array.from() or spread syntax for code-point-aware operations, or Intl.Segmenter for grapheme clusters (what users perceive as "characters"):

// Correct emoji-aware truncation
function truncate(str, maxCodePoints) {
  return [...str].slice(0, maxCodePoints).join('');
}
truncate('Hello πŸ˜€πŸŒ', 7)  // 'Hello πŸ˜€' (not 'Hello πŸ˜€οΏ½')

UTF-8 vs UTF-16: When Does It Matter?

UTF-8 UTF-16
ASCII/English 1 byte/char (optimal) 2 bytes/char (2Γ— waste)
CJK text 3 bytes/char 2 bytes/char (more compact)
Emoji 4 bytes/char 4 bytes/char (surrogate pair)
Web standard Yes (98%+ of pages) Rare
Self-synchronizing Yes No (can misalign on odd byte)

For most applications, UTF-8 is the right default. UTF-16 only wins for predominantly CJK text, and even then the complexity rarely justifies it.

Try It

Working with text encoding means constantly converting between representations β€” characters to code points to byte sequences. DailyToolbox's free text tools run entirely in your browser: inspect Unicode code points, convert between encodings, and debug the invisible characters (BOM, zero-width spaces) that cause so many production bugs.

πŸ‘‰ Explore Text Tools β†’

FAQ

Is UTF-8 the same as Unicode?

No. Unicode is the character set β€” the mapping of abstract characters to code points (U+0041 = 'A'). UTF-8 is one encoding of Unicode β€” a specific way to represent those code points as bytes. Other encodings include UTF-16 and UTF-32. Think of Unicode as the dictionary and UTF-8 as one particular printing of it.

Why do some files start with invisible characters?

That's the BOM (Byte Order Mark, U+FEFF, bytes EF BB BF). Windows tools add it to mark the file as UTF-8. It's invisible in most editors but breaks parsers, shebangs, and string comparisons. Strip it on input with Python's utf-8-sig codec or an explicit check.

What's the maximum valid Unicode code point?

U+10FFFF. UTF-8's 4-byte pattern can theoretically encode up to U+1FFFFF (21 payload bits), but RFC 3629 restricts it to the Unicode range. Surrogate code points (U+D800–U+DFFF) are also invalid in UTF-8 β€” they're UTF-16 artifacts and must never appear in UTF-8 byte sequences.

Why does my emoji break when I truncate a string in JavaScript?

JavaScript's .length counts UTF-16 code units, not characters. Emoji like πŸ˜€ are stored as two code units (a surrogate pair). Truncating at an odd index splits the pair, producing a lone surrogate (οΏ½). Use [...str] to work with code points instead.

#unicode#encoding#utf-8#text-processing#web-development
AS
Written by Alex Sun

Alex Sun is the developer behind Daily Toolbox. He writes these guides while building the tools themselves β€” every claim tested against the real thing.

Try the free tools mentioned above

Explore Text Tools β†’