URL Encoding Explained: Percent-Encoding, Reserved Characters, and When to Encode

Every developer has seen %20 and knows it means "space." Far fewer know why some characters get encoded and others don't, why + means space in a query string but a literal plus in a path, or why encoding an already-encoded string produces the %2520 bugs that haunt API integrations. URL encoding is simple in principle and treacherous in the details.

The Problem: URLs Have a Limited Alphabet

A URL is defined (RFC 3986) as a sequence of characters from a restricted set. The URL syntax itself uses characters like :, /, ?, #, &, and = as delimiters — they structure the URL into scheme, authority, path, query, and fragment. If your data contains those characters literally, the URL becomes ambiguous: is that & a query-parameter separator or part of a search term?

Percent-encoding solves this: any byte that isn't allowed (or is ambiguous) in a given position is replaced by % followed by two hexadecimal digits representing the byte value. A space (byte 0x20) becomes %20. The € sign (U+20AC, UTF-8 bytes E2 82 AC) becomes %E2%82%AC — note that non-ASCII characters are first encoded as UTF-8 bytes, then each byte is percent-encoded.

Which Characters Must Be Encoded?

RFC 3986 divides characters into precise categories:

Unreserved — never encode: A-Z a-z 0-9 - _ . ~. These are always safe, everywhere in a URL.

Reserved — encode when they're data, not delimiters:

• Gen-delims: : / ? # [ ] @
• Sub-delims: ! $ & ' ( ) * + , ; =

A reserved character keeps its literal meaning when used as a delimiter (? starts the query string). But when it appears inside data — a search term containing &, a filename containing # — it must be percent-encoded so parsers don't misread it.

Everything else — always encode: spaces, control characters, non-ASCII characters, and characters like % itself (a literal percent sign in data becomes %25, or it would be misread as the start of an escape sequence).

The practical rule: encode the data, not the structure. Build the URL's skeleton with literal delimiters, then percent-encode each dynamic value before inserting it.

from urllib.parse import quote, urlencode

# Wrong: encoding the whole URL destroys its structure
quote("https://example.com/search?q=fish & chips")
# 'https%3A//example.com/search%3Fq%3Dfish%20%26%20chips'  -- broken

# Right: encode only the value
urlencode({"q": "fish & chips"})
# 'q=fish+%26+chips'

The + Trap: Queries Are Not Paths

Here is the single most misunderstood detail: in the query string of application/x-www-form-urlencoded data, + means space. This convention comes from HTML forms, not from RFC 3986. Everywhere else in the URL — the path, the fragment — a + is a literal plus sign.

Consequences:

• ?q=fish+chips → the server reads fish chips.
• /files/C++ → the path is literally C++, not C . If you "helpfully" convert + to space in a path, you corrupt it.
• A literal plus in a query value must be encoded as %2B: ?q=C%2B%2B → C++.

Libraries handle this correctly only if you use the right function: Python's urllib.parse.quote_plus encodes spaces as + (for form-style queries), while quote encodes them as %20 (for paths). JavaScript's encodeURIComponent uses %20; there is no built-in + variant. Mixing these up is a perennial source of bugs.

UTF-8 First, Then Percent-Encode

Non-ASCII text follows a two-step process:

1. Encode the string as UTF-8 bytes.
2. Percent-encode every byte outside the unreserved set.

Example: café → UTF-8 bytes 63 61 66 C3 A9 → caf%C3%A9. The é (U+00E9) is two bytes in UTF-8, hence two %XX sequences.

Older systems sometimes used other encodings (Latin-1, Shift_JIS) in this step, which is why legacy URLs with %E9 for é still surface — that was the Latin-1 byte, not UTF-8. Modern standards (WHATWG URL spec, used by all browsers) mandate UTF-8. If you see mojibake in a decoded URL, mismatched assumptions about this step are the usual cause.

Worked Example: Decoding by Hand

Given q=%E2%82%AC100+%26+free:

1. Split on %: bytes are E2 82 AC, then literal 100, then +, then 26 → byte 0x26 = &, then +, then free.
2. The byte sequence E2 82 AC is valid UTF-8 for U+20AC (€).
3. In a form-encoded query, + decodes to space.
4. Result: q=€100 & free.

Common Mistakes

1. Double encoding. Encoding an already-encoded string: %20 → %2520 (the % becomes %25). The server decodes once and gets the literal text %20 instead of a space. This happens when a value passes through two encoding layers — e.g., a framework encodes automatically and you encode manually too. Rule: encode exactly once, at the boundary where data enters the URL.

2. Encoding the whole URL. As shown above, quote() on a full URL mangles the delimiters. Encode components, then assemble.

3. Forgetting to encode # in data. A # starts the fragment — everything after it never reaches the server. A filename like report#2.pdf in a path must be report%232.pdf, or the server sees report.

4. Treating + as space in paths. Only form-encoded query strings use +-for-space. In paths, decode %20 but leave + alone.

5. Decoding with the wrong charset. Percent-decoded bytes are UTF-8 by modern convention. Decoding them as Latin-1 turns %C3%A9 into é instead of é. If your stack lets you configure the charset, set UTF-8 explicitly.

6. Case sensitivity of hex digits. %2f and %2F are equivalent, but string comparison of URLs treats them as different. Normalize to uppercase hex when comparing or caching URLs.

Debugging Walkthrough: "The API Gets a Different Value Than I Sent"

1. Log the raw URL before it leaves your code — not the decoded parameters, the actual bytes on the wire. Most bugs are visible here.
2. Count the encoding layers. Put a breakpoint (or print) at each layer: your code, the HTTP client, any proxy, the framework's router, the framework's parameter parser. Find where the value changes.
3. Check for double encoding: search the raw URL for %25. If present, something encoded twice.
4. Check the + handling: if spaces arrive as literal + characters, the server decoded a path-style value with query rules, or vice versa.
5. Verify the charset: decode the raw bytes as UTF-8 manually. If the result is mojibake, the sender used a different encoding — or decoded with one.

Try It

Percent-encoding is easiest to learn by doing. DailyToolbox's free URL encoder/decoder runs entirely in your browser — encode tricky strings, decode %XX sequences, and see exactly which bytes your input produces.

👉 Encode a URL →

FAQ

What's the difference between encodeURI and encodeURIComponent in JavaScript?

encodeURI encodes a complete URI — it leaves delimiters like ://, ?, &, # intact. encodeURIComponent encodes a component (a single value) — it encodes those delimiters too. Use encodeURIComponent for query values and path segments; use encodeURI rarely, and never on data.

Should I encode the / in a path segment?

Yes, if the slash is data rather than a path separator. A filename a/b as a single segment must be a%2Fb, otherwise the server routes it as two segments. Some frameworks (notably older Java servlet containers) reject or mishandle %2F in paths for security reasons — test yours.

Why do browsers show unicode in the address bar instead of %XX?

Browsers decode percent-encoded UTF-8 for display but send the encoded form on the wire. What you see as https://example.com/café is transmitted as https://example.com/caf%C3%A9. Copy-pasting from the address bar may give you either form depending on the browser.

Is %20 or + correct for a space?

Both, in different places. In application/x-www-form-urlencoded query strings, + is the traditional encoding for space (HTML forms). In paths, fragments, and raw RFC 3986 contexts, space must be %20. When in doubt, %20 is accepted everywhere; + is only safe in form-encoded queries.

Can percent-encoding cause security issues?

Yes. Encoded slashes (%2F), dots (%2E), and null bytes (%00) have been used for path-traversal and filter-bypass attacks when one layer decodes and another doesn't. The defense: decode exactly once, as early as possible, then validate the decoded value against an allowlist before use.