Python Unicode and UTF-8: Code Points, Characters, and Bytes
Introduction
Python Unicode strings represent text as code points; UTF-8 encodes that text into bytes. A name can therefore fit a character limit and exceed a byte limit. Reading the same bytes with the wrong decoder can produce garbled text without raising an exception.
This lesson opens Strings in PCAP-31-03 (18% of the score, 8 of 40 items). It covers representation in objective 3.1 and ord/chr from objective 3.2. You will trace escape sequences, count encoded bytes, and identify encoding failures before moving on to string operations.
Learning objectives
After this lesson you can:
- Explain the relation between characters, code points and bytes.
- Distinguish ASCII, Unicode and the encodings UTF-8, UTF-16 and Latin-1.
- Use
ord()andchr()to convert between characters and code points. - Write characters with the escape sequences
\n,\t,\\,\',\",\xhh,\uhhhhand\Uhhhhhhhh, and use raw strings. - Convert text to bytes with
.encode()and back with.decode(), predicting the length and content. - Recognise
UnicodeEncodeErrorandUnicodeDecodeErrorand fix them.
Characters, code points and encodings
Computers store numbers. To store text, each character is given a number, called its code point. A table that maps characters to code points is a character set. An encoding is the rule that turns code points into a sequence of bytes for storage or transmission. Keeping these three ideas separate solves most exam problems.
| Concept | Meaning | Example |
|---|---|---|
| Character | a symbol such as a letter, digit or emoji | A, é, € |
| Code point | the number assigned to the character | 65, 233, 8364 |
| Encoding | rule that writes a code point as bytes | UTF-8 writes 8364 as three bytes |
| Byte | a value from 0 to 255 | 0xE2 |
ASCII
ASCII is the old standard with 128 code points (0 to 127). It contains English letters, digits, punctuation and control codes. Useful values are:
| Characters | Code points |
|---|---|
| space | 32 |
0 to 9 |
48 to 57 |
A to Z |
65 to 90 |
a to z |
97 to 122 |
newline \n |
10 |
Two facts follow: the ASCII lowercase letters are exactly 32 above their uppercase partners, and digits are ordered, so ord("7") - ord("0") equals 7.
Unicode
Unicode defines a code-point space from 0 to 0x10FFFF, covering writing systems, symbols, and emoji. Not every possible code point is assigned a character. A displayed character can also combine several code points. Its first 128 code points are identical to ASCII, and the next 128 match Latin-1. Python 3 strings (str) are sequences of Unicode code points, independent of any encoding. Therefore len("€") is 1, however many bytes the symbol needs later.
UTF-8 and other encodings
UTF-8 is the dominant encoding. It uses a variable number of bytes per code point:
| Code point range | Bytes in UTF-8 | Example |
|---|---|---|
| 0 to 127 | 1 | A |
| 128 to 2047 | 2 | é |
| 2048 to 65535, excluding surrogate code points | 3 | € |
| 65536 to 0x10FFFF | 4 | 😀 |
ASCII text has identical bytes in ASCII and UTF-8. UTF-16 represents Unicode scalar values using two or four bytes, with a possible byte-order mark at the start of an encoded stream. Latin-1 uses one byte for each of the first 256 code points. Python’s default for str.encode() and bytes.decode() is UTF-8. Surrogate code points U+D800–U+DFFF are not Unicode scalar values; the default strict UTF-8 encoder rejects them even though a Python string can contain them.
ord() and chr()
ord(c) receives a string of length one and returns its code point. chr(n) receives an integer and returns the character.
print(ord("A")) # 65
print(ord("a")) # 97
print(ord("€")) # 8364
print(chr(66)) # B
print(chr(8364)) # €
print(hex(ord("é"))) # 0xe9They are inverse functions: chr(ord(c)) == c. For a str argument, ord() needs exactly one code point: an empty or longer string raises TypeError. It also accepts a single-byte bytes or bytearray. chr() accepts integers from 0 through 0x10FFFF; values outside that range raise ValueError.
Arithmetic on code points is a classic exam topic:
print(ord("a") - ord("A")) # 32
print(chr(ord("a") - 32)) # A
print(chr(ord("a") + 1)) # b
print("".join(chr(ord(c) + 1) for c in "HAL")) # IBMWorked example: a simple letter shift
Problem. Write a function that shifts each ASCII lowercase letter three positions forward, wrapping from z to a, and leaves all other characters unchanged. What does shift("xyz, ok") return?
Step 1: map to numbers. Subtract ord("a") to get 0 to 25, add 3, take the remainder modulo 26, then add ord("a") back.
def shift(text):
out = []
for c in text:
if "a" <= c <= "z":
n = (ord(c) - ord("a") + 3) % 26
out.append(chr(n + ord("a")))
else:
out.append(c)
return "".join(out)Step 2: apply it to x. ord("x") is 120, so the index is 23. Adding 3 gives 26, and 26 % 26 is 0, which is a. Likewise y becomes b and z becomes c.
Step 3: other characters. The comma and the space stay unchanged, o becomes r and k becomes n.
Solution. shift("xyz, ok") returns "abc, rn".
Why other designs fail. Without % 26, x would become {, the character after z in ASCII. Forgetting the condition "a" <= c <= "z" would also shift the comma and space.
Escape sequences
A string literal can contain special characters written with a backslash.
| Sequence | Meaning | Code point |
|---|---|---|
\n |
newline | 10 |
\t |
tab | 9 |
\\ |
one backslash | 92 |
\' and \" |
quote characters | 39 and 34 |
\xhh |
character with 2 hex digits | up to 255 |
\uhhhh |
character with 4 hex digits | up to 65535 |
\Uhhhhhhhh |
character with 8 hex digits | any code point |
\N{name} |
character by its Unicode name | depends |
\ooo |
up to three octal digits | keep values at most 255 |
print("a\tb") # a b (a tab between)
print(len("a\tb")) # 3
print("\x41", "A", "\101") # A A A
print("€", "\N{EURO SIGN}") # € €
print(len("\\n")) # 2 (a backslash and the letter n)
print(len("\n")) # 1 (a single newline character)The character-producing escapes in the table each contribute one code point, so len("a\tb") is 3. Octal escapes above 255 produce a SyntaxWarning from Python 3.12; use Unicode escapes for larger values. The text "\\n" has two characters, the backslash and n, because the first backslash escapes the second.
Raw strings
A string prefixed with r keeps backslashes literally, which is convenient for Windows paths and patterns.
print(r"a\tb") # a\tb
print(len(r"a\tb")) # 4A raw string cannot end in an odd number of backslashes, because the last backslash would escape the closing quote.
Strings versus bytes
str holds text as code points. bytes holds raw numbers from 0 to 255. Converting between them requires an encoding:
s = "café"
b = s.encode("utf-8")
print(b) # b'caf\xc3\xa9'
print(len(s)) # 4
print(len(b)) # 5
print(b.decode("utf-8")) # caféThe letter é has code point 233 and takes two bytes in UTF-8 (0xC3 0xA9), so the text has 4 characters but the byte sequence has 5 elements.
Properties of bytes:
- A literal is written
b"..."and may contain only ASCII characters directly. - Indexing a
bytesobject returns an integer:b"caf\xc3\xa9"[0]is 99, not"c". - A
bytesobject is immutable;bytearrayis its mutable relative. strandbytesare different types, sob"a" == "a"isFalse.
print("€".encode("utf-8")) # b'\xe2\x82\xac'
print(len("€".encode("utf-8"))) # 3
print("😀".encode("utf-8")) # b'\xf0\x9f\x98\x80'
print(len("😀")) # 1
print("café".encode("latin-1")) # b'caf\xe9'
print("Zoë".encode("utf-8")) # b'Zo\xc3\xab'Encoding and decoding errors
Errors appear when the chosen encoding cannot handle the data.
"€".encode("ascii") # UnicodeEncodeError: no ASCII code for the euro sign
b"caf\xe9".decode("utf-8") # UnicodeDecodeError: \xe9 is not valid UTF-8 here
b"caf\xe9".decode("latin-1") # 'café' (valid in Latin-1)Both are subclasses of UnicodeError, which derives from ValueError, so except ValueError handles them. The method accepts an errors argument:
| Value | Effect when a character cannot be encoded |
|---|---|
"strict" (default) |
raise an error |
"ignore" |
drop the character |
"replace" |
substitute ? when encoding |
print("€".encode("ascii", errors="replace")) # b'?'
print("€".encode("ascii", errors="ignore")) # b''Use the documented encoding of the input and choose an output encoding that represents the required text. Replacement and ignoring policies lose information. Review text-file encodings and stream handling when applying the same rule to files.
Worked example: counting characters and bytes
Problem. A form field is limited to 6 bytes in the database. For the text "Zoë 😀" (letters Z, o, ë, a space and an emoji), how many characters and bytes are there in UTF-8, and does it fit?
Step 1: count characters. The text contains Z, o, ë, space, emoji: 5 code points, so len(text) is 5.
Step 2: count bytes. Z and o need 1 byte each (2 bytes). ë has code point 235, so 2 bytes. The space needs 1 byte. The emoji has a code point above 65535, so 4 bytes. The total is 2 + 2 + 1 + 4 = 9.
Step 3: verify with code. len(text.encode("utf-8")) returns 9.
Solution. There are 5 characters and 9 bytes. The 5 characters would pass a limit of 6 characters, but the 9 bytes exceed a limit of 6 bytes, so the text does not fit.
Why other answers are wrong. Saying 5 bytes counts characters instead of bytes, and wrongly concludes that the text fits. Saying 8 bytes treats the emoji as 3 bytes instead of 4. Both mistakes come from skipping the table of UTF-8 sizes.
Tracing drill: characters, numbers and bytes
Trace the following lines before reading the answers.
text = "Wö"
data = text.encode("utf-8")
print(len(text), len(data))
print(list(data))
print(data[1])
print(bytes([87, 195]).decode("latin-1"))
digits = "407"
print(sum(ord(c) - ord("0") for c in digits))texthas two characters,Wandö. The letterWis code point 87 (one byte);öis code point 246, which needs two bytes in UTF-8 (195and182). The first line prints2 3.list(data)shows the integers of the bytes:[87, 195, 182].- Indexing a
bytesobject returns an integer, sodata[1]prints195, not a letter. - Decoding the two bytes
87and195with Latin-1 always succeeds, because every byte value is a valid Latin-1 character: the result isWÃ. With UTF-8 the same call would fail, because195starts a two-byte sequence that is cut short, and Python would raiseUnicodeDecodeError. - Subtracting
ord("0")turns each digit character into its numeric value: 4 + 0 + 7 = 11.
The lesson of the drill is that the same bytes mean different text under different encodings, and that Python never guesses the encoding for you. A decoder either succeeds with the wrong text (Latin-1 accepts everything) or fails loudly (UTF-8 rejects invalid sequences). Failing loudly is often the better outcome because the problem is noticed during testing. On the exam, always ask two questions about a conversion: which encoding is named, and whether every byte or character is valid in it.
Using what you know
Detecting letters without methods. Because uppercase letters occupy a continuous range, you can test them by code point:
def is_upper_ascii(c):
return 65 <= ord(c) <= 90Sorting depends on code points. Python compares strings lexicographically by code point. Among the ASCII characters in this example, digits come before uppercase letters, then lowercase letters:
print(sorted(["b", "B", "a", "A", "1"])) # ['1', 'A', 'B', 'a', 'b']
print("Z" < "a") # True, because 90 < 97Combining characters. "é" can be a single code point (233) or e followed by a combining accent ("é", two code points). They look equal but "é" == "é" is False and the second has length 2. This representation difference explains why visual equality does not imply string equality. Unicode normalization can address canonical equivalents; it is separate from changing case. See Unicode-aware character tests for character predicates that operate on the actual code points.
Common pitfalls and exam tips
lencounts code points, not bytes. Encode first when bytes matter.- Indexing
bytesgives integers, indexingstrgives one-character strings. ordneeds exactly one character;chrneeds an integer."\\n"is two characters;"\n"is one.\xhh,\uhhhhand\Uhhhhhhhhneed exactly 2, 4 and 8 hex digits.- An encoding mismatch can fail or silently garble text. A permissive decoder such as Latin-1 accepts every byte value, which does not make its interpretation correct.
- ASCII ordering is not linguistic ordering. ASCII capitals sort before ASCII lowercase letters; other scripts and accents follow their own code points.
- UTF-8 is the default of
.encode()and.decode().
Exam tip. When a question mixes characters and bytes, write two lines: first the code points, then the byte count per character in the chosen encoding (1, 2, 3 or 4 for UTF-8). The answers follow from these numbers.
Mini case study: the garbled customer file
A shop imports a customer list written by an old accounting system. A name that should read José appears as José after import, and a later export crashes on some rows.
with open("customers.txt", "rb") as source:
raw = source.read() # bytes read from the file
text = raw.decode("latin-1")
with open("export.txt", "w", encoding="ascii") as out:
out.write(text)Task. Explain both symptoms and propose a fix.
Solution.
- The file was actually written in UTF-8, where
éis the two bytes0xC3 0xA9. Decoding them with Latin-1 reads them as two separate characters,Ãand©. This explainsJosé: the decoder does not fail, because every byte is valid in Latin-1, but the result is wrong. - The export opens the output file with the ASCII encoding, which has no code points above 127. Any accented letter or symbol raises
UnicodeEncodeError. - The fix is to decode with the real encoding and to write with an encoding that supports all characters:
with open("customers.txt", "rb") as source:
text = source.read().decode("utf-8")
with open("export.txt", "w", encoding="utf-8") as out:
out.write(text)If the accounting system really produced Latin-1, the decode step would use "latin-1" instead. The key rule is to know the encoding of each file and to state it explicitly.
Key terms
- Code point: the number identifying a character in Unicode.
- Encoding: the rule that converts code points to bytes.
- ASCII: 128-character set, code points 0 to 127.
- Unicode: universal character set used by Python
str. - UTF-8: variable-length encoding using 1 to 4 bytes per code point.
- Escape sequence: backslash notation for special characters.
- Raw string: literal with
rprefix that keeps backslashes. bytes: immutable sequence of integers from 0 to 255.
Summary checklist
- I can separate character, code point and byte.
- I know the ASCII values of digits, capitals and lowercase letters.
- I can use
ord()andchr()and compute shifts. - I know the main escape sequences and the length of strings that contain them.
- I can encode and decode with UTF-8 and predict byte counts.
- I recognise
UnicodeEncodeErrorandUnicodeDecodeError. - I know that indexing
bytesreturns integers.
Official sources
-
PCAP-31-03 exam syllabus, objectives 3.1 and 3.2: https://pythoninstitute.org/pcap-31-03-exam-syllabus (opens in a new tab)
-
Python Unicode HOWTO: https://docs.python.org/3.14/howto/unicode.html (opens in a new tab)
-
Built-in functions
ordandchr: https://docs.python.org/3.14/library/functions.html (opens in a new tab) -
String and bytes literals: https://docs.python.org/3.14/reference/lexical_analysis.html#string-and-bytes-literals (opens in a new tab)
-
Codec policies and surrogate handling: https://docs.python.org/3.14/library/codecs.html#error-handlers (opens in a new tab)
