Skip to content

Python Unicode, UTF-8, and String Operations

Step 15 of 41

Python Unicode and UTF-8: Code Points, Characters, and Bytes

Introduction

Python Unicode strings represent text as code points; UTF-8 encodes that text into bytes. A name can therefore fit a character limit and exceed a byte limit. Reading the same bytes with the wrong decoder can produce garbled text without raising an exception.

This lesson opens Strings in PCAP-31-03 (18% of the score, 8 of 40 items). It covers representation in objective 3.1 and ord/chr from objective 3.2. You will trace escape sequences, count encoded bytes, and identify encoding failures before moving on to string operations.

Learning objectives

After this lesson you can:

  1. Explain the relation between characters, code points and bytes.
  2. Distinguish ASCII, Unicode and the encodings UTF-8, UTF-16 and Latin-1.
  3. Use ord() and chr() to convert between characters and code points.
  4. Write characters with the escape sequences \n, \t, \\, \', \", \xhh, \uhhhh and \Uhhhhhhhh, and use raw strings.
  5. Convert text to bytes with .encode() and back with .decode(), predicting the length and content.
  6. Recognise UnicodeEncodeError and UnicodeDecodeError and fix them.

Characters, code points and encodings

Computers store numbers. To store text, each character is given a number, called its code point. A table that maps characters to code points is a character set. An encoding is the rule that turns code points into a sequence of bytes for storage or transmission. Keeping these three ideas separate solves most exam problems.

Concept Meaning Example
Character a symbol such as a letter, digit or emoji A, é, €
Code point the number assigned to the character 65, 233, 8364
Encoding rule that writes a code point as bytes UTF-8 writes 8364 as three bytes
Byte a value from 0 to 255 0xE2

ASCII

ASCII is the old standard with 128 code points (0 to 127). It contains English letters, digits, punctuation and control codes. Useful values are:

Characters Code points
space 32
0 to 9 48 to 57
A to Z 65 to 90
a to z 97 to 122
newline \n 10

Two facts follow: the ASCII lowercase letters are exactly 32 above their uppercase partners, and digits are ordered, so ord("7") - ord("0") equals 7.

Unicode

Unicode defines a code-point space from 0 to 0x10FFFF, covering writing systems, symbols, and emoji. Not every possible code point is assigned a character. A displayed character can also combine several code points. Its first 128 code points are identical to ASCII, and the next 128 match Latin-1. Python 3 strings (str) are sequences of Unicode code points, independent of any encoding. Therefore len("€") is 1, however many bytes the symbol needs later.

UTF-8 and other encodings

UTF-8 is the dominant encoding. It uses a variable number of bytes per code point:

Code point range Bytes in UTF-8 Example
0 to 127 1 A
128 to 2047 2 é
2048 to 65535, excluding surrogate code points 3 €
65536 to 0x10FFFF 4 😀

ASCII text has identical bytes in ASCII and UTF-8. UTF-16 represents Unicode scalar values using two or four bytes, with a possible byte-order mark at the start of an encoded stream. Latin-1 uses one byte for each of the first 256 code points. Python’s default for str.encode() and bytes.decode() is UTF-8. Surrogate code points U+D800–U+DFFF are not Unicode scalar values; the default strict UTF-8 encoder rejects them even though a Python string can contain them.

ord() and chr()

ord(c) receives a string of length one and returns its code point. chr(n) receives an integer and returns the character.

Python
print(ord("A"))      # 65
print(ord("a"))      # 97
print(ord("€"))      # 8364
print(chr(66))       # B
print(chr(8364))     # €
print(hex(ord("é"))) # 0xe9

They are inverse functions: chr(ord(c)) == c. For a str argument, ord() needs exactly one code point: an empty or longer string raises TypeError. It also accepts a single-byte bytes or bytearray. chr() accepts integers from 0 through 0x10FFFF; values outside that range raise ValueError.

Arithmetic on code points is a classic exam topic:

Python
print(ord("a") - ord("A"))            # 32
print(chr(ord("a") - 32))             # A
print(chr(ord("a") + 1))              # b
print("".join(chr(ord(c) + 1) for c in "HAL"))   # IBM

Worked example: a simple letter shift

Problem. Write a function that shifts each ASCII lowercase letter three positions forward, wrapping from z to a, and leaves all other characters unchanged. What does shift("xyz, ok") return?

Step 1: map to numbers. Subtract ord("a") to get 0 to 25, add 3, take the remainder modulo 26, then add ord("a") back.

Python
def shift(text):
    out = []
    for c in text:
        if "a" <= c <= "z":
            n = (ord(c) - ord("a") + 3) % 26
            out.append(chr(n + ord("a")))
        else:
            out.append(c)
    return "".join(out)

Step 2: apply it to x. ord("x") is 120, so the index is 23. Adding 3 gives 26, and 26 % 26 is 0, which is a. Likewise y becomes b and z becomes c.

Step 3: other characters. The comma and the space stay unchanged, o becomes r and k becomes n.

Solution. shift("xyz, ok") returns "abc, rn".

Why other designs fail. Without % 26, x would become {, the character after z in ASCII. Forgetting the condition "a" <= c <= "z" would also shift the comma and space.

Escape sequences

A string literal can contain special characters written with a backslash.

Sequence Meaning Code point
\n newline 10
\t tab 9
\\ one backslash 92
\' and \" quote characters 39 and 34
\xhh character with 2 hex digits up to 255
\uhhhh character with 4 hex digits up to 65535
\Uhhhhhhhh character with 8 hex digits any code point
\N{name} character by its Unicode name depends
\ooo up to three octal digits keep values at most 255
Python
print("a\tb")              # a    b   (a tab between)
print(len("a\tb"))         # 3
print("\x41", "A", "\101")   # A A A
print("€", "\N{EURO SIGN}")  # € €
print(len("\\n"))          # 2   (a backslash and the letter n)
print(len("\n"))           # 1   (a single newline character)

The character-producing escapes in the table each contribute one code point, so len("a\tb") is 3. Octal escapes above 255 produce a SyntaxWarning from Python 3.12; use Unicode escapes for larger values. The text "\\n" has two characters, the backslash and n, because the first backslash escapes the second.

Raw strings

A string prefixed with r keeps backslashes literally, which is convenient for Windows paths and patterns.

Python
print(r"a\tb")             # a\tb
print(len(r"a\tb"))        # 4

A raw string cannot end in an odd number of backslashes, because the last backslash would escape the closing quote.

Strings versus bytes

str holds text as code points. bytes holds raw numbers from 0 to 255. Converting between them requires an encoding:

Python
s = "café"
b = s.encode("utf-8")
print(b)             # b'caf\xc3\xa9'
print(len(s))        # 4
print(len(b))        # 5
print(b.decode("utf-8"))   # café

The letter é has code point 233 and takes two bytes in UTF-8 (0xC3 0xA9), so the text has 4 characters but the byte sequence has 5 elements.

Properties of bytes:

  • A literal is written b"..." and may contain only ASCII characters directly.
  • Indexing a bytes object returns an integer: b"caf\xc3\xa9"[0] is 99, not "c".
  • A bytes object is immutable; bytearray is its mutable relative.
  • str and bytes are different types, so b"a" == "a" is False.
Python
print("€".encode("utf-8"))        # b'\xe2\x82\xac'
print(len("€".encode("utf-8")))   # 3
print("😀".encode("utf-8"))        # b'\xf0\x9f\x98\x80'
print(len("😀"))                  # 1
print("café".encode("latin-1"))   # b'caf\xe9'
print("Zoë".encode("utf-8"))      # b'Zo\xc3\xab'

Encoding and decoding errors

Errors appear when the chosen encoding cannot handle the data.

Python
"€".encode("ascii")           # UnicodeEncodeError: no ASCII code for the euro sign
b"caf\xe9".decode("utf-8")    # UnicodeDecodeError: \xe9 is not valid UTF-8 here
b"caf\xe9".decode("latin-1")  # 'café'  (valid in Latin-1)

Both are subclasses of UnicodeError, which derives from ValueError, so except ValueError handles them. The method accepts an errors argument:

Value Effect when a character cannot be encoded
"strict" (default) raise an error
"ignore" drop the character
"replace" substitute ? when encoding
Python
print("€".encode("ascii", errors="replace"))   # b'?'
print("€".encode("ascii", errors="ignore"))    # b''

Use the documented encoding of the input and choose an output encoding that represents the required text. Replacement and ignoring policies lose information. Review text-file encodings and stream handling when applying the same rule to files.

Worked example: counting characters and bytes

Problem. A form field is limited to 6 bytes in the database. For the text "Zoë 😀" (letters Z, o, ë, a space and an emoji), how many characters and bytes are there in UTF-8, and does it fit?

Step 1: count characters. The text contains Z, o, ë, space, emoji: 5 code points, so len(text) is 5.

Step 2: count bytes. Z and o need 1 byte each (2 bytes). ë has code point 235, so 2 bytes. The space needs 1 byte. The emoji has a code point above 65535, so 4 bytes. The total is 2 + 2 + 1 + 4 = 9.

Step 3: verify with code. len(text.encode("utf-8")) returns 9.

Solution. There are 5 characters and 9 bytes. The 5 characters would pass a limit of 6 characters, but the 9 bytes exceed a limit of 6 bytes, so the text does not fit.

Why other answers are wrong. Saying 5 bytes counts characters instead of bytes, and wrongly concludes that the text fits. Saying 8 bytes treats the emoji as 3 bytes instead of 4. Both mistakes come from skipping the table of UTF-8 sizes.

Tracing drill: characters, numbers and bytes

Trace the following lines before reading the answers.

Python
text = "Wö"
data = text.encode("utf-8")
print(len(text), len(data))
print(list(data))
print(data[1])
print(bytes([87, 195]).decode("latin-1"))
digits = "407"
print(sum(ord(c) - ord("0") for c in digits))
  1. text has two characters, W and ö. The letter W is code point 87 (one byte); ö is code point 246, which needs two bytes in UTF-8 (195 and 182). The first line prints 2 3.
  2. list(data) shows the integers of the bytes: [87, 195, 182].
  3. Indexing a bytes object returns an integer, so data[1] prints 195, not a letter.
  4. Decoding the two bytes 87 and 195 with Latin-1 always succeeds, because every byte value is a valid Latin-1 character: the result is WÃ. With UTF-8 the same call would fail, because 195 starts a two-byte sequence that is cut short, and Python would raise UnicodeDecodeError.
  5. Subtracting ord("0") turns each digit character into its numeric value: 4 + 0 + 7 = 11.

The lesson of the drill is that the same bytes mean different text under different encodings, and that Python never guesses the encoding for you. A decoder either succeeds with the wrong text (Latin-1 accepts everything) or fails loudly (UTF-8 rejects invalid sequences). Failing loudly is often the better outcome because the problem is noticed during testing. On the exam, always ask two questions about a conversion: which encoding is named, and whether every byte or character is valid in it.

Using what you know

Detecting letters without methods. Because uppercase letters occupy a continuous range, you can test them by code point:

Python
def is_upper_ascii(c):
    return 65 <= ord(c) <= 90

Sorting depends on code points. Python compares strings lexicographically by code point. Among the ASCII characters in this example, digits come before uppercase letters, then lowercase letters:

Python
print(sorted(["b", "B", "a", "A", "1"]))   # ['1', 'A', 'B', 'a', 'b']
print("Z" < "a")                          # True, because 90 < 97

Combining characters. "é" can be a single code point (233) or e followed by a combining accent ("é", two code points). They look equal but "é" == "é" is False and the second has length 2. This representation difference explains why visual equality does not imply string equality. Unicode normalization can address canonical equivalents; it is separate from changing case. See Unicode-aware character tests for character predicates that operate on the actual code points.

Common pitfalls and exam tips

  • len counts code points, not bytes. Encode first when bytes matter.
  • Indexing bytes gives integers, indexing str gives one-character strings.
  • ord needs exactly one character; chr needs an integer.
  • "\\n" is two characters; "\n" is one.
  • \xhh, \uhhhh and \Uhhhhhhhh need exactly 2, 4 and 8 hex digits.
  • An encoding mismatch can fail or silently garble text. A permissive decoder such as Latin-1 accepts every byte value, which does not make its interpretation correct.
  • ASCII ordering is not linguistic ordering. ASCII capitals sort before ASCII lowercase letters; other scripts and accents follow their own code points.
  • UTF-8 is the default of .encode() and .decode().

Exam tip. When a question mixes characters and bytes, write two lines: first the code points, then the byte count per character in the chosen encoding (1, 2, 3 or 4 for UTF-8). The answers follow from these numbers.

Mini case study: the garbled customer file

A shop imports a customer list written by an old accounting system. A name that should read José appears as José after import, and a later export crashes on some rows.

Python
with open("customers.txt", "rb") as source:
    raw = source.read()                     # bytes read from the file
text = raw.decode("latin-1")
with open("export.txt", "w", encoding="ascii") as out:
    out.write(text)

Task. Explain both symptoms and propose a fix.

Solution.

  1. The file was actually written in UTF-8, where é is the two bytes 0xC3 0xA9. Decoding them with Latin-1 reads them as two separate characters, à and ©. This explains José: the decoder does not fail, because every byte is valid in Latin-1, but the result is wrong.
  2. The export opens the output file with the ASCII encoding, which has no code points above 127. Any accented letter or symbol raises UnicodeEncodeError.
  3. The fix is to decode with the real encoding and to write with an encoding that supports all characters:
Python
with open("customers.txt", "rb") as source:
    text = source.read().decode("utf-8")
with open("export.txt", "w", encoding="utf-8") as out:
    out.write(text)

If the accounting system really produced Latin-1, the decode step would use "latin-1" instead. The key rule is to know the encoding of each file and to state it explicitly.

Key terms

  • Code point: the number identifying a character in Unicode.
  • Encoding: the rule that converts code points to bytes.
  • ASCII: 128-character set, code points 0 to 127.
  • Unicode: universal character set used by Python str.
  • UTF-8: variable-length encoding using 1 to 4 bytes per code point.
  • Escape sequence: backslash notation for special characters.
  • Raw string: literal with r prefix that keeps backslashes.
  • bytes: immutable sequence of integers from 0 to 255.

Summary checklist

  • I can separate character, code point and byte.
  • I know the ASCII values of digits, capitals and lowercase letters.
  • I can use ord() and chr() and compute shifts.
  • I know the main escape sequences and the length of strings that contain them.
  • I can encode and decode with UTF-8 and predict byte counts.
  • I recognise UnicodeEncodeError and UnicodeDecodeError.
  • I know that indexing bytes returns integers.

Official sources