Skip to content

Python Text and Binary Files and Resource Cleanup

Step 37 of 41

Python Binary Files: bytearray, readinto, errno, and Safe Replacement

Introduction

Python binary files preserve bytes without text decoding or newline translation. A bytearray provides mutable storage, and readinto can reuse that storage across reads. The returned count identifies which bytes are new; the rest of a buffer can contain an earlier result.

This lesson continues PCAP-31-03 objectives 5.4 and 5.5, with exception handling from 2.1, in Comprehensions, Lambdas, Closures, and I/O (22% of the score, 9 of 40 items). You will trace byte counts, diagnose filesystem failures, and replace an output only after its new content has been written.

Learning objectives

After this lesson you can:

  1. Create a bytearray from a size, a list of integers or a bytes object, and index, slice and modify it.
  2. Explain the differences between bytes and bytearray, and between text and binary files.
  3. Read and write binary files, including in chunks, and use readinto() with a buffer.
  4. Predict the count returned by write(), the bytes returned by read(), and the count returned by readinto().
  5. Use the OSError attributes errno, strerror and filename, and the errno module constants.
  6. Select the correct exception subclass and order except clauses correctly.
  7. Distinguish decoding errors from operating-system errors and avoid partial updates of a file.

bytes and bytearray

A byte is an integer from 0 to 255. Python has two sequence types for bytes:

Type Mutable? Literal or constructor Item type
bytes No b"abc", bytes([65, 66]) int
bytearray Yes bytearray(b"abc"), bytearray(5) int

Creating a bytearray:

Python
ba = bytearray(5)
print(ba, len(ba))

ba = bytearray(b"abc")
ba[0] = 65
ba.append(100)
print(ba, ba[1], list(ba))

b2 = bytearray([72, 105])
print(b2, b2.decode())

Output:

Text
bytearray(b'\x00\x00\x00\x00\x00') 5
bytearray(b'Abcd') 98 [65, 98, 99, 100]
bytearray(b'Hi') Hi

bytearray(5) makes five bytes with value zero: an integer argument is a size, not a content. A list of integers creates one byte per value. Indexing returns an integer, so ba[1] is 98 (the code of "b"), not b"b" or "b". Slicing returns a new bytearray.

Values outside 0 to 255 are rejected, and bytes objects cannot be modified:

Python
try:
    ba[0] = 256
except ValueError as e:
    print(e)

try:
    bytes(b"ab")[0] = 1
except TypeError as e:
    print(e)

Output:

Text
byte must be in range(0, 256)
'bytes' object does not support item assignment

Other operations behave like a list of small integers: append, extend, pop, del ba[i], slice assignment, reverse(), concatenation with +, comparison with bytes:

Python
b = bytearray(b"hello")
b.extend(b"!!")
b.reverse()
print(b)
print(bytearray(b"abc") == b"abc", bytes(bytearray(b"ab")))

Output:

Text
bytearray(b'!!olleh')
True b'ab'

To move between text and bytes, use str.encode() and bytes.decode() (or bytearray.decode()). The length in bytes may differ from the length in characters:

Python
print(len(bytearray("é", "utf-8")), len("é"))

Output:

Text
2 1

Binary files

Add b to the mode: "rb", "wb", "ab". In binary mode the stream deals only with bytes-like objects. There is no encoding, no newline translation and no str.

Python
data = bytearray(range(250, 256))
with open("p.bin", "wb") as f:
    print(f.write(data))

with open("p.bin", "rb") as f:
    print(list(f.read(2)), f.read(100) == bytes([252, 253, 254, 255]), f.read())

Output:

Text
6
[250, 251] True b''

write() returns the number of bytes written (6). read(2) returns a bytes object with two bytes; list() shows the integers. read(100) asks for more than what is left and simply returns the four remaining bytes. A read at the end of the file returns an empty bytes object, b'', which is falsy. This is the standard way to detect the end in a loop.

Writing a str to a binary file is a TypeError, and writing bytes to a text file is also a TypeError. Opening the same file with the wrong kind of mode is a frequent mistake.

Reading into a buffer with readinto()

readinto(buffer) fills an existing mutable buffer, such as a bytearray, and returns the number of bytes actually read. It reuses the caller’s data buffer rather than returning a fresh bytes payload on each read. These examples use blocking regular files; raw non-blocking stream behavior requires separate handling.

Python
with open("p.bin", "rb") as f:
    buf = bytearray(4)
    k = f.readinto(buf)
    print(k, buf)
    print(f.readinto(buf), buf)

Output:

Text
4 bytearray(b'\xfa\xfb\xfc\xfd')
2 bytearray(b'\xfe\xff\xfc\xfd')

The first call fills all four positions. The second finds only two bytes left, returns 2, and overwrites only the first two positions of the buffer. The last two bytes of the buffer, \xfc\xfd, are left over from the previous read. Always use the returned count (for example buf[:k]) and never assume the whole buffer is new.

Copying in chunks

For distinct source and destination files, a bounded-memory copy uses chunks. Opening the destination with wb replaces its prior contents. This example uses ordinary blocking, buffered files:

Python
def copy(src, dst, size=4):
    if type(size) is not int or size <= 0:
        raise ValueError("chunk size must be a positive integer")
    total = 0
    with open(src, "rb") as s, open(dst, "wb") as t:
        while True:
            chunk = s.read(size)
            if not chunk:
                break
            total += t.write(chunk)
    return total

print(copy("p.bin", "q.bin"))

Output:

Text
6

The loop reads blocks of 4 bytes, then 2 bytes, then b'', which ends it. The sum of the values returned by write is the size of the file.

OSError and errno

Errors from the operating system are reported by OSError or by one of its subclasses. An OSError from a system call commonly carries the following information. Some attributes can be None, for example on an exception created manually. See the Python exception hierarchy for matching its subclasses:

Attribute Meaning Example
errno Numeric error code 2 on common platforms for a missing file
strerror Message from the system 'No such file or directory'
filename Path involved, if any 'missing/x.bin'

The module errno defines symbolic names such as errno.ENOENT (no such file or directory), errno.EEXIST (file exists), errno.EACCES (permission denied) and errno.EISDIR (is a directory). Comparing with the name is more readable and more portable than comparing with a number. os.strerror(code) converts a code to a message.

Python
import errno
import os
import tempfile

with tempfile.TemporaryDirectory() as directory:
    try:
        with open(os.path.join(directory, "missing", "x.bin"), "wb"):
            pass
    except OSError as e:
        print(type(e).__name__, e.errno == errno.ENOENT)

    try:
        with open(directory, "rb"):
            pass
    except OSError as e:
        print(type(e).__name__, e.errno == errno.EISDIR)

A typical POSIX result is:

Text
FileNotFoundError True
IsADirectoryError True

On Windows, opening a directory commonly raises PermissionError with EACCES instead. Match the actual exception and symbolic code rather than assuming this second line is portable.

The first attempt fails because the directory missing does not exist. Python maps common error codes to subclasses:

Exception Typical errno Typical cause
FileNotFoundError ENOENT Missing file or directory in the path
FileExistsError EEXIST Mode "x" or os.mkdir on an existing path
PermissionError EACCES, EPERM No right to read or write
IsADirectoryError EISDIR Opening a directory as a file
NotADirectoryError ENOTDIR A path component is a file

All of them inherit from OSError. In Python 3, IOError and EnvironmentError are only other names for OSError: IOError is OSError is True.

Order of except clauses

Python tests the clauses from top to bottom and runs the first one that matches. A handler for the base class placed first would catch everything and make the specific one unreachable:

Python
try:
    open("missing/x.bin", "wb")
except OSError:
    print("os first")
except FileNotFoundError:
    print("fnf second")

Output:

Text
os first

Put the specific subclasses first and the general OSError last. Remember also that error numbers differ between operating systems, so the portable approach is to catch the subclass (or compare with errno constants), and not to hard-code a number in a condition.

Decoding failures are not OSError

Reading bytes that are not valid for the chosen encoding raises UnicodeDecodeError. It is a subclass of ValueError, not of OSError:

Python
with open("t.txt", "wb") as f:
    f.write(b"\xff\xfe")

try:
    with open("t.txt", encoding="utf-8") as f:
        f.read()
except UnicodeDecodeError as e:
    print(type(e).__name__, issubclass(UnicodeDecodeError, ValueError), issubclass(UnicodeDecodeError, OSError))

Output:

Text
UnicodeDecodeError True False

The file was found and opened correctly; its bytes are invalid for the selected decoder. Compare Python text streams and open modes before treating every read failure as an OSError. A handler that catches only OSError would not catch it. You can decide how to react: open in binary mode and decode explicitly, pass errors="replace" to replace invalid bytes with the character �, or reject the file with a clear message.

Avoiding partial updates

Opening a file with "w" truncates it before anything is written. If the program fails halfway, the old content is gone and the new content is incomplete. A safer pattern writes a temporary file first and replaces the target only after the write succeeds:

Python
import os
import tempfile
from pathlib import Path


def safe_write(path, text):
    target = Path(path)
    temporary = None
    try:
        with tempfile.NamedTemporaryFile(
            mode="w", encoding="utf-8", newline="\n",
            dir=target.parent, prefix="." + target.name + ".",
            suffix=".tmp", delete=False
        ) as stream:
            temporary = stream.name
            stream.write(text)
        os.replace(temporary, target)
    finally:
        if temporary is not None:
            try:
                os.unlink(temporary)
            except FileNotFoundError:
                pass


safe_write("r.txt", "v1")
safe_write("r.txt", "v2")
with open("r.txt", encoding="utf-8") as stream:
    print(stream.read())

Output:

Text
v2

The temporary file has a unique name in the destination directory, keeping replacement on the same filesystem. It is closed before os.replace, which also matters on Windows. A failure before replacement preserves the old target, and the finally block attempts to remove the temporary file. Successful replacement removes the temporary name, so that cleanup tolerates FileNotFoundError. Other cleanup failures propagate.

Successful replacement is atomic; that visibility guarantee is distinct from durable storage after a crash and does not coordinate competing writers.

A permission problem shows up when you open a read-only file for writing as an ordinary user: PermissionError with errno.EACCES. Be aware that a privileged account can bypass permissions, so a test that works for you may behave differently in production; that is why you should handle the exception rather than predict it.

Worked example: copy a payload and diagnose a failing destination

Statement. Predict what the following program prints.

Python
import errno

payload = bytearray(b"ID") + bytearray([0, 255, 10])

with open("in.bin", "wb") as f:
    written = f.write(payload)

def copy_to(dst):
    try:
        with open("in.bin", "rb") as s, open(dst, "wb") as t:
            return t.write(s.read())
    except FileNotFoundError:
        return "no folder"
    except OSError as e:
        return f"os error {e.errno == errno.EACCES}"

print(written, copy_to("out.bin"), copy_to("nofolder/out.bin"))

Step 1: build the payload. bytearray(b"ID") has 2 bytes. bytearray([0, 255, 10]) has 3 bytes. The + operator concatenates them into a bytearray of 5 bytes.

Step 2: first write. write(payload) accepts a bytes-like object and returns 5. The variable written is 5.

Step 3: first copy. in.bin opens, out.bin is created in the current directory, read() returns 5 bytes, and write returns 5. The return inside the with block ends the function, and both files are closed on the way out.

Step 4: second copy. The destination path points to a folder that does not exist. The open(dst, "wb") raises FileNotFoundError. The first except matches, so the function returns "no folder". The second handler is not reached.

Text
5 5 no folder

Why other readings fail. An answer of 3 for the first copy forgets the two bytes of ID. An answer that prints os error False for the last call puts the OSError handler first in mind; but FileNotFoundError is a subclass handled by the earlier clause. Expecting an exception to escape from copy_to ignores that all OSError cases are caught.

Common pitfalls and exam tips

Pitfall What happens Remedy
bytearray(5) read as the content 5 Five zero bytes An integer is a size
Indexing a bytearray and expecting b"a" You get an integer Use ba[0:1] for a one-byte bytearray, or bytes(ba[0:1]) for bytes
Writing str to a binary file TypeError Encode first with .encode()
Assuming readinto() fills the whole buffer Old bytes remain after a short read Use the returned count
except OSError before except FileNotFoundError Specific handler is unreachable Put subclasses first
Catching OSError for a decoding problem UnicodeDecodeError escapes Handle it separately (ValueError)
Comparing errno with a literal number Not portable Use errno.ENOENT and similar
Writing directly to the final file Partial update on failure Write a temporary file, then os.replace

Exam tip. Check the first argument of bytearray(...): an int means a size, a list means contents, a bytes object means a copy of that content.

Exam tip. The end of a binary file is signalled by an empty bytes object. In loops, while chunk: is the normal idiom.

Exam tip. In a question about exceptions with several except lines, find the first clause that matches the raised class or one of its parents. The remaining clauses are skipped.

Exam tip. Do not confuse length in bytes with length in characters when non-ASCII characters are written.

End-of-lesson case study

Situation. A tool stores a small header and a name, then reads into an eight-byte buffer. A tester reports a surprising result.

Python
name = "Zoë"
raw = bytearray(b"H1") + bytearray(name.encode("utf-8"))

with open("h.bin", "wb") as f:
    n = f.write(raw)

buf = bytearray(b"????????")
with open("h.bin", "rb") as f:
    k = f.readinto(buf)

print(n, k, buf)

Predict the output and explain the surprise.

Solution, commented. The text "Zoë" has 3 characters but 4 bytes in UTF-8, because ë takes two bytes (\xc3\xab). The header b"H1" adds 2 bytes, so raw has 6 bytes and write returns 6. The buffer is 8 bytes long and was filled with question marks. readinto finds only 6 bytes in the file, fills the first six positions and returns 6; the last two question marks remain. The printed result is:

Text
6 6 bytearray(b'H1Zo\xc3\xab??')

The surprise is the two question marks at the end: they are not part of the data but leftovers of the previous buffer content. The correct way to use the buffer is buf[:k], which has exactly six bytes. A second remark: if the developer had computed the payload size with len(name) + 2, the value would have been 5, one byte too small, because len(name) counts characters and not bytes.

Key terms

  • bytes: an immutable sequence of integers from 0 to 255.
  • bytearray: a mutable sequence of integers from 0 to 255.
  • Binary mode: opening a file with b so that data are bytes, with no encoding.
  • Chunk: a block of bytes read or written in one operation.
  • readinto(): reads into an existing mutable buffer and returns the number of bytes read.
  • OSError: the base class for errors reported by the operating system.
  • errno: the numeric error code of an OSError, with symbolic names in the errno module.
  • Atomic replacement: replacing a file in one step with os.replace so that readers never see a half-written file.

Summary checklist

  • I can create and modify a bytearray, and I know that indexing returns an integer.
  • I know that bytes is immutable and bytearray is mutable.
  • I can read and write binary files and detect the end with an empty result.
  • I can predict the byte counts of write(), including with non-ASCII text.
  • I know that readinto() can leave old bytes in a buffer after a short read.
  • I can use errno, strerror and filename, and the errno constants.
  • I can order except clauses from the specific to the general.
  • I know UnicodeDecodeError is a ValueError, not an OSError, and how to avoid partial updates.

Official sources