Trace Module Discovery and Import Failures

Introduction

The previous lesson showed which names an import statement binds. This lesson answers the question that comes just before: which file does Python actually load when you write import something? This knowledge belongs to the Modules and Packages section of PCAP-31-03 (12% of the score, 6 of 40 items), objective 1.1, and it is the background you need for the package questions of objective 1.5 later in the course.

Many real-world import problems are not caused by a typo in your code. They are caused by Python finding a different file than the one you intended: a local file that hides a standard-library module, an old copy in another folder, or a package directory that is missing from the search list. Being able to read a traceback and say "Python loaded the wrong file, here is why" is a practical debugging skill, and it is also a skill the exam rewards through short code-analysis questions.

By the end you will be able to describe the search order, use sys.path and sys.modules to inspect it, qualify names of nested modules, and diagnose a project whose file name shadows a standard-library module.

Learning objectives

After this lesson you can:

1. Explain what happens the first time and each later time a module is imported.
2. Describe how sys.path controls the search and what its first entry is.
3. Use dotted names to import a module that lives inside another one (for example os.path).
4. Explain why a local file named like a standard-library module changes program behaviour.
5. Read a traceback top to bottom and identify the failing import line and the exception type.
6. Apply safe habits that avoid shadowing without editing any global Python installation paths.

What import really does

When Python executes import tools, it performs these steps:

1. Check the cache. The dictionary sys.modules maps module names to module objects that are already loaded. If 'tools' is a key, Python reuses that object and does nothing else.
2. Find the module. If it is not cached, Python searches for it using the places listed in sys.path (and a few built-in mechanisms for modules compiled into the interpreter).
3. Load and execute. Python creates a module object and runs the file's top-level code once, from the first line to the last.
4. Cache it. The module object is stored in sys.modules.
5. Bind the name. The import statement binds a name in the importing namespace, following the rules of the previous lesson.

Step 3 has a surprising consequence: top-level statements in a module run when it is imported, and only the first time.

Suppose greeting.py contains:

print("greeting is being loaded")

def hello():
    return "hello"

and the main script does:

import greeting
import greeting
print(greeting.hello())

The output is:

greeting is being loaded
hello

The message appears once. The second import greeting finds the cached module in sys.modules and only re-binds the name. This is the reason you can safely import the same module from many different files in a project: the code runs one time, and every importer shares the same module object (and therefore the same data).

You can look at the cache yourself:

import sys
import math

print('math' in sys.modules)     # True
print(sys.modules['math'] is math)   # True

The search path: sys.path

sys.path is an ordinary Python list of strings. Each string names a directory (or an archive) where Python looks for modules. The search goes from the first entry to the last, and the first match wins.

A typical content is:

1. The directory of the script being run (or an empty string when you run Python interactively or with -c).
2. Directories listed in the PYTHONPATH environment variable, if it is set.
3. The standard-library directories of the installation.
4. The site-packages directory, where third-party packages installed with pip live.

import sys

for entry in sys.path:
    print(entry)

The exact output depends on your machine, which is why the exam will not ask you for it. What you must know is the principle: the script's own folder is searched first. That ordering is what makes local files so powerful, and so dangerous.

Because sys.path is a list, a script can modify it:

import sys

sys.path.append("/opt/company/shared")      # searched last
sys.path.insert(0, "/home/anna/experiments") # searched first

Appending adds a directory at the end, inserting at index 0 places it before everything else. Use this sparingly. Editing the search list inside a program is a fallback for special situations; it makes behaviour depend on the file system layout and is harder to reproduce than a normal project structure.

Python also keeps compiled bytecode in __pycache__ folders to load modules faster. You can ignore them for the exam; they do not change which file is found.

Qualified names for nested modules

Modules can be grouped inside other modules or packages. A dotted name describes the path to the inner module. The standard library offers a familiar example, os.path:

import os.path

print(os.path.basename("/var/log/app.log"))   # app.log

Two points often tested:

• After import os.path, the name bound in your namespace is os (the top-level one), and you reach the inner module as os.path. There is no separate name path.
• from os import path binds path directly, and os is not bound.

import os.path
print(os.getcwd() is not None)    # True: os is bound
print(path)                       # NameError: name 'path' is not defined

from os import path
print(path.basename("/tmp/x.txt"))  # x.txt
print(os.getcwd())                  # NameError: name 'os' is not defined

You can also give the whole chain an alias: import os.path as osp binds only osp.

The same dotted rule applies to packages you write yourself, which you will meet in chapter 3.

When the wrong file wins: shadowing by local modules

Because the script's folder is first on sys.path, a file in your project that has the same name as a standard-library module is found before the library one. Python does not warn you. It simply loads your file.

Worked example: a project that contains random.py

A developer creates a folder dice_game with two files.

random.py (written as an experiment):

print("loading local random")

def choice(items):
    return items[0]

main.py:

import random

print(random.choice([5, 6, 7]))
print(random.randint(1, 3))

The developer runs python main.py from inside dice_game and gets:

loading local random
5
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    print(random.randint(1, 3))
AttributeError: module 'random' has no attribute 'randint'

Step 1: read the traceback from the bottom. The final line gives the exception type and message: AttributeError: module 'random' has no attribute 'randint'. The type is not ImportError; the import succeeded, so the failure happened later, when the attribute was accessed.

Step 2: ask why a module lacks randint. The real random module defines randint, so the loaded module cannot be the real one. The message says "module 'random'", so the name is a module, just not the expected one.

Step 3: notice the unexpected output. The text loading local random was printed, and choice returned 5, the first item, instead of a random one. That is the developer's own random.py running.

Step 4: connect the dots with sys.path. The script's folder is first in the search list, so random.py located there was found before the standard library.

Step 5: confirm. Printing random.__file__ shows the path of the loaded file (for example /home/anna/dice_game/random.py), which proves which module was used.

Fix. Rename the local file to dice_helpers.py. Also delete any leftover __pycache__ copy, since stale bytecode from the old name can remain in the folder, though it no longer matters once the source file is gone.

Why other fixes are worse. Adding sys.path.append for the standard library would not help: the local directory is still first. Reinstalling Python does nothing, because the interpreter is not broken. Editing the installation's search paths changes every project on the machine for a problem that belongs to a single folder.

A subtle variant

The shadowing can also occur indirectly. If your project contains random.py and you import statistics, which itself imports random, then the standard-library module statistics receives your file. You may see a traceback pointing inside the library, with the root cause in your folder. Always look at the file names in the traceback: a path that mixes library files and your own project files is a strong hint.

Another variant: a script named like the module it imports

A script saved as math.py that runs import math loads itself (or another local math.py) instead of the library, so math.sqrt fails with AttributeError. The habit that prevents all of these cases is to avoid file names that match anything in the standard library. If in doubt, run python -c "import name; print(name.__file__)" from a different folder: it shows which file Python would pick there.

Reading a traceback for import errors

A traceback lists the call chain from the oldest call at the top to the failing line at the bottom. For imports, three exception types matter:

ModuleNotFoundError is a subclass of ImportError, so except ImportError: catches both, which makes it possible to provide a fallback:

try:
    import fastjson as json
except ImportError:
    import json

print(json.dumps({"a": 1}))   # {"a": 1}

If fastjson is not installed, the program falls back to the standard json module. Notice the alias: whichever import succeeds, the rest of the program uses the name json. This is a legitimate and common pattern, and an exam-worthy one because it combines imports, aliases and exceptions.

Inspecting where a module comes from

A few attributes help you diagnose a problem without altering anything:

• module.__name__: the module's name, for example 'random'.
• module.__file__: the path of the source file for modules loaded from files (built-in modules compiled into the interpreter may lack it).
• sys.path: where Python searched.
• sys.modules: what has already been loaded.

import json

print(json.__name__)    # json
print(json.__file__)    # e.g. /usr/lib/python3.11/json/__init__.py

A short diagnostic routine for any suspicious import:

import random, sys

print(random.__file__)
print(sys.path[0])

If the first path starts with the second, a local file is being used.

Comparing the failure modes

Common pitfalls and exam tips

• Exam tip: "Which statement is executed when a module is imported twice?" The top-level code runs only the first time; later imports use sys.modules.
• Exam tip: sys.path is a list, so list methods such as append, insert and indexing work on it. The first match wins.
• Pitfall: Believing that running python main.py searches the current working directory. For a script, Python puts the script's own directory first; for python -c or an interactive session it puts an empty entry that means the current directory.
• Pitfall: Assuming that an AttributeError on a module means the library is broken. First print module.__file__.
• Pitfall: Naming your own files test.py, random.py, math.py, string.py or json.py.
• Pitfall: Believing import os.path creates a name path. It binds os.
• Exam tip: When two answers differ only by exception type, ask whether the import failed (ImportError, ModuleNotFoundError) or a later attribute access failed (AttributeError).
• Pitfall: Changing the global installation (copying files into the standard-library folder or editing site configuration) to "fix" one project. Fix the project instead.

Mini case study: the report that always gives the same answer

A data analyst writes report.py that picks 3 random customers for a satisfaction call. After a refactoring, the same three customers are chosen every time. The folder now looks like this:

calls/
    report.py
    random.py      # created last week to test a fixed ordering
    customers.py

report.py:

import random
from customers import ALL

print(random.sample(ALL, 3))

The run prints:

Traceback (most recent call last):
  File "report.py", line 4, in <module>
    print(random.sample(ALL, 3))
AttributeError: module 'random' has no attribute 'sample'

Questions. Why does the error appear, which line causes it, and what is the minimal fix? Would a try/except around the import help?

Solution.

• The error appears because import random on line 1 loads calls/random.py, which lacks sample. The script's folder is first on sys.path.
• The failing line is line 4, where the attribute is used, but the cause is the file name. The traceback shows the symptom.
• The minimal fix is to rename random.py (for example to fixed_order.py) and re-run. No code in report.py has to change.
• A try/except ImportError would not help: the import succeeds, since a module named random is found. The exception type is AttributeError, raised later. Catching AttributeError would only hide the problem.

An extra check proves the diagnosis: python -c "import random; print(random.__file__)" run from a different directory prints the standard-library path, while the same command from calls/ prints the local file.

Key terms

• sys.path: the list of locations searched for modules, first match wins.
• sys.modules: the cache of modules already imported.
• Module search: the process of locating a module's file from its name.
• Shadowing (of a module): a local file hiding a library module of the same name.
• Dotted name: a path to a nested module, such as os.path.
• ModuleNotFoundError: raised when no module of that name can be found.
• __file__: attribute holding the path of the file a module was loaded from.
• Traceback: the report of the call chain that ended in an exception.

Summary checklist

• I can describe the five steps of an import and where the cache fits in.
• I know that a module's top-level code runs only once.
• I can explain that the script's folder comes first in sys.path.
• I can predict what import a.b binds and what from a import b binds.
• I can tell ModuleNotFoundError, ImportError and AttributeError apart in a traceback.
• I can diagnose a local file that shadows a library module and fix it by renaming.
• I can use __file__, sys.path and sys.modules to inspect an import.

Official sources

• Python Institute, PCAP-31-0x exam syllabus (objective 1.1): https://pythoninstitute.org/pcap-exam-syllabus (opens in a new tab)
• Python documentation, the import system and sys.path: https://docs.python.org/3/reference/import.html (opens in a new tab)
• Python documentation, the sys module: https://docs.python.org/3/library/sys.html (opens in a new tab)
• Python tutorial, modules: https://docs.python.org/3/tutorial/modules.html (opens in a new tab)