Skip to content

Python Comprehensions, Lambdas, and Lazy Iterators

Step 30 of 41

Python List Comprehensions: Filters, Nested Loops, and Matrices

Introduction

Python list comprehensions build a list from an iterable using a transformation and optional filters. The position of if matters: [x for x in data if x > 0] selects values, while [x if x > 0 else 0 for x in data] replaces rejected values without dropping positions.

This lesson covers objective 5.1 of PCAP-31-03 in Comprehensions, Lambdas, Closures, and I/O (22% of the score, 9 of 40 items). You will expand nested comprehensions into loops, trace result order, and distinguish an independent matrix row from a shared reference.

Learning objectives

After this lesson you can:

  1. Write a list comprehension equivalent to a loop that builds a list, and the reverse.
  2. Distinguish the filtering if (after the for) from the conditional expression a if c else b (before the for).
  3. Predict the result of nested comprehensions with two for clauses and the order of the generated items.
  4. Flatten and transpose a two-dimensional list.
  5. Explain variable scope inside a comprehension.
  6. Recognise set and dictionary comprehensions and generator expressions as relatives of the list form.
  7. Decide when a loop is clearer than a comprehension.

The anatomy of a list comprehension

The general form is:

Text
[ expression  for item in iterable  if condition ]

The if condition part is optional. Python takes each item from the iterable, tests the condition if there is one, evaluates the expression, and collects the values in a new list.

Python
nums = [3, 8, 5, 12, 7]

squares = []
for n in nums:
    squares.append(n * n)

print(squares == [n * n for n in nums])   # True
print([n * n for n in nums])              # [9, 64, 25, 144, 49]

The loop and the comprehension produce the same list. The comprehension is read like a sentence: "the square of n, for each n in nums". The original list is not modified.

Two points are often tested:

  • The comprehension always creates a new list, even if the expression is simply n. [n for n in nums] is a shallow copy.
  • The result has one item per accepted element. For a finite source that is not changed during iteration, the unfiltered result has one item per source element. An iterable need not support len.

Two different if: filtering versus choosing

The word if can appear in two places, and the place decides the meaning.

An if after the for is a filter. It decides whether an element participates at all. Elements that fail the test are dropped, so the result can be shorter.

Python
print([n for n in nums if n > 5])           # [8, 12, 7]
print([n * 2 for n in nums if n % 2 == 1])  # [6, 10, 14]

An if ... else before the for is a conditional expression. It chooses between two values for every element. Nothing is dropped, so the length is unchanged.

Python
print([n if n > 5 else 0 for n in nums])    # [0, 8, 0, 12, 7]

A conditional expression needs both branches; [n if n > 5 for n in nums] is a SyntaxError. You can combine both forms:

Python
print(["even" if n % 2 == 0 else "odd" for n in nums if n > 4])
# ['even', 'odd', 'even', 'odd']

Here the filter keeps 8, 5, 12, 7, and the conditional expression labels each one.

Form Position Effect on length Else required?
[x for x in s if c] after for may shrink no
[a if c else b for x in s] before for same as source yes
both together both places may shrink yes for the front part

A truthiness shortcut also works in filters: [x for x in range(5) if x] gives [1, 2, 3, 4], because 0 is falsy and is dropped. In front of the for, [x if x else "z" for x in [0, 1, 2]] gives ['z', 1, 2].

Nested comprehensions: two for clauses

A comprehension may contain several for clauses. They are read from left to right, as nested loops, with the first clause as the outer loop.

Python
print([(a, b) for a in range(2) for b in "xy"])
# [(0, 'x'), (0, 'y'), (1, 'x'), (1, 'y')]

Expanded, this is:

Python
result = []
for a in range(2):
    for b in "xy":
        result.append((a, b))

The second clause can depend on the first:

Python
print([(a, b) for a in range(3) for b in range(a)])
# [(1, 0), (2, 0), (2, 1)]

For a = 0, range(0) is empty, so nothing is produced. For a = 1, b is 0. For a = 2, b is 0 and 1.

A filter can follow any clause. Its position decides when it is tested:

Python
m = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
print([x for row in m for x in row if x % 2 == 0])   # [2, 4, 6, 8]
print([x for row in m if sum(row) > 6 for x in row]) # [4, 5, 6, 7, 8, 9]

In the first, the filter tests each x. In the second, it tests each row before the inner loop starts, so the first row (sum 6) is skipped as a whole.

A reliable reading method

To trace any comprehension, rewrite it as loops:

  1. Write one for line per for clause, in the same order, each indented one level deeper.
  2. Write each if at the level where it appears in the text.
  3. Put append(expression) at the deepest level.
  4. Run the loops by hand.

This method removes the guesswork. Never try to read a nested comprehension from right to left.

Flattening and transposing a two-dimensional list

Two classic uses of nested structure must be kept apart.

Flattening uses two for clauses in one list. The result is a single flat list:

Python
print([x for row in m for x in row])
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

A comprehension inside a comprehension builds a list of lists. The inner comprehension is the expression of the outer one:

Python
print([[r[i] for r in m] for i in range(3)])
# [[1, 4, 7], [2, 5, 8], [3, 6, 9]]

For this rectangular, three-column matrix, this is its transpose. Other widths require adjusting the column range; a ragged row can raise IndexError: the outer clause chooses the column i, and the inner clause collects item i from each row r. A frequent exam contrast is [[x for x in row] for row in m] (still 3 rows, a copy of the structure) versus [x for row in m for x in row] (one flat list).

Python
print([[y for y in range(x)] for x in range(4)])
# [[], [0], [0, 1], [0, 1, 2]]

Another important idea concerns building a grid. The expression [[0] * 2] * 2 repeats the same inner list twice, whereas a comprehension creates new lists:

Python
rows = [[0] * 2] * 2
rows[0][0] = 1
print(rows)            # [[1, 0], [1, 0]]

rows = [[0] * 2 for _ in range(2)]
rows[0][0] = 1
print(rows)            # [[1, 0], [0, 0]]

In the first case both rows are the same object, so a change appears twice. In the second, each pass of the loop evaluates [0] * 2 again, producing independent rows. The variable name _ is a convention for "value not used".

Scope: the loop variable stays inside

In Python 3, the variable of a list comprehension exists only inside the comprehension. It does not overwrite a variable of the same name outside.

Python
i = 10
l = [i for i in range(3)]
print(i, l)            # 10 [0, 1, 2]

After the comprehension, i is still 10. (In a for loop it would have become 2.) The iteration variable stays isolated even though surrounding names remain readable. Assignment expressions have separate binding rules and can bind a containing-scope name. Lambdas may retain a reference to the comprehension’s iteration variable; see late binding in comprehension callbacks.

Relatives: set, dictionary and generator forms

The same syntax with different brackets builds other objects:

Python
print(sorted({n % 3 for n in nums}))          # [0, 1, 2]
print({w: len(w) for w in ["ab", "cde"]})     # {'ab': 2, 'cde': 3}
g = (n for n in nums)
print(type(g).__name__, sum(g), sum(g))       # generator 35 0
Syntax Result Notes
[e for x in s] list built immediately
{e for x in s} set duplicates removed, order not guaranteed
{k: v for x in s} dict later keys overwrite earlier equal keys
(e for x in s) generator lazy; can be consumed only once

The last line shows the subtle point: the first sum(g) consumes the generator; the second finds it empty and returns 0. Parentheses around a comprehension do not create a tuple. For a tuple use tuple(...).

When not to use a comprehension

A comprehension should build a collection from a clear rule. Use a plain loop when:

  • the body has side effects (printing, writing, modifying other objects); a comprehension used only for its side effects builds a useless list;
  • the logic needs several statements, try/except, or more than two nested clauses;
  • readability suffers: a line you cannot explain at a glance should be split.
Python
words = ["Tea", "coffee", "Milk", "juice"]
print([w.upper() for w in words if w[0].islower()])   # ['COFFEE', 'JUICE']

This is a good comprehension: one source, one test, one transformation.

Worked example: filtering a warehouse stock report

Statement. A warehouse stores the quantity of each product by shelf. A report must list every non-empty shelf as a pair (product, quantity), count them, and compute a dictionary of products whose total is greater than 2. Predict the output.

Python
stock = [("A", [4, 0, 7]), ("B", [0, 0, 2]), ("C", [5, 5, 0])]

flat = [(name, q) for name, qs in stock for q in qs if q > 0]
print(flat, len(flat))
print([q for _, qs in stock for q in qs])
print({name: sum(qs) for name, qs in stock if sum(qs) > 2})

Step 1: the first comprehension. Rewrite it as loops: outer loop over stock, unpacking each tuple into name and qs; inner loop over qs; the filter q > 0 at the inner level; append((name, q)) at the deepest level.

Step 2: run it. Product A: quantities 4, 0, 7 give ("A", 4) and ("A", 7) (zero dropped). Product B: only 2 is positive, so ("B", 2). Product C: 5, 5, 0 give ("C", 5) twice.

Step 3: the second line. The same two clauses without a filter and with q alone as the expression return all nine quantities in order: [4, 0, 7, 0, 0, 2, 5, 5, 0].

Step 4: the dictionary. The totals are A: 11, B: 2, C: 10. The filter keeps totals greater than 2, so B is excluded.

Text
[('A', 4), ('A', 7), ('B', 2), ('C', 5), ('C', 5)] 5
[4, 0, 7, 0, 0, 2, 5, 5, 0]
{'A': 11, 'C': 10}

Why other readings fail. An answer with a single element per product forgets that the inner loop runs once per quantity. An answer containing ('A', 0) has misread the filter as a conditional expression. Listing B in the dictionary wrongly keeps a total equal to 2, but the condition is strictly greater than 2.

Common pitfalls and exam tips

Pitfall What happens Remedy
[x if c for x in s] SyntaxError A conditional expression needs else
Using the filter if when values must be replaced Items are dropped instead of replaced Put a if c else b before the for
Reading nested for clauses right to left Wrong order of results Read as loops, first clause outer
Flattening with [[x for x in row] for row in m] Still a list of lists Use [x for row in m for x in row]
[[0] * n] * k for a grid Rows share one list [[0] * n for _ in range(k)]
Expecting the loop variable to survive It does not leak in Python 3 Use a normal loop if you need it afterwards
Parentheses to make a tuple Creates a generator Use tuple(...)
Reusing a generator Second pass is empty Create it again or build a list

Exam tip. Count the clauses. One for with no if keeps the length; an if after for may shorten it; a conditional expression never changes it.

Exam tip. When two for clauses appear, the number of results is the sum of inner iterations for each outer element, not the product of the sizes unless the inner range is independent.

Exam tip. A list comprehension creates a new outer list. Replacing or appending its entries does not change the source’s outer structure, but mutating an element shared by both is visible through either list. Review Python object identity and shallow copies before assuming a shallow copy isolates nested state.

End-of-lesson case study

Situation. A teacher stores scores per class and wants a flat list of passing scores (at least 50), plus a list that replaces failing scores with None. A colleague wrote the code below. Predict the output and describe the mistake if the goal was "flat list of passing scores".

Python
classes = [[40, 70], [55, 30, 90], []]

a = [score for group in classes for score in group if score >= 50]
b = [[score if score >= 50 else None for score in group] for group in classes]
print(a)
print(b)

Solution, commented. The assignment to a is valid. Flattening gives [70, 55, 90]: the clauses are read as an outer loop over classes and an inner loop over scores, and the filter drops 40 and 30. The empty class contributes nothing.

The assignment to b is also valid. The conditional expression sits in front of the inner for, so no score is removed, and the nested structure is preserved: [[None, 70], [55, None, 90], []]. The empty list stays as an empty list.

A proposed alternative is invalid:

Python
c = [score if score >= 50 for group in classes for score in group]

The conditional expression lacks else. If this invalid line were placed in the same script as the valid code, the script would fail to compile and neither print would run. To keep only passing scores, use the trailing filter already shown in a; to retain positions, use the complete conditional expression in b.

Key terms

  • List comprehension: an expression that builds a new list from an iterable, with optional filtering.
  • Filter clause: the if after the for; drops elements.
  • Conditional expression: a if c else b; chooses a value, requires both branches.
  • Nested comprehension: several for clauses (flattening) or a comprehension used as an expression (list of lists).
  • Generator expression: a comprehension in parentheses that yields values lazily, once.
  • Comprehension scope: the loop variable stays inside the comprehension.

Summary checklist

  • I can translate a comprehension into loops and back.
  • I can tell the filtering if from the if ... else expression and predict the length of the result.
  • I read multiple for clauses from left to right, first as the outer loop.
  • I can flatten and transpose a two-dimensional list.
  • I know why [[0] * n] * k is dangerous and how to fix it.
  • I know the loop variable does not leak outside a comprehension in Python 3.
  • I can recognise set, dictionary and generator forms and the single-use nature of generators.
  • I choose a loop when side effects or complex logic are involved.

Official sources