Home

03 Testing Fundamentals: Testing Theory, Coverage, and Requirements

December 2025 (9689 Words, 54 Minutes)

lecture testing coverage test-theory white-box black-box branch-coverage requirements test-adequacy

1. Introduction: Your Tests Pass, But Are They Enough?

Where you are so far:

From Chapter 03 (Testing Fundamentals), you learned how to write unit tests using pytest, design tests using equivalence classes and boundary value analysis, and understand the testing pyramid (70% unit, 20% module, 10% E2E).

From Chapter 03 (TDD and CI), you learned how to add test coverage measurement to your CI pipeline using pytest-cov, and you learned that coverage ≠ correctness. You saw that achieving 70-80% coverage is a good target, but you also learned that high coverage doesn’t guarantee bug-free software.

But here’s the gap:

You know how to measure coverage (run pytest --cov), but you don’t know:

Consider this scenario:

You’re working on the Road Profile Viewer project. The geometry.py module has a function find_intersection() that calculates where a camera ray intersects with a road profile. You run the tests:

$ pytest tests/test_geometry.py -v
==== test session starts ====
tests/test_geometry.py::test_find_intersection_finds_intersection_for_normal_angle PASSED [100%]

==== 1 passed in 0.15s ====

✅ All tests pass! Ship it, right?

Then you check coverage:

$ pytest tests/test_geometry.py --cov=road_profile_viewer.geometry --cov-report=term-missing
==== test session starts ====
tests/test_geometry.py::test_find_intersection_finds_intersection_for_normal_angle PASSED [100%]

---------- coverage: platform win32, python 3.12.0 -----------
Name                                  Stmts   Miss  Cover   Missing
-------------------------------------------------------------------
road_profile_viewer\geometry.py          45     18    60%   45, 98, 109-110, 123-127, 138
-------------------------------------------------------------------

Only 60% coverage! And you have 18 missing lines - lines that were never executed by your test suite.

Today’s question: How do you systematically design tests to cover those missing lines? More importantly, should you cover them, and what does coverage tell you about the quality of your tests?

2. Learning Objectives

By the end of this lecture, you will:

  1. Understand the theoretical framework for testing: Program, Input Domain, Test Suite, Model, and Coverage Criterion
  2. Explain different types of code coverage: Statement (C0) and Branch (C1) coverage
  3. Systematically design tests to achieve specific coverage goals (e.g., 100% branch coverage)
  4. Recognize that equivalence classes and boundary values are also coverage criteria over an input-domain model
  5. Understand the limitations of coverage metrics
  6. Apply an iterative spiral between requirements, tests, and coverage to refine both

What you WILL learn:

What you WON’T learn:

3. The Testing Theory: Program, Model, Test Suite, Coverage

Before we dive into code, let’s establish a theoretical vocabulary. This framework will help you think clearly about testing and coverage for the rest of your career.

3.1 Core Definitions

Connecting to Chapter 03 (Testing Fundamentals): The Infinite Input Problem

In Chapter 03 (Testing Fundamentals), you learned about equivalence classes as a practical technique for dealing with infinite input domains. Remember the reciprocal_sum() example? You couldn’t test all possible combinations of (x, y, z) values, so you partitioned the input space into equivalence classes (positive sum, negative sum, zero sum) and tested one representative from each.

That was test design thinking. Now we’re going one level deeper: How do we formalize this? How do we measure if our tests are “good enough”? How do we compare different testing strategies?

This is where testing theory comes in. Let’s build up the formal vocabulary, starting with a simple example.

Example: A Simple Function to Start With

Before tackling find_intersection(), let’s start with something trivial:

def classify_number(x: float) -> str:
    """Classify a number as positive, negative, or zero."""
    if x > 0:
        return "positive"
    elif x < 0:
        return "negative"
    else:
        return "zero"

This function has discrete output values (only 3 possible strings) and clear decision logic (perfect for learning the theory).

Now let’s define the formal terms using this example:

Program \(P\)

The software artifact you’re testing.

Input Domain \(D\)

The set of all possible inputs to the program.

For classify_number():

\(D = \mathbb{R}\) (all real numbers)

For find_intersection():

\(D = \{(x_{road}, y_{road}, angle, camera_x, camera_y) \lvert x_{road}, y_{road} \in \mathbb{R}^n, angle \in \mathbb{R}, camera_x, camera_y \in \mathbb{R}\}\)

Both are infinite! You can’t test every possible input. This is exactly the problem Chapter 03 (Testing Fundamentals)’s equivalence classes addressed.

Test Case \(t\)

A single element from the input domain: \(t \in D\)

For classify_number(): \(t = 5.0\) or \(t = -3.14\) or \(t = 0.0\)

For find_intersection(): t = (np.array([0, 10, 20]), np.array([0, 2, 4]), 10.0, 0.0, 10.0)

Test Suite \(T\)

A finite subset of the input domain: \(T \subseteq D\)

Your test suite is the collection of all test cases you actually run.

For classify_number(), a good test suite might be:

\(T = \{5.0, -3.14, 0.0\}\) → \(\vert T \vert = 3\)

For find_intersection(), you currently have:

\(\vert T \vert = 1\) (only one test!)

Connecting Back to Equivalence Classes

Notice how \(T = \{5.0, -3.14, 0.0\}\) for classify_number() maps directly to equivalence classes from Chapter 03 (Testing Fundamentals):

Equivalence classes are one way to systematically choose a finite \(T\) from an infinite \(D\). But they’re not the only way! That’s what we’re exploring in this lecture.

3.2 Model of a Program

Here’s the crucial insight: Coverage is always coverage of a model.

A model \(M(P)\) is an abstraction of the program’s behavior. But what exactly is \(M(P)\)? Is it a function? A set? A graph?

Answer: It depends on the model type! Different models represent programs as different mathematical objects.

Structural Models (white-box)

These models are derived from the program’s source code structure:

Semantic Models (black-box)

These models are derived from the program’s intended behavior:

Why does this matter? Because \(M(P)\) is the foundation for defining test requirements \(R_C(P)\). Let’s see concrete examples.

Example 1: CFG for classify_number() - The Complete Model

A Control Flow Graph (CFG) is a graphical representation of all paths that might be executed through a program. Each node represents a statement or decision, and each edge represents control flow.

The Model \(M_{CFG}(\text{classify_number})\):

For the CFG model, \(M(P)\) is a directed graph \(G = (V, E)\) where:

\(V = \{\text{Start}, \text{Check1}, \text{Check2}, \text{Positive}, \text{Negative}, \text{Zero}, \text{End}\}\)

\(E = \{(\text{Start}, \text{Check1}), (\text{Check1}, \text{Positive}), (\text{Check1}, \text{Check2}), (\text{Check2}, \text{Negative}), (\text{Check2}, \text{Zero}), (\text{Positive}, \text{End}), (\text{Negative}, \text{End}), (\text{Zero}, \text{End})\}\)

That’s what M(P) actually is - a concrete set of vertices and edges!

Here’s the visual representation of \(M_{CFG}(\text{classify_number})\):

graph TD
    Start([Start]) --> Check1{x > 0?}
    Check1 -->|True| Positive[Return 'positive']
    Check1 -->|False| Check2{x < 0?}
    Check2 -->|True| Negative[Return 'negative']
    Check2 -->|False| Zero[Return 'zero']
    Positive --> End([End])
    Negative --> End
    Zero --> End

From Model to Test Requirements:

Now that we have \(M_{CFG}(\text{classify_number})\), we can define different coverage criteria. Let’s preview two common ones (we’ll define them precisely in Section 4):

Statement Coverage (C₀):

\(R_{C_0}(P) = V \setminus \{\text{Start}, \text{End}\} = \{\text{Check1}, \text{Check2}, \text{Positive}, \text{Negative}, \text{Zero}\}\)

Each requirement means “execute this node at least once.” So \(\vert R_{C_0}(P) \vert = 5\) test requirements.

Branch Coverage (C₁):

\(R_{C_1}(P) = E = \{(\text{Start}, \text{Check1}), (\text{Check1}, \text{Positive}), (\text{Check1}, \text{Check2}), …\}\)

Each requirement means “traverse this edge at least once.” So \(\vert R_{C_1}(P) \vert = 8\) test requirements.

For now, just focus on understanding how \(R_C(P)\) is derived from \(M(P)\) - the specific meanings of C₀ and C₁ will become clear in Section 4.

Key insight: There are 3 possible execution paths through this CFG:

  1. \(x > 0\) → return “positive” (executes edges: Start→Check1, Check1→Positive, Positive→End)
  2. \(x \leq 0\) and \(x < 0\) → return “negative” (executes edges: Start→Check1, Check1→Check2, Check2→Negative, Negative→End)
  3. \(x \leq 0\) and \(x \geq 0\) → return “zero” (executes edges: Start→Check1, Check1→Check2, Check2→Zero, Zero→End)

Our test suite \(T = \{5.0, -3.14, 0.0\}\) executes all 3 paths, covering all 8 edges! This is 100% branch coverage of the CFG.

Does \(M(P)\) matter for students? YES! Understanding that:

…helps you understand WHY different coverage criteria exist and how to design tests systematically.

Example 2: CFG for find_intersection() (Real-World Complexity)

Now let’s see why the simple example was necessary. Here’s the CFG for the actual find_intersection() function you’re testing:

graph TD
    Start([Start]) --> B1{cos angle ≈ 0?}
    B1 -->|True| R1[Return None]
    B1 -->|False| Loop[For each segment]
    Loop --> B2{Segment behind<br/>camera?}
    B2 -->|True| Loop
    B2 -->|False| B3{Ray crosses<br/>segment?}
    B3 -->|False| Loop
    B3 -->|True| B4{Parallel lines?}
    B4 -->|True| SetT[Set t=0]
    B4 -->|False| SetT
    SetT --> Calc[Calculate intersection]
    Calc --> R2[Return intersection]
    Loop -->|No more segments| R3[Return None]
    R1 --> End([End])
    R2 --> End
    R3 --> End

Notice the difference in complexity! find_intersection() has:

Your single existing test only explores one path through this CFG. Many branches remain untested!

Example 3: Call Graph (For Multi-Function Programs)

A Call Graph shows which functions call which other functions. For example, if you had:

def main():
    data = load_data()
    result = process(data)
    save_result(result)

The Call Graph would be:

graph LR
    main[main] -->|calls| load[load_data]
    main -->|calls| proc[process]
    main -->|calls| save[save_result]

Coverage criteria can target this too: “Did you test all function calls?” For this simple example, you’d need tests that ensure main() actually invokes all three helper functions.

Example 4: Data Flow Graph (Tracking Variable Usage)

A Data Flow Graph tracks how data flows through variables. Consider:

def compute(x):
    y = x * 2      # Definition of y
    z = y + 1      # Use of y, Definition of z
    return z       # Use of z

Data flow coverage asks: “Did you test all definition-use pairs?” For example:

Summary: Why Models Matter

Different models lead to different coverage criteria:

You choose the model based on what bugs you’re trying to catch!

3.3 Coverage Criterion and Test Requirements

A coverage criterion \(C\) is a rule that defines a finite set of test requirements (also called test obligations):

\(R_C(P) = \{r_1, r_2, …, r_n\}\)

Each requirement \(r_i\) is something that should be “covered” by your test suite.

A test case \(t \in D\) satisfies a requirement \(r \in R_C(P)\) if, when you execute \(P\) on \(t\), the execution exhibits whatever behavior \(r\) describes. For example:

Example: Test requirements for find_intersection()

If we use statement coverage as our criterion \(C_{stmt}\), then:

\(R_{C_{stmt}}(\text{find_intersection}) = \lbrace\text{stmt}_1, \text{stmt}_2, …, \text{stmt}_{22}\rbrace\)

Each requirement \(\text{stmt}_i\) means “execute statement \(i\) at least once.”

Our single existing test satisfies approximately 60% of these requirements (only ~13 out of 22 statements).

3.4 The Adequacy Framework

Now let’s formalize coverage measurement and adequacy.

Coverage of a Test Suite

The coverage of a test suite \(T\) with respect to criterion \(C\) is the set of requirements satisfied by \(T\):

\(\text{cov}_C(T, P) = \{ r \in R_C(P) \lvert \exists t \in T \text{ such that } t \text{ satisfies } r \}\)

The coverage percentage is:

\(\frac{\vert\text{cov}_C(T, P)\vert}{\vert R_C(P)\vert} \times 100\%\)

Test Adequacy

A test suite \(T\) is adequate with respect to criterion \(C\) if and only if:

\(\text{cov}_C(T, P) = R_C(P)\)

In other words, \(T\) is adequate when it satisfies all requirements. This corresponds to “100% coverage.”

Example: Adequacy for find_intersection()

Currently:

Coverage: \(\frac{13}{22} \times 100\% \approx 59\%\)

\(T\) is NOT adequate for \(C_{stmt}\) because \(\text{cov}_{C_{stmt}}(T, P) \neq R_{C_{stmt}}(P)\).

Properties of Good Coverage Criteria

A useful coverage criterion should have these properties:

1. Finity (Finiteness)

\(R_C(P)\) must be finite, otherwise you could never achieve adequacy.

2. Monotonicity

If \(T \subseteq T’\), then \(\text{cov}_C(T, P) \subseteq \text{cov}_C(T’, P)\).

Adding tests cannot reduce coverage. This makes intuitive sense!

Example:

3. Non-triviality

We don’t want \(R_C(P) = \emptyset\) (that would make adequacy meaningless).

Every program should have at least some requirements to satisfy.

Other Considerations:

3.5 Subsumption: Comparing Coverage Criteria

Given two coverage criteria \(C_1\) and \(C_2\), we say:

\(C_1\) subsumes \(C_2\) if and only if: For every program \(P\) and every test suite \(T\): If \(T\) is adequate for \(C_1\), then \(T\) is also adequate for \(C_2\).

What does subsumption mean in practice?

If \(C_1\) subsumes \(C_2\), then:

Subsumption hierarchy (preview):

Path Coverage
     ↓ subsumes
Branch Coverage (C1)
     ↓ subsumes
Statement Coverage (C0)
     ↓ subsumes
Function Coverage

Formal example: Branch subsumes Statement

Let’s prove that branch coverage (\(C_1\)) subsumes statement coverage (\(C_0\)) for find_intersection().

Proof sketch:

Consider the CFG \(G = (V, E)\) where:

Statement coverage requirements: \(R_{C_0}(P) = V\)

Each requirement is “execute node \(v\) at least once.”

Branch coverage requirements: \(R_{C_1}(P) = E\)

Each requirement is “traverse edge \(e\) at least once.”

Key observation: Every edge \(e = (v_1, v_2) \in E\) has:

If you traverse edge \(e\), you must execute both \(v_1\) and \(v_2\).

Therefore: If a test suite \(T\) covers all edges \(E\), it must cover all nodes \(V\).

Formally: If \(\text{cov}{C_1}(T, P) = E\), then \(\text{cov}{C_0}(T, P) = V\).

Conclusion: \(C_1\) (branch) subsumes \(C_0\) (statement). ✅

Example with find_intersection():

Suppose you achieve 100% branch coverage:

So 100% branch coverage automatically gives you 100% statement coverage for these statements!

But the converse is NOT true!

You could have 100% statement coverage without 100% branch coverage:

Bad test suite example:

def test_only_normal_case():
    # Only tests the "False" branch of every decision
    x, y, dist = find_intersection(
        np.array([0, 10, 20]),
        np.array([0, 2, 4]),
        10.0, 0.0, 10.0
    )
    assert x is not None

This test might execute many statements (high C0), but it never tests:

So: High C0 ≠ High C1, but High C1 → High C0.

Why subsumption matters:

  1. Guides criterion selection: Choose stronger criteria when you can afford them
  2. Avoid redundant work: Don’t measure both C1 and C0 (C1 is sufficient)
  3. Theoretical foundations: Helps compare different testing strategies formally

Other subsumption relationships (future topics):

Connection to Chapter 03 (Testing Fundamentals):

Remember in Chapter 03 (Testing Fundamentals), you learned about equivalence classes as an input-domain partitioning technique. Equivalence class coverage is also a coverage criterion:

\(R_{EC}(P) = \lbrace EC_1, EC_2, …, EC_n\rbrace\)

Each requirement is “test at least one representative from equivalence class \(EC_i\).”

But equivalence class coverage and branch coverage are incomparable (neither subsumes the other)! You need both:

We’ll see this in detail in Section 6.

4. Code Coverage: Statement (C0) and Branch (C1)

Now let’s make this concrete with the two most common coverage criteria.

4.1 Statement Coverage (C0)

Model: Control Flow Graph (nodes = statements)

Test Requirements: Execute every statement at least once

\(R_{C0}(P) = \{\text{statement}_1, \text{statement}_2, …, \text{statement}_n\}\)

Example for find_intersection():

def find_intersection(
    x_road: NDArray[np.float64],
    y_road: NDArray[np.float64],
    angle_degrees: float,
    camera_x: float = 0,
    camera_y: float = 1.5,
) -> tuple[float | None, float | None, float | None]:
    angle_rad = -np.deg2rad(angle_degrees)              # Statement 1

    if np.abs(np.cos(angle_rad)) < 1e-10:               # Statement 2 (decision)
        return None, None, None                         # Statement 3

    slope = np.tan(angle_rad)                           # Statement 4

    for i in range(len(x_road) - 1):                    # Statement 5 (loop)
        x1, y1 = x_road[i], y_road[i]                   # Statement 6
        x2, y2 = x_road[i + 1], y_road[i + 1]           # Statement 7

        if x2 <= camera_x:                              # Statement 8 (decision)
            continue                                    # Statement 9

        ray_y1 = camera_y + slope * (x1 - camera_x)     # Statement 10
        ray_y2 = camera_y + slope * (x2 - camera_x)     # Statement 11

        diff1 = ray_y1 - y1                             # Statement 12
        diff2 = ray_y2 - y2                             # Statement 13

        if diff1 * diff2 <= 0:                          # Statement 14 (decision)
            if abs(diff2 - diff1) < 1e-10:              # Statement 15 (decision)
                t = 0                                   # Statement 16
            else:
                t = diff1 / (diff1 - diff2)             # Statement 17

            x_intersect = x1 + t * (x2 - x1)            # Statement 18
            y_intersect = y1 + t * (y2 - y1)            # Statement 19

            distance = np.sqrt(...)                     # Statement 20

            return x_intersect, y_intersect, distance   # Statement 21

    return None, None, None                             # Statement 22

Statement coverage requires executing all 22 statements at least once.

What does 60% coverage mean?

From the coverage report earlier: 45 statements, 18 missed = 60%

This means 18 statements (like statements 3, 9, 16, 22 above) were never executed by your single test.

4.2 Branch Coverage (C1)

Model: Control Flow Graph (edges = branches from decisions)

Test Requirements: Exercise every decision outcome (True and False) at least once

\(R_{C1}(P) = \{\text{branch}_{1T}, \text{branch}_{1F}, \text{branch}_{2T}, \text{branch}_{2F}, …\}\)

Why is this different from C0?

Consider this code:

if condition:
    do_something()
do_next_thing()

With C0, you only need to execute do_next_thing() - you could skip the if entirely!

With C1, you need to test BOTH:

Branch coverage table for find_intersection():

Branch ID Decision True Path False Path Current Coverage
B1 np.abs(np.cos(angle_rad)) < 1e-10 Vertical ray → return None Normal ray → continue ❌ Only False covered
B2 x2 <= camera_x Segment behind camera → skip Segment in front → check intersection ❌ Only False covered
B3 diff1 * diff2 <= 0 Ray crosses segment → calculate intersection Ray doesn't cross → continue ✅ Both covered (lucky!)
B4 abs(diff2 - diff1) < 1e-10 Parallel lines → t=0 Normal intersection → calculate t ❌ Only False covered
B5 Loop iterations At least one iteration Zero iterations (empty array) ❌ Zero iterations not tested
B6 Return path after loop No intersection found → return None Intersection found earlier → return values ❌ No intersection path not tested

4.3 C1 Subsumes C0

Claim: If you achieve 100% branch coverage, you automatically achieve 100% statement coverage.

Proof by example:

To cover both branches of if condition: do_something():

Both branches require executing the if statement itself ✅

So every statement is executed. QED.

But the converse is NOT true: You can have 100% C0 without 100% C1 (by only testing one branch of each decision).

5. Systematic Test Design for Branch Coverage

Now comes the practical part: How do you systematically design tests to achieve 100% C1 coverage for find_intersection()?

5.1 Step-by-Step Methodology

Step 1: Identify all decision points

Go through the code and list every if, for, while, and decision:

  1. if np.abs(np.cos(angle_rad)) < 1e-10 (vertical ray check)
  2. for i in range(len(x_road) - 1) (loop - 0 vs 1+ iterations)
  3. if x2 <= camera_x (behind camera check)
  4. if diff1 * diff2 <= 0 (intersection check)
  5. if abs(diff2 - diff1) < 1e-10 (parallel check)
  6. Return paths (early return vs loop completion)

Step 2: For each decision, design inputs for BOTH outcomes

Use your CFG or mental trace to figure out:

Step 3: Check loop coverage

Step 4: Check all return paths

5.2 Worked Example 1: Vertical Ray Test

Branch: B1 (True path) - np.abs(np.cos(angle_rad)) < 1e-10

Analysis: This is True when the ray is vertical (angle = 90° or 270°), because \(\cos(90°) = 0\).

Test:

import numpy as np
from road_profile_viewer.geometry import find_intersection

def test_vertical_ray_returns_none():
    """
    Test that find_intersection() returns None for a vertical ray.

    Branch coverage: B1 True path
    Equivalence class: Vertical rays (cos(angle) ≈ 0)
    """
    # Arrange: Create any road profile (doesn't matter - won't intersect)
    x_road = np.array([0, 10, 20], dtype=np.float64)
    y_road = np.array([0, 1, 2], dtype=np.float64)

    # Vertical ray (downward): angle = 90°
    angle = 90.0
    camera_x = 5.0
    camera_y = 10.0

    # Act
    x, y, dist = find_intersection(x_road, y_road, angle, camera_x, camera_y)

    # Assert: Vertical ray should return None (by design)
    assert x is None, "Vertical ray should return None for x"
    assert y is None, "Vertical ray should return None for y"
    assert dist is None, "Vertical ray should return None for distance"

Coverage gained: ✅ B1 True path, Statement 3

5.3 Worked Example 2: Road Segment Behind Camera

Branch: B2 (True path) - x2 <= camera_x

Analysis: This is True when the entire road is behind the camera position.

Test:

def test_road_entirely_behind_camera_returns_none():
    """
    Test that find_intersection() returns None when all road segments
    are behind the camera position.

    Branch coverage: B2 True path, B6 True path (loop completes without finding intersection)
    Equivalence class: All road x-coordinates <= camera_x
    """
    # Arrange: Road at x = [0, 5, 10], camera at x = 15
    x_road = np.array([0, 5, 10], dtype=np.float64)
    y_road = np.array([0, 1, 2], dtype=np.float64)

    angle = 45.0  # Downward angle
    camera_x = 15.0  # Camera is at x=15 (all road is behind at x <= 10)
    camera_y = 5.0

    # Act
    x, y, dist = find_intersection(x_road, y_road, angle, camera_x, camera_y)

    # Assert: No intersection because road is entirely behind camera
    assert x is None
    assert y is None
    assert dist is None

Coverage gained: ✅ B2 True path, ✅ B6 True path (no intersection found), Statement 9, Statement 22

5.4 Worked Example 3: Parallel Lines Case

Branch: B4 (True path) - abs(diff2 - diff1) < 1e-10

Analysis: This happens when the ray is parallel to a road segment. This is a numerical stability edge case.

Test:

def test_parallel_ray_to_road_segment():
    """
    Test intersection when ray is parallel to a road segment.

    Branch coverage: B4 True path
    Equivalence class: Ray slope equals road segment slope
    Boundary case: Numerical stability when differences are near zero
    """
    # Arrange: Horizontal road segment
    x_road = np.array([0, 10, 20], dtype=np.float64)
    y_road = np.array([5, 5, 3], dtype=np.float64)  # First segment is horizontal at y=5

    # Horizontal ray at y=5 (parallel to first segment)
    angle = 0.0  # Horizontal ray
    camera_x = 5.0
    camera_y = 5.0  # Camera exactly on the road line

    # Act
    x, y, dist = find_intersection(x_road, y_road, angle, camera_x, camera_y)

    # Assert: Should find intersection (using t=0 for parallel case)
    # The implementation chooses the start of the segment
    assert x is not None
    assert y is not None
    assert dist is not None

Coverage gained: ✅ B4 True path, Statement 16

5.5 Student Exercises

Now it’s your turn! Design tests for these remaining branches:

Exercise 1: Design a test for B5 False path (zero loop iterations)

Hint: What happens if x_road and y_road have length 0 or 1?

Exercise 2: Design a test for B3 False path (ray doesn’t cross segment)

Hint: Position the road below the camera and aim the ray upward.

Exercise 3: Design a test that finds an intersection on the second segment, not the first

Hint: Make the first segment miss, and the second segment hit.

Exercise 4: Design a test for a boundary case: intersection exactly at a road vertex

Hint: Make the ray pass through x_road[i] exactly.

6. Equivalence Classes and Boundary Values as Coverage Criteria

Remember equivalence classes and boundary value analysis from Chapter 03 (Testing Fundamentals)? They seemed like informal heuristics. But now you have the theoretical framework to understand them properly:

Equivalence classes are a coverage criterion over an input-domain model.

6.1 The Input-Domain Model

Model: Partition the input domain \(D\) into equivalence classes \(EC_1, EC_2, …, EC_n\)

Where:

Test Requirements:

\(R_{EC}(P) = \lbrace EC_1, EC_2, …, EC_n\rbrace\)

Coverage: Test at least one representative value from each equivalence class.

6.2 Equivalence Classes for find_intersection()

EC ID Equivalence Class Description Representative Value Expected Behavior
EC1 Normal intersection Ray hits road in front of camera, single intersection angle=10°, camera above road, road in front Returns (x, y, dist) with valid values
EC2 No intersection Ray misses road entirely (e.g., ray points upward) angle=-45°, road below camera Returns (None, None, None)
EC3 Vertical ray Ray is vertical (angle = 90° or 270°) angle=90° Returns (None, None, None)
EC4 Road behind camera All road segments have x ≤ camera_x x_road=[0,10], camera_x=20 Returns (None, None, None)
EC5 Multiple intersections Ray could hit multiple segments (return first) Wavy road with ray crossing multiple segments Returns first intersection found
EC6 Intersection at vertex Ray passes exactly through a road vertex Ray aimed at x_road[i], y_road[i] Returns vertex coordinates
EC7 Parallel ray and segment Ray slope equals road segment slope Horizontal ray, horizontal road segment Returns intersection (uses t=0)
EC8 Empty road x_road and y_road have length < 2 x_road=[], y_road=[] Returns (None, None, None)
EC9 Intersection at camera Ray intersects road exactly at camera position camera_x=5, camera_y=2, road passes through (5,2) Returns (5, 2, 0) - distance is zero

6.3 Boundary Value Analysis

Model: Identify boundaries between equivalence classes

Test Requirements: Test values at, just above, and just below each boundary

Boundaries for find_intersection():

Parameter Boundary Test Values
angle_degrees 0° (horizontal), 90° (vertical down), 180°, 270° (vertical up) -0.1°, 0°, 0.1°, 89.9°, 90°, 90.1°, ...
camera_y Exactly on road (y = y_road[i]) y_road[i] - ε, y_road[i], y_road[i] + ε
camera_x Exactly at road start (x_road[0]) or end (x_road[-1]) x_road[0] - ε, x_road[0], x_road[0] + ε
x_road length Empty (0), single point (1), two points (2) len=0, len=1, len=2, len=3
Numerical threshold np.abs(np.cos(angle_rad)) < 1e-10 Values near 1e-10 (numerical stability)
Numerical threshold abs(diff2 - diff1) < 1e-10 Values near 1e-10 (parallel detection)

6.4 EC/BVA vs Branch Coverage

Key insight: Equivalence class coverage and branch coverage are complementary, not competing!

You need both!

Example: You could achieve 100% C1 without testing EC9 (intersection at camera position with distance=0). This might expose a bug in distance calculation that C1 alone wouldn’t find.

Conversely, you could test all ECs but miss the if abs(diff2 - diff1) < 1e-10 branch if your parallel ray test doesn’t trigger the exact numerical threshold.

7. Limitations of Coverage

Now for the uncomfortable truth: 100% coverage does NOT mean bug-free software.

7.1 Limitation 1: Correct Structure, Wrong Logic

Scenario: You achieve 100% C1 coverage, but there’s a bug in the formula.

Example:

# BUG: Wrong distance formula (should be sqrt, not just sum)
distance = (x_intersect - camera_x) + (y_intersect - camera_y)  # ❌ WRONG!

Test that passes despite the bug:

def test_finds_intersection():
    # ... test code that exercises this line ...
    assert dist is not None  # ✅ Passes (dist is *a* number, just the wrong one!)

What went wrong?

Lesson: Coverage tells you what you executed, not what you verified.

7.2 Limitation 2: Missing Test Assertions

Scenario: Your test executes code but doesn’t check the result.

Example:

def test_weak_assertion():
    # Arrange
    x_road = np.array([0, 10, 20], dtype=np.float64)
    y_road = np.array([0, 2, 4], dtype=np.float64)

    # Act
    x, y, dist = find_intersection(x_road, y_road, 10.0, 0.0, 10.0)

    # Assert: Weak assertion!
    assert x is not None  # ❌ Doesn't check if x is CORRECT!

Coverage report: 100% C1 ✅

Bug detection: 0% - this test would pass even if x is completely wrong!

Lesson: From Appendix A - weak assertions are a common anti-pattern in LLM-generated tests.

7.3 Limitation 3: Equivalence Classes Defined Wrong

Scenario: You achieve 100% EC coverage, but you missed an important equivalence class.

Example: Your EC partition includes:

But you forgot:

What happens?

Lesson: Coverage is only as good as your model. If your EC model is incomplete, 100% coverage is meaningless.

7.4 Limitation 4: Coverage Doesn’t Detect Mutation

Consider this code:

if diff1 * diff2 <= 0:  # Original
    # find intersection

A mutation (bug introduced by changing operator):

if diff1 * diff2 < 0:  # Mutated (changed <= to <)
    # find intersection

Impact: The case where diff1 * diff2 == 0 (ray exactly touches segment) is now missed.

Your tests:

def test_intersection_found():
    # ... test where diff1 * diff2 = -5.0 (clearly < 0)
    assert x is not None  # ✅ Passes

Coverage: 100% C1 ✅ (both True and False branches covered by other tests)

Bug detection: 0% - the test doesn’t specifically check the == 0 boundary!

Lesson: Coverage tells you which lines executed, not whether your tests would catch bugs. This is where mutation testing (future topic) helps.

8. Wait… What Are We Even Testing Against?

Let’s pause and reflect on what we’ve been doing in this course so far.

What you’ve learned:

What we’ve never properly addressed:

When you write a test like this:

def test_finds_intersection_for_normal_angle() -> None:
    """Test that find_intersection() returns a valid intersection..."""
    x, y, dist = find_intersection(x_road, y_road, angle=10.0, camera_x=0.0, camera_y=10.0)
    assert x is not None  # ← But... what makes this "correct"?
    assert dist > 0       # ← Says who?

The uncomfortable question: How do you know what the function should do?

8.1 The Implicit Requirements We’ve Been Using

Up until now, our “requirements” have been hidden in three places:

1. Function Signatures:

def find_intersection(
    x_road: NDArray[np.float64],
    y_road: NDArray[np.float64],
    angle_degrees: float,
    camera_x: float = 0,
    camera_y: float = 1.5,
) -> tuple[float | None, float | None, float | None]:

Implicit requirement: “Returns three values: x, y, distance. Any of them can be None.”

2. Docstrings:

"""
Find the intersection point between the camera ray and the road profile.

Returns:
    tuple of (float, float, float) or (None, None, None)
        x, y coordinates of intersection and distance from camera,
        or None if no intersection
"""

Implicit requirement: “Returns None if no intersection is found.”

3. Code Comments:

# Skip if this segment is behind the camera
if x2 <= camera_x:
    continue

Implicit requirement: “Segments behind the camera should be ignored.”

The problem: These implicit requirements are incomplete, ambiguous, and scattered.

You’ve been testing code without knowing what it’s supposed to do!

8.2 How Coverage Gaps Reveal Missing Requirements

Let’s see what happens when we try to improve coverage on find_intersection().

Starting Point: The Initial Test

def test_finds_intersection_for_normal_angle():
    # Simple case: downward ray, upward road
    x, y, dist = find_intersection(
        x_road=np.array([0, 10, 20, 30], dtype=np.float64),
        y_road=np.array([0, 2, 4, 6], dtype=np.float64),
        angle_degrees=10.0,
        camera_x=0.0,
        camera_y=10.0
    )
    assert x is not None

Coverage: 60% C1 (you check the coverage report)

Discovery 1: Uncovered Line 98

Coverage report shows line 98 is not covered:

if np.abs(np.cos(angle_rad)) < 1e-10:
    return None, None, None  # ← Line 98 not covered

Your question: “Wait, what condition triggers this? Let me read the code…”

(You analyze: cos(angle_rad) ≈ 0 happens when angle_rad ≈ 90° or 270° - vertical rays)

Your next question: “Okay, but what should happen for vertical rays? Is returning None correct behavior?”

The problem: There’s no specification that says what to do for vertical rays!

You could reasonably argue:

This coverage gap reveals a missing requirement.

Discovery 2: Uncovered Line 109

You write a test for vertical rays and check coverage again. Now line 109 is exposed:

if x2 <= camera_x:
    continue  # ← Line 109 not covered

Your question: “When would x2 <= camera_x? Oh, when road segments are behind the camera.”

Your next question: “Should segments behind the camera be ignored? Or should it raise an error?”

Again, there’s no specification!

Discovery 3: Uncovered Line 123

Coverage report shows another gap:

if abs(diff2 - diff1) < 1e-10:
    # Parallel lines
    t = 0  # ← Line 123 not covered

Your question: “When are the ray and road segment parallel? And why does it set t = 0? Is that correct?”

At this point, you realize:

“I can’t write good tests because I don’t actually know what this function is supposed to do in all these edge cases. I need a systematic specification of the behavior!”

8.3 From Implicit to Explicit: The Need for Requirements

What you’ve just experienced is the natural emergence of requirements engineering from testing practice.

The Testing → Requirements Spiral:

1. Write initial tests (based on implicit understanding)
      ↓
2. Measure coverage (C1, EC)
      ↓
3. Find uncovered code paths
      ↓
4. Ask: "What should happen here?"
      ↓
5. Realize: The specification is unclear/missing
      ↓
6. Write down explicit requirement
      ↓
7. Design test for that requirement
      ↓
8. Repeat from step 2

Example: Explicit Requirements Emerging from Coverage Analysis

After three iterations of coverage analysis, you’ve discovered these requirements need to be explicit:

Scenario Implicit (Before) Explicit (After)
Vertical rays (undefined - check code to guess) REQ-1: For angles where \(\lvert\cos(\theta)\rvert < 10^{-10}\), return (None, None, None)
Segments behind camera (code comment only) REQ-2: Only consider road segments where x > camera_x. Segments at or behind the camera shall be ignored.
Parallel ray/segment (unclear magic number) REQ-3: When ray and road segment are parallel (\(\lvert\Delta_{diff}\rvert < 10^{-10}\)), use first endpoint as intersection point.
No intersection found (docstring mentions it) REQ-4: If no road segment intersects the ray, return (None, None, None)
Multiple intersections (completely unspecified!) REQ-5: If multiple road segments intersect the ray, return the first intersection encountered (smallest distance from camera).

Notice what happened:

8.4 Tests as Executable Specifications

Here’s a powerful realization:

Your tests ARE requirements, just written in executable form.

Compare:

Traditional requirement document:

“REQ-1: The system shall return None for vertical camera rays.”

Executable specification (pytest test):

def test_vertical_ray_returns_none():
    """REQ-1: Vertical rays shall return None."""
    x, y, dist = find_intersection(
        x_road=np.array([0, 10], dtype=np.float64),
        y_road=np.array([0, 5], dtype=np.float64),
        angle_degrees=90.0,  # Vertical ray
        camera_x=0.0,
        camera_y=10.0
    )
    assert x is None
    assert y is None
    assert dist is None

Which one is better?

Traditional Requirement Test as Specification
Can be ambiguous Unambiguous (executes!)
Can become outdated Fails if code doesn't match
Passive (someone must read it) Active (runs on every commit)
Separate from code Lives with the code

This is the bridge between testing and requirements engineering:

8.5 Where Do We Go From Here?

You’ve now reached an important insight:

Testing and Requirements Engineering are two sides of the same coin.

What you’ve learned so far:

What comes next (future lectures):

For now, remember this:

When you see a coverage gap, don’t just add a test to increase the percentage. Ask what requirement the gap reveals. Write that requirement down explicitly. Then write the test.

The spiral continues:

Coverage Gap → Requirement Question → Explicit Specification → Test Design → Coverage Improvement
     ↑                                                                              ↓
     └──────────────────────────────────────────────────────────────────────────────┘

This iterative process is not a bug—it’s a feature. It’s how real software projects discover what they’re actually supposed to do.

9. Summary

Let’s consolidate what you’ve learned.

Concept Definition Key Insight Practical Takeaway
Program P The software artifact under test Testing is always about a specific program Be clear about scope (function? module? system?)
Input Domain D Set of all possible inputs to P Usually infinite - you can't test everything Need systematic strategies to sample D
Test Suite T Finite subset of D that you actually test T ⊆ D, and |T| << |D| Quality of T matters more than size
Model M(P) Abstraction of P's structure or behavior Coverage is always coverage of a model Different models → different test requirements
Coverage Criterion C Rule defining test requirements R_C(P) C defines what "adequate" testing means Choose C based on risk and cost
Statement Coverage (C0) Execute every statement at least once Weakest structural criterion Baseline - aim higher than C0
Branch Coverage (C1) Exercise every decision outcome (T/F) C1 subsumes C0 Good target for unit tests (70-80%+ C1)
Equivalence Classes Partition of input domain EC is a coverage criterion over input model Use ECs + C1 together (complementary)
Boundary Values Values at edges of equivalence classes Bugs cluster at boundaries Always test boundaries explicitly
Coverage Limitations 100% coverage ≠ bug-free Coverage measures execution, not verification Need strong assertions + good model
Requirements ↔ Tests Spiral Iterative refinement process Testing discovers requirements gaps Use coverage to guide requirement elicitation

10. Reflection Questions

Before you move on, spend a few minutes thinking about these questions. Write down your answers - this will deepen your understanding.

Question 1: Models and Coverage

What is the model behind statement coverage (C0)? What is the model behind equivalence class coverage?

How are these models different, and why does that matter for testing?

Question 2: Explaining Coverage to Stakeholders

Imagine a project manager says: “Great, we have 100% branch coverage! That means no bugs, right?”

How would you explain why this is not true? Give a specific example using geometry.py.

Question 3: Designing New Equivalence Classes

Look at the function calculate_ray_line() from geometry.py (not shown in detail here, but it calculates ray coordinates given an angle).

Design three equivalence classes for the angle_degrees parameter. For each class:

Question 4: Coverage-Driven Requirements

You run coverage analysis on find_intersection() and discover that the line t = 0 (in the parallel lines branch) is never executed.

What question should you ask about the requirements? What test would you write after clarifying the requirement?

Question 5: Beyond Unit Testing

This lecture focused on unit testing coverage. How would you apply the concepts of Program, Model, Test Suite, and Coverage Criterion to:

What would be different, and what would stay the same?

11. Further Reading

If you want to dive deeper into testing theory and coverage:

Books:

Papers:

Tools:

Standards:

Next Topics (Future Lectures or Projects):

12. Appendix: Solution Sketch for Exercises

Here’s one possible set of tests that achieve higher coverage for find_intersection(). These are not the only solutions - there are many ways to achieve coverage!

12.1 Exercise 1 Solution: Zero Loop Iterations

def test_empty_road_array_returns_none():
    """
    Test that find_intersection() returns None for an empty road array.

    Branch coverage: B5 False path (zero loop iterations)
    Equivalence class: EC8 (empty road)
    """
    # Arrange: Empty arrays
    x_road = np.array([], dtype=np.float64)
    y_road = np.array([], dtype=np.float64)

    # Act
    x, y, dist = find_intersection(x_road, y_road, 45.0, 0.0, 5.0)

    # Assert
    assert x is None
    assert y is None
    assert dist is None

12.2 Exercise 2 Solution: Ray Doesn’t Cross Segment

def test_ray_misses_road_entirely():
    """
    Test that find_intersection() returns None when ray misses all road segments.

    Branch coverage: B3 False path (multiple times), B6 True path
    Equivalence class: EC2 (no intersection)
    """
    # Arrange: Road below, ray pointing upward
    x_road = np.array([0, 10, 20], dtype=np.float64)
    y_road = np.array([0, 1, 2], dtype=np.float64)  # Road at low y values

    # Upward ray (negative angle = upward from horizontal)
    angle = -45.0
    camera_x = 5.0
    camera_y = 10.0  # Camera high above road

    # Act
    x, y, dist = find_intersection(x_road, y_road, angle, camera_x, camera_y)

    # Assert: Ray points upward, road is below - no intersection
    assert x is None
    assert y is None
    assert dist is None

12.3 Exercise 3 Solution: Intersection on Second Segment

def test_intersection_on_second_segment():
    """
    Test that find_intersection() finds intersection on the second road segment,
    not the first (first segment should miss).

    Branch coverage: B3 False (first segment), B3 True (second segment)
    Equivalence class: EC5 (multiple segments, selective intersection)
    """
    # Arrange: Two segments, ray only hits the second one
    # First segment: (0,0) to (10,0) - horizontal at y=0
    # Second segment: (10,0) to (20,10) - upward slope
    x_road = np.array([0, 10, 20], dtype=np.float64)
    y_road = np.array([0, 0, 10], dtype=np.float64)

    # Ray from camera at (15, 20) pointing downward at 45°
    angle = 45.0
    camera_x = 15.0
    camera_y = 20.0

    # Act
    x, y, dist = find_intersection(x_road, y_road, angle, camera_x, camera_y)

    # Assert: Should find intersection on second segment
    assert x is not None
    assert y is not None
    assert 10 < x < 20, "Intersection should be on second segment (x between 10 and 20)"
    assert 0 < y < 10, "Intersection should be on second segment (y between 0 and 10)"
    assert dist > 0

12.4 Exercise 4 Solution: Intersection at Road Vertex

def test_intersection_at_road_vertex():
    """
    Test intersection when ray passes exactly through a road vertex.

    Branch coverage: B3 True (with diff1 or diff2 exactly 0)
    Equivalence class: EC6 (intersection at vertex)
    Boundary case: diff1 * diff2 = 0 (ray exactly at vertex)
    """
    # Arrange: Road with vertex at (10, 5)
    x_road = np.array([0, 10, 20], dtype=np.float64)
    y_road = np.array([0, 5, 10], dtype=np.float64)

    # Calculate angle so ray passes through (10, 5)
    # Camera at (0, 10), target (10, 5)
    # Slope = (5 - 10) / (10 - 0) = -5/10 = -0.5
    # angle = arctan(-0.5) ≈ -26.57°
    # But remember: angle is measured downward, and we negate it in the code
    # So we want angle such that tan(-angle_rad) = -0.5
    # This means angle ≈ 26.57° (downward)

    camera_x = 0.0
    camera_y = 10.0
    angle = np.rad2deg(np.arctan(0.5))  # ≈ 26.57°

    # Act
    x, y, dist = find_intersection(x_road, y_road, angle, camera_x, camera_y)

    # Assert: Should find intersection at vertex (10, 5)
    assert x is not None
    assert y is not None
    # Use pytest.approx for floating-point comparison
    from pytest import approx
    assert x == approx(10.0, abs=0.01), "Should intersect at x=10"
    assert y == approx(5.0, abs=0.01), "Should intersect at y=5"
    assert dist == approx(np.sqrt(100 + 25), abs=0.01), "Distance should be sqrt(125)"

Note: These are example solutions. You might design different tests that achieve the same coverage. The key is to:

  1. Understand which branch/EC you’re targeting
  2. Design input that triggers that specific behavior
  3. Write strong assertions that verify correctness (not just “is not None”)
© 2026 Dominik Mueller   •  Powered by Soopr   •  Theme  Moonwalk