25. Keeping It Working: Continuous Integration and the Tests the Early Chapters Never Had¶

Mountain goats in space helmets on the Moon

What you will understand: what "continuous integration" is, how to build a small one for this book (a script plus a workflow file), how to check that it fails when it should, and why the honest description of a CI setup says what has actually run it and what has not. Along the way this chapter fills a gap listed since Chapter 15: Chapters 1, 2 and 9 had never had automated tests.

What you need to know first: Chapter 15 (the lit test suite and what FileCheck does), Chapters 21 to 24 (the suite has grown to 99 tests and the compiler to its newest build). Nothing about GitHub or CI is assumed.

Compile and run

cd <repository root>
pip install -r requirements.txt           # lit, mkdocs: the Python tools (see below for the system packages)
./ci.sh                                   # everything CI does: tools, workflow check, build, tests, docs (about 80 s with a warm build)
docs/part25/code/ci_failure_demo.sh       # run CI against an old compiler on purpose: it must FAIL
docs/part25/code/chapter_tests_mutation.sh   # damage the early chapters' files on purpose: the new tests must notice
cd docs/part15/code && ./run_lit.sh       # just the test suite (102 tests)

The system packages are clang-18 llvm-18-dev llvm-18-tools libmlir-18-dev mlir-18-tools cmake make. Every listing and output on this page comes from these commands.

Why this page shows FAIL lines, and why that is good

Two outputs here come from negative controls: CI run against a deliberately old compiler, and the new tests run against deliberately damaged example files. There a FAIL, "CI FAILED" or "caught by" is the expected, wanted result. The normal CI run and the baseline lines must pass, and they do.

Primer: what continuous integration is¶

A test suite only protects a project if somebody runs it. Continuous integration (CI) means a machine runs it for you, automatically, every time the code changes: on every push to the main branch, and on every proposed change (a pull request) before anyone merges it. If anything fails, the change is marked red where everyone can see it, and the author hears about it within minutes instead of a reader finding it weeks later.

CI is a small idea with two parts. The script does the work: build the thing, run the tests, report pass or fail through its exit status (0 for success, anything else for failure; that number is the only thing the machinery looks at). The workflow file tells a service (here GitHub Actions) when to run the script and on what kind of computer: a fresh, empty virtual machine, which is the point. A fresh machine has none of your leftover files and none of the packages you installed years ago and forgot, so a successful run there proves the project builds from nothing.

For this book that matters more than usual. The tests depend on LLVM and MLIR version 18, on cmake, on lit, on a particular FileCheck, and on files built by earlier chapters. Every one of those is something that can be installed on the author's machine and missing, or the wrong version, everywhere else.

The script: `ci.sh`¶

The script is the single source of truth. Everything CI does is in it, and you can run exactly the same thing on your own machine:

#!/bin/sh
# ci.sh: everything continuous integration does, in one script you can run on your own machine.
#   1. check the tools are installed          4. run the whole test suite (docs/part15/code/run_lit.sh)
#   2. check the workflow file                5. build the documentation site (mkdocs build --strict)
#   3. build the newest compiler (docs/part44/code/build.sh)
# Exit status 0 means every step passed; 1 means at least one failed (the summary at the end says which).
# Knobs: CI_SKIP_BUILD=1 reuses an existing build; CI_SKIP_DOCS=1 skips the site build; MG_OPT=/path/to/mg-opt tests a different build.
ROOT=$(cd "$(dirname "$0")" && pwd); cd "$ROOT"
FAILED=""; SUMMARY=""
step() {  # step "name" command...   (runs the command, records pass/fail and the time it took)
  name=$1; shift
  echo "::group::$name"; echo "=== $name"
  start=$(date +%s)
  if "$@"; then result=PASS; else result=FAIL; FAILED="$FAILED|$name"; fi
  secs=$(( $(date +%s) - start ))
  echo "::endgroup::"; echo "=== $name: $result (${secs}s)"
  SUMMARY="$SUMMARY$(printf '  %-4s %-40s %4ss' "$result" "$name" "$secs")
"
}
skipped() { SUMMARY="$SUMMARY$(printf '  SKIP %-40s' "$1")
"; }

check_tools() {
  missing=""
  for t in clang-18 clang++-18 mlir-opt-18 mlir-translate-18 mlir-cpu-runner-18 FileCheck-18 not-18 valgrind callgrind_annotate cmake make python3 lit mkdocs; do
    command -v $t >/dev/null 2>&1 || missing="$missing $t"
  done
  [ -z "$missing" ] || { echo "missing tools:$missing"; return 1; }
  echo "all tools found: $(clang-18 --version | head -1)"
}
build_compiler() {
  if [ -n "$CI_SKIP_BUILD" ] && [ -x docs/part44/code/build/mg-opt ]; then echo "CI_SKIP_BUILD set: reusing docs/part44/code/build/mg-opt"; return 0; fi
  docs/part44/code/build.sh && [ -x docs/part44/code/build/mg-opt ]
}
run_suite() { docs/part15/code/run_lit.sh; }
build_docs() { mkdocs build --strict; }

step "check tools" check_tools
step "check workflow file" python3 .github/check_ci.py
if [ -n "$MG_OPT" ] && [ -n "$CI_SKIP_BUILD" ]; then skipped "build the compiler (testing MG_OPT=$MG_OPT)"
else step "build the compiler" build_compiler; fi
case "$FAILED" in *"build the compiler"*|*"check tools"*) skipped "run the test suite (a step it depends on failed)";; *) step "run the test suite" run_suite;; esac
if [ -n "$CI_SKIP_DOCS" ]; then skipped "build the documentation"; else step "build the documentation" build_docs; fi

echo; echo "CI summary:"; printf '%s' "$SUMMARY"
if [ -n "$FAILED" ]; then echo "CI FAILED in: $(echo "$FAILED" | sed 's/^|//; s/|/, /g')"; exit 1; fi
echo "CI passed."

Read it as five steps. It checks that the tools exist (so a missing package produces a clear message, not a mysterious failure inside a test), checks the workflow file (next section), builds the newest compiler (docs/part24/code/build.sh), runs the whole suite (run_lit.sh), and builds the documentation with mkdocs build --strict, which fails on any broken link or missing embedded file. Each step records pass or fail and its time; at the end the script prints a summary and exits 1 if anything failed. A failed build skips the test step instead of running tests against a compiler that does not exist.

A real run, from a warm build directory:

CI summary:
  PASS check tools                                 0s
  PASS check workflow file                         0s
  PASS build the compiler                         67s
  PASS run the test suite                          7s
  PASS build the documentation                     5s
CI passed.

(The summary is the last few lines of the run's output; above it, in the real output, are the build log, the 102 test results and the documentation build.) On a fresh runner the compiler build takes several minutes, not the one minute shown here; the workflow caches it (below).

Knobs for local use: CI_SKIP_BUILD=1 reuses an existing build, CI_SKIP_DOCS=1 skips the site build, and MG_OPT=/path/to/mg-opt tests a different compiler build.

The workflow file¶

# Continuous integration for the Mountain Goat book: build the compiler, run the whole test suite, and build the site.
# The work is done by ./ci.sh, which you can run on your own machine; this file only prepares an Ubuntu 24.04 runner for it.
# History: this file was written in an environment that could not run GitHub Actions, then pushed. Its first real run (commit d8b1c2e)
# passed on a fresh runner: compiler built in 46 s, 102 tests passed in 24 s, docs built in 4 s. Chapter 25 records what that shows
# and what it does not (for example, the cache was saved by that run but a cache RESTORE had not yet been observed).
name: ci

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  suite:
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4

      - name: Install LLVM/MLIR 18 and build tools
        run: |
          sudo apt-get update
          sudo apt-get install -y clang-18 llvm-18-dev llvm-18-tools libmlir-18-dev mlir-18-tools cmake make valgrind

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: requirements.txt

      - name: Install Python tools (lit, mkdocs)
        run: pip install -r requirements.txt

      # The compiler takes a few minutes to build; reuse it when no source file that goes into it has changed.
      - name: Cache the compiler build
        id: cache
        uses: actions/cache@v4
        with:
          path: docs/part44/code/build
          key: mg-opt-${{ runner.os }}-${{ hashFiles('docs/part44/code/*.cpp', 'docs/part44/code/CMakeLists.txt', 'docs/part42/code/*.cpp', 'docs/part38/code/*.cpp', 'docs/part32/code/*.cpp', 'docs/part32/code/CMakeLists.txt', 'docs/part32/code/MgOps.td', 'docs/part3/code/*', 'docs/part14/code/*', 'docs/part19/code/AssertToStderr.cpp') }}

      # An exact cache hit means no source that goes into the compiler changed, so the restored build is current and ci.sh skips building.
      # (Restoring the cache alone is not enough: build.sh recopies the sources, which makes `make` think everything changed and rebuild it.)
      - name: Build, test, and build the docs
        run: ./ci.sh
        env:
          CI_SKIP_BUILD: ${{ steps.cache.outputs.cache-hit == 'true' && '1' || '' }}

What each part does. on: says when: on every push to main, on every pull request, and on demand (workflow_dispatch adds a "Run workflow" button). concurrency cancels an older run of the same branch when a newer push arrives, so a burst of pushes does not queue up. runs-on: ubuntu-24.04 is the fresh machine; Ubuntu 24.04 is the platform this book has used throughout, and it provides LLVM and MLIR 18 as packages. The install step gets exactly the system packages the book has needed. setup-python provides a Python without the "externally managed" restrictions of the system one, and pip install -r requirements.txt installs the pinned versions of lit and mkdocs-material (the same versions the book was written with). The cache step saves the compiler build between runs, keyed on a hash of every source file that goes into it: when none changed, the several-minute build becomes a no-op, and when any changed the key changes and a fresh build runs. Finally the one step that matters: ./ci.sh.

.github/check_ci.py is the check from step 2 of the script. It parses the workflow, confirms it triggers on pushes to main and on pull requests, confirms the packages above are installed, and confirms that every file and path the workflow mentions exists. It catches the mistakes you can catch without GitHub (a typo in a path, a deleted script):

#!/usr/bin/env python3
"""Sanity-check .github/workflows/ci.yml without GitHub: it must parse, trigger on push to main and on pull requests, and every
file and script it mentions must exist in the repository. (A parse is not a run: see Chapter 25 for what this does not establish.)"""
import os, re, sys, yaml
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
path = os.path.join(root, ".github", "workflows", "ci.yml")
text = open(path).read()
doc = yaml.safe_load(text)
problems = []
triggers = doc.get(True) or doc.get("on")                      # YAML 1.1 reads the key `on` as the boolean True
if "pull_request" not in triggers: problems.append("workflow does not run on pull requests")
if "main" not in (triggers.get("push") or {}).get("branches", []): problems.append("workflow does not run on pushes to main")
job = doc["jobs"]["suite"]
if not job["runs-on"].startswith("ubuntu-24"): problems.append("expected an ubuntu-24.04 runner (LLVM/MLIR 18 packages)")
steps = job["steps"]
runs = "\n".join(s.get("run", "") for s in steps)
if "./ci.sh" not in runs: problems.append("no step runs ./ci.sh")
for pkg in ("clang-18", "llvm-18-dev", "llvm-18-tools", "libmlir-18-dev", "mlir-18-tools", "cmake"):
    if pkg not in runs: problems.append(f"package {pkg} is not installed by the workflow")
# every path under docs/ that the workflow mentions (cache path and hash globs) must have a real directory behind it
for m in set(re.findall(r"docs/[A-Za-z0-9_./-]+", text)):
    base = m.split("*")[0].rstrip("/")
    if not os.path.exists(os.path.join(root, base)) and not os.path.exists(os.path.join(root, os.path.dirname(base))):
        problems.append(f"workflow mentions {m}, which does not exist")
for f in ("ci.sh", "requirements.txt", "docs/part44/code/build.sh", "docs/part15/code/run_lit.sh"):
    if not os.path.exists(os.path.join(root, f)): problems.append(f"missing file the workflow depends on: {f}")
if problems:
    print("CI workflow check FAILED:"); [print("  -", p) for p in problems]; sys.exit(1)
print(f"CI workflow check passed: {len(steps)} steps, triggers {sorted(k for k in triggers)}, runner {job['runs-on']}")

What was established, and what was not¶

This section was first written before the workflow had ever run, because the environment this book is written in can push code but could not start or watch GitHub Actions. It said so, listed what was and was not established, and told the reader to look at the Actions tab. The push of this chapter then triggered the workflow's first real run, and it can be checked, so here is the record, in two parts.

Before the first run, established: ci.sh passes from start to finish on the author's machine (shown above); check_ci.py finds nothing wrong with the workflow; and ci.sh fails with exit status 1 when the tests fail (the demonstration below). Not established: that GitHub accepts the file, that the package names resolve on a fresh Ubuntu 24.04 runner, that the cached build is restored, and that the run finishes within its time limit.

The first real run (the ci workflow, run number 1, on the push that carried this chapter, commit d8b1c2e, read back through the GitHub API) passed. From its job log:

=== check tools ... PASS (3 s)        === check workflow file ... PASS (0 s)
=== build the compiler ... PASS (46 s)    (from nothing: the cache was empty)
=== run the test suite ... PASS (24 s)    Total Discovered Tests: 102   Passed: 102 (100.00%)
=== build the documentation ... PASS (4 s)
CI passed.      whole job: about two minutes

That settles three of the four open items: GitHub accepted the file (a parse is not validation, but the run exists with every step listed), the package names resolve on a fresh runner (the install step took 25 seconds and everything after it found its tools), and the run is far inside the 45-minute limit. It also shows something the local run could not: the compiler builds from nothing in 46 seconds on a GitHub runner, much faster than the several minutes I had guessed above. The test suite took 24 seconds there (the Chapter 9 test, which builds a C++ program against MLIR, finished last).

Runs 2 and 3: the cache, and a bug in my own workflow. The push that recorded the paragraph above only touched documentation, so the compiler sources were unchanged and the cache key matched. Run 2 passed, and its log says Cache hit occurred on the primary key mg-opt-Linux-…, not saving cache: the cache was restored. But the summary also said PASS build the compiler 35s. A restored cache that still takes 35 seconds to "build" is a cache doing nothing. The reason is in build.sh: it copies the source files into a tree/ directory on every run, so every file has a fresh modification time, newer than the restored object files, and make correctly decides everything is out of date and rebuilds it. (The 35 seconds against the first run's 46 is runner variance, not a saving.)

The fix is to use the information the cache step already gives: an exact hit on a key that is a hash of all the compiler's sources means the restored build is current. The step now has an id, and the ./ci.sh step sets CI_SKIP_BUILD=1 when steps.cache.outputs.cache-hit is true (the script already knew how to reuse an existing build). Run 3, on the commit that made that change, shows it working in its log:

PASS check tools            1s       PASS run the test suite     24s
PASS check workflow file    1s       PASS build the documentation 4s
PASS build the compiler     0s       CI passed.      (whole job: 66 s, against 95 s for run 2)
Cache hit occurred on the primary key mg-opt-Linux-…, not saving cache.

So the cache restore is established, and so is the fix. The lesson is a general one about caching: a cache can hit and still save nothing, and the only way to know is to look at what the cached step cost afterwards. This one would have gone on looking healthy ("Cache restored") while saving nothing.

Still not established:

Stability over time. Three runs, all passing, are evidence that the setup works, not that it will keep working: a new runner image, a changed package version or a flaky test would show up only over many runs.
A failing run on GitHub. Every GitHub run so far passed. That ci.sh exits 1 and so would turn a run red is shown locally (below), but a red run has not been seen on GitHub itself.
The warning the runs print. GitHub reports that actions/checkout@v4, actions/cache@v4 and actions/setup-python@v5 target Node.js 20, which is deprecated, and is forcing them onto Node.js 24. They work; the versions will need bumping eventually, and the workflow is not changed for that here.

CI must be able to fail¶

A CI script that always says "passed" is worse than none. So, as in Chapter 15, there is a negative control: run ci.sh against the old Chapter 14 compiler, which lacks most of what the tests check.

#!/bin/sh
# READ THIS FIRST: this script runs ci.sh against an OLDER compiler build (Chapter 14's) on purpose.
#   The failures and the "CI FAILED" line in its output are the EXPECTED result: they show that CI really does fail (exit status 1)
#   when the tests fail, instead of reporting success no matter what. The normal run (ci_run_out.txt) must pass.
# Needs docs/part14/code/build.sh to have been run. Output: ci_failure_demo_out.txt
echo "NOTE: this script deliberately tests an old compiler build. FAIL lines and 'CI FAILED' below are EXPECTED: they show CI can fail."
ROOT=$(cd "$(dirname "$0")/../../.." && pwd)
cd "$ROOT"
CI_SKIP_BUILD=1 CI_SKIP_DOCS=1 MG_OPT=$ROOT/docs/part14/code/build/mg-opt ./ci.sh > /tmp/ci_demo.$$ 2>&1
status=$?
echo "--- a few of the failing tests (the old build lacks features the current tests check):"
grep -E "^FAIL:" /tmp/ci_demo.$$ | sed 's/IR Forge \/ Mountain Goat :: //; s/ ([0-9]* of [0-9]*)//' | sort -u | head -8
echo "--- the suite's own totals:"
grep -E "^  (Passed|Failed)" /tmp/ci_demo.$$
echo "--- ci.sh's summary:"
sed -n '/^CI summary:/,$p' /tmp/ci_demo.$$
echo "exit status of ci.sh: $status   (0 = CI passes, 1 = CI fails; here 1 is the expected result)"
rm -f /tmp/ci_demo.$$

NOTE: this script deliberately tests an old compiler build. FAIL lines and 'CI FAILED' below are EXPECTED: they show CI can fail.
--- a few of the failing tests (the old build lacks features the current tests check):
FAIL: assert-stderr/guard-nested-assert.mlir
FAIL: assert-stderr/message-bytes-exact.mlir
FAIL: assert-stderr/message-on-stderr.mlir
FAIL: assert-stderr/nested-assert-runs.mlir
FAIL: assert-stderr/rewrite-structure.mlir
FAIL: assert-stderr/run-twice-no-collision.mlir
FAIL: assert-stderr/static-unchanged.mlir
FAIL: assert-stderr/valid-shapes-still-pass.mlir
--- the suite's own totals:
  Passed: 53 (51.96%)
  Failed: 49 (48.04%)
--- ci.sh's summary:
CI summary:
  PASS check tools                                 0s
  PASS check workflow file                         0s
  SKIP build the compiler (testing MG_OPT=/home/user/ir_forge_compiler_from_scratch/docs/part14/code/build/mg-opt)
  FAIL run the test suite                          2s
  SKIP build the documentation                 
CI FAILED in: run the test suite
exit status of ci.sh: 1   (0 = CI passes, 1 = CI fails; here 1 is the expected result)

The suite fails, the summary shows FAIL run the test suite, and ci.sh exits with status 1, which is exactly what a CI service reads as "red". The step that tests the compiler build is skipped here on purpose (the old build is supplied from outside), and the documentation step is skipped to keep the demonstration short.

The tests the early chapters never had¶

The suite had no test for Chapters 1, 2 or 9. Chapter 1's page even said so ("the scf.for demonstration here is untested beyond the single recorded run shown above"). Three tests in test/chapters/ now run those chapters' own files:

Test	What it runs
`ch01-pipeline`	Chapter 1's `sum.mlir`: parse and verify it, lower it with the four passes, check that the structured loop is gone and the single memref argument became five scalars, translate to LLVM IR, compile with the chapter's `harness.c`, run, and check the output `sum_array({1,2,3,4,5}) = 15`
`ch02-example-programs`	Chapter 2's three example programs with the current compiler: the valid one still parses and prints its operations; the transpose with the wrong shape and the add with mismatched ranks are rejected with the exact messages. (Chapter 2's own compiler accepted the mismatched add; Chapter 3 added the check, so rejecting it now is later, correct behavior.)
`ch09-execution-engine`	Chapter 9's C++ program, built from the chapter's own `CMakeLists.txt` in a temporary directory and run on Chapter 8's lowered IR through MLIR's JIT, checking the printed matrix `6 8 / 10 12`. This one takes the longest (about a minute on a fresh build) because it builds a C++ program against MLIR

These tests read the chapters' files in place (through a %docs substitution), so a chapter's page, its files and its test cannot drift apart. They also mean that damage to those files now has a consequence, which chapter_tests_mutation.sh checks by making each file wrong, one at a time (Chapter 1's loop subtracts instead of adds; Chapter 2's valid program loses its transpose; Chapter 2's "bad" transpose is made valid; Chapter 9's lowered add subtracts):

#!/bin/sh
# READ THIS FIRST: this script edits the early chapters' own example files to be WRONG, one at a time, and re-runs the new
#   chapter tests (test/chapters). A "caught by" line (a test failed) is the EXPECTED, wanted result: it shows the test can notice that
#   mistake. "NOT CAUGHT" would be a test gap. Every file is restored afterwards. Output: chapter_tests_mutation_out.txt
HERE=$(cd "$(dirname "$0")" && pwd); D=$HERE/../..; LIT=$D/part15/code/run_lit.sh; W=$HERE/work/mut; mkdir -p $W
echo "NOTE: this script deliberately damages example files. 'caught by' lines are EXPECTED: they show the tests can detect the damage."
run() {  # label baseline?
  out=$(MG_TEST_TMP=$W/out sh $LIT --filter ':: chapters/' 2>&1); fails=$(echo "$out" | grep '^FAIL' | sed 's/.*:: //; s/ (.*//' | tr '\n' ' ')
  if [ "$2" = baseline ]; then [ -z "$fails" ] && echo "$1: all tests pass (expected)" || echo "$1: UNEXPECTED FAILURES: $fails"
  else [ -n "$fails" ] && echo "$1: caught by $fails" || echo "$1: NOT CAUGHT (a test gap)"; fi
}
mutate() {  # label file sed-expr
  cp "$2" $W/backup; sed -i "$3" "$2"
  if cmp -s "$2" $W/backup; then echo "$1: MUTATION DID NOT APPLY"; else run "$1"; fi
  cp $W/backup "$2"
}
run "baseline (nothing damaged)" baseline
mutate "Chapter 1: the loop subtracts instead of adds (answer is no longer 15)" $D/part1/code/sum.mlir 's/arith.addi %sum, %val/arith.subi %sum, %val/'
mutate "Chapter 2: the valid program loses its transpose" $D/part2/code/mountain_goat.mlir '/mg.transpose/d;s/mg.print %3/mg.print %2/'
mutate "Chapter 2: the 'bad' transpose is made valid (so it is no longer rejected)" $D/part2/code/bad_transpose.mlir 's/to tensor<2x3xf64>/to tensor<3x2xf64>/'
mutate "Chapter 9: the lowered add subtracts instead (JIT output changes)" $D/part9/code/add_tensors_llvm.mlir 's/llvm.fadd/llvm.fsub/'
run "restored (baseline again)" baseline

NOTE: this script deliberately damages example files. 'caught by' lines are EXPECTED: they show the tests can detect the damage.
baseline (nothing damaged): all tests pass (expected)
Chapter 1: the loop subtracts instead of adds (answer is no longer 15): caught by chapters/ch01-pipeline.mlir 
Chapter 2: the valid program loses its transpose: caught by chapters/ch02-example-programs.mlir 
Chapter 2: the 'bad' transpose is made valid (so it is no longer rejected): caught by chapters/ch02-example-programs.mlir 
Chapter 9: the lowered add subtracts instead (JIT output changes): caught by chapters/ch09-execution-engine.mlir 
restored (baseline again): all tests pass (expected)

Every damage is caught by the test for that chapter, and nothing else fails. The full suite, 102 tests, with the newest compiler:

Testing Time: 5.93s

Total Discovered Tests: 102
  Passed: 102 (100.00%)

Limits and what is not established¶

Three real runs so far (above), all passed; no failing run has been seen on GitHub, and stability over many runs is not yet observed.
The documentation step only checks that the site builds. It does not check that the pages say true things; the tests and the "FAIL is expected" notes exist for that.
Only the newest compiler build is built and tested. The older chapters' own builds (Chapters 2 to 7, 13, 14, 19, 20, 21, 22) are not rebuilt by CI; their build.sh scripts were run once when written. A change that broke only an old chapter's build would not be noticed.
The negative controls and mutation scripts are not run by CI (they take minutes and several rebuild the compiler). They are evidence about the tests, run by hand and recorded.
No performance checks. CI deliberately does not run the benchmarks (Chapters 23 and 24); timings on shared runners are too noisy to gate on.
No GPU or hardware-specific testing. The GPU path is compile-only everywhere, including here.
One platform: Ubuntu 24.04 with LLVM 18. Nothing here says the book builds on macOS, Windows, or other LLVM versions.

Reproducing¶

pip install -r requirements.txt
./ci.sh
docs/part25/code/ci_failure_demo.sh > docs/part25/code/ci_failure_demo_out.txt
docs/part25/code/chapter_tests_mutation.sh > docs/part25/code/chapter_tests_mutation_out.txt
cd docs/part15/code && ./run_lit.sh        # 102 tests

Chapter summary¶

Continuous integration is a script (the work; its exit status is the verdict) plus a workflow file (when and where to run it). ci.sh and .github/workflows/ci.yml implement both for this book.
ci.sh checks the tools, checks the workflow, builds the newest compiler, runs the 102-test suite and builds the docs with --strict; it exits 1 on any failure, and a failed build skips the tests that depend on it.
It was shown able to fail: against an old compiler, 49 tests fail and ci.sh exits 1.
Three new tests give Chapters 1, 2 and 9 automated coverage of their own files; four deliberate damages to those files are all caught.
The chapter first said the workflow had not been run by GitHub rather than claiming CI works; its first real run then passed (102 tests, compiler built from nothing in 46 s), and a second and third run showed the cache restoring but initially saving nothing (fixed). A failing run on GitHub and long-run stability are still to be observed.

Self-check questions¶

Each answer is collapsed; try the question first.

Why is the CI work in a script rather than written directly in the workflow file?

Answer

A script can be run on your own machine, so a failure in CI can be reproduced and fixed without pushing and waiting. The workflow file is then only about when and where (triggers, a fresh machine, installing packages), and it does not tie the project to one CI service: another service could call the same script.
What does "exit status" have to do with CI, and why does it matter that ci.sh exits 1 on failure?

Answer

A CI service does not read your test output; it looks at the exit status of the step it ran. Zero means success and anything else means failure. If ci.sh printed failures but still exited 0, CI would show green over failing tests. The failure demonstration exists to show it exits 1 when tests fail.
Why does CI run on a fresh machine, and what kind of problem does that expose?

Answer

A fresh machine has none of your leftover files, none of the packages installed long ago, and none of the environment variables you set, so it shows whether the project builds from nothing. It exposes dependencies you forgot you had: a package only installed on your machine, a file only present in your working directory, an older tool version.
Before the workflow's first real run, what had been established about it, and what changed afterwards?

Answer

Before: the script it calls passed locally, a parse of the workflow found no inconsistency (triggers, packages, paths), and the script failed when tests failed. Not established: that GitHub accepted the file, that the packages installed on a fresh runner, that the cache restored, and that the run finished in time. The environment could not run GitHub Actions. The first real run then passed (102 tests, compiler built from nothing in 46 s), settling acceptance, packages and time; only the cache restore and long-run stability remained open.
Why is the compiler build cached, and what decides when the cache is used?

Answer

Building the compiler takes several minutes on a fresh machine, so rebuilding it on every push wastes time. The cache stores the build directory under a key that is a hash of every source file that goes into the compiler; if none changed, the key matches and the build is restored, and if any changed the key differs, so a fresh build runs and a new cache is saved. A hit alone does not skip the build, though: in this repository build.sh recopies the sources, which makes make rebuild everything, so the workflow also tells ci.sh to skip building on an exact hit. The first version of the workflow had the cache and still took 35 seconds to build; only reading the timing in the log showed it.
Why does ci.sh skip the test step when the build step fails?

Answer

Running the tests against a compiler that was not built would produce dozens of unrelated-looking failures that hide the real cause. Skipping the dependent step keeps the failure report short and points at the first thing that broke.
Chapter 2's compiler accepted the mismatched add, but the new Chapter 2 test expects it to be rejected. Is the test wrong?

Answer

No. The test runs Chapter 2's example files with the current compiler, not with Chapter 2's. Chapter 3 added the shape check, so rejecting that program is the later, correct behavior, and a test expecting acceptance would assert a bug. The test says so in its comment.
Why are the mutation scripts and benchmarks not run by CI?

Answer

The mutation scripts take minutes and several rebuild the compiler for each injected bug; they are evidence about the quality of the tests, not checks that must pass on every push. The benchmarks measure speed, which on shared runners is too noisy to gate on: a test that fails when a neighbor is busy teaches people to ignore red. They are run by hand and their results recorded.
What kind of mistake would the check_ci.py script catch, and what kind would it miss?

Answer

It catches mistakes you can see without GitHub: invalid YAML, a missing trigger, a deleted script, a path in the cache key that does not exist, a forgotten package. It misses anything only GitHub knows: whether an action version exists, whether a package name resolves on the runner image, whether the syntax is valid by GitHub's own rules. A successful parse is not validation.
The chapter was written and committed before the workflow's first run, and said so. Why not wait and write it afterwards, and what would have been the right response had the run failed?

Answer

Waiting would have hidden the real order of events and tempted the text to claim more than had been known at the time; stating what was unknown, then recording what the run showed, keeps the two apart and makes the unknowns checkable. Had the run failed, the failure would have been a finding: read the log, fix the workflow or the script, note which of the "not established" items it turned out to be, and rerun. A run that passes does not close every question (cache restore and stability remain), which is why the section still lists what is open.