cc-libs/docs/adrs/adr-0009-layered-test-timeouts.md

ADR 0009: Layered Test Timeouts

Status

Accepted

Date

2026-06-08

Context

ADR 0005 made CraftOS-PC the local harness and ADR 0007 split libtest (cases and assertions) from runtest (suite orchestration). The only timeout in that harness was the shell watchdog in the Justfile test: recipe: it kill -TERMs the whole CraftOS-PC process after TRAP_CCLIBS_TEST_TIMEOUT_SECONDS.

That single layer is coarse. It kills the entire process, so it cannot say which case hung, it produces one generic message, and it cannot tell a cooperatively-blocked event loop (the common failure: waiting on an event or sleep that never resolves) apart from a genuinely wedged process. A per-case timeout inside Lua is both finer and faster, but the shell watchdog is still needed for the cases Lua cannot interrupt.

Decision

Run two independent timeout layers, ordered so the finer one fires first.

Layer 1 — libtest per-case timeout (primary). /apis/libtest.lua races each test case against a timer with parallel.waitForAny(runner, timer). The default is DEFAULT_TIMEOUT_SECONDS = 3. When the timer wins, the case fails with a distinct message containing the token libtest timeout and, in --verbose, an extra TIMEOUT … (libtest) diagnostic. --timeout <seconds> overrides the default; --no-timeout disables the layer. /programs/runtest.lua forwards both flags to each case script. This only interrupts cases that yield (the usual hang); a non-yielding CPU loop cannot be preempted in ComputerCraft.

Layer 2 — shell watchdog (backstop). The Justfile test: recipe keeps its existing TRAP_CCLIBS_TEST_TIMEOUT_SECONDS watchdog unchanged, as an independent double-check. Its default matches the libtest default (.env.test ships 3; the recipe falls back to 3) so libtest should fire first for yielding cases in normal runs and the watchdog only catches what Lua cannot — a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its SIGTERM message is worded differently from the libtest timeout message, so the two layers are never confused.

How To Write Tests Properly

• Normal tests live in tests/*.lua, use /apis/libtest.lua, and must finish under the libtest timeout. runtest auto-discovers them; just test runs the suite.
• Never commit a hanging or intentionally-slow test to tests/: it would fail every run.
• Intentionally-slow fixtures that exercise the harness itself live in tests/harness/. runtest discovery skips subdirectories, so they never run with the normal suite; they are driven only by dedicated recipes (just test-timeout-lua, just test-timeout-shell, aggregated by just test-timeout).
• Use --no-timeout only for harness fixtures that must outlive the libtest layer to prove the shell watchdog, never for ordinary tests.

Consequences

• A hung case now fails in ~3s with a per-case message instead of taking down the whole process anonymously.
• just test-timeout is a self-asserting harness regression guard wired into just ci. It chains test-timeout-lua (Layer 1: libtest cancels the slow case immediately with --timeout 0, before the shell backstop) and test-timeout-shell (Layer 2: the TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS watchdog, default 1, kills the slow case with libtest bypassed). Both drive a single tests/harness/slow-case.lua fixture; the tight timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself is covered against regressions on every ci.
• libtest stays a normal ComputerCraft program: parallel and sleep are sandbox globals, so the timeout works in CraftOS-PC and in-game alike.

Future Work

• Per-case timing in --verbose output if slow-but-passing cases become hard to spot.
• A libtest-level marker for "expected timeout" if more harness fixtures appear.