# ADR 0009: Layered Test Timeouts

## Status

Accepted

## Date

2026-06-08

## Context

ADR 0005 made CraftOS-PC the local harness and ADR 0007 split `libtest` (cases and
assertions) from `runtest` (suite orchestration). The only timeout in that harness was the
shell watchdog in the `Justfile` `test:` recipe: it `kill -TERM`s the whole CraftOS-PC
process after `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS`.

That single layer is coarse. It kills the entire process, so it cannot say *which* case
hung, it produces one generic message, and it cannot tell a cooperatively-blocked event
loop (the common failure: waiting on an event or `sleep` that never resolves) apart from a
genuinely wedged process. A per-case timeout inside Lua is both finer and faster, but the
shell watchdog is still needed for the cases Lua cannot interrupt.

## Decision

Run two independent timeout layers, ordered so the finer one fires first.

**Layer 1 — `libtest` per-case timeout (primary).** `/apis/libtest.lua` races each test
case against a timer with `parallel.waitForAny(runner, timer)`. The default is
`DEFAULT_TIMEOUT_SECONDS = 3`. When the timer wins, the case fails with a distinct message
containing the token `libtest timeout` and, in `--verbose`, an extra `TIMEOUT … (libtest)`
diagnostic. `--timeout <seconds>` overrides the default; `--no-timeout` disables the layer.
`/programs/runtest.lua` forwards both flags to each case script. This only interrupts cases
that yield (the usual hang); a non-yielding CPU loop cannot be preempted in ComputerCraft.

**Layer 2 — shell watchdog (backstop).** The `Justfile` `test:` recipe keeps its existing
`TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` watchdog unchanged, as an independent double-check. Its
default matches the libtest default (`.env.test` ships `3`; the recipe falls back to `3`)
so libtest should fire first for yielding cases in normal runs and the watchdog only catches
what Lua cannot — a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its
SIGTERM message is worded differently from the `libtest timeout` message, so the two layers
are never confused.

## How To Write Tests Properly

- Normal tests live in `tests/*.lua`, use `/apis/libtest.lua`, and must finish under the
  libtest timeout. `runtest` auto-discovers them; `just test` runs the suite.
- Never commit a hanging or intentionally-slow test to `tests/`: it would fail every run.
- Intentionally-slow fixtures that exercise the harness itself live in `tests/harness/`.
  `runtest` discovery skips subdirectories, so they never run with the normal suite; they
  are driven only by dedicated recipes (`just test-timeout-lua`, `just test-timeout-shell`,
  aggregated by `just test-timeout`).
- Use `--no-timeout` only for harness fixtures that must outlive the libtest layer to prove
  the shell watchdog, never for ordinary tests.

## Consequences

- A hung case now fails in ~3s with a per-case message instead of taking down the whole
  process anonymously.
- `just test-timeout` is a self-asserting harness regression guard wired into `just ci`. It
  chains `test-timeout-lua` (Layer 1: libtest cancels the slow case immediately with
  `--timeout 0`, before the shell backstop) and `test-timeout-shell` (Layer 2: the
  `TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS` watchdog, default `1`, kills the slow case with
  libtest bypassed). Both drive a single `tests/harness/slow-case.lua` fixture; the tight
  timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself
  is covered against regressions on every `ci`.
- `libtest` stays a normal ComputerCraft program: `parallel` and `sleep` are sandbox
  globals, so the timeout works in CraftOS-PC and in-game alike.

## Future Work

- Per-case timing in `--verbose` output if slow-but-passing cases become hard to spot.
- A `libtest`-level marker for "expected timeout" if more harness fixtures appear.