ADR 0007: Test Framework — libtest, runtest, and Layered Timeouts

Status

Accepted

Date

2026-06-08

Context

ADR-0005 made CraftOS-PC the local harness. The first behavior test, tests/eventloop.lua, proved the harness can exercise ComputerCraft APIs headlessly, but it also duplicated test-runner concerns directly in the script: collecting named cases, per-case progress in verbose mode, fail-fast messaging, success-marker emission, and process shutdown. Those details are easy to copy incorrectly.

A blocked eventloop test then showed two more harness needs:

The shell harness needs a timeout and captured output so agentic debugging can proceed without manual interruption.
A single shell watchdog is coarse. kill -TERM on the whole CraftOS-PC process cannot say which case hung, produces one generic message, and cannot tell a cooperatively-blocked event loop (the common failure — waiting on an event or sleep that never resolves) apart from a genuinely wedged process. A per-case timeout inside Lua is both finer and faster, but the shell watchdog is still needed for the cases Lua cannot interrupt.

At the same time, tests in this repository are still ComputerCraft programs. They should be useful in CraftOS-PC and in-game, not only inside a host-side shell loop.

Decision

1. `libtest` is the per-case helper

apis/libtest.lua is the repository's lightweight ComputerCraft test helper. Tests under tests/ require it with an absolute ComputerCraft path:

local createLibTest = require('/apis/libtest');
local testlib = createLibTest({ ... });

testlib.test('example', function()
  testlib.assertEquals(1 + 1, 2);
end);

testlib.run();

libtest intentionally stays small. It provides named cases, assertEquals, assertTrue, assertErrors, optional case-status report output consumed by the suite runner, and failure reporting. It must stay usable from normal ComputerCraft programs in CraftOS-PC or in-game.

2. `runtest` owns suite orchestration; the Justfile stays minimal

/programs/runtest.lua owns suite-level concerns: test discovery under /tests, invoking each script with shell.run, grouped --pretty output, --verbose runner diagnostics, the __TRAPOS_TEST_OK__ success marker (printed only after the full suite passes), and optional shutdown when the host harness asks with --shutdown. runtest can run inside CraftOS-PC or in-game when /tests and dependencies are present.

The Justfile launches CraftOS-PC, mounts repository directories, enforces the process timeout, checks for the success marker, and prints runner output files. It does not know about individual test files or cases. Verbose mode is reserved for debugging and agent work loops; --pretty is the normal human-readable mode.

3. Layered test timeouts

Two independent timeout layers, ordered so the finer one fires first.

Layer 1 — libtest per-case timeout (primary). apis/libtest.lua races each test case against a timer with parallel.waitForAny(runner, timer). The default is DEFAULT_TIMEOUT_SECONDS = 3. When the timer wins, the case fails with a distinct message containing the token libtest timeout and, in --verbose, an extra TIMEOUT … (libtest) diagnostic. --timeout <seconds> overrides the default; --no-timeout disables the layer. runtest forwards both flags to each case script. This only interrupts cases that yield (the usual hang); a non-yielding CPU loop cannot be preempted in ComputerCraft.

Layer 2 — shell watchdog (backstop). The Justfile test: recipe keeps its TRAP_CCLIBS_TEST_TIMEOUT_SECONDS watchdog as an independent double-check. Its default matches the libtest default (.env.test ships 3; the recipe falls back to 3) so libtest fires first for yielding cases in normal runs and the watchdog only catches what Lua cannot — a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its SIGTERM message is worded differently from the libtest timeout message, so the two layers are never confused.

How to write tests properly

Normal tests live in tests/*.lua, use /apis/libtest.lua, and must finish under the libtest timeout. runtest auto-discovers them; just test runs the suite.
Never commit a hanging or intentionally-slow test to tests/: it would fail every run.
Intentionally-slow fixtures that exercise the harness itself live in tests/harness/. runtest discovery skips subdirectories, so they never run with the normal suite; they are driven only by dedicated recipes (just test-timeout-lua, just test-timeout-shell, aggregated by just test-timeout).
Use --no-timeout only for harness fixtures that must outlive the libtest layer to prove the shell watchdog, never for ordinary tests.

Consequences

New deterministic behavior should get as many useful CraftOS-PC tests as practical. Tests that require human validation, such as complex turtle motion, in-game UX feel, or visual approval, may be skipped, but deterministic pieces should still get unit-style non-regression coverage.
Test scripts remain normal ComputerCraft programs, not standalone Lua tests. They can run through just test, /programs/runtest.lua, or direct in-game execution when copied with their dependencies.
libtest lives under /apis and ships in trapos-test, so it can be required consistently in the mounted CraftOS-PC environment.
The __TRAPOS_TEST_OK__ marker remains the single shell-level success contract and is owned by runtest.
A hung case fails in ~3s with a per-case message instead of taking down the whole process anonymously.
just test-timeout is a self-asserting harness regression guard wired into just ci. It chains test-timeout-lua (Layer 1: libtest cancels the slow case immediately with --timeout 0, before the shell backstop) and test-timeout-shell (Layer 2: the TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS watchdog, default 1, kills the slow case with libtest bypassed). Both drive a single tests/harness/slow-case.lua fixture; the tight timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself is covered against regressions on every ci.
libtest stays a normal ComputerCraft program: parallel and sleep are sandbox globals, so the timeout works in CraftOS-PC and in-game alike.
Host-specific concerns remain outside production Lua code.

Future Work

Add more assertions only when tests need them; avoid growing a large framework.
Add test selection filters when the suite grows.
Add runner-level or per-case timing in --verbose output if slow-but-passing cases become hard to spot.
A libtest-level marker for "expected timeout" if more harness fixtures appear.
Explore GitHub Actions with the Linux CraftOS-PC AppImage after local coverage is broader.

6.8 KiB Raw Blame History