80 lines
6.8 KiB
Markdown
80 lines
6.8 KiB
Markdown
# ADR 0007: Test Framework — libtest, runtest, and Layered Timeouts
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Date
|
|
|
|
2026-06-08
|
|
|
|
## Context
|
|
|
|
[ADR-0005](adr-0005-craftos-pc-harness-and-probes.md) made CraftOS-PC the local harness. The first behavior test, `tests/eventloop.lua`, proved the harness can exercise ComputerCraft APIs headlessly, but it also duplicated test-runner concerns directly in the script: collecting named cases, per-case progress in verbose mode, fail-fast messaging, success-marker emission, and process shutdown. Those details are easy to copy incorrectly.
|
|
|
|
A blocked eventloop test then showed two more harness needs:
|
|
|
|
- The shell harness needs a timeout and captured output so agentic debugging can proceed without manual interruption.
|
|
- A single shell watchdog is coarse. `kill -TERM` on the whole CraftOS-PC process cannot say *which* case hung, produces one generic message, and cannot tell a cooperatively-blocked event loop (the common failure — waiting on an event or `sleep` that never resolves) apart from a genuinely wedged process. A per-case timeout inside Lua is both finer and faster, but the shell watchdog is still needed for the cases Lua cannot interrupt.
|
|
|
|
At the same time, tests in this repository are still ComputerCraft programs. They should be useful in CraftOS-PC and in-game, not only inside a host-side shell loop.
|
|
|
|
## Decision
|
|
|
|
### 1. `libtest` is the per-case helper
|
|
|
|
[`apis/libtest.lua`](../../apis/libtest.lua) is the repository's lightweight ComputerCraft test helper. Tests under `tests/` require it with an absolute ComputerCraft path:
|
|
|
|
```lua
|
|
local createLibTest = require('/apis/libtest');
|
|
local testlib = createLibTest({ ... });
|
|
|
|
testlib.test('example', function()
|
|
testlib.assertEquals(1 + 1, 2);
|
|
end);
|
|
|
|
testlib.run();
|
|
```
|
|
|
|
`libtest` intentionally stays small. It provides named cases, `assertEquals`, `assertTrue`, `assertErrors`, optional case-status report output consumed by the suite runner, and failure reporting. It must stay usable from normal ComputerCraft programs in CraftOS-PC or in-game.
|
|
|
|
### 2. `runtest` owns suite orchestration; the Justfile stays minimal
|
|
|
|
[`/programs/runtest.lua`](../../programs/runtest.lua) owns suite-level concerns: test discovery under `/tests`, invoking each script with `shell.run`, grouped `--pretty` output, `--verbose` runner diagnostics, the `__TRAPOS_TEST_OK__` success marker (printed only after the full suite passes), and optional shutdown when the host harness asks with `--shutdown`. `runtest` can run inside CraftOS-PC or in-game when `/tests` and dependencies are present.
|
|
|
|
The `Justfile` launches CraftOS-PC, mounts repository directories, enforces the process timeout, checks for the success marker, and prints runner output files. It does not know about individual test files or cases. Verbose mode is reserved for debugging and agent work loops; `--pretty` is the normal human-readable mode.
|
|
|
|
### 3. Layered test timeouts
|
|
|
|
Two independent timeout layers, ordered so the finer one fires first.
|
|
|
|
**Layer 1 — `libtest` per-case timeout (primary).** [`apis/libtest.lua`](../../apis/libtest.lua) races each test case against a timer with `parallel.waitForAny(runner, timer)`. The default is `DEFAULT_TIMEOUT_SECONDS = 3`. When the timer wins, the case fails with a distinct message containing the token `libtest timeout` and, in `--verbose`, an extra `TIMEOUT … (libtest)` diagnostic. `--timeout <seconds>` overrides the default; `--no-timeout` disables the layer. `runtest` forwards both flags to each case script. This only interrupts cases that yield (the usual hang); a non-yielding CPU loop cannot be preempted in ComputerCraft.
|
|
|
|
**Layer 2 — shell watchdog (backstop).** The `Justfile` `test:` recipe keeps its `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` watchdog as an independent double-check. Its default matches the libtest default (`.env.test` ships `3`; the recipe falls back to `3`) so libtest fires first for yielding cases in normal runs and the watchdog only catches what Lua cannot — a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its SIGTERM message is worded differently from the `libtest timeout` message, so the two layers are never confused.
|
|
|
|
### How to write tests properly
|
|
|
|
- Normal tests live in `tests/*.lua`, use `/apis/libtest.lua`, and must finish under the libtest timeout. `runtest` auto-discovers them; `just test` runs the suite.
|
|
- Never commit a hanging or intentionally-slow test to `tests/`: it would fail every run.
|
|
- Intentionally-slow fixtures that exercise the harness itself live in `tests/harness/`. `runtest` discovery skips subdirectories, so they never run with the normal suite; they are driven only by dedicated recipes (`just test-timeout-lua`, `just test-timeout-shell`, aggregated by `just test-timeout`).
|
|
- Use `--no-timeout` only for harness fixtures that must outlive the libtest layer to prove the shell watchdog, never for ordinary tests.
|
|
|
|
## Consequences
|
|
|
|
- New deterministic behavior should get as many useful CraftOS-PC tests as practical. Tests that require human validation, such as complex turtle motion, in-game UX feel, or visual approval, may be skipped, but deterministic pieces should still get unit-style non-regression coverage.
|
|
- Test scripts remain normal ComputerCraft programs, not standalone Lua tests. They can run through `just test`, `/programs/runtest.lua`, or direct in-game execution when copied with their dependencies.
|
|
- `libtest` lives under `/apis` and ships in `trapos-test`, so it can be required consistently in the mounted CraftOS-PC environment.
|
|
- The `__TRAPOS_TEST_OK__` marker remains the single shell-level success contract and is owned by `runtest`.
|
|
- A hung case fails in ~3s with a per-case message instead of taking down the whole process anonymously.
|
|
- `just test-timeout` is a self-asserting harness regression guard wired into `just ci`. It chains `test-timeout-lua` (Layer 1: libtest cancels the slow case immediately with `--timeout 0`, before the shell backstop) and `test-timeout-shell` (Layer 2: the `TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS` watchdog, default `1`, kills the slow case with libtest bypassed). Both drive a single `tests/harness/slow-case.lua` fixture; the tight timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself is covered against regressions on every `ci`.
|
|
- `libtest` stays a normal ComputerCraft program: `parallel` and `sleep` are sandbox globals, so the timeout works in CraftOS-PC and in-game alike.
|
|
- Host-specific concerns remain outside production Lua code.
|
|
|
|
## Future Work
|
|
|
|
- Add more assertions only when tests need them; avoid growing a large framework.
|
|
- Add test selection filters when the suite grows.
|
|
- Add runner-level or per-case timing in `--verbose` output if slow-but-passing cases become hard to spot.
|
|
- A `libtest`-level marker for "expected timeout" if more harness fixtures appear.
|
|
- Explore GitHub Actions with the Linux CraftOS-PC AppImage after local coverage is broader.
|