test(harness): use test env for timeouts

This commit is contained in:
Guillaume ARM 2026-06-08 19:22:32 +02:00
parent b87dafc666
commit 7b19de7945
5 changed files with 25 additions and 25 deletions

View File

@ -1 +0,0 @@
TRAP_CCLIBS_TEST_TIMEOUT_SECONDS=3

5
.env.test Normal file
View File

@ -0,0 +1,5 @@
# Host-side watchdog for the normal `just test` CraftOS-PC process.
TRAP_CCLIBS_TEST_TIMEOUT_SECONDS=3
# Dedicated `just test-timeout` fixture timings.
TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS=1

View File

@ -12,7 +12,7 @@ After cloning the repository, run:
just install
```
This creates `.env` from `.env.sample` when needed and installs the local Git hooks: pre-commit runs `just test`, and pre-push runs `just ci`.
This installs the local Git hooks: pre-commit runs `just test`, and pre-push runs `just ci`.
`just ci` is the full local verification entry point. Today it verifies that `craftos --version` reports v2.8.3 or newer, runs `just check` for `luacheck`, runs `just test` for CraftOS-PC headless tests, and runs the harness timeout regression guard. Use `just craftos` to launch CraftOS-PC with repo-local save data under `.craftos` and read-only mounts for `/trapos`, `/apis`, `/programs`, `/servers`, and `/startup`. `just repl` opens the same environment with `--cli` for human interactive use only; LLM agents must not run it. Use the CraftOS-PC glossary when adjusting `--headless`, `--exec`, `--script`, `--rom`, or `--mount-*` usage.
@ -23,6 +23,6 @@ Tests live under `tests/` and run inside CraftOS-PC through `just test`, which l
Test timeouts run in two independent layers (see [`docs/adrs/adr-0009-layered-test-timeouts.md`](docs/adrs/adr-0009-layered-test-timeouts.md)):
- **libtest per-case timeout (primary).** Each `libtest` case is cancelled after `3` seconds by default, failing with a distinct `libtest timeout` message. Override with `--timeout <seconds>`, or disable with `--no-timeout` (both forwarded by `runtest` to each case). This catches a single hung case quickly without taking down the whole run.
- **Shell watchdog (backstop).** The whole CraftOS-PC process is killed if it does not finish within `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` (`.env.sample` ships `7`; the recipe falls back to `7`). Keep it above the `3`s libtest default so libtest fires first; the watchdog only catches what Lua cannot interrupt. Override in `.env` for slower local probes.
- **Shell watchdog (backstop).** The whole CraftOS-PC process is killed if it does not finish within `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` (`.env.test` ships `3`; the recipe falls back to `3`). Keep it at or above the `3`s libtest default so libtest fires first; the watchdog only catches what Lua cannot interrupt. Override in your shell for slower local probes.
`just test-timeout` is a self-asserting regression guard for these layers and runs automatically as part of `just ci`. It chains `just test-timeout-lua` (proves libtest cancels a slow case at 0.1s, before a 2s shell backstop) and `just test-timeout-shell` (proves the 1s shell watchdog kills a slow case run with `--no-timeout`). Both drive the `tests/harness/slow-case.lua` fixture, which is never picked up by the normal `just test` suite (`runtest` skips `tests/` subdirectories).
`just test-timeout` is a self-asserting regression guard for these layers and runs automatically as part of `just ci`. It chains `just test-timeout-lua` (proves libtest cancels a slow case immediately with `--timeout 0`, before the shell backstop) and `just test-timeout-shell` (proves the `TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS` watchdog, default `1`, kills a slow case run with `--no-timeout`). Both drive the `tests/harness/slow-case.lua` fixture, which is never picked up by the normal `just test` suite (`runtest` skips `tests/` subdirectories).

View File

@ -6,14 +6,7 @@ default:
@just --list
# Install local development tooling.
install: init-env install-git-hooks check-install
# Create a local environment file when one does not exist.
init-env:
@if [ ! -f .env ]; then \
cp .env.sample .env; \
printf '%s\n' 'Created .env from .env.sample'; \
fi
install: install-git-hooks check-install
# Install Git hooks for this repository.
install-git-hooks:
@ -109,10 +102,10 @@ ci *args: check-craftos check
# Run CraftOS-PC headless integration tests. Pass `--pretty` for grouped output.
test *args:
@if [ -f .env ]; then set -a; . ./.env; set +a; fi; \
@if [ -f .env.test ]; then set -a; . ./.env.test; set +a; fi; \
pretty=0; \
verbose=0; \
timeout_seconds="${TRAP_CCLIBS_TEST_TIMEOUT_SECONDS:-7}"; \
timeout_seconds="${TRAP_CCLIBS_TEST_TIMEOUT_SECONDS:-3}"; \
case "$timeout_seconds" in ''|*[!0-9]*) printf '%s\n' 'TRAP_CCLIBS_TEST_TIMEOUT_SECONDS must be a positive integer' >&2; exit 1 ;; esac; \
if [ "$timeout_seconds" -lt 1 ]; then printf '%s\n' 'TRAP_CCLIBS_TEST_TIMEOUT_SECONDS must be >= 1' >&2; exit 1; fi; \
for a in {{args}}; do case "$a" in --pretty) pretty=1 ;; --verbose|-v) pretty=1; verbose=1 ;; esac; done; \
@ -163,6 +156,7 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
#!/usr/bin/env bash
set -uo pipefail
repo='{{justfile_directory()}}'
if [ -f "$repo/.env.test" ]; then set -a; . "$repo/.env.test"; set +a; fi
rom_arg=""
if [ "$(uname -s)" = "Darwin" ]; then
rom_arg="--rom /Applications/CraftOS-PC.app/Contents/Resources"
@ -172,9 +166,10 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
data_dir="$(mktemp -d)"
output_path="$data_dir/computer/0/trapos-test-output"
exec_code="shell.run('/programs/runtest.lua', '{{script}}', '--verbose', '--output', '/trapos-test-output', {{extra_flag}} '--shutdown')"
shell_timeout="{{shell_timeout}}"
craftos --directory "$data_dir" --headless $rom_arg $mount_arg --exec "$exec_code" >"$tmp" 2>&1 &
pid="$!"
( sleep {{shell_timeout}}; kill -TERM "$pid" >/dev/null 2>&1 ) &
( sleep "$shell_timeout"; kill -TERM "$pid" >/dev/null 2>&1 ) &
watchdog="$!"
wait "$pid" >/dev/null 2>&1
status="$?"
@ -194,7 +189,7 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
;;
shell)
if [ "$status" -eq 143 ]; then
printf '%s\n' "${green}OK${reset} shell watchdog killed the run after {{shell_timeout}}s (status 143; libtest timeout bypassed)"; \
printf '%s\n' "${green}OK${reset} shell watchdog killed the run after ${shell_timeout}s (status 143; libtest timeout bypassed)"; \
else
printf '%s\n' "${red}FAIL${reset} expected the shell watchdog to kill the run (status=$status)" >&2
rc=1
@ -212,11 +207,11 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
# Prove the libtest (Lua) timeout layer: libtest cancels the slow case quickly,
# before the shell watchdog backstop can fire.
test-timeout-lua: (_timeout-fixture "/tests/harness/slow-case.lua" "2" "'--timeout', '0.1'," "lua")
test-timeout-lua: (_timeout-fixture "/tests/harness/slow-case.lua" "${TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS:-1}" "'--timeout', '0'," "lua")
# Prove the shell watchdog backstop: the slow case runs with the libtest timeout
# bypassed (--no-timeout), so the shell watchdog kills the whole process.
test-timeout-shell: (_timeout-fixture "/tests/harness/slow-case.lua" "1" "'--no-timeout'," "shell")
test-timeout-shell: (_timeout-fixture "/tests/harness/slow-case.lua" "${TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS:-1}" "'--no-timeout'," "shell")
# Fast regression guard for both timeout layers. Wired into `ci`.
test-timeout: test-timeout-lua test-timeout-shell

View File

@ -35,11 +35,11 @@ that yield (the usual hang); a non-yielding CPU loop cannot be preempted in Comp
**Layer 2 — shell watchdog (backstop).** The `Justfile` `test:` recipe keeps its existing
`TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` watchdog unchanged, as an independent double-check. Its
default sits *above* the libtest default (`.env.sample` ships `7`; the recipe falls back to
`7`) so libtest fires first in normal runs and the watchdog only catches what Lua cannot —
a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its SIGTERM message
is worded differently from the `libtest timeout` message, so the two layers are never
confused.
default matches the libtest default (`.env.test` ships `3`; the recipe falls back to `3`)
so libtest should fire first for yielding cases in normal runs and the watchdog only catches
what Lua cannot — a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its
SIGTERM message is worded differently from the `libtest timeout` message, so the two layers
are never confused.
## How To Write Tests Properly
@ -58,8 +58,9 @@ confused.
- A hung case now fails in ~3s with a per-case message instead of taking down the whole
process anonymously.
- `just test-timeout` is a self-asserting harness regression guard wired into `just ci`. It
chains `test-timeout-lua` (Layer 1: libtest cancels the slow case at 0.1s, before a 2s
shell backstop) and `test-timeout-shell` (Layer 2: the 1s watchdog kills the slow case with
chains `test-timeout-lua` (Layer 1: libtest cancels the slow case immediately with
`--timeout 0`, before the shell backstop) and `test-timeout-shell` (Layer 2: the
`TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS` watchdog, default `1`, kills the slow case with
libtest bypassed). Both drive a single `tests/harness/slow-case.lua` fixture; the tight
timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself
is covered against regressions on every `ci`.