test(harness): run timeout-layer checks in ci
- Add fast `test-timeout` guard chaining `test-timeout-lua`/`-shell` - Wire `test-timeout` into `just ci` for harness regression coverage - Consolidate fixtures into single `tests/harness/slow-case.lua` - Tighten timeouts (libtest 2s / shell watchdog 3s) so each check is ~3s - Update ADR-0009 and DEVELOPMENT.md for the new recipes and ci wiring
This commit is contained in:
parent
e5d8e16c5a
commit
3c4a8876b9
@ -23,4 +23,4 @@ Test timeouts run in two independent layers (see [`docs/adrs/adr-0009-layered-te
|
|||||||
- **libtest per-case timeout (primary).** Each `libtest` case is cancelled after `3` seconds by default, failing with a distinct `libtest timeout` message. Override with `--timeout <seconds>`, or disable with `--no-timeout` (both forwarded by `runtest` to each case). This catches a single hung case quickly without taking down the whole run.
|
- **libtest per-case timeout (primary).** Each `libtest` case is cancelled after `3` seconds by default, failing with a distinct `libtest timeout` message. Override with `--timeout <seconds>`, or disable with `--no-timeout` (both forwarded by `runtest` to each case). This catches a single hung case quickly without taking down the whole run.
|
||||||
- **Shell watchdog (backstop).** The whole CraftOS-PC process is killed if it does not finish within `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` (`.env.sample` ships `7`; the recipe falls back to `7`). Keep it above the `3`s libtest default so libtest fires first; the watchdog only catches what Lua cannot interrupt. Override in `.env` for slower local probes.
|
- **Shell watchdog (backstop).** The whole CraftOS-PC process is killed if it does not finish within `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` (`.env.sample` ships `7`; the recipe falls back to `7`). Keep it above the `3`s libtest default so libtest fires first; the watchdog only catches what Lua cannot interrupt. Override in `.env` for slower local probes.
|
||||||
|
|
||||||
`just test-timeout-5s` and `just test-timeout-10s` are self-asserting harness checks for these layers (fixtures under `tests/harness/`, not part of `just test`): the first proves libtest cancels a 5s case at 3s, the second proves the shell watchdog kills a 10s case run with `--no-timeout`.
|
`just test-timeout` is a self-asserting regression guard for these layers and runs automatically as part of `just ci`. It chains `just test-timeout-lua` (proves libtest cancels a slow case at 2s, before a generous 5s shell backstop) and `just test-timeout-shell` (proves the 3s shell watchdog kills a slow case run with `--no-timeout`). Both drive the `tests/harness/slow-case.lua` fixture, which is never picked up by the normal `just test` suite (`runtest` skips `tests/` subdirectories).
|
||||||
|
|||||||
16
Justfile
16
Justfile
@ -100,6 +100,7 @@ repl:
|
|||||||
# Local CI entry point used by Git hooks. Pass args through to `test`.
|
# Local CI entry point used by Git hooks. Pass args through to `test`.
|
||||||
ci *args: check-craftos check
|
ci *args: check-craftos check
|
||||||
@just test {{args}}
|
@just test {{args}}
|
||||||
|
@just test-timeout
|
||||||
|
|
||||||
# Run CraftOS-PC headless integration tests. Pass `--pretty` for grouped output.
|
# Run CraftOS-PC headless integration tests. Pass `--pretty` for grouped output.
|
||||||
test *args:
|
test *args:
|
||||||
@ -204,13 +205,16 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
|
|||||||
rm -rf "$data_dir"
|
rm -rf "$data_dir"
|
||||||
exit "$rc"
|
exit "$rc"
|
||||||
|
|
||||||
# Prove the libtest (Lua) timeout layer: a 5s case is cancelled at the 3s
|
# Prove the libtest (Lua) timeout layer: libtest cancels the slow case at 2s,
|
||||||
# default, before the generous 30s shell watchdog can fire.
|
# before the generous 5s shell watchdog backstop can fire.
|
||||||
test-timeout-5s: (_timeout-fixture "/tests/harness/timeout-5s.lua" "30" "" "lua")
|
test-timeout-lua: (_timeout-fixture "/tests/harness/slow-case.lua" "5" "'--timeout', '2'," "lua")
|
||||||
|
|
||||||
# Prove the shell watchdog backstop: a 10s case runs with the libtest timeout
|
# Prove the shell watchdog backstop: the slow case runs with the libtest timeout
|
||||||
# bypassed (--no-timeout), so the 5s shell watchdog kills the whole process.
|
# bypassed (--no-timeout), so the 3s shell watchdog kills the whole process.
|
||||||
test-timeout-10s: (_timeout-fixture "/tests/harness/timeout-10s.lua" "5" "'--no-timeout'," "shell")
|
test-timeout-shell: (_timeout-fixture "/tests/harness/slow-case.lua" "3" "'--no-timeout'," "shell")
|
||||||
|
|
||||||
|
# Fast regression guard for both timeout layers. Wired into `ci`.
|
||||||
|
test-timeout: test-timeout-lua test-timeout-shell
|
||||||
|
|
||||||
# Lint all Lua source with luacheck.
|
# Lint all Lua source with luacheck.
|
||||||
check: check-luacheck
|
check: check-luacheck
|
||||||
|
|||||||
@ -48,7 +48,8 @@ confused.
|
|||||||
- Never commit a hanging or intentionally-slow test to `tests/`: it would fail every run.
|
- Never commit a hanging or intentionally-slow test to `tests/`: it would fail every run.
|
||||||
- Intentionally-slow fixtures that exercise the harness itself live in `tests/harness/`.
|
- Intentionally-slow fixtures that exercise the harness itself live in `tests/harness/`.
|
||||||
`runtest` discovery skips subdirectories, so they never run with the normal suite; they
|
`runtest` discovery skips subdirectories, so they never run with the normal suite; they
|
||||||
are driven only by dedicated recipes (`just test-timeout-5s`, `just test-timeout-10s`).
|
are driven only by dedicated recipes (`just test-timeout-lua`, `just test-timeout-shell`,
|
||||||
|
aggregated by `just test-timeout`).
|
||||||
- Use `--no-timeout` only for harness fixtures that must outlive the libtest layer to prove
|
- Use `--no-timeout` only for harness fixtures that must outlive the libtest layer to prove
|
||||||
the shell watchdog, never for ordinary tests.
|
the shell watchdog, never for ordinary tests.
|
||||||
|
|
||||||
@ -56,10 +57,12 @@ confused.
|
|||||||
|
|
||||||
- A hung case now fails in ~3s with a per-case message instead of taking down the whole
|
- A hung case now fails in ~3s with a per-case message instead of taking down the whole
|
||||||
process anonymously.
|
process anonymously.
|
||||||
- The two `test-timeout-*` recipes are self-asserting harness tests: `test-timeout-5s`
|
- `just test-timeout` is a self-asserting harness regression guard wired into `just ci`. It
|
||||||
proves Layer 1 (libtest cancels a 5s case before the watchdog), `test-timeout-10s` proves
|
chains `test-timeout-lua` (Layer 1: libtest cancels the slow case at 2s, before a generous
|
||||||
Layer 2 (the watchdog kills a 10s case with libtest bypassed). They are intentionally
|
5s shell backstop) and `test-timeout-shell` (Layer 2: the 3s watchdog kills the slow case
|
||||||
excluded from `ci`/`test`.
|
with libtest bypassed). Both drive a single `tests/harness/slow-case.lua` fixture; the
|
||||||
|
tight timeouts — not the fixture's sleep length — decide which layer fires, so each check
|
||||||
|
finishes in ~3s and the harness itself is covered against regressions on every `ci`.
|
||||||
- `libtest` stays a normal ComputerCraft program: `parallel` and `sleep` are sandbox
|
- `libtest` stays a normal ComputerCraft program: `parallel` and `sleep` are sandbox
|
||||||
globals, so the timeout works in CraftOS-PC and in-game alike.
|
globals, so the timeout works in CraftOS-PC and in-game alike.
|
||||||
|
|
||||||
|
|||||||
16
tests/harness/slow-case.lua
Normal file
16
tests/harness/slow-case.lua
Normal file
@ -0,0 +1,16 @@
|
|||||||
|
-- Harness fixture (NOT auto-discovered: lives under tests/harness/).
|
||||||
|
-- A single case that sleeps well past any harness timeout. The driving recipe
|
||||||
|
-- decides which layer cancels it: `test-timeout-lua` lets libtest fire via
|
||||||
|
-- `--timeout`, `test-timeout-shell` bypasses libtest (`--no-timeout`) so the
|
||||||
|
-- shell watchdog kills the process. See `just test-timeout`.
|
||||||
|
local createLibTest = require('/apis/libtest');
|
||||||
|
|
||||||
|
local testlib = createLibTest({ ... });
|
||||||
|
|
||||||
|
testlib.test('sleeps past the harness timeout', function()
|
||||||
|
testlib.log('about to sleep 10s; the active harness timeout should cancel this first');
|
||||||
|
sleep(10);
|
||||||
|
testlib.assertTrue(true); -- never reached: a timeout layer cancels first
|
||||||
|
end);
|
||||||
|
|
||||||
|
testlib.run();
|
||||||
@ -1,14 +0,0 @@
|
|||||||
-- Harness fixture (NOT auto-discovered: lives under tests/harness/).
|
|
||||||
-- Sleeps 10s with the libtest timeout bypassed (--no-timeout), proving the
|
|
||||||
-- shell watchdog is still the backstop. Driven by `just test-timeout-10s`.
|
|
||||||
local createLibTest = require('/apis/libtest');
|
|
||||||
|
|
||||||
local testlib = createLibTest({ ... });
|
|
||||||
|
|
||||||
testlib.test('sleeps past the shell watchdog', function()
|
|
||||||
testlib.log('about to sleep 10s with the libtest timeout disabled');
|
|
||||||
sleep(10);
|
|
||||||
testlib.assertTrue(true); -- never reached: shell watchdog kills the process
|
|
||||||
end);
|
|
||||||
|
|
||||||
testlib.run();
|
|
||||||
@ -1,14 +0,0 @@
|
|||||||
-- Harness fixture (NOT auto-discovered: lives under tests/harness/).
|
|
||||||
-- Sleeps past the 3s libtest default timeout but under the 7s shell watchdog,
|
|
||||||
-- proving the libtest layer cancels the case first. Driven by `just test-timeout-5s`.
|
|
||||||
local createLibTest = require('/apis/libtest');
|
|
||||||
|
|
||||||
local testlib = createLibTest({ ... });
|
|
||||||
|
|
||||||
testlib.test('sleeps past the libtest timeout', function()
|
|
||||||
testlib.log('about to sleep 5s; libtest should cancel this case at 3s');
|
|
||||||
sleep(5);
|
|
||||||
testlib.assertTrue(true); -- never reached: libtest cancels first
|
|
||||||
end);
|
|
||||||
|
|
||||||
testlib.run();
|
|
||||||
Loading…
Reference in New Issue
Block a user