test(harness): use test env for timeouts
This commit is contained in:
parent
b87dafc666
commit
7b19de7945
@ -1 +0,0 @@
|
|||||||
TRAP_CCLIBS_TEST_TIMEOUT_SECONDS=3
|
|
||||||
5
.env.test
Normal file
5
.env.test
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
# Host-side watchdog for the normal `just test` CraftOS-PC process.
|
||||||
|
TRAP_CCLIBS_TEST_TIMEOUT_SECONDS=3
|
||||||
|
|
||||||
|
# Dedicated `just test-timeout` fixture timings.
|
||||||
|
TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS=1
|
||||||
@ -12,7 +12,7 @@ After cloning the repository, run:
|
|||||||
just install
|
just install
|
||||||
```
|
```
|
||||||
|
|
||||||
This creates `.env` from `.env.sample` when needed and installs the local Git hooks: pre-commit runs `just test`, and pre-push runs `just ci`.
|
This installs the local Git hooks: pre-commit runs `just test`, and pre-push runs `just ci`.
|
||||||
|
|
||||||
`just ci` is the full local verification entry point. Today it verifies that `craftos --version` reports v2.8.3 or newer, runs `just check` for `luacheck`, runs `just test` for CraftOS-PC headless tests, and runs the harness timeout regression guard. Use `just craftos` to launch CraftOS-PC with repo-local save data under `.craftos` and read-only mounts for `/trapos`, `/apis`, `/programs`, `/servers`, and `/startup`. `just repl` opens the same environment with `--cli` for human interactive use only; LLM agents must not run it. Use the CraftOS-PC glossary when adjusting `--headless`, `--exec`, `--script`, `--rom`, or `--mount-*` usage.
|
`just ci` is the full local verification entry point. Today it verifies that `craftos --version` reports v2.8.3 or newer, runs `just check` for `luacheck`, runs `just test` for CraftOS-PC headless tests, and runs the harness timeout regression guard. Use `just craftos` to launch CraftOS-PC with repo-local save data under `.craftos` and read-only mounts for `/trapos`, `/apis`, `/programs`, `/servers`, and `/startup`. `just repl` opens the same environment with `--cli` for human interactive use only; LLM agents must not run it. Use the CraftOS-PC glossary when adjusting `--headless`, `--exec`, `--script`, `--rom`, or `--mount-*` usage.
|
||||||
|
|
||||||
@ -23,6 +23,6 @@ Tests live under `tests/` and run inside CraftOS-PC through `just test`, which l
|
|||||||
Test timeouts run in two independent layers (see [`docs/adrs/adr-0009-layered-test-timeouts.md`](docs/adrs/adr-0009-layered-test-timeouts.md)):
|
Test timeouts run in two independent layers (see [`docs/adrs/adr-0009-layered-test-timeouts.md`](docs/adrs/adr-0009-layered-test-timeouts.md)):
|
||||||
|
|
||||||
- **libtest per-case timeout (primary).** Each `libtest` case is cancelled after `3` seconds by default, failing with a distinct `libtest timeout` message. Override with `--timeout <seconds>`, or disable with `--no-timeout` (both forwarded by `runtest` to each case). This catches a single hung case quickly without taking down the whole run.
|
- **libtest per-case timeout (primary).** Each `libtest` case is cancelled after `3` seconds by default, failing with a distinct `libtest timeout` message. Override with `--timeout <seconds>`, or disable with `--no-timeout` (both forwarded by `runtest` to each case). This catches a single hung case quickly without taking down the whole run.
|
||||||
- **Shell watchdog (backstop).** The whole CraftOS-PC process is killed if it does not finish within `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` (`.env.sample` ships `7`; the recipe falls back to `7`). Keep it above the `3`s libtest default so libtest fires first; the watchdog only catches what Lua cannot interrupt. Override in `.env` for slower local probes.
|
- **Shell watchdog (backstop).** The whole CraftOS-PC process is killed if it does not finish within `TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` (`.env.test` ships `3`; the recipe falls back to `3`). Keep it at or above the `3`s libtest default so libtest fires first; the watchdog only catches what Lua cannot interrupt. Override in your shell for slower local probes.
|
||||||
|
|
||||||
`just test-timeout` is a self-asserting regression guard for these layers and runs automatically as part of `just ci`. It chains `just test-timeout-lua` (proves libtest cancels a slow case at 0.1s, before a 2s shell backstop) and `just test-timeout-shell` (proves the 1s shell watchdog kills a slow case run with `--no-timeout`). Both drive the `tests/harness/slow-case.lua` fixture, which is never picked up by the normal `just test` suite (`runtest` skips `tests/` subdirectories).
|
`just test-timeout` is a self-asserting regression guard for these layers and runs automatically as part of `just ci`. It chains `just test-timeout-lua` (proves libtest cancels a slow case immediately with `--timeout 0`, before the shell backstop) and `just test-timeout-shell` (proves the `TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS` watchdog, default `1`, kills a slow case run with `--no-timeout`). Both drive the `tests/harness/slow-case.lua` fixture, which is never picked up by the normal `just test` suite (`runtest` skips `tests/` subdirectories).
|
||||||
|
|||||||
23
Justfile
23
Justfile
@ -6,14 +6,7 @@ default:
|
|||||||
@just --list
|
@just --list
|
||||||
|
|
||||||
# Install local development tooling.
|
# Install local development tooling.
|
||||||
install: init-env install-git-hooks check-install
|
install: install-git-hooks check-install
|
||||||
|
|
||||||
# Create a local environment file when one does not exist.
|
|
||||||
init-env:
|
|
||||||
@if [ ! -f .env ]; then \
|
|
||||||
cp .env.sample .env; \
|
|
||||||
printf '%s\n' 'Created .env from .env.sample'; \
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Install Git hooks for this repository.
|
# Install Git hooks for this repository.
|
||||||
install-git-hooks:
|
install-git-hooks:
|
||||||
@ -109,10 +102,10 @@ ci *args: check-craftos check
|
|||||||
|
|
||||||
# Run CraftOS-PC headless integration tests. Pass `--pretty` for grouped output.
|
# Run CraftOS-PC headless integration tests. Pass `--pretty` for grouped output.
|
||||||
test *args:
|
test *args:
|
||||||
@if [ -f .env ]; then set -a; . ./.env; set +a; fi; \
|
@if [ -f .env.test ]; then set -a; . ./.env.test; set +a; fi; \
|
||||||
pretty=0; \
|
pretty=0; \
|
||||||
verbose=0; \
|
verbose=0; \
|
||||||
timeout_seconds="${TRAP_CCLIBS_TEST_TIMEOUT_SECONDS:-7}"; \
|
timeout_seconds="${TRAP_CCLIBS_TEST_TIMEOUT_SECONDS:-3}"; \
|
||||||
case "$timeout_seconds" in ''|*[!0-9]*) printf '%s\n' 'TRAP_CCLIBS_TEST_TIMEOUT_SECONDS must be a positive integer' >&2; exit 1 ;; esac; \
|
case "$timeout_seconds" in ''|*[!0-9]*) printf '%s\n' 'TRAP_CCLIBS_TEST_TIMEOUT_SECONDS must be a positive integer' >&2; exit 1 ;; esac; \
|
||||||
if [ "$timeout_seconds" -lt 1 ]; then printf '%s\n' 'TRAP_CCLIBS_TEST_TIMEOUT_SECONDS must be >= 1' >&2; exit 1; fi; \
|
if [ "$timeout_seconds" -lt 1 ]; then printf '%s\n' 'TRAP_CCLIBS_TEST_TIMEOUT_SECONDS must be >= 1' >&2; exit 1; fi; \
|
||||||
for a in {{args}}; do case "$a" in --pretty) pretty=1 ;; --verbose|-v) pretty=1; verbose=1 ;; esac; done; \
|
for a in {{args}}; do case "$a" in --pretty) pretty=1 ;; --verbose|-v) pretty=1; verbose=1 ;; esac; done; \
|
||||||
@ -163,6 +156,7 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
repo='{{justfile_directory()}}'
|
repo='{{justfile_directory()}}'
|
||||||
|
if [ -f "$repo/.env.test" ]; then set -a; . "$repo/.env.test"; set +a; fi
|
||||||
rom_arg=""
|
rom_arg=""
|
||||||
if [ "$(uname -s)" = "Darwin" ]; then
|
if [ "$(uname -s)" = "Darwin" ]; then
|
||||||
rom_arg="--rom /Applications/CraftOS-PC.app/Contents/Resources"
|
rom_arg="--rom /Applications/CraftOS-PC.app/Contents/Resources"
|
||||||
@ -172,9 +166,10 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
|
|||||||
data_dir="$(mktemp -d)"
|
data_dir="$(mktemp -d)"
|
||||||
output_path="$data_dir/computer/0/trapos-test-output"
|
output_path="$data_dir/computer/0/trapos-test-output"
|
||||||
exec_code="shell.run('/programs/runtest.lua', '{{script}}', '--verbose', '--output', '/trapos-test-output', {{extra_flag}} '--shutdown')"
|
exec_code="shell.run('/programs/runtest.lua', '{{script}}', '--verbose', '--output', '/trapos-test-output', {{extra_flag}} '--shutdown')"
|
||||||
|
shell_timeout="{{shell_timeout}}"
|
||||||
craftos --directory "$data_dir" --headless $rom_arg $mount_arg --exec "$exec_code" >"$tmp" 2>&1 &
|
craftos --directory "$data_dir" --headless $rom_arg $mount_arg --exec "$exec_code" >"$tmp" 2>&1 &
|
||||||
pid="$!"
|
pid="$!"
|
||||||
( sleep {{shell_timeout}}; kill -TERM "$pid" >/dev/null 2>&1 ) &
|
( sleep "$shell_timeout"; kill -TERM "$pid" >/dev/null 2>&1 ) &
|
||||||
watchdog="$!"
|
watchdog="$!"
|
||||||
wait "$pid" >/dev/null 2>&1
|
wait "$pid" >/dev/null 2>&1
|
||||||
status="$?"
|
status="$?"
|
||||||
@ -194,7 +189,7 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
|
|||||||
;;
|
;;
|
||||||
shell)
|
shell)
|
||||||
if [ "$status" -eq 143 ]; then
|
if [ "$status" -eq 143 ]; then
|
||||||
printf '%s\n' "${green}OK${reset} shell watchdog killed the run after {{shell_timeout}}s (status 143; libtest timeout bypassed)"; \
|
printf '%s\n' "${green}OK${reset} shell watchdog killed the run after ${shell_timeout}s (status 143; libtest timeout bypassed)"; \
|
||||||
else
|
else
|
||||||
printf '%s\n' "${red}FAIL${reset} expected the shell watchdog to kill the run (status=$status)" >&2
|
printf '%s\n' "${red}FAIL${reset} expected the shell watchdog to kill the run (status=$status)" >&2
|
||||||
rc=1
|
rc=1
|
||||||
@ -212,11 +207,11 @@ _timeout-fixture script shell_timeout extra_flag expect: check-install
|
|||||||
|
|
||||||
# Prove the libtest (Lua) timeout layer: libtest cancels the slow case quickly,
|
# Prove the libtest (Lua) timeout layer: libtest cancels the slow case quickly,
|
||||||
# before the shell watchdog backstop can fire.
|
# before the shell watchdog backstop can fire.
|
||||||
test-timeout-lua: (_timeout-fixture "/tests/harness/slow-case.lua" "2" "'--timeout', '0.1'," "lua")
|
test-timeout-lua: (_timeout-fixture "/tests/harness/slow-case.lua" "${TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS:-1}" "'--timeout', '0'," "lua")
|
||||||
|
|
||||||
# Prove the shell watchdog backstop: the slow case runs with the libtest timeout
|
# Prove the shell watchdog backstop: the slow case runs with the libtest timeout
|
||||||
# bypassed (--no-timeout), so the shell watchdog kills the whole process.
|
# bypassed (--no-timeout), so the shell watchdog kills the whole process.
|
||||||
test-timeout-shell: (_timeout-fixture "/tests/harness/slow-case.lua" "1" "'--no-timeout'," "shell")
|
test-timeout-shell: (_timeout-fixture "/tests/harness/slow-case.lua" "${TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS:-1}" "'--no-timeout'," "shell")
|
||||||
|
|
||||||
# Fast regression guard for both timeout layers. Wired into `ci`.
|
# Fast regression guard for both timeout layers. Wired into `ci`.
|
||||||
test-timeout: test-timeout-lua test-timeout-shell
|
test-timeout: test-timeout-lua test-timeout-shell
|
||||||
|
|||||||
@ -35,11 +35,11 @@ that yield (the usual hang); a non-yielding CPU loop cannot be preempted in Comp
|
|||||||
|
|
||||||
**Layer 2 — shell watchdog (backstop).** The `Justfile` `test:` recipe keeps its existing
|
**Layer 2 — shell watchdog (backstop).** The `Justfile` `test:` recipe keeps its existing
|
||||||
`TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` watchdog unchanged, as an independent double-check. Its
|
`TRAP_CCLIBS_TEST_TIMEOUT_SECONDS` watchdog unchanged, as an independent double-check. Its
|
||||||
default sits *above* the libtest default (`.env.sample` ships `7`; the recipe falls back to
|
default matches the libtest default (`.env.test` ships `3`; the recipe falls back to `3`)
|
||||||
`7`) so libtest fires first in normal runs and the watchdog only catches what Lua cannot —
|
so libtest should fire first for yielding cases in normal runs and the watchdog only catches
|
||||||
a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its SIGTERM message
|
what Lua cannot — a non-yielding loop, a wedged libtest, or a deliberately bypassed case. Its
|
||||||
is worded differently from the `libtest timeout` message, so the two layers are never
|
SIGTERM message is worded differently from the `libtest timeout` message, so the two layers
|
||||||
confused.
|
are never confused.
|
||||||
|
|
||||||
## How To Write Tests Properly
|
## How To Write Tests Properly
|
||||||
|
|
||||||
@ -58,8 +58,9 @@ confused.
|
|||||||
- A hung case now fails in ~3s with a per-case message instead of taking down the whole
|
- A hung case now fails in ~3s with a per-case message instead of taking down the whole
|
||||||
process anonymously.
|
process anonymously.
|
||||||
- `just test-timeout` is a self-asserting harness regression guard wired into `just ci`. It
|
- `just test-timeout` is a self-asserting harness regression guard wired into `just ci`. It
|
||||||
chains `test-timeout-lua` (Layer 1: libtest cancels the slow case at 0.1s, before a 2s
|
chains `test-timeout-lua` (Layer 1: libtest cancels the slow case immediately with
|
||||||
shell backstop) and `test-timeout-shell` (Layer 2: the 1s watchdog kills the slow case with
|
`--timeout 0`, before the shell backstop) and `test-timeout-shell` (Layer 2: the
|
||||||
|
`TRAP_CCLIBS_TEST_TIMEOUT_WATCHDOG_SECONDS` watchdog, default `1`, kills the slow case with
|
||||||
libtest bypassed). Both drive a single `tests/harness/slow-case.lua` fixture; the tight
|
libtest bypassed). Both drive a single `tests/harness/slow-case.lua` fixture; the tight
|
||||||
timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself
|
timeouts — not the fixture's sleep length — decide which layer fires, so the harness itself
|
||||||
is covered against regressions on every `ci`.
|
is covered against regressions on every `ci`.
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user