Adds scripts/test-convert-outputs.sh, the output half of the regression eval
(the install half landed in #828), and wires it plus the previously
un-wired test-agent-selection.sh (#779) into CI.
Why an eval at all: every converter bug so far passed lint and the existing
tests while the installed product was broken. #778 shipped a double-wrapped
description that was valid YAML, so a wrapper check passed; #817 dropped a
whole tool from --parallel and every remaining tool looked fine. Those are
invariant violations, not syntax errors.
Layer A (no history needed), for every agent x every converted tool:
round-trip parsed(generated).description == source description
strict-parse every generated frontmatter / TOML / YAML parses with a real
parser (kimi/vibe carry only an identifier: id == slug and the
prose file exists; aider/windsurf: "## Name" + description line)
count every tool emits exactly one output per roster agent
source every SOURCE frontmatter strict-parses and carries no leaked
quote — the desktop app reads sources with js-yaml (#473)
Layer B: scripts/convert-outputs.sha256, one aggregate hash per tool plus
divisions.json / tools.json / runbooks.json. A flipped line means outputs or a
contract changed; --update regenerates deliberately so review sees the blast
radius. Date-stable (no generated file embeds a date).
The expected side is derived by an INDEPENDENT strict parse of each source,
never by lib.sh's get_field: the generator uses get_field, so an expected
value derived the same way would move with a get_field bug and hide it —
which is exactly how #778 stayed invisible. That independence found two
shipping defects on the first green run:
- get_field returned only the first line of a multi-line plain scalar.
Three healthcare agents write their description as an indented
continuation; every generated output for them shipped it truncated
mid-sentence while the app showed the whole thing. get_field now folds
continuation lines the way YAML does (newline -> single space).
- get_field stripped only "field: " (one space). The same three files use
column-aligned frontmatter (name: X), so their generated names
carried leading whitespace in every tool's output. Plain-scalar padding
is now trimmed.
After both fixes get_field agrees with PyYAML on name and description for
all 273 sources. Acceptance: re-introducing #778's double-wrap, a dropped
tool, a divisions.json change, and an unquoted source each fail the eval
(the double-wrap via Layer A round-trip, not only the manifest).
Refs #778#817#473#810#826#828#779
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Michael SitarzewskiandClaude Fable 5.1