mirror of
https://github.com/msitarzewski/agency-agents.git
synced 2026-09-04 01:00:51 +03:00
Michael SitarzewskiandClaude Fable 5.1
04eadbd3e5
Adds scripts/test-convert-outputs.sh, the output half of the regression eval (the install half landed in #828), and wires it plus the previously un-wired test-agent-selection.sh (#779) into CI. Why an eval at all: every converter bug so far passed lint and the existing tests while the installed product was broken. #778 shipped a double-wrapped description that was valid YAML, so a wrapper check passed; #817 dropped a whole tool from --parallel and every remaining tool looked fine. Those are invariant violations, not syntax errors. Layer A (no history needed), for every agent x every converted tool: round-trip parsed(generated).description == source description strict-parse every generated frontmatter / TOML / YAML parses with a real parser (kimi/vibe carry only an identifier: id == slug and the prose file exists; aider/windsurf: "## Name" + description line) count every tool emits exactly one output per roster agent source every SOURCE frontmatter strict-parses and carries no leaked quote — the desktop app reads sources with js-yaml (#473) Layer B: scripts/convert-outputs.sha256, one aggregate hash per tool plus divisions.json / tools.json / runbooks.json. A flipped line means outputs or a contract changed; --update regenerates deliberately so review sees the blast radius. Date-stable (no generated file embeds a date). The expected side is derived by an INDEPENDENT strict parse of each source, never by lib.sh's get_field: the generator uses get_field, so an expected value derived the same way would move with a get_field bug and hide it — which is exactly how #778 stayed invisible. That independence found two shipping defects on the first green run: - get_field returned only the first line of a multi-line plain scalar. Three healthcare agents write their description as an indented continuation; every generated output for them shipped it truncated mid-sentence while the app showed the whole thing. get_field now folds continuation lines the way YAML does (newline -> single space). - get_field stripped only "field: " (one space). The same three files use column-aligned frontmatter (name: X), so their generated names carried leading whitespace in every tool's output. Plain-scalar padding is now trimmed. After both fixes get_field agrees with PyYAML on name and description for all 273 sources. Acceptance: re-introducing #778's double-wrap, a dropped tool, a divisions.json change, and an unquoted source each fail the eval (the double-wrap via Layer A round-trip, not only the manifest). Refs #778 #817 #473 #810 #826 #828 #779 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>