test(convert): regression eval for generated outputs + app contracts; fix two get_field bugs it found (#829)

Adds scripts/test-convert-outputs.sh, the output half of the regression eval
(the install half landed in #828), and wires it plus the previously
un-wired test-agent-selection.sh (#779) into CI.

Why an eval at all: every converter bug so far passed lint and the existing
tests while the installed product was broken. #778 shipped a double-wrapped
description that was valid YAML, so a wrapper check passed; #817 dropped a
whole tool from --parallel and every remaining tool looked fine. Those are
invariant violations, not syntax errors.

Layer A (no history needed), for every agent x every converted tool:
  round-trip   parsed(generated).description == source description
  strict-parse every generated frontmatter / TOML / YAML parses with a real
               parser (kimi/vibe carry only an identifier: id == slug and the
               prose file exists; aider/windsurf: "## Name" + description line)
  count        every tool emits exactly one output per roster agent
  source       every SOURCE frontmatter strict-parses and carries no leaked
               quote — the desktop app reads sources with js-yaml (#473)
Layer B: scripts/convert-outputs.sha256, one aggregate hash per tool plus
divisions.json / tools.json / runbooks.json. A flipped line means outputs or a
contract changed; --update regenerates deliberately so review sees the blast
radius. Date-stable (no generated file embeds a date).

The expected side is derived by an INDEPENDENT strict parse of each source,
never by lib.sh's get_field: the generator uses get_field, so an expected
value derived the same way would move with a get_field bug and hide it —
which is exactly how #778 stayed invisible. That independence found two
shipping defects on the first green run:

- get_field returned only the first line of a multi-line plain scalar.
  Three healthcare agents write their description as an indented
  continuation; every generated output for them shipped it truncated
  mid-sentence while the app showed the whole thing. get_field now folds
  continuation lines the way YAML does (newline -> single space).
- get_field stripped only "field: " (one space). The same three files use
  column-aligned frontmatter (name:        X), so their generated names
  carried leading whitespace in every tool's output. Plain-scalar padding
  is now trimmed.

After both fixes get_field agrees with PyYAML on name and description for
all 273 sources. Acceptance: re-introducing #778's double-wrap, a dropped
tool, a divisions.json change, and an unquoted source each fail the eval
(the double-wrap via Layer A round-trip, not only the manifest).

Refs #778 #817 #473 #810 #826 #828 #779

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Michael Sitarzewski
2026-09-03 07:58:13 -05:00
committed by GitHub
co-authored by Claude Fable 5.1
parent e88946efa3
commit 04eadbd3e5
4 changed files with 357 additions and 10 deletions
+16 -10
View File
@@ -20,17 +20,23 @@
get_field() {
local field="$1" file="$2"
awk -v f="$field" '
/^---$/ { fm++; next }
fm == 1 && $0 ~ "^" f ": " {
sub("^" f ": ", "")
# A quoted YAML scalar carries its quotes as delimiters, not content.
# Strip one matching outer pair and unescape, so a quoted source value
# never double-wraps when a converter re-quotes it (\047 is a literal
# apostrophe; this program sits inside shell single quotes).
if ($0 ~ /^".*"$/) { $0 = substr($0, 2, length($0) - 2); gsub(/\\"/, "\"", $0); gsub(/\\\\/, "\\", $0) }
else if ($0 ~ /^\047.*\047$/) { $0 = substr($0, 2, length($0) - 2); gsub(/\047\047/, "\047", $0) }
print; exit
# A quoted YAML scalar carries its quotes as delimiters, not content:
# strip one matching outer pair and unescape (\047 is a literal apostrophe;
# this program sits inside shell single quotes). A plain scalar may also
# continue onto indented lines; YAML folds those into one line joined by
# single spaces, and so do we — otherwise the generated description is
# silently truncated to its first line (three healthcare agents were).
function emit(v) {
sub(/^[ \t]+/, "", v); sub(/[ \t]+$/, "", v) # YAML: plain-scalar padding is not content
if (v ~ /^".*"$/) { v = substr(v, 2, length(v) - 2); gsub(/\\"/, "\"", v); gsub(/\\\\/, "\\", v) }
else if (v ~ /^\047.*\047$/) { v = substr(v, 2, length(v) - 2); gsub(/\047\047/, "\047", v) }
print v; printed = 1; exit
}
/^---$/ { fm++; if (fm == 2 && found) emit(val); next }
fm == 1 && !found && $0 ~ "^" f ": " { sub("^" f ": ", ""); val = $0; found = 1; next }
fm == 1 && found && /^[ \t]+[^ \t]/ { sub(/^[ \t]+/, ""); val = val " " $0; next }
fm == 1 && found { emit(val) }
END { if (found && !printed) emit(val) }
' "$file"
}