mirror of
https://github.com/msitarzewski/agency-agents.git
synced 2026-09-04 09:10:50 +03:00
test(convert): regression eval for generated outputs + app contracts; fix two get_field bugs it found (#829)
Adds scripts/test-convert-outputs.sh, the output half of the regression eval (the install half landed in #828), and wires it plus the previously un-wired test-agent-selection.sh (#779) into CI. Why an eval at all: every converter bug so far passed lint and the existing tests while the installed product was broken. #778 shipped a double-wrapped description that was valid YAML, so a wrapper check passed; #817 dropped a whole tool from --parallel and every remaining tool looked fine. Those are invariant violations, not syntax errors. Layer A (no history needed), for every agent x every converted tool: round-trip parsed(generated).description == source description strict-parse every generated frontmatter / TOML / YAML parses with a real parser (kimi/vibe carry only an identifier: id == slug and the prose file exists; aider/windsurf: "## Name" + description line) count every tool emits exactly one output per roster agent source every SOURCE frontmatter strict-parses and carries no leaked quote — the desktop app reads sources with js-yaml (#473) Layer B: scripts/convert-outputs.sha256, one aggregate hash per tool plus divisions.json / tools.json / runbooks.json. A flipped line means outputs or a contract changed; --update regenerates deliberately so review sees the blast radius. Date-stable (no generated file embeds a date). The expected side is derived by an INDEPENDENT strict parse of each source, never by lib.sh's get_field: the generator uses get_field, so an expected value derived the same way would move with a get_field bug and hide it — which is exactly how #778 stayed invisible. That independence found two shipping defects on the first green run: - get_field returned only the first line of a multi-line plain scalar. Three healthcare agents write their description as an indented continuation; every generated output for them shipped it truncated mid-sentence while the app showed the whole thing. get_field now folds continuation lines the way YAML does (newline -> single space). - get_field stripped only "field: " (one space). The same three files use column-aligned frontmatter (name: X), so their generated names carried leading whitespace in every tool's output. Plain-scalar padding is now trimmed. After both fixes get_field agrees with PyYAML on name and description for all 273 sources. Acceptance: re-introducing #778's double-wrap, a dropped tool, a divisions.json change, and an unquoted source each fail the eval (the double-wrap via Layer A round-trip, not only the manifest). Refs #778 #817 #473 #810 #826 #828 #779 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
e88946efa3
commit
04eadbd3e5
+16
-10
@@ -20,17 +20,23 @@
|
||||
get_field() {
|
||||
local field="$1" file="$2"
|
||||
awk -v f="$field" '
|
||||
/^---$/ { fm++; next }
|
||||
fm == 1 && $0 ~ "^" f ": " {
|
||||
sub("^" f ": ", "")
|
||||
# A quoted YAML scalar carries its quotes as delimiters, not content.
|
||||
# Strip one matching outer pair and unescape, so a quoted source value
|
||||
# never double-wraps when a converter re-quotes it (\047 is a literal
|
||||
# apostrophe; this program sits inside shell single quotes).
|
||||
if ($0 ~ /^".*"$/) { $0 = substr($0, 2, length($0) - 2); gsub(/\\"/, "\"", $0); gsub(/\\\\/, "\\", $0) }
|
||||
else if ($0 ~ /^\047.*\047$/) { $0 = substr($0, 2, length($0) - 2); gsub(/\047\047/, "\047", $0) }
|
||||
print; exit
|
||||
# A quoted YAML scalar carries its quotes as delimiters, not content:
|
||||
# strip one matching outer pair and unescape (\047 is a literal apostrophe;
|
||||
# this program sits inside shell single quotes). A plain scalar may also
|
||||
# continue onto indented lines; YAML folds those into one line joined by
|
||||
# single spaces, and so do we — otherwise the generated description is
|
||||
# silently truncated to its first line (three healthcare agents were).
|
||||
function emit(v) {
|
||||
sub(/^[ \t]+/, "", v); sub(/[ \t]+$/, "", v) # YAML: plain-scalar padding is not content
|
||||
if (v ~ /^".*"$/) { v = substr(v, 2, length(v) - 2); gsub(/\\"/, "\"", v); gsub(/\\\\/, "\\", v) }
|
||||
else if (v ~ /^\047.*\047$/) { v = substr(v, 2, length(v) - 2); gsub(/\047\047/, "\047", v) }
|
||||
print v; printed = 1; exit
|
||||
}
|
||||
/^---$/ { fm++; if (fm == 2 && found) emit(val); next }
|
||||
fm == 1 && !found && $0 ~ "^" f ": " { sub("^" f ": ", ""); val = $0; found = 1; next }
|
||||
fm == 1 && found && /^[ \t]+[^ \t]/ { sub(/^[ \t]+/, ""); val = val " " $0; next }
|
||||
fm == 1 && found { emit(val) }
|
||||
END { if (found && !printed) emit(val) }
|
||||
' "$file"
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user