mirror of
https://github.com/msitarzewski/agency-agents.git
synced 2026-09-04 01:00:51 +03:00
Michael SitarzewskiandClaude Fable 5.1
04eadbd3e5
Adds scripts/test-convert-outputs.sh, the output half of the regression eval (the install half landed in #828), and wires it plus the previously un-wired test-agent-selection.sh (#779) into CI. Why an eval at all: every converter bug so far passed lint and the existing tests while the installed product was broken. #778 shipped a double-wrapped description that was valid YAML, so a wrapper check passed; #817 dropped a whole tool from --parallel and every remaining tool looked fine. Those are invariant violations, not syntax errors. Layer A (no history needed), for every agent x every converted tool: round-trip parsed(generated).description == source description strict-parse every generated frontmatter / TOML / YAML parses with a real parser (kimi/vibe carry only an identifier: id == slug and the prose file exists; aider/windsurf: "## Name" + description line) count every tool emits exactly one output per roster agent source every SOURCE frontmatter strict-parses and carries no leaked quote — the desktop app reads sources with js-yaml (#473) Layer B: scripts/convert-outputs.sha256, one aggregate hash per tool plus divisions.json / tools.json / runbooks.json. A flipped line means outputs or a contract changed; --update regenerates deliberately so review sees the blast radius. Date-stable (no generated file embeds a date). The expected side is derived by an INDEPENDENT strict parse of each source, never by lib.sh's get_field: the generator uses get_field, so an expected value derived the same way would move with a get_field bug and hide it — which is exactly how #778 stayed invisible. That independence found two shipping defects on the first green run: - get_field returned only the first line of a multi-line plain scalar. Three healthcare agents write their description as an indented continuation; every generated output for them shipped it truncated mid-sentence while the app showed the whole thing. get_field now folds continuation lines the way YAML does (newline -> single space). - get_field stripped only "field: " (one space). The same three files use column-aligned frontmatter (name: X), so their generated names carried leading whitespace in every tool's output. Plain-scalar padding is now trimmed. After both fixes get_field agrees with PyYAML on name and description for all 273 sources. Acceptance: re-introducing #778's double-wrap, a dropped tool, a divisions.json change, and an unquoted source each fail the eval (the double-wrap via Layer A round-trip, not only the manifest). Refs #778 #817 #473 #810 #826 #828 #779 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
33 lines
977 B
YAML
33 lines
977 B
YAML
name: Check Tools Consistency
|
|
|
|
# Runs on every PR (no path filter on purpose): a new or renamed tool must trip
|
|
# this check even when nobody touched tools.json or the install/convert scripts.
|
|
on:
|
|
pull_request:
|
|
push:
|
|
branches: [main]
|
|
|
|
jobs:
|
|
check-tools:
|
|
name: tools.json is the single source of truth
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Validate tool set
|
|
run: |
|
|
chmod +x scripts/check-tools.sh
|
|
./scripts/check-tools.sh
|
|
|
|
- name: Validate generated Hermes plugin
|
|
run: python3 scripts/check-hermes-plugin.py
|
|
|
|
- name: Validate converted YAML frontmatter
|
|
run: bash scripts/test-convert-frontmatter.sh
|
|
|
|
- name: Validate converted outputs (round-trip, strict parse, counts, drift)
|
|
run: bash scripts/test-convert-outputs.sh
|
|
|
|
- name: Validate agent selection (install.sh --agent / --agents-file)
|
|
run: bash scripts/test-agent-selection.sh
|