From 54573734191790031cfe294c98d2a6b002136956 Mon Sep 17 00:00:00 2001 From: Michael Sitarzewski Date: Thu, 3 Sep 2026 09:23:28 -0500 Subject: [PATCH] docs(contributing): tell contributors about the output eval, and why it's there (#830) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the regression eval landed in #829 to the two places contributors actually read: item 8 of the tool-integration checklist (beside #772's install-suite item 7) and a new item 7 under "Before Submitting", the path every agent contributor takes. Adding or editing an agent changes the generated product and so flips the drift manifest; the text says so plainly, frames --update as the expected next step rather than a failure, and asks for the refreshed manifest to be committed alongside the change. Closes with a short note on why the checks exist: people are building remarkable things on these agents and thousands rely on them daily, so a slip in one converter reaches all of them at once — running the suite is how we keep that smooth for everyone downstream. Refs #829 #828 #772 Co-authored-by: Claude Fable 5.1 --- CONTRIBUTING.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d136a503..47708dcc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -245,6 +245,15 @@ Want agency-agents to install into a new tool (a CLI, editor, or agent runtime)? contract: where files land, that `--path` beats the tool's env var, that `--division` / `--agent` / `--agents-file` filter, that `--dry-run` writes nothing, and that paths with spaces survive. CI runs it on Linux and macOS. +8. **Run `./scripts/test-convert-outputs.sh`** — it must pass. It regenerates + every tool's output into a scratch directory and checks the *product*, not + the syntax: every agent's description round-trips intact, every generated + file parses with a real YAML/TOML parser, every tool emits exactly one output + per agent, and every source file parses the way the desktop app reads it. + When you've changed a converter on purpose it will report **manifest drift** + — that's expected. Look over what changed, run it again with `--update`, and + commit the refreshed `scripts/convert-outputs.sha256` so reviewers can see + the blast radius at a glance. CI runs it on every PR. If your PR commits the converted output (the generated `integrations//*` files), CI and review will ask you to remove it and add the `.gitignore` rule instead. @@ -301,6 +310,9 @@ We love ambitious ideas — a [Discussion](https://github.com/msitarzewski/agenc 4. **Define Metrics**: Include specific, measurable success criteria 5. **Proofread**: Check for typos, formatting issues, clarity 6. **Check it's original**: Run `./scripts/check-agent-originality.sh path/to/your-agent.md`. It compares your agent against the whole roster and flags near-duplicates (a swapped country/platform name won't fool it). A new agent should be genuinely new — if you're localizing for a market, make the platforms, tactics, and examples actually different, not a find-replace. +7. **Check it comes through every tool intact**: Run `./scripts/test-convert-outputs.sh`. It regenerates every tool's output and confirms your agent survives each converter — description round-tripped, files parsing, nothing dropped — and that its frontmatter parses the way the desktop app reads it. Adding or editing an agent changes the generated product, so it will report **manifest drift**; that's expected, not a failure on your part. Look over the change, run it again with `--update`, and commit the refreshed `scripts/convert-outputs.sha256` alongside your agent. CI runs the same check. + +A word on why these checks exist. People are building genuinely remarkable things on top of these agents, and thousands rely on them every day across a dozen different tools. That's wonderful — and it means a small slip in one converter, or a stray quote in one file, quietly reaches all of them at once. Running the suite locally is how we keep that smooth for everyone downstream. It takes about a minute, and it means your work arrives exactly as you wrote it, in every tool, for everyone. Thank you for taking the extra step — it's a real kindness to people you'll never meet. ### Submitting Your PR