ci: add description-quality and collision gates

The description is the only signal an agent sees at discovery time, so
overlapping descriptions cause misrouting. Nothing in CI checked for that.

- tools/lint-descriptions.py enforces name==folder, description <=1024
  chars, terminal punctuation (a truncation canary), a trigger clause, a
  negative trigger, and a 500-line body cap. Pre-existing failures are
  grandfathered in tools/lint-baseline.json so this blocks new debt only;
  the baseline may shrink and never grow.
- tools/detect-collisions.py scores every description pair by TF-IDF
  cosine and ratchets the count of unreviewed near-duplicates. It strips
  negative-trigger clauses before vectorizing: those name the sibling
  skill on purpose, so scoring them would make correct disambiguation
  raise a pair's similarity.
- wire both into validate-skills.yml, along with agentskills conformance,
  an index.json freshness check, and a guard that fails the build if a
  regex frontmatter parser is reintroduced.
- broaden the path filters from tools/validate-skill.py to tools/**, as
  noted when #105 merged.

All five gates verified to fail on deliberately broken input.
This commit is contained in:
Mahipal
2026-08-23 17:15:12 +02:00
parent 796d96c413
commit a81b233649
5 changed files with 1459 additions and 2 deletions
+37 -2
View File
@@ -4,12 +4,12 @@ on:
push:
paths:
- 'skills/**'
- 'tools/validate-skill.py'
- 'tools/**'
- '.github/workflows/validate-skills.yml'
pull_request:
paths:
- 'skills/**'
- 'tools/validate-skill.py'
- 'tools/**'
- '.github/workflows/validate-skills.yml'
workflow_dispatch:
@@ -20,12 +20,47 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Install dependencies
run: pip install pyyaml
# All frontmatter is parsed by tools/skill_frontmatter.py (PyYAML). Any
# reintroduced regex parser silently truncates multi-line descriptions --
# that bug shipped 604/817 broken descriptions before it was caught.
- name: Guard against hand-rolled YAML parsers
run: |
if grep -rnE '(re\.(search|match|compile)\([^)]*description|^\s*description:.*\(\.\*\))' \
tools/ --include='*.py' ; then
echo "::error::Regex-based frontmatter parsing detected. Use tools/skill_frontmatter.py."
exit 1
fi
echo "OK: no regex frontmatter parsers"
# Single source of truth: tools/validate-skill.py validates required
# frontmatter fields, kebab-case name, description length, subdomain, and
# tag count. (Previously this step duplicated a weaker inline parser.)
- name: Validate SKILL.md frontmatter
run: python3 tools/validate-skill.py --all
# agentskills.io conformance: name==directory, 1..1024 description,
# reserved-word ban, angle-bracket injection check.
- name: Validate agentskills.io conformance
run: python3 tools/validate-agentskills.py --strict
# index.json is generated; a PR that changes a description must regenerate it.
- name: Check index.json is current
run: python3 tools/generate-index.py --check
# Description quality gate. Pre-existing failures are grandfathered in
# tools/lint-baseline.json so this blocks NEW debt only; the baseline is
# allowed to shrink and never to grow.
- name: Lint descriptions
run: python3 tools/lint-descriptions.py --all --stats
# Ratchet: the number of unreviewed near-duplicate description pairs may
# never increase. Lower this cap as disambiguation lands.
- name: Detect skill collisions
run: python3 tools/detect-collisions.py --max-unreviewed 55
- name: Check for duplicate skill names
run: |
python3 << 'EOF'