docs(web-test): спецификация регресс-движка + чистка regress.md

Новый канонический документ docs/web-test-regression-spec.md —
техническое описание движка регрессионных тестов: CLI, формат
тест-модулей, ctx-контракт, утверждения, три уровня хуков
(инфра/тест/контекст), конфиг, контексты Playwright и режимы
изоляции, форматы отчётов (JSON/Allure/JUnit), обнаружение тестов,
ошибки/таймауты/повторы, анализ результатов, глоссарий.

Документ предназначен для CI-интеграторов, ручного редактирования
сгенерированных тестов и сопровождения самого движка. Без дорожной
карты и внутренних self-тестов — только публичный контракт.

regress.md в скилле почищен: добавлены контракт ctx и список
утверждений (раньше модели приходилось читать исходники), срезаны
дубликаты с SKILL.md (live recon, паттерны catalog/document),
переформулированы анти-паттерны под специфику регресс-движка.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nick Shirokov
2026-05-18 23:01:38 +03:00
co-authored by Claude Opus 4.7
parent e93185c18b
commit f91b569564
2 changed files with 1127 additions and 162 deletions
+149 -162
View File
@@ -15,7 +15,6 @@ Tests live next to the project they cover (not inside the skill). Convention: `t
| Goal | Mode |
|------|------|
| Explore a form, prototype a single step, debug one selector | `exec` (interactive session) |
| **Walk through a scenario live before committing it as a test** | `exec` first, then `test` |
| Reproduce a bug as a failing test before fixing it | `test` |
| Cover a feature so future changes are checked automatically | `test` |
| Run the project's regression on a new build | `test` |
@@ -25,73 +24,15 @@ Don't write a `.test.mjs` for a one-shot user request. Don't drive a regression
## Before writing tests — recon
Two layers, in order. Don't skip either.
Two layers, in order.
### 1. Static recon — metadata
**1. Static recon — metadata.** Never invent identifiers. For every metadata object the user mentions, run the matching info skill first: `/meta-info` (attributes/tabular sections), `/form-info` (form layout), `/skd-info` (DCS), `/mxl-info` (templates), `/role-info` (rights), `/subsystem-info` (composition / command interface). If the user names objects you can't find — stop and ask.
Never invent identifiers. For every metadata object the user mentions (or that you decide to cover), run the matching info skill first:
**2. Live recon — interactive walkthrough.** For any non-trivial scenario, walk the path live in `exec` mode before transcribing it. Metadata tells you what exists; the live walkthrough tells you what actually happens. Capture from `getFormState()`: exact button names (`'Провести и закрыть'`, not `'Сохранить'`), table section names for multi-grid forms, required fields, places where a real async wait is needed. Then transcribe the working sequence into `*.test.mjs`, wrapping logical chunks in `step('...', async () => { ... })`.
| Object type | Skill |
|-------------|-------|
| Catalog/document/register attributes, tabular sections | `/meta-info` |
| Form layout — fields, buttons, tabs, tables | `/form-info` |
| DCS report — fields, parameters, filters | `/skd-info` |
| Spreadsheet template areas/parameters | `/mxl-info` |
| Role rights / restrictions | `/role-info` |
| Subsystem composition / command interface | `/subsystem-info` |
The mechanics of `exec` / `getFormState` / `fillFields` / `clickElement` are in [SKILL.md](SKILL.md) — read it before recon if you haven't already.
This gives the real Russian field labels, command names, column headers, table-section names. Without it, fuzzy matching will silently land on the wrong element, or fail with no useful diagnostic.
If the user names objects you cannot find: stop and ask. Do not guess.
### 2. Live recon — interactive walkthrough
For any non-trivial scenario, walk the path live in `exec` mode before writing it down. Metadata tells you what exists; the live walkthrough tells you what actually happens — which button posts the document, which dialog 1C raises, how the form looks after `clickElement('Создать')`, what fields are required, where `wait()` is genuinely needed.
```bash
# Start a session (background).
node $RUN start http://localhost:9191/myapp/ru_RU
# Step the scenario interactively. After each step, inspect.
cat <<'EOF' | node $RUN exec -
await navigateSection('Склад');
const cmds = await getCommands();
console.log(cmds);
EOF
cat <<'EOF' | node $RUN exec -
await openCommand('Приходная накладная');
await clickElement('Создать');
const s = await getFormState();
console.log(JSON.stringify(s.fields.map(f => ({ name: f.name, label: f.label, required: f.required })), null, 2));
console.log('buttons:', s.buttons.map(b => b.name));
console.log('tables:', s.tables.map(t => ({ name: t.name, label: t.label, columns: t.columns })));
EOF
# Try the actions you plan to encode. If a step fails, fix and re-try
# before transcribing it.
cat <<'EOF' | node $RUN exec -
await fillFields({ 'Контрагент': 'ООО Север' });
await fillTableRow({ 'Номенклатура': 'Товар 01', 'Количество': '5' },
{ table: 'Товары', add: true });
await clickElement('Провести и закрыть');
console.log(JSON.stringify(await getFormState()));
EOF
# When done, stop the session (or leave it for the next test you write).
node $RUN stop
```
What to record from the walkthrough into the test:
- Exact button names (`'Провести и закрыть'`, not `'Сохранить'`).
- Field labels as 1C renders them (with possible non-breaking spaces — `fillFields` normalises, but be exact).
- Table section names from `getFormState().tables[].name`/`label` for multi-grid forms.
- Required `wait()` durations — only where a real async event happens (report generation, server-side calculation). Default actions await internally.
- The shape of `getFormState()` after each action — gives you the right `assert.equal(...)` paths.
After this, transcribe the working sequence into `*.test.mjs`, wrap each chunk in `step('...', async () => { ... })`, add assertions for the invariants you saw. Run the file once with `node $RUN test path/to/file.test.mjs` to confirm.
When live recon is overkill: trivial reads (`navigateSection` + `readTable` + assert non-empty), or scenarios you've already proven once in this session. When it's essential: anything with confirmation dialogs, posting/cancellation flows, reports with custom filters, multi-grid forms, or user-customised forms you've never seen.
When live recon is overkill: trivial reads (`navigateSection` + `readTable` + assert non-empty), or scenarios you've already proven once in this session. When it's essential: confirmation dialogs, posting/cancellation flows, reports with custom filters, multi-grid forms, user-customised forms.
## Suite layout
@@ -99,30 +40,20 @@ When live recon is overkill: trivial reads (`navigateSection` + `readTable` + as
```
tests/
web-test/ # engine self-tests (reserved if our repo layout)
<app-name>/ # application regression — one per solution
_hooks.mjs
webtest.config.mjs
_allure/ # optional static Allure config
01-login/
02-counterparties/
...
<another-app>/ # second solution, fully isolated
_hooks.mjs
...
```
`<app-name>` is the project/extension slug (`acc-payroll`, `erp-customisation`, etc.). Pick something stable and pass it on the CLI:
```bash
node $RUN test tests/<app-name>/
```
Inside the application subfolder, organize by **feature**, not by metadata kind. Numeric prefixes on both folder and file enforce run order (discovery is alphabetic by full path).
Inside the application subfolder, organize by **feature**, not by metadata kind. Numeric prefixes on both folder and file enforce run order — discovery walks recursively and sorts files by full relative path; entries starting with `_` or `.` are skipped (so `_hooks.mjs`, `_allure/` won't be picked up as tests).
```
tests/<app-name>/
_hooks.mjs # stand prep + cross-cutting hooks (optional)
webtest.config.mjs # url, contexts, defaults (optional)
01-login/
01-open-base.test.mjs
02-section-navigation.test.mjs
@@ -132,15 +63,11 @@ tests/<app-name>/
03-goods-receipt/
01-fill.test.mjs
02-post.test.mjs
03-unpost.test.mjs
04-balance-report/
01-generate.test.mjs
02-warehouse-filter.test.mjs
05-approval-process/
01-end-to-end.test.mjs # multi-user
```
Per-folder `_hooks.mjs` / `webtest.config.mjs` inside the application subfolder are NOT supported. Only the application-root copies are loaded.
Per-folder `_hooks.mjs` / `webtest.config.mjs` inside the application subfolder are NOT supported — only the application-root copies are loaded.
## Test file anatomy
@@ -185,18 +112,95 @@ export default async function(ctx) {
}
```
The runner injects every `browser.mjs` export into `ctx` plus `assert`, `step`, `log`, `testInfo`, `testResult` (afterEach only). For multi-context tests, each context name is its own scoped namespace (`ctx.clerk.clickElement(...)` etc.) — `step`/`assert` stay top-level.
**Step names — in Russian, descriptive.** Step labels surface in the console output, in JSON/JUnit, and as Allure step nodes. Russian-speaking QA reads them. Use a full action phrase (`'Создать нового контрагента'`), not a tag (`'create'`) and not a transliteration. Same applies to `export const name` and `displayName` in `webtest.config.mjs`.
**Step names — in Russian, descriptive.** Step labels surface in the console output, in JSON/JUnit, and as Allure step nodes. Russian-speaking QA reads them. Use a full action phrase (`'Создать нового контрагента'`, `'Проверить наличие документа в списке'`), not a tag (`'create'`, `'verify'`) and not a transliteration. Same applies to `export const name` and `displayName` in `webtest.config.mjs`.
## `ctx` contract
The runner injects every `browser.mjs` export into `ctx` (all 1C action functions auto-detect platform errors — see SKILL.md), plus the test utilities below.
### Test utilities
```js
step(name, fn) // async wrapper. Records start/stop. Nested calls supported.
// On throw: marks the step failed, re-throws.
// On screenshot='every-step': captures after fn().
log(...args) // adds a line to ctx.testInfo's output (goes into JSON / Allure
// attachment). Use instead of console.log inside tests.
assert.* // see "Assertions" below
```
### `ctx.testInfo` (always set, read-only)
```js
{
name, // 'Навигация по разделам' (with params substituted)
file, // '01-navigation.test.mjs' (basename)
filePath, // relative path inside testDir
tags, // ['nav', 'smoke']
timeout, // ms
attempt, // 1..maxAttempts (1-based)
maxAttempts, // 1 + retry
param, // { ... } | undefined (only when export const params is set)
contexts: { // mirrors config.contexts; includes custom fields like displayName
clerk: { url, isolation, displayName, ... },
manager: { ... },
},
primaryContext, // 'clerk' — name of the context active at test entry
// (= t.context for single, t.contexts[0] for multi)
}
```
### `ctx.testResult` (only in `afterEach`)
```js
{
status, // 'passed' | 'failed'
duration, // ms
attempts, // attempts actually executed
error, // { message, step?, screenshot? } | null
steps, // array of step results (each: { name, start, stop, status, error?, steps[] })
}
```
### Context shape
- **Single-context (default or `export const context = 'manager'`):** all API on `ctx` top-level — `ctx.clickElement(...)`, `ctx.getFormState()`, etc.
- **Multi-context (`export const contexts = ['clerk', 'manager']`):** each name is its own scoped namespace — `ctx.clerk.clickElement(...)`, `ctx.manager.fillFields(...)`. `step`, `assert`, `log`, `testInfo` stay top-level. Scoped methods auto-switch the active page before each call.
## Assertions
All on `ctx.assert`. Throw `AssertionError` with `.message`, `.actual`, `.expected`. No dependencies.
```js
// generic
assert.ok(value, msg?) // truthy
assert.equal(actual, expected, msg?) // ===
assert.notEqual(actual, expected, msg?) // !==
assert.deepEqual(actual, expected, msg?) // JSON-compare
assert.includes(haystack, needle, msg?) // string.includes / array.includes
assert.match(string, regex, msg?) // regex.test(string)
await assert.throws(asyncFn, msg?) // passes if fn throws (use await)
// 1C-specific — operate on getFormState() / readTable() output
assert.formHasField(state, 'Контрагент', msg?) // state.fields[name] exists
assert.formTitle(state, expected, msg?) // state.title includes expected
assert.tableHasRow(table, predicate, msg?) // predicate: object (partial match) or fn(row) => bool
// object form: { 'Наименование': 'Тест' }
// fn form: r => r['Сумма'] > 100
assert.tableRowCount(table, expected, msg?) // table.rows.length === expected
assert.noErrors(state, msg?) // !state.errors
```
Beyond these, just use plain JS (`throw new Error(...)`) — there's no custom matcher extension API. The 1C-specific helpers are the ones worth preferring over hand-rolled equivalents because their error messages name the actual fields/rows present, which speeds up triage.
## webtest.config.mjs
```js
export default {
// Single-context: just url.
// Single-context shorthand:
url: 'http://localhost:9191/myapp/ru_RU',
// OR multi-context: named contexts. Each test picks via `context`/`contexts` exports.
// OR multi-context:
// contexts: {
// clerk: { url: 'http://localhost:9191/myapp-clerk/ru_RU', displayName: 'Кладовщик' },
// manager: { url: 'http://localhost:9191/myapp-manager/ru_RU', displayName: 'Менеджер' },
@@ -205,7 +209,7 @@ export default {
timeout: 30000,
retries: 0,
screenshot: 'on-failure',
screenshot: 'on-failure', // 'every-step' | 'off'
record: false,
// Severity → tags mapping for Allure. Each tag at most one bucket.
@@ -217,7 +221,7 @@ export default {
};
```
CLI flags override config. Recommend latin context IDs + Russian `displayName` for video badges.
CLI flags override config. Use latin context IDs + Russian `displayName` for ergonomics — `ctx.testInfo.contexts.clerk.displayName` is friendlier than mixed-case Cyrillic keys.
## _hooks.mjs
@@ -228,74 +232,44 @@ import { execSync } from 'child_process';
// Infra — runs once around the whole suite.
export async function prepare({ hookArgs, log, config }) {
// Restore DB, publish to Apache, build EPF, etc.
// hookArgs = everything after `--` on the CLI. Parse yourself.
if (hookArgs.includes('--rebuild-stand')) { /* full rebuild */ }
// Use idempotent hash-locks to skip work on warm starts.
// hookArgs: everything after `--` on the CLI, as a string[]. Parse yourself.
const force = hookArgs.includes('--rebuild-stand');
const dataArg = hookArgs.find(a => a.startsWith('--data='))?.slice('--data='.length);
log('preparing stand, force=', force, 'data=', dataArg);
// Idempotent hash-locks on inputs (config sources, EPF spec, DB dump) keep
// warm starts to a liveness probe.
}
export async function cleanup({ log, config }) {
// Tear down or leave the stand running. Choose per project.
}
export async function cleanup({ log, config }) { /* optional */ }
// Testlevel — runs with browser ctx.
export async function beforeAll(ctx) { /* once after first context opens */ }
export async function afterAll(ctx) { /* once before final teardown */ }
export async function beforeEach(ctx) { /* ctx.testInfo is set */ }
export async function afterEach(ctx) { /* ctx.testResult is set */ }
export async function afterEach(ctx) { /* ctx.testInfo + ctx.testResult set */ }
// Per-context — runs whenever a context is created/closed.
export async function afterOpenContext(ctx, name, spec) { /* spec = config.contexts[name] */ }
export async function beforeCloseContext(ctx, name, spec) { }
```
Built-in state reset (`dismissPendingErrors` + close all forms) runs after `afterEach` automatically. Don't reimplement it.
Built-in state reset (`dismissPendingErrors` + close all forms) runs after `afterEach` automatically. Don't reimplement it in `afterEach`.
Pass hook args after `--`:
```bash
node $RUN test tests/<app-name>/ --bail -- --rebuild-stand --data=demo
└─runner─┘ └────── hookArgs ─────────┘
```
**Where to put data setup:**
- DB restore, publication, EPF build → `prepare()`. Make it idempotent (hash-locks on inputs — config sources, EPF spec, DB dump) so warm starts skip everything but a liveness probe.
- Test-specific seed data (the document this test will edit, the counterparty it expects) → per-test `setup`.
- DB restore, publication, EPF build → `prepare()`. Make it idempotent (hash-locks).
- Test-specific seed data → per-test `setup`.
- Shared session-wide warmup → `beforeAll`.
## Ready-to-paste patterns
### Catalog full cycle
```js
await step('Создать контрагента', async () => {
await navigateSection('Продажи');
await openCommand('Контрагенты');
await clickElement('Создать');
await fillFields({ 'Наименование': 'ТД Тест', 'ИНН': '7707083893' });
await clickElement('Записать и закрыть');
});
await step('Проверить наличие в списке', async () => {
const t = await readTable({ maxRows: 50 });
assert.tableHasRow(t, { 'Наименование': 'ТД Тест' });
});
await step('Удалить контрагента и подтвердить удаление', async () => {
await clickElement('ТД Тест');
const page = await getPage();
await page.keyboard.press('Delete');
await clickElement('Да');
});
```
### Document create + post
```js
const marker = 'Тест-' + Date.now();
await openCommand('Приходная накладная');
await clickElement('Создать');
await fillFields({ 'Контрагент': 'ООО Север', 'Комментарий': marker });
await fillTableRow(
{ 'Номенклатура': 'Товар 01', 'Количество': '5', 'Цена': '100' },
{ table: 'Товары', add: true }
);
await clickElement('Провести и закрыть');
// Verify: re-open list, filter or scan, assert by `marker`.
```
Use a unique marker (`Date.now()` or random suffix) so re-runs don't collide. Identify your own row by it, not by position or natural keys that may already exist in the DB.
A minimal CRUD shape is in *Test file anatomy* above — use it as the rhythm for catalog/document tests, swapping in the right section/command/fields. The patterns below cover what's specific to the regression engine, not the browser API (those live in SKILL.md).
### DCS report
@@ -335,7 +309,7 @@ export default async function({ clerk, manager, step, assert }) {
});
await step('Кладовщик видит новый статус', async () => {
const s = await clerk.getFormState();
assert.equal(s.fields.find(f => f.name === 'Статус')?.value, 'Утверждён');
assert.equal(s.fields['Статус']?.value, 'Утверждён');
});
await step('Освободить сессию кладовщика', async () => {
await manager.closeContext('clerk'); // free a 1C license for the next test
@@ -343,7 +317,7 @@ export default async function({ clerk, manager, step, assert }) {
}
```
License caveat: stock 1C allows ~2 web sessions concurrently. Close contexts you no longer need before the next multi-user test starts.
Close contexts you no longer need (`manager.closeContext('clerk')`) before the next multi-user test starts — frees a 1C web-client license and stops the previous role from holding state.
### Failing-test repro
@@ -356,37 +330,50 @@ export default async function({ openCommand, clickElement, getFormState, assert,
await clickElement('Создать');
await clickElement('Провести');
const s = await getFormState();
assert.ok(s.errorModal || s.fields.find(f => f.name === 'Контрагент')?.required,
assert.ok(s.errorModal || s.fields['Контрагент']?.required,
'Должна быть ошибка валидации или поле помечено обязательным');
}
```
Write it red first, hand it to the user, fix the underlying issue, re-run green.
### Parameterised test
```js
export const name = 'Заполнение поля {type}';
export const params = [
{ type: 'String', field: 'Наименование', value: 'Тест' },
{ type: 'Number', field: 'Цена', value: '100.50' },
{ type: 'Date', field: 'ДатаПоступления', value: '01.01.2024' },
];
export default async function({ fillFields, getFormState, assert }, { type, field, value }) {
await fillFields({ [field]: value });
const state = await getFormState();
assert.equal(state.fields[field]?.value, String(value));
}
```
Each `params` entry becomes its own test in the report. `{key}` placeholders in `name` get substituted; without placeholders, a `[index]` suffix is added. `ctx.testInfo.param` carries the current row.
## Running
```bash
node $RUN test tests/<app-name>/ # full app suite
node $RUN test tests/<app-name>/03-goods-receipt/ # one feature folder
node $RUN test tests/<app-name>/ # full app suite
node $RUN test tests/<app-name>/03-goods-receipt/ # one feature folder
node $RUN test tests/<app-name>/02-counterparties/01-create.test.mjs # one file
node $RUN test tests/<app-name>/ --tags=smoke # by tag (intersection)
node $RUN test tests/<app-name>/ --grep='накладн' # by name regex
node $RUN test tests/<app-name>/ --bail --retry=1 # stop on first fail, allow 1 retry
node $RUN test tests/<app-name>/ --tags=smoke # by tag (intersection)
node $RUN test tests/<app-name>/ --grep='накладн' # by name regex
node $RUN test tests/<app-name>/ --bail --retry=1 # stop on first fail, allow 1 retry
node $RUN test tests/<app-name>/ --report=allure-results --format=allure --report-dir=allure-results
node $RUN test tests/<app-name>/ -- --rebuild-stand # everything after `--` goes to hooks
node $RUN test tests/<app-name>/ -- --rebuild-stand # after `--` hookArgs
```
Default report is JSON when `--report=…` is given. Allure needs `--format=allure` + a directory. JUnit similarly with `--format=junit`.
### Allure static config — `_allure/` directory
### Allure static config — `_allure/`
The runner copies `<testDir>/_allure/` into the report directory before generating Allure output. Standard Allure convention applies — three files are typically used:
- **`categories.json`** — failure classification. Always emit this when setting up a suite, with 1C-specific patterns: license pool exhaustion (`Не обнаружено свободной лицензии`), 1C application errors (`ВызватьИсключение|Произошла ошибка|…`), navigation/element lookup misses, runner timeouts, assertion failures.
- **`environment.properties`** — `key=value` lines for the Environment widget. Useful when the suite runs across builds/branches (URL, 1C platform version, git branch, configuration version). Often emitted dynamically by `prepare()` rather than committed as a static file.
- **`executor.json`** — CI metadata (Jenkins URL, GitHub run ID, etc.). Only relevant when the suite runs on a CI server; for local runs, skip it.
Discovery skips the underscored directory, so it never collides with tests.
The runner copies `<testDir>/_allure/` into the report directory before generating Allure output. Drop in `categories.json` (regex-based failure classification — useful for 1C-specific buckets: license pool exhaustion, platform exceptions, runner timeouts, assertion failures), `environment.properties` (optional, often emitted dynamically by `prepare()`), `executor.json` (CI metadata, skip locally). The underscore prefix keeps the directory out of test discovery.
## Severity guidance
@@ -404,26 +391,26 @@ Don't promote everything to `critical` — it loses signal in the Allure dashboa
## Anti-patterns
- **Sleeps as a substitute for assertions.** `wait(5)` after `openCommand` is fine; `wait(30)` because something flakes is a bug — find what state you can wait on with `getFormState` instead.
- **Sleeps as a substitute for assertions.** `wait(5)` after `openCommand` is fine; `wait(30)` because something flakes is a bug — wait on `getFormState` instead.
- **Retry as a substitute for understanding.** "Not found" twice means the data isn't there or the label is wrong. Don't loop.
- **Raw DOM via `getPage().$$(...)`.** Use `getFormState`, `readTable`, `readSpreadsheet`. Raw selectors break across 1C platform versions.
- **`clickElement('×')` or `clickElement('Закрыть')`** to dismiss a form. Use `closeForm({ save: true|false })` — handles confirmation correctly.
- **Position-based row identification** (`rows[0]`) when the DB has shared seed data. Filter by unique marker or label instead.
- **Skipping recon** because "I know what this catalog looks like." You don't — the project's customisation almost certainly differs from a stock config.
- **Position-based row identification** (`rows[0]`) when the DB has shared seed data. Filter by a unique marker (`Date.now()` suffix) instead.
- **Hand-writing reset code in `afterEach`.** The runner already closes forms and dismisses errors after the hook.
- **Cross-test state assumptions.** Each test must start from the desktop and seed its own data. Order-of-execution coupling is a regression-suite trap.
- **`tags: ['smoke']` on a 90-second test.** Smoke means fast.
- **Hand-writing reset code** in `afterEach`. The runner already closes forms and dismisses errors.
- **Cross-test state assumptions.** Each test must start from desktop and seed its own data. Order-of-execution coupling is a regression-suite trap.
- **Skipping recon** because "I know what this catalog looks like." The project's customisation almost certainly differs from stock.
(General browser-API anti-patterns — raw DOM, `clickElement('Закрыть')` instead of `closeForm()` — live in SKILL.md.)
## After a run — failure triage
1. Scan the JSON or Allure summary for `failed`.
2. For each failure, read `error.message` + `error.step` + screenshot (saved next to the report).
2. For each failure, read `error.message` + `error.step` + screenshot.
3. If `error.onecError.stack` is present — it's a 1C exception, look at the platform trace.
4. Classify:
- **Test bug** — selector wrong, expectation wrong, race with no anchor → fix the test.
- **Application bug** — actual misbehaviour reproduced → report to the user with the failing step name and the platform stack.
- **Stand flake** — Apache timeout, login form not loading, license shortage → fix the hook idempotency or session-cleanup logic, not the test.
5. After fixes, re-run only the affected files (`node $RUN test tests/03-goods-receipt/`) before the full suite.
5. After fixes, re-run only the affected files before the full suite.
Report back to the user with the classification, not raw failure dumps.