claudekit/skills/testing-anti-patterns/references/anti-pattern-catalog.md

Testing Anti-Pattern Catalog

Quick reference of common testing anti-patterns. Each entry includes: what it looks like, why it's a problem, and how to fix it.

---

1. The Ice Cream Cone

Symptom: Most tests are E2E or integration tests. Few or no unit tests. The test pyramid is inverted.

Root cause: Tests were written after the feature, following user flows instead of testing isolated logic. Or the code is tightly coupled, making unit tests hard to write.

Impact: Test suite is slow, brittle, and expensive to maintain. Failures are hard to diagnose because tests cover too much at once.

Fix: Refactor toward the test pyramid. Extract business logic into pure functions and unit test them. Reserve E2E tests for critical user flows only. Target ratio: 70% unit, 20% integration, 10% E2E.

---

2. The Mockery

Symptom: Tests mock so aggressively that they're testing the mocks, not the actual code. The thing under test has all its dependencies replaced.

Root cause: Code has too many dependencies, or the developer equates "isolated" with "mock everything."

Impact: Tests pass even when the real code is broken. Refactoring breaks every test because mocks are coupled to implementation details.

Fix: Only mock external boundaries (network, database, filesystem, clock). Use real implementations for internal collaborators. If you need too many mocks, the code has too many dependencies — refactor first.

---

3. The Slow Suite

Symptom: Test suite takes more than a few minutes to run. Developers skip tests locally and only run them in CI.

Root cause: Too many integration/E2E tests, tests that hit real databases or network, no test parallelization, expensive setup/teardown.

Impact: Developers stop running tests, bugs slip through, CI becomes a bottleneck.

Fix:

• Profile the suite to find the slowest tests
• Replace slow integration tests with fast unit tests where possible
• Use in-memory databases for integration tests
• Parallelize test execution
• Target: unit suite under 30 seconds, full suite under 5 minutes

---

4. The Flaky Test

Symptom: Test passes most of the time but fails unpredictably. Re-running usually makes it pass.

Root cause: Race conditions, time-dependent logic, shared mutable state between tests, reliance on external services, non-deterministic ordering.

Impact: Team loses trust in tests. "Oh that one's flaky" becomes an excuse to ignore real failures. CI results become meaningless.

Fix:

• Isolate the flaky test and run it 100 times to confirm flakiness
• Check for: shared state, date/time usage, async timing, test ordering
• Fix the root cause (don't just add retries)
• Quarantine truly unfixable flaky tests while investigating

---

5. The Assertion-Free Test

Symptom: Test runs code but doesn't assert anything meaningful. It only checks that no exception was thrown.

# Bad — this tests nothing useful
def test_process_data():
    process_data(sample_input)  # No assertion

Root cause: Test was written to hit a coverage target rather than verify behavior.

Impact: False sense of security. Code "has tests" but bugs go undetected.

Fix: Every test must assert on the outcome. Ask: "What behavior am I verifying?" If you can't answer, the test isn't testing anything.

# Good — asserts the actual behavior
def test_process_data_calculates_total():
    result = process_data(sample_input)
    assert result.total == 42.0

---

6. The Copy-Paste Test

Symptom: Test file has blocks of nearly identical code repeated with minor variations. Tests are long and look like each other.

Root cause: Developer tested a new case by copying an existing test and tweaking values instead of extracting a pattern.

Impact: Maintenance nightmare. A change to the interface requires updating dozens of near-identical tests. Easy to introduce subtle bugs in copies.

Fix: Use parameterized tests for variations on the same behavior:

# Python — pytest.mark.parametrize
@pytest.mark.parametrize("input,expected", [
    ("hello", "HELLO"),
    ("", ""),
    ("123", "123"),
])
def test_to_upper(input, expected):
    assert to_upper(input) == expected

// TypeScript — test.each (vitest/jest)
test.each([
  ["hello", "HELLO"],
  ["", ""],
  ["123", "123"],
])("to_upper(%s) returns %s", (input, expected) => {
  expect(toUpper(input)).toBe(expected);
});

---

7. The Time Bomb

Symptom: Test passes today but will fail on a future date, or fails on certain days/times (new year, month boundary, DST change, leap year).

Root cause: Test uses Date.now(), new Date(), or similar without controlling the clock. Assertions are hardcoded to specific dates.

Impact: Sudden failures on specific dates. CI breaks on January 1, or during DST transitions.

Fix: Always inject or mock the clock:

# Python — freeze time
from freezegun import freeze_time

@freeze_time("2025-06-15T12:00:00Z")
def test_expiry_check():
    assert is_expired(created_at="2025-06-14T12:00:00Z", ttl_hours=23)

// TypeScript — vitest fake timers
vi.useFakeTimers();
vi.setSystemTime(new Date("2025-06-15T12:00:00Z"));
expect(isExpired(createdAt, 23)).toBe(true);
vi.useRealTimers();

---

8. The Hidden Dependency

Symptom: Test passes locally but fails in CI, or fails when run in isolation but passes as part of the full suite.

Root cause: Test depends on external state that isn't set up by the test itself: a running database, a file on disk, an environment variable, output from a previous test, or global state modified by another test.

Impact: Tests are order-dependent, environment-dependent, and unreliable. Debugging failures requires understanding the entire test suite's execution order.

Fix:

• Each test must set up and tear down its own state
• Use fixtures (pytest fixtures, beforeEach/afterEach) for shared setup
• Run tests in random order to catch hidden dependencies

  pytest -p randomly    # Python
  vitest --sequence.shuffle  # vitest
  

• Never rely on test execution order

---

Quick Decision Table

Symptom	Likely Anti-Pattern	First Action
Tests are slow	Ice Cream Cone or Slow Suite	Profile, find the slowest tests
Tests break on refactor	The Mockery	Reduce mocks, test behavior not implementation
Tests fail randomly	Flaky Test	Isolate and run 100x
High coverage but bugs slip through	Assertion-Free Test	Audit assertions in coverage-targeted tests
Tests are hard to maintain	Copy-Paste Test	Extract parameterized tests
Tests fail on certain dates	Time Bomb	Inject/mock the clock
Tests fail in CI only	Hidden Dependency	Run locally in random order
Tests pass but code is clearly broken	The Mockery or Assertion-Free	Check what's actually being asserted