Slop Test Detector

Agents generate tests that compile, pass, and increase coverage — but catch zero bugs. This skill detects those patterns programmatically.

When to use: Auditing existing test files for quality. Validating generated tests before writing them to disk. Enforcing the // Defect: comment convention.

When not to use: Evaluating test strategy (what to test). Measuring code coverage. Reviewing production code quality.

Rationalizations (Do Not Skip)

What To Protect (Start Here)

Before auditing or generating tests, identify which slop patterns apply:

Assertion API Support

The detector recognizes node:assert, vitest/Jest expect(), and chai expect() patterns:

Chain modifiers .not, .resolves, .rejects and chai language chains (.to, .be, .have, .been, etc.) are handled. Multiline chains (matcher on next line) are supported. Commented-out expect() calls are detected. Chai property assertions (expect(x).to.be.true) are supported.

Included Utilities

import {
  analyzeTestFile,
  validateTestBlock,
  formatReport,
  formatReportJSON,
  getPreset,
  parseImports,
  parseTestFile,
} from './slop-detector.ts';

Configuration

Presets

Three built-in presets control which rules are active:

import { getPreset } from './slop-detector.ts';

const config = getPreset('balanced'); // default
const strict = getPreset('strict');   // all rules enforced

Custom Configuration

import { type SlopConfig, getPreset } from './slop-detector.ts';

const config: SlopConfig = {
  ...getPreset('balanced'),
  assertionEquivalents: ['assertLogEntry', 'assertNoLogsAbove'],
  scoreThreshold: 90,
};

• enabledRules: Set<SlopRule> — which rules produce findings
• assertionEquivalents: string[] — function names treated as assertions (prevents false empty_test_body)
• scoreThreshold: number — minimum passing score (for CI gating)

Suppression Comments

Suppress specific findings with a required reason:

// slop-ignore: tautological_assertion — intentional canary test for CI pipeline health
it("canary", () => {
  assert.ok(true);
});

• The — reason is required — suppressions without a reason are ignored
• Multiple rules: // slop-ignore: empty_test_body, truthiness_only — stub pending #1234
• Place on the line before it() or anywhere inside the test body

Core Workflow

Step 1: Audit Existing Files

import { readFileSync } from 'node:fs';

const source = readFileSync('my-module.spec.ts', 'utf-8');
const report = analyzeTestFile(source, 'my-module.spec.ts');
console.log(formatReport(report));

Step 2: Report Findings with Impact

When presenting findings, always include:

• Rule name — the specific slop pattern (e.g., tautological_assertion)
• Test name and line number — so the user can navigate to it
• Severity — must-fail (test is structurally broken) vs should-fail (quality concern)
• Why it matters — what production bug could slip through because of this pattern
• What clean tests should NOT be flagged — avoid false positives on well-written tests

If a test file has assertion-equivalent helpers (functions named assert*() or test*() that internally call assert.*), configure assertionEquivalents to prevent false empty_test_body findings.

Step 3: Validate During Generation

Before writing a generated test to disk, check for slop:

const findings = validateTestBlock(`
  // Defect: if the parser miscounts blocks then all downstream rules produce wrong results
  it("parses correctly", () => {
    assert.equal(result, expected);
  });
`);

if (findings.some(f => f.severity === 'must-fail')) {
  // Fix before writing
}

Step 4: Machine-Readable Output (CI)

const report = analyzeTestFile(source, filePath, getPreset('balanced'));
const json = formatReportJSON(report);
// Outputs structured JSON with filePath, score, summary, findings[]

Step 5: Interpret the Score

The score formula weights must-fail findings (1.0) higher than should-fail (0.3):

score = max(0, round(100 × (1 - weightedFindings / testCount)))

• 100: No slop detected
• 90-99: Minor should-fail findings (missing comments, no negative tests)
• Below 80: Significant quality issues — review must-fail findings first

Violation Rules

opt-in rules are only active in the strict preset. Use getPreset('strict') or add them to a custom enabledRules set.

Definition of Done

• analyzeTestFile() returns zero must-fail findings
• All should-fail findings are reviewed and either fixed or justified
• Score is 90 or above
• slop-detector.spec.ts itself passes all its own rules

Companion Skills

• observability-testing — Structured log assertions that the slop detector validates for completeness
• fault-injection-testing — Circuit breaker and retry tests where empty_test_body and no_negative_test rules are most likely to fire
• breaking-change-detector — Contract classification tests where duplicate_assertion_set detects copy-paste patterns
• model-based-testing — State machine transition matrix tests where no_input_variation catches redundant transition checks
• pairwise-test-coverage — Combinatorial test generation where truthiness_only and assert_return_type_only are common slop patterns

See patterns.md for all 12 patterns with before/after code examples.