john-slop/docs/evaluation.md

2.8 KiB

Evaluation corpus

Milestone 3 ships a small, reviewed characterization corpus under tests/fixtures/evaluation. Its text was authored for this repository and is covered by the MIT license. Labels name observable configured signals, never a presumed author or generation method.

With the embedded schema-version 1 catalog, the corpus currently reports:

Rule True positive False positive False negative Precision Recall
assistant-self-identification 1 0 0 100% 100%
assistant-capability-disclaimer 1 0 0 100% 100%
assistant-answer-preface 1 0 0 100% 100%
knowledge-cutoff-disclaimer 1 0 0 100% 100%
prompt-response-scaffolding 1 0 0 100% 100%
templated-conclusion 1 0 0 100% 100%
templated-filler 1 0 0 100% 100%
repeated-transition 1 0 0 100% 100%
source-placeholder-todo 1 0 0 100% 100%
source-tutorial-narration 1 0 0 100% 100%
typography-non-ascii-hyphen 1 0 0 100% 100%

Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0 false negatives. The clean-document false-positive rate is 0/5 (0%). The test suite recomputes these values from the manifest and fails on drift.

The same manifest pins evidence scores for regression characterization:

Corpus case Score
four assistant disclaimer/preface rules 60
one prompt/response marker 33
style signals 19
source-code signals 24
typography signal 17
each of five negative documents 0

These perfect values describe a tiny purpose-built regression set, not expected performance on real documents. They must not be used as an accuracy or authorship claim. Broader, independently sourced data and domain-specific review are required before tuning default weights or thresholds.

Default-policy sign-off

Each shipped rule has at least one explicitly labeled realistic near miss. The 0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged. The renewed review accepts lighter 0.90ⁿ correlation damping for distinct prompt-leakage rules while retaining 0.75ⁿ for other categories. The checked-in signoff.toml records that decision, acknowledges the corpus limitations, and contains the unchanged SHA-256 digest of assets/heuristics.toml.

Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold change invalidates the sign-off test. Renewal requires reviewing positive and near-miss cases, reporting changed per-rule/aggregate metrics, explaining the false-positive tradeoff, updating the digest and decision text, and obtaining explicit maintainer approval in the change review. A green test alone is not approval to tune defaults.