john-slop/docs/evaluation.md

66 lines
3.2 KiB
Markdown

# Evaluation corpus
Milestone 3 ships a small, reviewed characterization corpus under
`tests/fixtures/evaluation`. Its text was authored for this repository and is
covered by the MIT license. Labels name observable configured signals, never a
presumed author or generation method.
With the embedded schema-version 1 catalog, the corpus currently reports:
| Rule | True positive | False positive | False negative | Precision | Recall |
| --- | ---: | ---: | ---: | ---: | ---: |
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
| templated-filler | 1 | 0 | 0 | 100% | 100% |
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
| unicode-ascii-replacement | 2 | 0 | 0 | 100% | 100% |
| unicode-invisible-control | 1 | 0 | 0 | 100% | 100% |
Aggregate labeled-rule counts are 13 true positives, 0 false positives, and 0
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
suite recomputes these values from the manifest and fails on drift.
The same manifest pins evidence scores for regression characterization:
| Corpus case | Score |
| --- | ---: |
| four assistant disclaimer/preface rules | 60 |
| one prompt/response marker | 33 |
| style signals | 19 |
| source-code signals | 24 |
| typography signal | 17 |
| one ASCII lookalike and one invisible character | 28 |
| each of five negative documents | 0 |
These perfect values describe a tiny purpose-built regression set, not expected
performance on real documents. They must not be used as an accuracy or
authorship claim. Broader, independently sourced data and domain-specific review
are required before tuning default weights or thresholds.
## Default-policy sign-off
Each shipped rule has at least one explicitly labeled realistic near miss. The
initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds
unchanged. The prompt-leakage review accepted lighter `0.90ⁿ` correlation
damping for distinct prompt-leakage rules while retaining `0.75ⁿ` for other
categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings
replaceable by default, retains a non-mutating `review` policy, and keeps the
Gitea-compatible invisible/control profile diagnostic-only, with explicit
multilingual limitations. The checked-in `signoff.toml` records those decisions
and contains the SHA-256 digest of `config.toml`, `heuristics.toml`, and
`unicode.toml`. This binds the reviewed scoring behavior to both rule catalogs
and the default category policy.
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
change invalidates the sign-off test. Renewal requires reviewing positive and
near-miss cases, reporting changed per-rule/aggregate metrics, explaining the
false-positive tradeoff, updating the digest and decision text, and obtaining
explicit maintainer approval in the change review. A green test alone is not
approval to tune defaults.