john-slop/docs/evaluation.md

84 lines
4.2 KiB
Markdown

# Evaluation corpus
Milestone 3 ships a small, reviewed characterization corpus under
`tests/fixtures/evaluation`. Its text was authored for this repository and is
covered by the MIT license. Labels name observable configured signals, never a
presumed author or generation method.
With the embedded schema-version 1 catalog, the corpus currently reports:
| Rule | True positive | False positive | False negative | Precision | Recall |
| --- | ---: | ---: | ---: | ---: | ---: |
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
| assistant-follow-up-offer | 1 | 0 | 0 | 100% | 100% |
| assistant-revision-preface | 1 | 0 | 0 | 100% | 100% |
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
| templated-filler | 1 | 0 | 0 | 100% | 100% |
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
| editorial-insertion-placeholder | 1 | 0 | 0 | 100% | 100% |
| stock-helpful-closing | 1 | 0 | 0 | 100% | 100% |
| excess-vocabulary-cluster | 1 | 0 | 0 | 100% | 100% |
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
| source-omitted-code-placeholder | 1 | 0 | 0 | 100% | 100% |
| source-markdown-fence | 1 | 0 | 0 | 100% | 100% |
| unicode-ascii-replacement | 2 | 0 | 0 | 100% | 100% |
| unicode-invisible-control | 1 | 0 | 0 | 100% | 100% |
Aggregate labeled-rule counts are 20 true positives, 0 false positives, and 0
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
suite recomputes these values from the manifest and fails on drift.
The same manifest pins evidence scores for regression characterization:
| Corpus case | Score |
| --- | ---: |
| six assistant disclaimer/preface rules | 64 |
| one prompt/response marker | 33 |
| style signals | 25 |
| source-code signals | 33 |
| typography signal | 17 |
| one ASCII lookalike and one invisible character | 28 |
| each of five negative documents | 0 |
These perfect values describe a tiny purpose-built regression set, not expected
performance on real documents. They must not be used as an accuracy or
authorship claim. Broader, independently sourced data and domain-specific review
are required before tuning default weights or thresholds.
## Default-policy sign-off
Each shipped rule has at least one explicitly labeled realistic near miss. The
initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds
unchanged. The prompt-leakage review accepted lighter `0.90ⁿ` correlation
damping for distinct prompt-leakage rules while retaining `0.75ⁿ` for other
categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings
replaceable by default, retains a non-mutating `review` policy, and keeps the
Gitea-compatible invisible/control profile diagnostic-only, with explicit
multilingual limitations. The Milestone 25 review gives one-to-one fullwidth
ASCII compatibility folds precedence over visual skeletons under every policy.
The Milestone 26 review adds narrow response and unfinished-editing residue,
keeps population-level vocabulary and stock closings deliberately low-weight,
and excludes language-proficiency proxies.
The checked-in `signoff.toml` records those decisions
and contains the SHA-256 digest of `config.toml`, `heuristics.toml`, and
`unicode.toml`. This binds the reviewed scoring behavior to both rule catalogs
and the default category policy.
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
change invalidates the sign-off test. Renewal requires reviewing positive and
near-miss cases, reporting changed per-rule/aggregate metrics, explaining the
false-positive tradeoff, updating the digest and decision text, and obtaining
explicit maintainer approval in the change review. A green test alone is not
approval to tune defaults.
Runtime diagnostic catalogs are deployment policy and are not part of the
embedded-catalog digest. Their rules use the same validation and scoring
formula, but teams must evaluate their own added patterns and near misses for
their corpus rather than treating the embedded sign-off as validation of custom
rules.