4.2 KiB
Evaluation corpus
Milestone 3 ships a small, reviewed characterization corpus under
tests/fixtures/evaluation. Its text was authored for this repository and is
covered by the MIT license. Labels name observable configured signals, never a
presumed author or generation method.
With the embedded schema-version 1 catalog, the corpus currently reports:
| Rule | True positive | False positive | False negative | Precision | Recall |
|---|---|---|---|---|---|
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
| assistant-follow-up-offer | 1 | 0 | 0 | 100% | 100% |
| assistant-revision-preface | 1 | 0 | 0 | 100% | 100% |
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
| templated-filler | 1 | 0 | 0 | 100% | 100% |
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
| editorial-insertion-placeholder | 1 | 0 | 0 | 100% | 100% |
| stock-helpful-closing | 1 | 0 | 0 | 100% | 100% |
| excess-vocabulary-cluster | 1 | 0 | 0 | 100% | 100% |
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
| source-omitted-code-placeholder | 1 | 0 | 0 | 100% | 100% |
| source-markdown-fence | 1 | 0 | 0 | 100% | 100% |
| unicode-ascii-replacement | 2 | 0 | 0 | 100% | 100% |
| unicode-invisible-control | 1 | 0 | 0 | 100% | 100% |
Aggregate labeled-rule counts are 20 true positives, 0 false positives, and 0 false negatives. The clean-document false-positive rate is 0/5 (0%). The test suite recomputes these values from the manifest and fails on drift.
The same manifest pins evidence scores for regression characterization:
| Corpus case | Score |
|---|---|
| six assistant disclaimer/preface rules | 64 |
| one prompt/response marker | 33 |
| style signals | 25 |
| source-code signals | 33 |
| typography signal | 17 |
| one ASCII lookalike and one invisible character | 28 |
| each of five negative documents | 0 |
These perfect values describe a tiny purpose-built regression set, not expected performance on real documents. They must not be used as an accuracy or authorship claim. Broader, independently sourced data and domain-specific review are required before tuning default weights or thresholds.
Default-policy sign-off
Each shipped rule has at least one explicitly labeled realistic near miss. The
initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds
unchanged. The prompt-leakage review accepted lighter 0.90ⁿ correlation
damping for distinct prompt-leakage rules while retaining 0.75ⁿ for other
categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings
replaceable by default, retains a non-mutating review policy, and keeps the
Gitea-compatible invisible/control profile diagnostic-only, with explicit
multilingual limitations. The Milestone 25 review gives one-to-one fullwidth
ASCII compatibility folds precedence over visual skeletons under every policy.
The Milestone 26 review adds narrow response and unfinished-editing residue,
keeps population-level vocabulary and stock closings deliberately low-weight,
and excludes language-proficiency proxies.
The checked-in signoff.toml records those decisions
and contains the SHA-256 digest of config.toml, heuristics.toml, and
unicode.toml. This binds the reviewed scoring behavior to both rule catalogs
and the default category policy.
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold change invalidates the sign-off test. Renewal requires reviewing positive and near-miss cases, reporting changed per-rule/aggregate metrics, explaining the false-positive tradeoff, updating the digest and decision text, and obtaining explicit maintainer approval in the change review. A green test alone is not approval to tune defaults.
Runtime diagnostic catalogs are deployment policy and are not part of the embedded-catalog digest. Their rules use the same validation and scoring formula, but teams must evaluate their own added patterns and near misses for their corpus rather than treating the embedded sign-off as validation of custom rules.