john-slop/docs/evaluation.md

4.2 KiB

Evaluation corpus

Milestone 3 ships a small, reviewed characterization corpus under tests/fixtures/evaluation. Its text was authored for this repository and is covered by the MIT license. Labels name observable configured signals, never a presumed author or generation method.

With the embedded schema-version 1 catalog, the corpus currently reports:

Rule True positive False positive False negative Precision Recall
assistant-self-identification 1 0 0 100% 100%
assistant-capability-disclaimer 1 0 0 100% 100%
assistant-answer-preface 1 0 0 100% 100%
knowledge-cutoff-disclaimer 1 0 0 100% 100%
prompt-response-scaffolding 1 0 0 100% 100%
assistant-follow-up-offer 1 0 0 100% 100%
assistant-revision-preface 1 0 0 100% 100%
templated-conclusion 1 0 0 100% 100%
templated-filler 1 0 0 100% 100%
repeated-transition 1 0 0 100% 100%
editorial-insertion-placeholder 1 0 0 100% 100%
stock-helpful-closing 1 0 0 100% 100%
excess-vocabulary-cluster 1 0 0 100% 100%
source-placeholder-todo 1 0 0 100% 100%
source-tutorial-narration 1 0 0 100% 100%
source-omitted-code-placeholder 1 0 0 100% 100%
source-markdown-fence 1 0 0 100% 100%
unicode-ascii-replacement 2 0 0 100% 100%
unicode-invisible-control 1 0 0 100% 100%

Aggregate labeled-rule counts are 20 true positives, 0 false positives, and 0 false negatives. The clean-document false-positive rate is 0/5 (0%). The test suite recomputes these values from the manifest and fails on drift.

The same manifest pins evidence scores for regression characterization:

Corpus case Score
six assistant disclaimer/preface rules 64
one prompt/response marker 33
style signals 25
source-code signals 33
typography signal 17
one ASCII lookalike and one invisible character 28
each of five negative documents 0

These perfect values describe a tiny purpose-built regression set, not expected performance on real documents. They must not be used as an accuracy or authorship claim. Broader, independently sourced data and domain-specific review are required before tuning default weights or thresholds.

Default-policy sign-off

Each shipped rule has at least one explicitly labeled realistic near miss. The initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds unchanged. The prompt-leakage review accepted lighter 0.90ⁿ correlation damping for distinct prompt-leakage rules while retaining 0.75ⁿ for other categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings replaceable by default, retains a non-mutating review policy, and keeps the Gitea-compatible invisible/control profile diagnostic-only, with explicit multilingual limitations. The Milestone 25 review gives one-to-one fullwidth ASCII compatibility folds precedence over visual skeletons under every policy. The Milestone 26 review adds narrow response and unfinished-editing residue, keeps population-level vocabulary and stock closings deliberately low-weight, and excludes language-proficiency proxies. The checked-in signoff.toml records those decisions and contains the SHA-256 digest of config.toml, heuristics.toml, and unicode.toml. This binds the reviewed scoring behavior to both rule catalogs and the default category policy.

Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold change invalidates the sign-off test. Renewal requires reviewing positive and near-miss cases, reporting changed per-rule/aggregate metrics, explaining the false-positive tradeoff, updating the digest and decision text, and obtaining explicit maintainer approval in the change review. A green test alone is not approval to tune defaults.

Runtime diagnostic catalogs are deployment policy and are not part of the embedded-catalog digest. Their rules use the same validation and scoring formula, but teams must evaluate their own added patterns and near misses for their corpus rather than treating the embedded sign-off as validation of custom rules.