john-slop/docs/evaluation.md

32 lines
1.6 KiB
Markdown

# Evaluation corpus
Milestone 3 ships a small, reviewed characterization corpus under
`tests/fixtures/evaluation`. Its text was authored for this repository and is
covered by the MIT license. Labels name observable configured signals, never a
presumed author or generation method.
With the embedded schema-version 1 catalog, the corpus currently reports:
| Rule | True positive | False positive | False negative | Precision | Recall |
| --- | ---: | ---: | ---: | ---: | ---: |
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
| templated-filler | 1 | 0 | 0 | 100% | 100% |
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
| typography-non-ascii-hyphen | 1 | 0 | 0 | 100% | 100% |
Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0
false negatives. The clean-document false-positive rate is 0/1 (0%). The test
suite recomputes these values from the manifest and fails on drift.
These perfect values describe a tiny purpose-built regression set, not expected
performance on real documents. They must not be used as an accuracy or
authorship claim. Broader, independently sourced data and domain-specific review
are required before tuning default weights or thresholds.