47 lines
2.4 KiB
Markdown
47 lines
2.4 KiB
Markdown
# Evaluation corpus
|
|
|
|
Milestone 3 ships a small, reviewed characterization corpus under
|
|
`tests/fixtures/evaluation`. Its text was authored for this repository and is
|
|
covered by the MIT license. Labels name observable configured signals, never a
|
|
presumed author or generation method.
|
|
|
|
With the embedded schema-version 1 catalog, the corpus currently reports:
|
|
|
|
| Rule | True positive | False positive | False negative | Precision | Recall |
|
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
|
|
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
|
|
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
|
|
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
|
|
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
|
|
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
|
|
| templated-filler | 1 | 0 | 0 | 100% | 100% |
|
|
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
|
|
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
|
|
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
|
|
| typography-non-ascii-hyphen | 1 | 0 | 0 | 100% | 100% |
|
|
|
|
Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0
|
|
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
|
|
suite recomputes these values from the manifest and fails on drift.
|
|
|
|
These perfect values describe a tiny purpose-built regression set, not expected
|
|
performance on real documents. They must not be used as an accuracy or
|
|
authorship claim. Broader, independently sourced data and domain-specific review
|
|
are required before tuning default weights or thresholds.
|
|
|
|
## Default-policy sign-off
|
|
|
|
Each shipped rule has at least one explicitly labeled realistic near miss. The
|
|
0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged
|
|
and accepts them only as a conservative release baseline. The checked-in
|
|
`signoff.toml` records that decision, acknowledges the corpus limitations, and
|
|
contains the SHA-256 digest of `assets/heuristics.toml`.
|
|
|
|
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
|
|
change invalidates the sign-off test. Renewal requires reviewing positive and
|
|
near-miss cases, reporting changed per-rule/aggregate metrics, explaining the
|
|
false-positive tradeoff, updating the digest and decision text, and obtaining
|
|
explicit maintainer approval in the change review. A green test alone is not
|
|
approval to tune defaults.
|