test: renew prompt scoring sign-off

This commit is contained in:
Dandaedre 2026-09-29 21:00:29 +02:00
parent d6163076d9
commit 7d365af5df
Signed by: Dandaedre
GPG Key ID: 440C6592E8A26D99
9 changed files with 60 additions and 16 deletions

View File

@ -5,4 +5,4 @@ coverage = 92.19
[last_run_clean.run_duration]
secs = 29
nanos = 930030912
nanos = 350250460

View File

@ -45,8 +45,8 @@ Run line coverage with Cargo Tarpaulin:
cargo tarpaulin --config tarpaulin.toml
```
The configuration enforces a 90% floor. The current suite measured 92.16%
(1329/1442 lines) on 2026-09-29. The remaining lines are dominated by
The configuration enforces a 90% floor. The current suite measured 92.19%
(1333/1446 lines) on 2026-09-29. The remaining lines are dominated by
platform/I/O failure injection, malformed archive internals, coordinate
overflow, LSP logging/reload failures, and subprocess attribution. Do not add
brittle tests or production seams solely to make Tarpaulin count those paths.

View File

@ -25,6 +25,17 @@ Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
suite recomputes these values from the manifest and fails on drift.
The same manifest pins confidence scores for regression characterization:
| Corpus case | Score |
| --- | ---: |
| four assistant disclaimer/preface rules | 60 |
| one prompt/response marker | 33 |
| style signals | 19 |
| source-code signals | 24 |
| typography signal | 17 |
| each of five negative documents | 0 |
These perfect values describe a tiny purpose-built regression set, not expected
performance on real documents. They must not be used as an accuracy or
authorship claim. Broader, independently sourced data and domain-specific review
@ -33,10 +44,11 @@ are required before tuning default weights or thresholds.
## Default-policy sign-off
Each shipped rule has at least one explicitly labeled realistic near miss. The
0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged
and accepts them only as a conservative release baseline. The checked-in
0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged.
The renewed review accepts lighter `0.90ⁿ` correlation damping for distinct
prompt-leakage rules while retaining `0.75ⁿ` for other categories. The checked-in
`signoff.toml` records that decision, acknowledges the corpus limitations, and
contains the SHA-256 digest of `assets/heuristics.toml`.
contains the unchanged SHA-256 digest of `assets/heuristics.toml`.
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
change invalidates the sign-off test. Renewal requires reviewing positive and

View File

@ -20,8 +20,12 @@ Findings beyond the cap remain visible but do not increase the score.
Rules are grouped by category as correlated evidence families. The first
matching rule in a category receives its full occurrence-adjusted contribution;
later distinct rules receive successive factors of `0.75`. This prevents a
cluster of closely related phrases from being treated as independent proof.
later distinct prompt-leakage rules receive successive factors of `0.90`, while
style, typography, and source-code rules receive successive factors of `0.75`.
Prompt/response markers and copied assistant disclaimers are stronger concrete
artifacts than generic stylistic similarities, but the remaining damping still
prevents a cluster of related phrases from being treated as fully independent
proof.
If the resulting evidence sum is `E`, the displayed score is
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
@ -38,3 +42,8 @@ to explain the score.
Weights and thresholds are review policy, not calibrated probabilities. A
score of 75 does not mean a 75% chance that AI wrote the input.
The diagnostic showcase exercises five distinct prompt-leakage rules, including
two prompt/response markers. Together with its style and typography findings it
produces evidence score 71 (medium). This is a deterministic policy score, not
an estimated probability.

View File

@ -274,15 +274,19 @@ and builds without `fancy-diagnostics` retain the plain human fallback.
- [x] Give distinct prompt-leakage rules lighter correlation damping (`0.90ⁿ`)
while preserving `0.75ⁿ` damping for style, typography, and source-code
families and preserving per-rule occurrence damping/caps.
- [ ] Re-run the reviewed corpus, renew the existing default-policy sign-off,
- [x] Re-run the reviewed corpus, renew the existing default-policy sign-off,
and document the changed formula and showcase score without changing the
sign-off mechanism itself.
sign-off mechanism itself.
Acceptance: multiple independent prompt-leakage artifacts contribute more than
equally weighted correlated style signals; the diagnostic showcase score is
pinned, the evaluation corpus remains free of observed regressions, and all
quality/coverage gates pass.
Accepted on 2026-09-29: the showcase score is 71 (medium); reviewed positive
corpus scores are pinned at 60/33/19/24/17, all five negative documents remain
at zero, and matching metrics remain 11 TP, 0 FP, and 0 FN.
Optional follow-up, intentionally not required for this milestone:
- [ ] After observing real usage, choose one direction for the default-policy

View File

@ -15,6 +15,7 @@ struct Manifest {
struct Case {
path: String,
kind: String,
expected_score: u8,
expected: BTreeSet<String>,
#[serde(default)]
near_miss_for: BTreeSet<String>,
@ -48,8 +49,14 @@ fn reviewed_corpus_reports_per_rule_metrics_and_false_positive_rate() {
"prose" | "source-code" => AnalysisContext::for_path(&case.path),
other => panic!("unknown corpus kind {other}"),
};
let actual = detector
.analyze_with(&text, &context)
let analysis = detector.analyze_with(&text, &context);
assert_eq!(
analysis.confidence.score(),
case.expected_score,
"confidence score for {}",
case.path
);
let actual = analysis
.findings
.into_iter()
.map(|finding| finding.rule_id.to_string())
@ -120,9 +127,10 @@ fn reviewed_corpus_reports_per_rule_metrics_and_false_positive_rate() {
.collect::<String>();
assert_eq!(signoff.schema_version, 1);
assert_eq!(signoff.release, env!("CARGO_PKG_VERSION"));
assert_eq!(signoff.status, "accepted-for-release-baseline");
assert_eq!(signoff.status, "accepted-after-prompt-damping-review");
assert_eq!(signoff.catalog_sha256, catalog_hash);
assert!(signoff.limitations_acknowledged);
assert_eq!(signoff.reviewed_on, "2026-09-29");
assert!(signoff.decision.contains("unchanged"));
assert!(signoff.decision.contains("prompt-leakage"));
assert!(signoff.decision.contains("0.90"));
}

View File

@ -159,6 +159,7 @@ fn diagnostic_showcase_is_an_expected_threshold_failure() {
assert!(stdout.contains("assistant-self-identification"));
assert!(stdout.contains("prompt-response-scaffolding"));
assert!(stdout.contains("typography-non-ascii-hyphen"));
assert!(stdout.contains("confidence 71 (medium)"));
assert!(stdout.contains("advice"));
assert!(stdout.contains("warning"));
assert!(stdout.contains("error"));

View File

@ -4,6 +4,7 @@ license = "MIT"
[[cases]]
path = "assistant.md"
kind = "prose"
expected_score = 60
expected = [
"assistant-answer-preface",
"assistant-capability-disclaimer",
@ -14,32 +15,38 @@ expected = [
[[cases]]
path = "prompt.md"
kind = "prose"
expected_score = 33
expected = ["prompt-response-scaffolding"]
[[cases]]
path = "style.md"
kind = "prose"
expected_score = 19
expected = ["repeated-transition", "templated-conclusion", "templated-filler"]
[[cases]]
path = "example.rs"
kind = "source-code"
expected_score = 24
expected = ["source-placeholder-todo", "source-tutorial-narration"]
[[cases]]
path = "typography.md"
kind = "prose"
expected_score = 17
expected = ["typography-non-ascii-hyphen"]
[[cases]]
path = "negative.md"
kind = "prose"
expected_score = 0
expected = []
near_miss_for = ["typography-non-ascii-hyphen"]
[[cases]]
path = "negative-assistant.md"
kind = "prose"
expected_score = 0
expected = []
near_miss_for = [
"assistant-answer-preface",
@ -51,12 +58,14 @@ near_miss_for = [
[[cases]]
path = "negative-prompt.md"
kind = "prose"
expected_score = 0
expected = []
near_miss_for = ["prompt-response-scaffolding"]
[[cases]]
path = "negative-style.md"
kind = "prose"
expected_score = 0
expected = []
near_miss_for = [
"repeated-transition",
@ -67,5 +76,6 @@ near_miss_for = [
[[cases]]
path = "negative-source.rs"
kind = "source-code"
expected_score = 0
expected = []
near_miss_for = ["source-placeholder-todo", "source-tutorial-narration"]

View File

@ -1,7 +1,7 @@
schema_version = 1
release = "0.1.0"
reviewed_on = "2026-09-29"
status = "accepted-for-release-baseline"
status = "accepted-after-prompt-damping-review"
catalog_sha256 = "8617e56c1f874307fd967ef688c37d42efa750bfb833eb364b797dc588ce1fb0"
limitations_acknowledged = true
decision = "Keep the schema-version 1 default rules, weights, and thresholds unchanged. The reviewed characterization corpus has no observed false-positive cases; its small, authored sample does not establish real-world accuracy."
decision = "Accept category-specific correlation damping: prompt-leakage uses 0.90 per later distinct rule while other families retain 0.75. Rules, weights, thresholds, and the catalog hash remain unchanged. Reviewed corpus scores are pinned with 0/5 observed false-positive documents; the small authored sample does not establish real-world accuracy."