test: renew prompt scoring sign-off
This commit is contained in:
parent
d6163076d9
commit
7d365af5df
|
|
@ -5,4 +5,4 @@ coverage = 92.19
|
|||
|
||||
[last_run_clean.run_duration]
|
||||
secs = 29
|
||||
nanos = 930030912
|
||||
nanos = 350250460
|
||||
|
|
|
|||
|
|
@ -45,8 +45,8 @@ Run line coverage with Cargo Tarpaulin:
|
|||
cargo tarpaulin --config tarpaulin.toml
|
||||
```
|
||||
|
||||
The configuration enforces a 90% floor. The current suite measured 92.16%
|
||||
(1329/1442 lines) on 2026-09-29. The remaining lines are dominated by
|
||||
The configuration enforces a 90% floor. The current suite measured 92.19%
|
||||
(1333/1446 lines) on 2026-09-29. The remaining lines are dominated by
|
||||
platform/I/O failure injection, malformed archive internals, coordinate
|
||||
overflow, LSP logging/reload failures, and subprocess attribution. Do not add
|
||||
brittle tests or production seams solely to make Tarpaulin count those paths.
|
||||
|
|
|
|||
|
|
@ -25,6 +25,17 @@ Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0
|
|||
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
|
||||
suite recomputes these values from the manifest and fails on drift.
|
||||
|
||||
The same manifest pins confidence scores for regression characterization:
|
||||
|
||||
| Corpus case | Score |
|
||||
| --- | ---: |
|
||||
| four assistant disclaimer/preface rules | 60 |
|
||||
| one prompt/response marker | 33 |
|
||||
| style signals | 19 |
|
||||
| source-code signals | 24 |
|
||||
| typography signal | 17 |
|
||||
| each of five negative documents | 0 |
|
||||
|
||||
These perfect values describe a tiny purpose-built regression set, not expected
|
||||
performance on real documents. They must not be used as an accuracy or
|
||||
authorship claim. Broader, independently sourced data and domain-specific review
|
||||
|
|
@ -33,10 +44,11 @@ are required before tuning default weights or thresholds.
|
|||
## Default-policy sign-off
|
||||
|
||||
Each shipped rule has at least one explicitly labeled realistic near miss. The
|
||||
0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged
|
||||
and accepts them only as a conservative release baseline. The checked-in
|
||||
0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged.
|
||||
The renewed review accepts lighter `0.90ⁿ` correlation damping for distinct
|
||||
prompt-leakage rules while retaining `0.75ⁿ` for other categories. The checked-in
|
||||
`signoff.toml` records that decision, acknowledges the corpus limitations, and
|
||||
contains the SHA-256 digest of `assets/heuristics.toml`.
|
||||
contains the unchanged SHA-256 digest of `assets/heuristics.toml`.
|
||||
|
||||
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
|
||||
change invalidates the sign-off test. Renewal requires reviewing positive and
|
||||
|
|
|
|||
|
|
@ -20,8 +20,12 @@ Findings beyond the cap remain visible but do not increase the score.
|
|||
|
||||
Rules are grouped by category as correlated evidence families. The first
|
||||
matching rule in a category receives its full occurrence-adjusted contribution;
|
||||
later distinct rules receive successive factors of `0.75`. This prevents a
|
||||
cluster of closely related phrases from being treated as independent proof.
|
||||
later distinct prompt-leakage rules receive successive factors of `0.90`, while
|
||||
style, typography, and source-code rules receive successive factors of `0.75`.
|
||||
Prompt/response markers and copied assistant disclaimers are stronger concrete
|
||||
artifacts than generic stylistic similarities, but the remaining damping still
|
||||
prevents a cluster of related phrases from being treated as fully independent
|
||||
proof.
|
||||
|
||||
If the resulting evidence sum is `E`, the displayed score is
|
||||
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
|
||||
|
|
@ -38,3 +42,8 @@ to explain the score.
|
|||
|
||||
Weights and thresholds are review policy, not calibrated probabilities. A
|
||||
score of 75 does not mean a 75% chance that AI wrote the input.
|
||||
|
||||
The diagnostic showcase exercises five distinct prompt-leakage rules, including
|
||||
two prompt/response markers. Together with its style and typography findings it
|
||||
produces evidence score 71 (medium). This is a deterministic policy score, not
|
||||
an estimated probability.
|
||||
|
|
|
|||
8
tasks.md
8
tasks.md
|
|
@ -274,15 +274,19 @@ and builds without `fancy-diagnostics` retain the plain human fallback.
|
|||
- [x] Give distinct prompt-leakage rules lighter correlation damping (`0.90ⁿ`)
|
||||
while preserving `0.75ⁿ` damping for style, typography, and source-code
|
||||
families and preserving per-rule occurrence damping/caps.
|
||||
- [ ] Re-run the reviewed corpus, renew the existing default-policy sign-off,
|
||||
- [x] Re-run the reviewed corpus, renew the existing default-policy sign-off,
|
||||
and document the changed formula and showcase score without changing the
|
||||
sign-off mechanism itself.
|
||||
sign-off mechanism itself.
|
||||
|
||||
Acceptance: multiple independent prompt-leakage artifacts contribute more than
|
||||
equally weighted correlated style signals; the diagnostic showcase score is
|
||||
pinned, the evaluation corpus remains free of observed regressions, and all
|
||||
quality/coverage gates pass.
|
||||
|
||||
Accepted on 2026-09-29: the showcase score is 71 (medium); reviewed positive
|
||||
corpus scores are pinned at 60/33/19/24/17, all five negative documents remain
|
||||
at zero, and matching metrics remain 11 TP, 0 FP, and 0 FN.
|
||||
|
||||
Optional follow-up, intentionally not required for this milestone:
|
||||
|
||||
- [ ] After observing real usage, choose one direction for the default-policy
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ struct Manifest {
|
|||
struct Case {
|
||||
path: String,
|
||||
kind: String,
|
||||
expected_score: u8,
|
||||
expected: BTreeSet<String>,
|
||||
#[serde(default)]
|
||||
near_miss_for: BTreeSet<String>,
|
||||
|
|
@ -48,8 +49,14 @@ fn reviewed_corpus_reports_per_rule_metrics_and_false_positive_rate() {
|
|||
"prose" | "source-code" => AnalysisContext::for_path(&case.path),
|
||||
other => panic!("unknown corpus kind {other}"),
|
||||
};
|
||||
let actual = detector
|
||||
.analyze_with(&text, &context)
|
||||
let analysis = detector.analyze_with(&text, &context);
|
||||
assert_eq!(
|
||||
analysis.confidence.score(),
|
||||
case.expected_score,
|
||||
"confidence score for {}",
|
||||
case.path
|
||||
);
|
||||
let actual = analysis
|
||||
.findings
|
||||
.into_iter()
|
||||
.map(|finding| finding.rule_id.to_string())
|
||||
|
|
@ -120,9 +127,10 @@ fn reviewed_corpus_reports_per_rule_metrics_and_false_positive_rate() {
|
|||
.collect::<String>();
|
||||
assert_eq!(signoff.schema_version, 1);
|
||||
assert_eq!(signoff.release, env!("CARGO_PKG_VERSION"));
|
||||
assert_eq!(signoff.status, "accepted-for-release-baseline");
|
||||
assert_eq!(signoff.status, "accepted-after-prompt-damping-review");
|
||||
assert_eq!(signoff.catalog_sha256, catalog_hash);
|
||||
assert!(signoff.limitations_acknowledged);
|
||||
assert_eq!(signoff.reviewed_on, "2026-09-29");
|
||||
assert!(signoff.decision.contains("unchanged"));
|
||||
assert!(signoff.decision.contains("prompt-leakage"));
|
||||
assert!(signoff.decision.contains("0.90"));
|
||||
}
|
||||
|
|
|
|||
|
|
@ -159,6 +159,7 @@ fn diagnostic_showcase_is_an_expected_threshold_failure() {
|
|||
assert!(stdout.contains("assistant-self-identification"));
|
||||
assert!(stdout.contains("prompt-response-scaffolding"));
|
||||
assert!(stdout.contains("typography-non-ascii-hyphen"));
|
||||
assert!(stdout.contains("confidence 71 (medium)"));
|
||||
assert!(stdout.contains("advice"));
|
||||
assert!(stdout.contains("warning"));
|
||||
assert!(stdout.contains("error"));
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ license = "MIT"
|
|||
[[cases]]
|
||||
path = "assistant.md"
|
||||
kind = "prose"
|
||||
expected_score = 60
|
||||
expected = [
|
||||
"assistant-answer-preface",
|
||||
"assistant-capability-disclaimer",
|
||||
|
|
@ -14,32 +15,38 @@ expected = [
|
|||
[[cases]]
|
||||
path = "prompt.md"
|
||||
kind = "prose"
|
||||
expected_score = 33
|
||||
expected = ["prompt-response-scaffolding"]
|
||||
|
||||
[[cases]]
|
||||
path = "style.md"
|
||||
kind = "prose"
|
||||
expected_score = 19
|
||||
expected = ["repeated-transition", "templated-conclusion", "templated-filler"]
|
||||
|
||||
[[cases]]
|
||||
path = "example.rs"
|
||||
kind = "source-code"
|
||||
expected_score = 24
|
||||
expected = ["source-placeholder-todo", "source-tutorial-narration"]
|
||||
|
||||
[[cases]]
|
||||
path = "typography.md"
|
||||
kind = "prose"
|
||||
expected_score = 17
|
||||
expected = ["typography-non-ascii-hyphen"]
|
||||
|
||||
[[cases]]
|
||||
path = "negative.md"
|
||||
kind = "prose"
|
||||
expected_score = 0
|
||||
expected = []
|
||||
near_miss_for = ["typography-non-ascii-hyphen"]
|
||||
|
||||
[[cases]]
|
||||
path = "negative-assistant.md"
|
||||
kind = "prose"
|
||||
expected_score = 0
|
||||
expected = []
|
||||
near_miss_for = [
|
||||
"assistant-answer-preface",
|
||||
|
|
@ -51,12 +58,14 @@ near_miss_for = [
|
|||
[[cases]]
|
||||
path = "negative-prompt.md"
|
||||
kind = "prose"
|
||||
expected_score = 0
|
||||
expected = []
|
||||
near_miss_for = ["prompt-response-scaffolding"]
|
||||
|
||||
[[cases]]
|
||||
path = "negative-style.md"
|
||||
kind = "prose"
|
||||
expected_score = 0
|
||||
expected = []
|
||||
near_miss_for = [
|
||||
"repeated-transition",
|
||||
|
|
@ -67,5 +76,6 @@ near_miss_for = [
|
|||
[[cases]]
|
||||
path = "negative-source.rs"
|
||||
kind = "source-code"
|
||||
expected_score = 0
|
||||
expected = []
|
||||
near_miss_for = ["source-placeholder-todo", "source-tutorial-narration"]
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
schema_version = 1
|
||||
release = "0.1.0"
|
||||
reviewed_on = "2026-09-29"
|
||||
status = "accepted-for-release-baseline"
|
||||
status = "accepted-after-prompt-damping-review"
|
||||
catalog_sha256 = "8617e56c1f874307fd967ef688c37d42efa750bfb833eb364b797dc588ce1fb0"
|
||||
limitations_acknowledged = true
|
||||
decision = "Keep the schema-version 1 default rules, weights, and thresholds unchanged. The reviewed characterization corpus has no observed false-positive cases; its small, authored sample does not establish real-world accuracy."
|
||||
decision = "Accept category-specific correlation damping: prompt-leakage uses 0.90 per later distinct rule while other families retain 0.75. Rules, weights, thresholds, and the catalog hash remain unchanged. Reviewed corpus scores are pinned with 0/5 observed false-positive documents; the small authored sample does not establish real-world accuracy."
|
||||
|
|
|
|||
Loading…
Reference in New Issue