diff --git a/.run_status.toml b/.run_status.toml index c679ead..9c55f32 100644 --- a/.run_status.toml +++ b/.run_status.toml @@ -5,4 +5,4 @@ coverage = 92.19 [last_run_clean.run_duration] secs = 29 -nanos = 930030912 +nanos = 350250460 diff --git a/docs/development.md b/docs/development.md index 65b9e1b..09cb976 100644 --- a/docs/development.md +++ b/docs/development.md @@ -45,8 +45,8 @@ Run line coverage with Cargo Tarpaulin: cargo tarpaulin --config tarpaulin.toml ``` -The configuration enforces a 90% floor. The current suite measured 92.16% -(1329/1442 lines) on 2026-09-29. The remaining lines are dominated by +The configuration enforces a 90% floor. The current suite measured 92.19% +(1333/1446 lines) on 2026-09-29. The remaining lines are dominated by platform/I/O failure injection, malformed archive internals, coordinate overflow, LSP logging/reload failures, and subprocess attribution. Do not add brittle tests or production seams solely to make Tarpaulin count those paths. diff --git a/docs/evaluation.md b/docs/evaluation.md index d13a756..47df611 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -25,6 +25,17 @@ Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0 false negatives. The clean-document false-positive rate is 0/5 (0%). The test suite recomputes these values from the manifest and fails on drift. +The same manifest pins confidence scores for regression characterization: + +| Corpus case | Score | +| --- | ---: | +| four assistant disclaimer/preface rules | 60 | +| one prompt/response marker | 33 | +| style signals | 19 | +| source-code signals | 24 | +| typography signal | 17 | +| each of five negative documents | 0 | + These perfect values describe a tiny purpose-built regression set, not expected performance on real documents. They must not be used as an accuracy or authorship claim. Broader, independently sourced data and domain-specific review @@ -33,10 +44,11 @@ are required before tuning default weights or thresholds. ## Default-policy sign-off Each shipped rule has at least one explicitly labeled realistic near miss. The -0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged -and accepts them only as a conservative release baseline. The checked-in +0.1.0 review keeps all schema-version 1 rules, weights, and thresholds unchanged. +The renewed review accepts lighter `0.90ⁿ` correlation damping for distinct +prompt-leakage rules while retaining `0.75ⁿ` for other categories. The checked-in `signoff.toml` records that decision, acknowledges the corpus limitations, and -contains the SHA-256 digest of `assets/heuristics.toml`. +contains the unchanged SHA-256 digest of `assets/heuristics.toml`. Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold change invalidates the sign-off test. Renewal requires reviewing positive and diff --git a/docs/heuristics.md b/docs/heuristics.md index aa8c1dc..cb79880 100644 --- a/docs/heuristics.md +++ b/docs/heuristics.md @@ -20,8 +20,12 @@ Findings beyond the cap remain visible but do not increase the score. Rules are grouped by category as correlated evidence families. The first matching rule in a category receives its full occurrence-adjusted contribution; -later distinct rules receive successive factors of `0.75`. This prevents a -cluster of closely related phrases from being treated as independent proof. +later distinct prompt-leakage rules receive successive factors of `0.90`, while +style, typography, and source-code rules receive successive factors of `0.75`. +Prompt/response markers and copied assistant disclaimers are stronger concrete +artifacts than generic stylistic similarities, but the remaining damping still +prevents a cluster of related phrases from being treated as fully independent +proof. If the resulting evidence sum is `E`, the displayed score is `round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`, @@ -38,3 +42,8 @@ to explain the score. Weights and thresholds are review policy, not calibrated probabilities. A score of 75 does not mean a 75% chance that AI wrote the input. + +The diagnostic showcase exercises five distinct prompt-leakage rules, including +two prompt/response markers. Together with its style and typography findings it +produces evidence score 71 (medium). This is a deterministic policy score, not +an estimated probability. diff --git a/tasks.md b/tasks.md index 64917ad..3a66961 100644 --- a/tasks.md +++ b/tasks.md @@ -274,15 +274,19 @@ and builds without `fancy-diagnostics` retain the plain human fallback. - [x] Give distinct prompt-leakage rules lighter correlation damping (`0.90ⁿ`) while preserving `0.75ⁿ` damping for style, typography, and source-code families and preserving per-rule occurrence damping/caps. -- [ ] Re-run the reviewed corpus, renew the existing default-policy sign-off, +- [x] Re-run the reviewed corpus, renew the existing default-policy sign-off, and document the changed formula and showcase score without changing the - sign-off mechanism itself. +sign-off mechanism itself. Acceptance: multiple independent prompt-leakage artifacts contribute more than equally weighted correlated style signals; the diagnostic showcase score is pinned, the evaluation corpus remains free of observed regressions, and all quality/coverage gates pass. +Accepted on 2026-09-29: the showcase score is 71 (medium); reviewed positive +corpus scores are pinned at 60/33/19/24/17, all five negative documents remain +at zero, and matching metrics remain 11 TP, 0 FP, and 0 FN. + Optional follow-up, intentionally not required for this milestone: - [ ] After observing real usage, choose one direction for the default-policy diff --git a/tests/evaluation.rs b/tests/evaluation.rs index 826def7..9838870 100644 --- a/tests/evaluation.rs +++ b/tests/evaluation.rs @@ -15,6 +15,7 @@ struct Manifest { struct Case { path: String, kind: String, + expected_score: u8, expected: BTreeSet, #[serde(default)] near_miss_for: BTreeSet, @@ -48,8 +49,14 @@ fn reviewed_corpus_reports_per_rule_metrics_and_false_positive_rate() { "prose" | "source-code" => AnalysisContext::for_path(&case.path), other => panic!("unknown corpus kind {other}"), }; - let actual = detector - .analyze_with(&text, &context) + let analysis = detector.analyze_with(&text, &context); + assert_eq!( + analysis.confidence.score(), + case.expected_score, + "confidence score for {}", + case.path + ); + let actual = analysis .findings .into_iter() .map(|finding| finding.rule_id.to_string()) @@ -120,9 +127,10 @@ fn reviewed_corpus_reports_per_rule_metrics_and_false_positive_rate() { .collect::(); assert_eq!(signoff.schema_version, 1); assert_eq!(signoff.release, env!("CARGO_PKG_VERSION")); - assert_eq!(signoff.status, "accepted-for-release-baseline"); + assert_eq!(signoff.status, "accepted-after-prompt-damping-review"); assert_eq!(signoff.catalog_sha256, catalog_hash); assert!(signoff.limitations_acknowledged); assert_eq!(signoff.reviewed_on, "2026-09-29"); - assert!(signoff.decision.contains("unchanged")); + assert!(signoff.decision.contains("prompt-leakage")); + assert!(signoff.decision.contains("0.90")); } diff --git a/tests/example_fixtures.rs b/tests/example_fixtures.rs index de8a74c..95ee886 100644 --- a/tests/example_fixtures.rs +++ b/tests/example_fixtures.rs @@ -159,6 +159,7 @@ fn diagnostic_showcase_is_an_expected_threshold_failure() { assert!(stdout.contains("assistant-self-identification")); assert!(stdout.contains("prompt-response-scaffolding")); assert!(stdout.contains("typography-non-ascii-hyphen")); + assert!(stdout.contains("confidence 71 (medium)")); assert!(stdout.contains("advice")); assert!(stdout.contains("warning")); assert!(stdout.contains("error")); diff --git a/tests/fixtures/evaluation/manifest.toml b/tests/fixtures/evaluation/manifest.toml index 64a412c..d56d5d8 100644 --- a/tests/fixtures/evaluation/manifest.toml +++ b/tests/fixtures/evaluation/manifest.toml @@ -4,6 +4,7 @@ license = "MIT" [[cases]] path = "assistant.md" kind = "prose" +expected_score = 60 expected = [ "assistant-answer-preface", "assistant-capability-disclaimer", @@ -14,32 +15,38 @@ expected = [ [[cases]] path = "prompt.md" kind = "prose" +expected_score = 33 expected = ["prompt-response-scaffolding"] [[cases]] path = "style.md" kind = "prose" +expected_score = 19 expected = ["repeated-transition", "templated-conclusion", "templated-filler"] [[cases]] path = "example.rs" kind = "source-code" +expected_score = 24 expected = ["source-placeholder-todo", "source-tutorial-narration"] [[cases]] path = "typography.md" kind = "prose" +expected_score = 17 expected = ["typography-non-ascii-hyphen"] [[cases]] path = "negative.md" kind = "prose" +expected_score = 0 expected = [] near_miss_for = ["typography-non-ascii-hyphen"] [[cases]] path = "negative-assistant.md" kind = "prose" +expected_score = 0 expected = [] near_miss_for = [ "assistant-answer-preface", @@ -51,12 +58,14 @@ near_miss_for = [ [[cases]] path = "negative-prompt.md" kind = "prose" +expected_score = 0 expected = [] near_miss_for = ["prompt-response-scaffolding"] [[cases]] path = "negative-style.md" kind = "prose" +expected_score = 0 expected = [] near_miss_for = [ "repeated-transition", @@ -67,5 +76,6 @@ near_miss_for = [ [[cases]] path = "negative-source.rs" kind = "source-code" +expected_score = 0 expected = [] near_miss_for = ["source-placeholder-todo", "source-tutorial-narration"] diff --git a/tests/fixtures/evaluation/signoff.toml b/tests/fixtures/evaluation/signoff.toml index 019a205..3a8cbca 100644 --- a/tests/fixtures/evaluation/signoff.toml +++ b/tests/fixtures/evaluation/signoff.toml @@ -1,7 +1,7 @@ schema_version = 1 release = "0.1.0" reviewed_on = "2026-09-29" -status = "accepted-for-release-baseline" +status = "accepted-after-prompt-damping-review" catalog_sha256 = "8617e56c1f874307fd967ef688c37d42efa750bfb833eb364b797dc588ce1fb0" limitations_acknowledged = true -decision = "Keep the schema-version 1 default rules, weights, and thresholds unchanged. The reviewed characterization corpus has no observed false-positive cases; its small, authored sample does not establish real-world accuracy." +decision = "Accept category-specific correlation damping: prompt-leakage uses 0.90 per later distinct rule while other families retain 0.75. Rules, weights, thresholds, and the catalog hash remain unchanged. Reviewed corpus scores are pinned with 0/5 observed false-positive documents; the small authored sample does not establish real-world accuracy."