diff --git a/AGENTS.md b/AGENTS.md index 022bd2e..6046370 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,8 +7,8 @@ insufficiently reviewed text and source code. It reports concrete signals that are often associated with careless AI-assisted work. It must not claim to prove that a person used AI, identify a model, or infer authorship from writing style. -Treat a score as **confidence that configured signals are present**, not as the -probability that content is AI-generated. User-facing language must say +Treat the **low-care evidence score** as the bounded strength of configured +signals, not as the probability that content is AI-generated. User-facing language must say "suspicious", "low-care signal", or "needs review" rather than making an accusation. A result must always be explainable from the findings that produced it. @@ -77,7 +77,7 @@ struct Finding { struct Analysis { findings: Vec, - confidence: Confidence, + evidence_score: EvidenceScore, } struct TextEdit { @@ -96,7 +96,7 @@ Rule IDs are stable, unique, lowercase kebab-case identifiers. Findings are sorted by source identity, span start, then rule ID so output is deterministic. Never use a hash-map iteration order as output order. -## Heuristics and confidence +## Heuristics and evidence scoring Store data-driven heuristic definitions in `assets/heuristics.toml`, separately from replacement configuration. Compile and validate rules once at startup. @@ -116,7 +116,7 @@ Start with high-precision, independently testable signals: that merely narrate obvious code, using conservative language-specific rules. Avoid weak demographic, dialect, spelling, grammar, vocabulary, or -English-fluency proxies. Repetition alone must not trigger a high-confidence +English-fluency proxies. Repetition alone must not trigger a high evidence-band verdict. Give users a per-rule disable/weight override and path/language scopes. Scoring must be documented and deterministic. Use bounded weighted evidence, diff --git a/CHANGELOG.md b/CHANGELOG.md index fd3d2fc..d5cdcd7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,11 +6,18 @@ All notable changes to John Slop are recorded here. The format follows ## [Unreleased] +### Changed + +- Renamed the ambiguous confidence metric to low-care evidence score throughout + the Rust API, CLI, terminal output, and JSON schema version 2. Deprecated Rust + type aliases and `--fail-confidence` remain available for migration; the + scoring formula and values are unchanged. + ## [0.1.0] - 2026-09-29 ### Added -- Local, explainable heuristic analysis with bounded evidence confidence. +- Local, explainable heuristic analysis with a bounded low-care evidence score. - Layered embedded, user, named, explicit, and CLI configuration. - File, directory, ZIP, office-container, and stdin input adapters with limits. - Human, graphical Miette, concise, and versioned JSON detector output. diff --git a/README.md b/README.md index 4399a3a..77e9815 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ ZIP-based office documents. Eleven conservative, configurable rules cover copied assistant scaffolding, generic style residue, source-code placeholders, tutorial narration, and configured non-ASCII characters. Rules retain exact source spans, apply path/content scopes, and feed a bounded, explainable -confidence score. The detector renders plain or graphical Miette diagnostics, +low-care evidence score. The detector renders plain or graphical Miette diagnostics, concise records, or versioned JSON Lines. The formatter applies only configured Unicode-to-ASCII replacements, supports check/diff modes, and requires explicit opt-in for atomic file writes. The stdio LSP publishes those same replacement @@ -34,9 +34,10 @@ cargo run --bin john-slop-fmt -- --check path/to/document cargo run --bin john-slop-lsp ``` -Use `--fail-confidence 0..100` to make the detector exit with status `1` when a -document reaches a chosen evidence threshold. Confidence measures configured -signals, not the probability that AI wrote the content. +Use `--fail-score 0..100` to make the detector exit with status `1` when a +document reaches a chosen evidence threshold. The score measures configured +signals, not the probability that AI wrote the content. The former +`--fail-confidence` spelling remains an alias for migration. Formatter file mode prints transformed text without modifying the source. Use `--write` explicitly for atomic in-place replacement, or pipe editor buffers @@ -59,7 +60,7 @@ placeholders, tutorial narration, and Unicode punctuation covered by the ASCII replacement catalog. Every result includes the matching source span, rule ID, severity, message, and remediation help. -The confidence score aggregates those configured signals. It is not an +The low-care evidence score aggregates those configured signals. It is not an AI-authorship probability, and a clean result does not prove human authorship or good quality. Legitimate transcripts, tutorials, multilingual text, and typographic punctuation can match. Review the source context before acting. @@ -92,7 +93,7 @@ untrusted inputs, but John Slop is not a sandbox or malware scanner. - [Architecture and public library API](docs/architecture.md) - [Layered configuration](docs/configuration.md) - [Commented base and named configuration examples](examples/config/) -- [Heuristics, confidence, and applicability](docs/heuristics.md) +- [Heuristics, evidence scoring, and applicability](docs/heuristics.md) - [Evaluation corpus and current metrics](docs/evaluation.md) - [Limitations and responsible interpretation](docs/limitations.md) - [Development, tests, features, and coverage](docs/development.md) diff --git a/docs/README.md b/docs/README.md index 21b9f42..0b3540f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -20,7 +20,7 @@ the details needed to use, embed, and develop John Slop: precedence, stable-ID merges, validation, and inspection commands. - [Commented configuration examples](../examples/config/) provide user-wide and named partial overlays ready to copy. -- [Heuristics and confidence](heuristics.md) documents rule execution, +- [Heuristics and evidence scoring](heuristics.md) documents rule execution, applicability, evidence damping, normalization, and score interpretation. - [Evaluation corpus](evaluation.md) records fixture provenance, per-rule precision/recall, aggregate false positives, and benchmark caveats. @@ -39,7 +39,7 @@ the details needed to use, embed, and develop John Slop: - [Ambiguous and invisible Unicode references](unicode-confusables.md) records Gitea's locale-aware catalog, its upstream, and safe adoption boundaries. - [Decision records](decisions/) preserve durable product and engineering - choices, including confidence semantics and input limits. + choices, including evidence-score semantics and input limits. Release-hardening contracts remain specified in [tasks.md](../tasks.md) and [AGENTS.md](../AGENTS.md) until that milestone lands. diff --git a/docs/architecture.md b/docs/architecture.md index 42fc90d..440afc8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -19,7 +19,7 @@ The main modules have deliberately narrow responsibilities: - `config` owns the validated embedded replacement catalog and input limits; - `input` streams files, directory entries, and archive members as documents; - `detector` compiles configured matchers and path scopes once, then produces - explainable findings and bounded confidence; + explainable findings and a bounded low-care evidence score; - `diagnostic` binds findings to exact named snapshots as typed `miette` diagnostics; - `replacement` produces non-overlapping edits and validates/applies arbitrary @@ -44,7 +44,7 @@ The crate root exposes core entry points and presentation-neutral types: - `load_config` resolves the configuration embedded in the current build; - `read_inputs` returns a streaming iterator of per-document results; - `read_stdin` returns one bounded strict-UTF-8 standard-input document; -- `analyze_source` returns findings and evidence confidence for UTF-8 text; +- `analyze_source` returns findings and an evidence score for UTF-8 text; - `plan_replacements` returns safe edits against the same source snapshot; - `apply_replacements` validates boundaries and overlap, then applies edits in descending byte order while borrowing no-op input unchanged. diff --git a/docs/background.md b/docs/background.md index 09c958d..232aa1b 100644 --- a/docs/background.md +++ b/docs/background.md @@ -10,5 +10,5 @@ the kinds of review problems the project aims to help surface: - [Gitea/Unicode confusable-character source notes](unicode-confusables.md) Project-specific decisions take precedence over external guidance. See -[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and +[ADR 0001](decisions/0001-core-analysis-contracts.md) for evidence scoring and authorship semantics. diff --git a/docs/configuration.md b/docs/configuration.md index 28ff883..4eeeaed 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -70,8 +70,8 @@ wins at each byte offset, with stable rule ID ordering as the deterministic tie-break. Edits always refer to the original UTF-8 snapshot. Each heuristic also configures `severity` as `advice`, `warning`, or `error`. -Severity controls diagnostic presentation and summary counts; confidence still -comes from the independently configured weight and evidence formula. +Severity controls diagnostic presentation and summary counts; the evidence score +still comes from the independently configured weight and aggregation formula. ## Inspection diff --git a/docs/decisions/0001-core-analysis-contracts.md b/docs/decisions/0001-core-analysis-contracts.md index 96cffc8..078578b 100644 --- a/docs/decisions/0001-core-analysis-contracts.md +++ b/docs/decisions/0001-core-analysis-contracts.md @@ -7,16 +7,16 @@ John Slop needs stable semantics before its proof-of-concept modules are split into a library, CLI, formatter, and language server. In particular, consumers -need to know what a confidence score means, how source locations are represented, +need to know what the aggregate evidence score means, how source locations are represented, whether formatting is destructive by default, and whether archive members can be rewritten. ## Decision -### Confidence represents evidence, not authorship +### The low-care evidence score represents evidence, not authorship -The detector reports confidence that configured, observable low-care signals are -present. It does not report the probability that content is AI-generated and +The detector reports the bounded strength of configured, observable low-care +signals. It does not report the probability that content is AI-generated and must not present a score as an authorship determination. Every score must be decomposable into source-labelled findings from stable rule IDs. diff --git a/docs/decisions/0003-lsp-framework-and-protocol.md b/docs/decisions/0003-lsp-framework-and-protocol.md index b45f645..c781ab2 100644 --- a/docs/decisions/0003-lsp-framework-and-protocol.md +++ b/docs/decisions/0003-lsp-framework-and-protocol.md @@ -34,7 +34,7 @@ The initial server targets stable LSP 3.17 behavior with these assumptions: handling while replacement behavior is established; - the server owns snapshots of open documents only and never writes their URIs; - diagnostics and quick fixes cover configured ASCII replacements, not heuristic - authorship or confidence findings; + authorship or aggregate heuristic findings; - quick fixes use versioned workspace edits and are rejected when the request refers to a stale or unknown document version; - `workspace/didChangeConfiguration` reloads the same layered configuration used diff --git a/docs/editors.md b/docs/editors.md index 745bf54..de5d610 100644 --- a/docs/editors.md +++ b/docs/editors.md @@ -2,7 +2,7 @@ Both `john-slop-fmt` and `john-slop-lsp` are operational. The LSP publishes configured ASCII-replacement warnings and versioned quick fixes; it does not -publish the broader heuristic confidence findings. +publish the broader heuristic evidence findings. Install all three binaries somewhere on `PATH`, then confirm the editor-facing help is available: diff --git a/docs/evaluation.md b/docs/evaluation.md index 47df611..b143c96 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -25,7 +25,7 @@ Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0 false negatives. The clean-document false-positive rate is 0/5 (0%). The test suite recomputes these values from the manifest and fails on drift. -The same manifest pins confidence scores for regression characterization: +The same manifest pins evidence scores for regression characterization: | Corpus case | Score | | --- | ---: | diff --git a/docs/heuristics.md b/docs/heuristics.md index cb79880..e4558a5 100644 --- a/docs/heuristics.md +++ b/docs/heuristics.md @@ -1,4 +1,4 @@ -# Heuristics and confidence +# Heuristics and evidence scoring John Slop reports observable low-care signals. It does not classify authorship. Every contribution to the score is represented by a finding with an exact @@ -11,10 +11,10 @@ generated/vendor/minified paths decide whether each rule applies. Findings are deduplicated by rule and byte span, then sorted by start offset, end offset, and rule ID so every adapter observes the same order. -## Confidence formula +## Evidence-score formula For each rule in deterministic catalog order, only the configured -`occurrence_cap` contributes to confidence. Its first occurrence receives the +`occurrence_cap` contributes to the evidence score. Its first occurrence receives the full weight, then later occurrences receive one half, one quarter, and so on. Findings beyond the cap remain visible but do not increase the score. @@ -31,7 +31,7 @@ If the resulting evidence sum is `E`, the displayed score is `round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`, `medium`, and `high` thresholds name the resulting band. Evidence below `low` is explicitly reported as below threshold. Empty input and text with no -findings produce score zero and no confidence verdict. Binary, invalid UTF-8, +findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8, oversized, and unsupported inputs remain typed input errors; they are never reported as clean analyses. @@ -45,5 +45,5 @@ score of 75 does not mean a 75% chance that AI wrote the input. The diagnostic showcase exercises five distinct prompt-leakage rules, including two prompt/response markers. Together with its style and typography findings it -produces evidence score 71 (medium). This is a deterministic policy score, not +produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not an estimated probability. diff --git a/docs/limitations.md b/docs/limitations.md index 2bd67a8..554935e 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -41,7 +41,7 @@ underrepresented. File-extension inference covers a conservative common set and defaults unknown paths to prose. Teams should validate rules against their own reviewed material before changing enforcement thresholds. -## Confidence is not probability +## Evidence score is not probability The score is a deterministic compression of configured weights, occurrence caps, and category damping. Those values are policy choices and have not been @@ -74,7 +74,7 @@ containers, prose, or source structure. Recursive writes require explicit The LSP intentionally handles open editor snapshots only, uses full-document synchronization, and publishes configured ASCII-replacement warnings rather -than the broader heuristic/confidence catalog. It does not read or write a +than the broader heuristic catalog. It does not read or write a document URI, mutate archives, provide prose rewrites, or implement incremental range changes. Configuration reload errors retain the last valid configuration. diff --git a/docs/output.md b/docs/output.md index 06a8615..626b149 100644 --- a/docs/output.md +++ b/docs/output.md @@ -9,8 +9,8 @@ container, and findings are sorted by byte span and rule ID. `--output human` renders every finding as a plain `miette` diagnostic with the source name, exact excerpt and primary span, severity, stable rule code, -message, and remediation help. A final line per document reports the confidence -score/band and advice, warning, and error counts. The narratable renderer +message, and remediation help. A final line per document reports the low-care +evidence score/band and advice, warning, and error counts. The narratable renderer deliberately avoids ANSI escapes. ## Miette @@ -29,19 +29,19 @@ finding line contains the source identity, half-open UTF-8 byte range, severity, rule ID, and message. Recoverable input errors go to stderr in human and concise modes. -## JSON Lines schema version 1 +## JSON Lines schema version 2 `--output json` emits one compact JSON object per line. An analyzed document has `kind: "analysis"` and this shape: ```json { - "schema_version": 1, + "schema_version": 2, "kind": "analysis", "source": "article.md", "summary": { - "confidence_score": 17, - "confidence_band": "below-threshold", + "evidence_score": 17, + "evidence_band": "below-threshold", "advice": 1, "warnings": 0, "errors": 0, @@ -66,7 +66,7 @@ modes. } ``` -`confidence_band` is `null` when there is no evidence. Spans are zero-based, +`evidence_band` is `null` when there is no evidence. Spans are zero-based, half-open UTF-8 byte offsets into the exact source snapshot, not character or UTF-16 positions. `fix` is `null` for findings without a safe configured edit. @@ -74,7 +74,7 @@ A recoverable input failure has `kind: "input-error"`: ```json { - "schema_version": 1, + "schema_version": 2, "kind": "input-error", "source": "broken.txt", "message": "input broken.txt is not valid UTF-8" @@ -84,5 +84,7 @@ A recoverable input failure has `kind: "input-error"`: JSON records are the only stdout content in JSON mode and never contain ANSI escapes or logs. Fatal argument/configuration/setup failures can occur before the JSON stream starts; they return status 2, leave stdout empty, and report on -stderr. Additive fields may appear within schema version 1; incompatible changes -require a new schema version. +stderr. Additive fields may appear within schema version 2; incompatible changes +require a new schema version. Schema version 2 renamed the version 1 summary +fields `confidence_score` and `confidence_band` to `evidence_score` and +`evidence_band`; version 1 records are no longer emitted. diff --git a/docs/usage.md b/docs/usage.md index 28e71ce..e467a41 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -5,7 +5,7 @@ Run the detector over one or more paths: ```sh -cargo run --bin john-slop -- [--fail-confidence 0..100] ... +cargo run --bin john-slop -- [--fail-score 0..100] ... cargo run --bin john-slop -- --stdin [--stdin-filename article.md] ``` @@ -28,7 +28,7 @@ annotations when the default `fancy-diagnostics` feature is enabled; it falls back to human mode without that feature. Human mode renders uncolored plain source excerpts, exact labels, remediation help, and a summary. Concise mode emits stable single-line finding and summary records. JSON mode emits one object -per analyzed document with `schema_version: 1`, summary fields, exact byte spans, +per analyzed document with `schema_version: 2`, summary fields, exact byte spans, and every finding. Machine modes never emit ANSI escapes or logs on stdout. Findings cover the embedded or overridden heuristic catalog and retain exact @@ -37,14 +37,15 @@ for the human/concise formats and complete JSON Lines schema. Exit statuses are stable: -- `0`: every analyzed document is below the requested confidence threshold; +- `0`: every analyzed document is below the requested evidence-score threshold; - `1`: at least one document reaches the threshold; - `2`: arguments are invalid or an operational/input error prevents completion. -`--fail-confidence SCORE` selects an exact inclusive score threshold and +`--fail-score SCORE` selects an exact inclusive score threshold and defaults to 100. `--fail-level low|medium|high` instead uses the corresponding threshold from the fully resolved configuration. The two options are mutually -exclusive. +exclusive. The old `--fail-confidence` spelling remains a visible alias for +migration. Input failures are isolated per document whenever iteration can continue. Valid documents still emit all findings, while any partial input failure makes the @@ -53,8 +54,8 @@ concise input errors go to stderr. JSON mode emits versioned `input-error` records on stdout alongside `analysis` records and leaves stderr empty unless a fatal setup failure prevents the output contract from starting. -Confidence is a summary of observable configured evidence. It is not an -AI-authorship probability. See [Heuristics and confidence](heuristics.md) for +The low-care evidence score summarizes observable configured evidence. It is +not an AI-authorship probability. See [Heuristics and evidence scoring](heuristics.md) for the formula and [Limitations](limitations.md) before using thresholds in review workflows. @@ -77,7 +78,7 @@ formatted output. The executable fixture and protocol tests validate the library, detector, formatter, and LSP against the shared replacement contract. The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase) prints every current prose finding in graphical Miette form and intentionally -exits `1` when its low confidence threshold is reached. +exits `1` when its low evidence threshold is reached. ## Supported input policy @@ -140,7 +141,7 @@ It stores only currently open UTF-8 snapshots, their URI, language ID, and monotonically increasing version. On open or an accepted newer full-document change it publishes warnings only for enabled Unicode-to-ASCII replacements. Each diagnostic has source `john-slop`, a stable replacement-rule code, the -document version, and a UTF-16 range. Heuristic confidence findings are not +document version, and a UTF-16 range. Broader heuristic findings are not published by this initial server. Quick fixes are recomputed from the current snapshot with the shared replacement diff --git a/tasks.md b/tasks.md index db051c4..0434400 100644 --- a/tasks.md +++ b/tasks.md @@ -298,7 +298,7 @@ Optional follow-up, intentionally not required for this milestone: - [x] Rename the public domain model from ambiguous confidence terminology to `EvidenceScore`, `EvidenceBand`, and `EvidenceThresholds`; retain deprecated Rust type aliases and the legacy `--fail-confidence` CLI alias for migration. -- [ ] Publish the low-care evidence-score wording across terminal output, +- [x] Publish the low-care evidence-score wording across terminal output, examples, and documentation; rename `--fail-confidence` to `--fail-score` and version the JSON field rename as schema version 2. - [ ] Run the complete quality and coverage gates, refresh generated badges, and