docs: publish evidence score terminology

This commit is contained in:
Dandaedre 2026-09-29 21:08:03 +02:00
parent a4dd4e2c5f
commit 95d4067dc8
Signed by: Dandaedre
GPG Key ID: 440C6592E8A26D99
16 changed files with 64 additions and 53 deletions

View File

@ -7,8 +7,8 @@ insufficiently reviewed text and source code. It reports concrete signals that
are often associated with careless AI-assisted work. It must not claim to prove
that a person used AI, identify a model, or infer authorship from writing style.
Treat a score as **confidence that configured signals are present**, not as the
probability that content is AI-generated. User-facing language must say
Treat the **low-care evidence score** as the bounded strength of configured
signals, not as the probability that content is AI-generated. User-facing language must say
"suspicious", "low-care signal", or "needs review" rather than making an
accusation. A result must always be explainable from the findings that produced
it.
@ -77,7 +77,7 @@ struct Finding {
struct Analysis {
findings: Vec<Finding>,
confidence: Confidence,
evidence_score: EvidenceScore,
}
struct TextEdit {
@ -96,7 +96,7 @@ Rule IDs are stable, unique, lowercase kebab-case identifiers. Findings are
sorted by source identity, span start, then rule ID so output is deterministic.
Never use a hash-map iteration order as output order.
## Heuristics and confidence
## Heuristics and evidence scoring
Store data-driven heuristic definitions in `assets/heuristics.toml`, separately
from replacement configuration. Compile and validate rules once at startup.
@ -116,7 +116,7 @@ Start with high-precision, independently testable signals:
that merely narrate obvious code, using conservative language-specific rules.
Avoid weak demographic, dialect, spelling, grammar, vocabulary, or
English-fluency proxies. Repetition alone must not trigger a high-confidence
English-fluency proxies. Repetition alone must not trigger a high evidence-band
verdict. Give users a per-rule disable/weight override and path/language scopes.
Scoring must be documented and deterministic. Use bounded weighted evidence,

View File

@ -6,11 +6,18 @@ All notable changes to John Slop are recorded here. The format follows
## [Unreleased]
### Changed
- Renamed the ambiguous confidence metric to low-care evidence score throughout
the Rust API, CLI, terminal output, and JSON schema version 2. Deprecated Rust
type aliases and `--fail-confidence` remain available for migration; the
scoring formula and values are unchanged.
## [0.1.0] - 2026-09-29
### Added
- Local, explainable heuristic analysis with bounded evidence confidence.
- Local, explainable heuristic analysis with a bounded low-care evidence score.
- Layered embedded, user, named, explicit, and CLI configuration.
- File, directory, ZIP, office-container, and stdin input adapters with limits.
- Human, graphical Miette, concise, and versioned JSON detector output.

View File

@ -14,7 +14,7 @@ ZIP-based office documents. Eleven conservative, configurable rules cover
copied assistant scaffolding, generic style residue, source-code placeholders,
tutorial narration, and configured non-ASCII characters. Rules retain exact
source spans, apply path/content scopes, and feed a bounded, explainable
confidence score. The detector renders plain or graphical Miette diagnostics,
low-care evidence score. The detector renders plain or graphical Miette diagnostics,
concise records, or versioned JSON Lines. The formatter applies only configured
Unicode-to-ASCII replacements, supports check/diff modes, and requires explicit
opt-in for atomic file writes. The stdio LSP publishes those same replacement
@ -34,9 +34,10 @@ cargo run --bin john-slop-fmt -- --check path/to/document
cargo run --bin john-slop-lsp
```
Use `--fail-confidence 0..100` to make the detector exit with status `1` when a
document reaches a chosen evidence threshold. Confidence measures configured
signals, not the probability that AI wrote the content.
Use `--fail-score 0..100` to make the detector exit with status `1` when a
document reaches a chosen evidence threshold. The score measures configured
signals, not the probability that AI wrote the content. The former
`--fail-confidence` spelling remains an alias for migration.
Formatter file mode prints transformed text without modifying the source. Use
`--write` explicitly for atomic in-place replacement, or pipe editor buffers
@ -59,7 +60,7 @@ placeholders, tutorial narration, and Unicode punctuation covered by the ASCII
replacement catalog. Every result includes the matching source span, rule ID,
severity, message, and remediation help.
The confidence score aggregates those configured signals. It is not an
The low-care evidence score aggregates those configured signals. It is not an
AI-authorship probability, and a clean result does not prove human authorship or
good quality. Legitimate transcripts, tutorials, multilingual text, and
typographic punctuation can match. Review the source context before acting.
@ -92,7 +93,7 @@ untrusted inputs, but John Slop is not a sandbox or malware scanner.
- [Architecture and public library API](docs/architecture.md)
- [Layered configuration](docs/configuration.md)
- [Commented base and named configuration examples](examples/config/)
- [Heuristics, confidence, and applicability](docs/heuristics.md)
- [Heuristics, evidence scoring, and applicability](docs/heuristics.md)
- [Evaluation corpus and current metrics](docs/evaluation.md)
- [Limitations and responsible interpretation](docs/limitations.md)
- [Development, tests, features, and coverage](docs/development.md)

View File

@ -20,7 +20,7 @@ the details needed to use, embed, and develop John Slop:
precedence, stable-ID merges, validation, and inspection commands.
- [Commented configuration examples](../examples/config/) provide user-wide and
named partial overlays ready to copy.
- [Heuristics and confidence](heuristics.md) documents rule execution,
- [Heuristics and evidence scoring](heuristics.md) documents rule execution,
applicability, evidence damping, normalization, and score interpretation.
- [Evaluation corpus](evaluation.md) records fixture provenance, per-rule
precision/recall, aggregate false positives, and benchmark caveats.
@ -39,7 +39,7 @@ the details needed to use, embed, and develop John Slop:
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
Gitea's locale-aware catalog, its upstream, and safe adoption boundaries.
- [Decision records](decisions/) preserve durable product and engineering
choices, including confidence semantics and input limits.
choices, including evidence-score semantics and input limits.
Release-hardening contracts remain specified in [tasks.md](../tasks.md) and
[AGENTS.md](../AGENTS.md) until that milestone lands.

View File

@ -19,7 +19,7 @@ The main modules have deliberately narrow responsibilities:
- `config` owns the validated embedded replacement catalog and input limits;
- `input` streams files, directory entries, and archive members as documents;
- `detector` compiles configured matchers and path scopes once, then produces
explainable findings and bounded confidence;
explainable findings and a bounded low-care evidence score;
- `diagnostic` binds findings to exact named snapshots as typed `miette`
diagnostics;
- `replacement` produces non-overlapping edits and validates/applies arbitrary
@ -44,7 +44,7 @@ The crate root exposes core entry points and presentation-neutral types:
- `load_config` resolves the configuration embedded in the current build;
- `read_inputs` returns a streaming iterator of per-document results;
- `read_stdin` returns one bounded strict-UTF-8 standard-input document;
- `analyze_source` returns findings and evidence confidence for UTF-8 text;
- `analyze_source` returns findings and an evidence score for UTF-8 text;
- `plan_replacements` returns safe edits against the same source snapshot;
- `apply_replacements` validates boundaries and overlap, then applies edits in
descending byte order while borrowing no-op input unchanged.

View File

@ -10,5 +10,5 @@ the kinds of review problems the project aims to help surface:
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
Project-specific decisions take precedence over external guidance. See
[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and
[ADR 0001](decisions/0001-core-analysis-contracts.md) for evidence scoring and
authorship semantics.

View File

@ -70,8 +70,8 @@ wins at each byte offset, with stable rule ID ordering as the deterministic
tie-break. Edits always refer to the original UTF-8 snapshot.
Each heuristic also configures `severity` as `advice`, `warning`, or `error`.
Severity controls diagnostic presentation and summary counts; confidence still
comes from the independently configured weight and evidence formula.
Severity controls diagnostic presentation and summary counts; the evidence score
still comes from the independently configured weight and aggregation formula.
## Inspection

View File

@ -7,16 +7,16 @@
John Slop needs stable semantics before its proof-of-concept modules are split
into a library, CLI, formatter, and language server. In particular, consumers
need to know what a confidence score means, how source locations are represented,
need to know what the aggregate evidence score means, how source locations are represented,
whether formatting is destructive by default, and whether archive members can
be rewritten.
## Decision
### Confidence represents evidence, not authorship
### The low-care evidence score represents evidence, not authorship
The detector reports confidence that configured, observable low-care signals are
present. It does not report the probability that content is AI-generated and
The detector reports the bounded strength of configured, observable low-care
signals. It does not report the probability that content is AI-generated and
must not present a score as an authorship determination. Every score must be
decomposable into source-labelled findings from stable rule IDs.

View File

@ -34,7 +34,7 @@ The initial server targets stable LSP 3.17 behavior with these assumptions:
handling while replacement behavior is established;
- the server owns snapshots of open documents only and never writes their URIs;
- diagnostics and quick fixes cover configured ASCII replacements, not heuristic
authorship or confidence findings;
authorship or aggregate heuristic findings;
- quick fixes use versioned workspace edits and are rejected when the request
refers to a stale or unknown document version;
- `workspace/didChangeConfiguration` reloads the same layered configuration used

View File

@ -2,7 +2,7 @@
Both `john-slop-fmt` and `john-slop-lsp` are operational. The LSP publishes
configured ASCII-replacement warnings and versioned quick fixes; it does not
publish the broader heuristic confidence findings.
publish the broader heuristic evidence findings.
Install all three binaries somewhere on `PATH`, then confirm the editor-facing
help is available:

View File

@ -25,7 +25,7 @@ Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
suite recomputes these values from the manifest and fails on drift.
The same manifest pins confidence scores for regression characterization:
The same manifest pins evidence scores for regression characterization:
| Corpus case | Score |
| --- | ---: |

View File

@ -1,4 +1,4 @@
# Heuristics and confidence
# Heuristics and evidence scoring
John Slop reports observable low-care signals. It does not classify authorship.
Every contribution to the score is represented by a finding with an exact
@ -11,10 +11,10 @@ generated/vendor/minified paths decide whether each rule applies. Findings are
deduplicated by rule and byte span, then sorted by start offset, end offset, and
rule ID so every adapter observes the same order.
## Confidence formula
## Evidence-score formula
For each rule in deterministic catalog order, only the configured
`occurrence_cap` contributes to confidence. Its first occurrence receives the
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
full weight, then later occurrences receive one half, one quarter, and so on.
Findings beyond the cap remain visible but do not increase the score.
@ -31,7 +31,7 @@ If the resulting evidence sum is `E`, the displayed score is
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
is explicitly reported as below threshold. Empty input and text with no
findings produce score zero and no confidence verdict. Binary, invalid UTF-8,
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
oversized, and unsupported inputs remain typed input errors; they are never
reported as clean analyses.
@ -45,5 +45,5 @@ score of 75 does not mean a 75% chance that AI wrote the input.
The diagnostic showcase exercises five distinct prompt-leakage rules, including
two prompt/response markers. Together with its style and typography findings it
produces evidence score 71 (medium). This is a deterministic policy score, not
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
an estimated probability.

View File

@ -41,7 +41,7 @@ underrepresented. File-extension inference covers a conservative common set and
defaults unknown paths to prose. Teams should validate rules against their own
reviewed material before changing enforcement thresholds.
## Confidence is not probability
## Evidence score is not probability
The score is a deterministic compression of configured weights, occurrence
caps, and category damping. Those values are policy choices and have not been
@ -74,7 +74,7 @@ containers, prose, or source structure. Recursive writes require explicit
The LSP intentionally handles open editor snapshots only, uses full-document
synchronization, and publishes configured ASCII-replacement warnings rather
than the broader heuristic/confidence catalog. It does not read or write a
than the broader heuristic catalog. It does not read or write a
document URI, mutate archives, provide prose rewrites, or implement incremental
range changes. Configuration reload errors retain the last valid configuration.

View File

@ -9,8 +9,8 @@ container, and findings are sorted by byte span and rule ID.
`--output human` renders every finding as a plain `miette` diagnostic with the
source name, exact excerpt and primary span, severity, stable rule code,
message, and remediation help. A final line per document reports the confidence
score/band and advice, warning, and error counts. The narratable renderer
message, and remediation help. A final line per document reports the low-care
evidence score/band and advice, warning, and error counts. The narratable renderer
deliberately avoids ANSI escapes.
## Miette
@ -29,19 +29,19 @@ finding line contains the source identity, half-open UTF-8 byte range, severity,
rule ID, and message. Recoverable input errors go to stderr in human and concise
modes.
## JSON Lines schema version 1
## JSON Lines schema version 2
`--output json` emits one compact JSON object per line. An analyzed document has
`kind: "analysis"` and this shape:
```json
{
"schema_version": 1,
"schema_version": 2,
"kind": "analysis",
"source": "article.md",
"summary": {
"confidence_score": 17,
"confidence_band": "below-threshold",
"evidence_score": 17,
"evidence_band": "below-threshold",
"advice": 1,
"warnings": 0,
"errors": 0,
@ -66,7 +66,7 @@ modes.
}
```
`confidence_band` is `null` when there is no evidence. Spans are zero-based,
`evidence_band` is `null` when there is no evidence. Spans are zero-based,
half-open UTF-8 byte offsets into the exact source snapshot, not character or
UTF-16 positions. `fix` is `null` for findings without a safe configured edit.
@ -74,7 +74,7 @@ A recoverable input failure has `kind: "input-error"`:
```json
{
"schema_version": 1,
"schema_version": 2,
"kind": "input-error",
"source": "broken.txt",
"message": "input broken.txt is not valid UTF-8"
@ -84,5 +84,7 @@ A recoverable input failure has `kind: "input-error"`:
JSON records are the only stdout content in JSON mode and never contain ANSI
escapes or logs. Fatal argument/configuration/setup failures can occur before
the JSON stream starts; they return status 2, leave stdout empty, and report on
stderr. Additive fields may appear within schema version 1; incompatible changes
require a new schema version.
stderr. Additive fields may appear within schema version 2; incompatible changes
require a new schema version. Schema version 2 renamed the version 1 summary
fields `confidence_score` and `confidence_band` to `evidence_score` and
`evidence_band`; version 1 records are no longer emitted.

View File

@ -5,7 +5,7 @@
Run the detector over one or more paths:
```sh
cargo run --bin john-slop -- [--fail-confidence 0..100] <PATH>...
cargo run --bin john-slop -- [--fail-score 0..100] <PATH>...
cargo run --bin john-slop -- --stdin [--stdin-filename article.md]
```
@ -28,7 +28,7 @@ annotations when the default `fancy-diagnostics` feature is enabled; it falls
back to human mode without that feature. Human mode renders uncolored plain
source excerpts, exact labels, remediation help, and a summary. Concise mode
emits stable single-line finding and summary records. JSON mode emits one object
per analyzed document with `schema_version: 1`, summary fields, exact byte spans,
per analyzed document with `schema_version: 2`, summary fields, exact byte spans,
and every finding. Machine modes never emit ANSI escapes or logs on stdout.
Findings cover the embedded or overridden heuristic catalog and retain exact
@ -37,14 +37,15 @@ for the human/concise formats and complete JSON Lines schema.
Exit statuses are stable:
- `0`: every analyzed document is below the requested confidence threshold;
- `0`: every analyzed document is below the requested evidence-score threshold;
- `1`: at least one document reaches the threshold;
- `2`: arguments are invalid or an operational/input error prevents completion.
`--fail-confidence SCORE` selects an exact inclusive score threshold and
`--fail-score SCORE` selects an exact inclusive score threshold and
defaults to 100. `--fail-level low|medium|high` instead uses the corresponding
threshold from the fully resolved configuration. The two options are mutually
exclusive.
exclusive. The old `--fail-confidence` spelling remains a visible alias for
migration.
Input failures are isolated per document whenever iteration can continue. Valid
documents still emit all findings, while any partial input failure makes the
@ -53,8 +54,8 @@ concise input errors go to stderr. JSON mode emits versioned `input-error`
records on stdout alongside `analysis` records and leaves stderr empty unless a
fatal setup failure prevents the output contract from starting.
Confidence is a summary of observable configured evidence. It is not an
AI-authorship probability. See [Heuristics and confidence](heuristics.md) for
The low-care evidence score summarizes observable configured evidence. It is
not an AI-authorship probability. See [Heuristics and evidence scoring](heuristics.md) for
the formula and [Limitations](limitations.md) before using thresholds in review
workflows.
@ -77,7 +78,7 @@ formatted output. The executable fixture and protocol tests validate the
library, detector, formatter, and LSP against the shared replacement contract.
The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase)
prints every current prose finding in graphical Miette form and intentionally
exits `1` when its low confidence threshold is reached.
exits `1` when its low evidence threshold is reached.
## Supported input policy
@ -140,7 +141,7 @@ It stores only currently open UTF-8 snapshots, their URI, language ID, and
monotonically increasing version. On open or an accepted newer full-document
change it publishes warnings only for enabled Unicode-to-ASCII replacements.
Each diagnostic has source `john-slop`, a stable replacement-rule code, the
document version, and a UTF-16 range. Heuristic confidence findings are not
document version, and a UTF-16 range. Broader heuristic findings are not
published by this initial server.
Quick fixes are recomputed from the current snapshot with the shared replacement

View File

@ -298,7 +298,7 @@ Optional follow-up, intentionally not required for this milestone:
- [x] Rename the public domain model from ambiguous confidence terminology to
`EvidenceScore`, `EvidenceBand`, and `EvidenceThresholds`; retain deprecated
Rust type aliases and the legacy `--fail-confidence` CLI alias for migration.
- [ ] Publish the low-care evidence-score wording across terminal output,
- [x] Publish the low-care evidence-score wording across terminal output,
examples, and documentation; rename `--fail-confidence` to `--fail-score` and
version the JSON field rename as schema version 2.
- [ ] Run the complete quality and coverage gates, refresh generated badges, and