docs: publish evidence score terminology
This commit is contained in:
parent
a4dd4e2c5f
commit
95d4067dc8
10
AGENTS.md
10
AGENTS.md
|
|
@ -7,8 +7,8 @@ insufficiently reviewed text and source code. It reports concrete signals that
|
|||
are often associated with careless AI-assisted work. It must not claim to prove
|
||||
that a person used AI, identify a model, or infer authorship from writing style.
|
||||
|
||||
Treat a score as **confidence that configured signals are present**, not as the
|
||||
probability that content is AI-generated. User-facing language must say
|
||||
Treat the **low-care evidence score** as the bounded strength of configured
|
||||
signals, not as the probability that content is AI-generated. User-facing language must say
|
||||
"suspicious", "low-care signal", or "needs review" rather than making an
|
||||
accusation. A result must always be explainable from the findings that produced
|
||||
it.
|
||||
|
|
@ -77,7 +77,7 @@ struct Finding {
|
|||
|
||||
struct Analysis {
|
||||
findings: Vec<Finding>,
|
||||
confidence: Confidence,
|
||||
evidence_score: EvidenceScore,
|
||||
}
|
||||
|
||||
struct TextEdit {
|
||||
|
|
@ -96,7 +96,7 @@ Rule IDs are stable, unique, lowercase kebab-case identifiers. Findings are
|
|||
sorted by source identity, span start, then rule ID so output is deterministic.
|
||||
Never use a hash-map iteration order as output order.
|
||||
|
||||
## Heuristics and confidence
|
||||
## Heuristics and evidence scoring
|
||||
|
||||
Store data-driven heuristic definitions in `assets/heuristics.toml`, separately
|
||||
from replacement configuration. Compile and validate rules once at startup.
|
||||
|
|
@ -116,7 +116,7 @@ Start with high-precision, independently testable signals:
|
|||
that merely narrate obvious code, using conservative language-specific rules.
|
||||
|
||||
Avoid weak demographic, dialect, spelling, grammar, vocabulary, or
|
||||
English-fluency proxies. Repetition alone must not trigger a high-confidence
|
||||
English-fluency proxies. Repetition alone must not trigger a high evidence-band
|
||||
verdict. Give users a per-rule disable/weight override and path/language scopes.
|
||||
|
||||
Scoring must be documented and deterministic. Use bounded weighted evidence,
|
||||
|
|
|
|||
|
|
@ -6,11 +6,18 @@ All notable changes to John Slop are recorded here. The format follows
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- Renamed the ambiguous confidence metric to low-care evidence score throughout
|
||||
the Rust API, CLI, terminal output, and JSON schema version 2. Deprecated Rust
|
||||
type aliases and `--fail-confidence` remain available for migration; the
|
||||
scoring formula and values are unchanged.
|
||||
|
||||
## [0.1.0] - 2026-09-29
|
||||
|
||||
### Added
|
||||
|
||||
- Local, explainable heuristic analysis with bounded evidence confidence.
|
||||
- Local, explainable heuristic analysis with a bounded low-care evidence score.
|
||||
- Layered embedded, user, named, explicit, and CLI configuration.
|
||||
- File, directory, ZIP, office-container, and stdin input adapters with limits.
|
||||
- Human, graphical Miette, concise, and versioned JSON detector output.
|
||||
|
|
|
|||
13
README.md
13
README.md
|
|
@ -14,7 +14,7 @@ ZIP-based office documents. Eleven conservative, configurable rules cover
|
|||
copied assistant scaffolding, generic style residue, source-code placeholders,
|
||||
tutorial narration, and configured non-ASCII characters. Rules retain exact
|
||||
source spans, apply path/content scopes, and feed a bounded, explainable
|
||||
confidence score. The detector renders plain or graphical Miette diagnostics,
|
||||
low-care evidence score. The detector renders plain or graphical Miette diagnostics,
|
||||
concise records, or versioned JSON Lines. The formatter applies only configured
|
||||
Unicode-to-ASCII replacements, supports check/diff modes, and requires explicit
|
||||
opt-in for atomic file writes. The stdio LSP publishes those same replacement
|
||||
|
|
@ -34,9 +34,10 @@ cargo run --bin john-slop-fmt -- --check path/to/document
|
|||
cargo run --bin john-slop-lsp
|
||||
```
|
||||
|
||||
Use `--fail-confidence 0..100` to make the detector exit with status `1` when a
|
||||
document reaches a chosen evidence threshold. Confidence measures configured
|
||||
signals, not the probability that AI wrote the content.
|
||||
Use `--fail-score 0..100` to make the detector exit with status `1` when a
|
||||
document reaches a chosen evidence threshold. The score measures configured
|
||||
signals, not the probability that AI wrote the content. The former
|
||||
`--fail-confidence` spelling remains an alias for migration.
|
||||
|
||||
Formatter file mode prints transformed text without modifying the source. Use
|
||||
`--write` explicitly for atomic in-place replacement, or pipe editor buffers
|
||||
|
|
@ -59,7 +60,7 @@ placeholders, tutorial narration, and Unicode punctuation covered by the ASCII
|
|||
replacement catalog. Every result includes the matching source span, rule ID,
|
||||
severity, message, and remediation help.
|
||||
|
||||
The confidence score aggregates those configured signals. It is not an
|
||||
The low-care evidence score aggregates those configured signals. It is not an
|
||||
AI-authorship probability, and a clean result does not prove human authorship or
|
||||
good quality. Legitimate transcripts, tutorials, multilingual text, and
|
||||
typographic punctuation can match. Review the source context before acting.
|
||||
|
|
@ -92,7 +93,7 @@ untrusted inputs, but John Slop is not a sandbox or malware scanner.
|
|||
- [Architecture and public library API](docs/architecture.md)
|
||||
- [Layered configuration](docs/configuration.md)
|
||||
- [Commented base and named configuration examples](examples/config/)
|
||||
- [Heuristics, confidence, and applicability](docs/heuristics.md)
|
||||
- [Heuristics, evidence scoring, and applicability](docs/heuristics.md)
|
||||
- [Evaluation corpus and current metrics](docs/evaluation.md)
|
||||
- [Limitations and responsible interpretation](docs/limitations.md)
|
||||
- [Development, tests, features, and coverage](docs/development.md)
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ the details needed to use, embed, and develop John Slop:
|
|||
precedence, stable-ID merges, validation, and inspection commands.
|
||||
- [Commented configuration examples](../examples/config/) provide user-wide and
|
||||
named partial overlays ready to copy.
|
||||
- [Heuristics and confidence](heuristics.md) documents rule execution,
|
||||
- [Heuristics and evidence scoring](heuristics.md) documents rule execution,
|
||||
applicability, evidence damping, normalization, and score interpretation.
|
||||
- [Evaluation corpus](evaluation.md) records fixture provenance, per-rule
|
||||
precision/recall, aggregate false positives, and benchmark caveats.
|
||||
|
|
@ -39,7 +39,7 @@ the details needed to use, embed, and develop John Slop:
|
|||
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
|
||||
Gitea's locale-aware catalog, its upstream, and safe adoption boundaries.
|
||||
- [Decision records](decisions/) preserve durable product and engineering
|
||||
choices, including confidence semantics and input limits.
|
||||
choices, including evidence-score semantics and input limits.
|
||||
|
||||
Release-hardening contracts remain specified in [tasks.md](../tasks.md) and
|
||||
[AGENTS.md](../AGENTS.md) until that milestone lands.
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ The main modules have deliberately narrow responsibilities:
|
|||
- `config` owns the validated embedded replacement catalog and input limits;
|
||||
- `input` streams files, directory entries, and archive members as documents;
|
||||
- `detector` compiles configured matchers and path scopes once, then produces
|
||||
explainable findings and bounded confidence;
|
||||
explainable findings and a bounded low-care evidence score;
|
||||
- `diagnostic` binds findings to exact named snapshots as typed `miette`
|
||||
diagnostics;
|
||||
- `replacement` produces non-overlapping edits and validates/applies arbitrary
|
||||
|
|
@ -44,7 +44,7 @@ The crate root exposes core entry points and presentation-neutral types:
|
|||
- `load_config` resolves the configuration embedded in the current build;
|
||||
- `read_inputs` returns a streaming iterator of per-document results;
|
||||
- `read_stdin` returns one bounded strict-UTF-8 standard-input document;
|
||||
- `analyze_source` returns findings and evidence confidence for UTF-8 text;
|
||||
- `analyze_source` returns findings and an evidence score for UTF-8 text;
|
||||
- `plan_replacements` returns safe edits against the same source snapshot;
|
||||
- `apply_replacements` validates boundaries and overlap, then applies edits in
|
||||
descending byte order while borrowing no-op input unchanged.
|
||||
|
|
|
|||
|
|
@ -10,5 +10,5 @@ the kinds of review problems the project aims to help surface:
|
|||
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
|
||||
|
||||
Project-specific decisions take precedence over external guidance. See
|
||||
[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and
|
||||
[ADR 0001](decisions/0001-core-analysis-contracts.md) for evidence scoring and
|
||||
authorship semantics.
|
||||
|
|
|
|||
|
|
@ -70,8 +70,8 @@ wins at each byte offset, with stable rule ID ordering as the deterministic
|
|||
tie-break. Edits always refer to the original UTF-8 snapshot.
|
||||
|
||||
Each heuristic also configures `severity` as `advice`, `warning`, or `error`.
|
||||
Severity controls diagnostic presentation and summary counts; confidence still
|
||||
comes from the independently configured weight and evidence formula.
|
||||
Severity controls diagnostic presentation and summary counts; the evidence score
|
||||
still comes from the independently configured weight and aggregation formula.
|
||||
|
||||
## Inspection
|
||||
|
||||
|
|
|
|||
|
|
@ -7,16 +7,16 @@
|
|||
|
||||
John Slop needs stable semantics before its proof-of-concept modules are split
|
||||
into a library, CLI, formatter, and language server. In particular, consumers
|
||||
need to know what a confidence score means, how source locations are represented,
|
||||
need to know what the aggregate evidence score means, how source locations are represented,
|
||||
whether formatting is destructive by default, and whether archive members can
|
||||
be rewritten.
|
||||
|
||||
## Decision
|
||||
|
||||
### Confidence represents evidence, not authorship
|
||||
### The low-care evidence score represents evidence, not authorship
|
||||
|
||||
The detector reports confidence that configured, observable low-care signals are
|
||||
present. It does not report the probability that content is AI-generated and
|
||||
The detector reports the bounded strength of configured, observable low-care
|
||||
signals. It does not report the probability that content is AI-generated and
|
||||
must not present a score as an authorship determination. Every score must be
|
||||
decomposable into source-labelled findings from stable rule IDs.
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ The initial server targets stable LSP 3.17 behavior with these assumptions:
|
|||
handling while replacement behavior is established;
|
||||
- the server owns snapshots of open documents only and never writes their URIs;
|
||||
- diagnostics and quick fixes cover configured ASCII replacements, not heuristic
|
||||
authorship or confidence findings;
|
||||
authorship or aggregate heuristic findings;
|
||||
- quick fixes use versioned workspace edits and are rejected when the request
|
||||
refers to a stale or unknown document version;
|
||||
- `workspace/didChangeConfiguration` reloads the same layered configuration used
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Both `john-slop-fmt` and `john-slop-lsp` are operational. The LSP publishes
|
||||
configured ASCII-replacement warnings and versioned quick fixes; it does not
|
||||
publish the broader heuristic confidence findings.
|
||||
publish the broader heuristic evidence findings.
|
||||
|
||||
Install all three binaries somewhere on `PATH`, then confirm the editor-facing
|
||||
help is available:
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ Aggregate labeled-rule counts are 11 true positives, 0 false positives, and 0
|
|||
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
|
||||
suite recomputes these values from the manifest and fails on drift.
|
||||
|
||||
The same manifest pins confidence scores for regression characterization:
|
||||
The same manifest pins evidence scores for regression characterization:
|
||||
|
||||
| Corpus case | Score |
|
||||
| --- | ---: |
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Heuristics and confidence
|
||||
# Heuristics and evidence scoring
|
||||
|
||||
John Slop reports observable low-care signals. It does not classify authorship.
|
||||
Every contribution to the score is represented by a finding with an exact
|
||||
|
|
@ -11,10 +11,10 @@ generated/vendor/minified paths decide whether each rule applies. Findings are
|
|||
deduplicated by rule and byte span, then sorted by start offset, end offset, and
|
||||
rule ID so every adapter observes the same order.
|
||||
|
||||
## Confidence formula
|
||||
## Evidence-score formula
|
||||
|
||||
For each rule in deterministic catalog order, only the configured
|
||||
`occurrence_cap` contributes to confidence. Its first occurrence receives the
|
||||
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
|
||||
full weight, then later occurrences receive one half, one quarter, and so on.
|
||||
Findings beyond the cap remain visible but do not increase the score.
|
||||
|
||||
|
|
@ -31,7 +31,7 @@ If the resulting evidence sum is `E`, the displayed score is
|
|||
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
|
||||
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
|
||||
is explicitly reported as below threshold. Empty input and text with no
|
||||
findings produce score zero and no confidence verdict. Binary, invalid UTF-8,
|
||||
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
|
||||
oversized, and unsupported inputs remain typed input errors; they are never
|
||||
reported as clean analyses.
|
||||
|
||||
|
|
@ -45,5 +45,5 @@ score of 75 does not mean a 75% chance that AI wrote the input.
|
|||
|
||||
The diagnostic showcase exercises five distinct prompt-leakage rules, including
|
||||
two prompt/response markers. Together with its style and typography findings it
|
||||
produces evidence score 71 (medium). This is a deterministic policy score, not
|
||||
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
|
||||
an estimated probability.
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ underrepresented. File-extension inference covers a conservative common set and
|
|||
defaults unknown paths to prose. Teams should validate rules against their own
|
||||
reviewed material before changing enforcement thresholds.
|
||||
|
||||
## Confidence is not probability
|
||||
## Evidence score is not probability
|
||||
|
||||
The score is a deterministic compression of configured weights, occurrence
|
||||
caps, and category damping. Those values are policy choices and have not been
|
||||
|
|
@ -74,7 +74,7 @@ containers, prose, or source structure. Recursive writes require explicit
|
|||
|
||||
The LSP intentionally handles open editor snapshots only, uses full-document
|
||||
synchronization, and publishes configured ASCII-replacement warnings rather
|
||||
than the broader heuristic/confidence catalog. It does not read or write a
|
||||
than the broader heuristic catalog. It does not read or write a
|
||||
document URI, mutate archives, provide prose rewrites, or implement incremental
|
||||
range changes. Configuration reload errors retain the last valid configuration.
|
||||
|
||||
|
|
|
|||
|
|
@ -9,8 +9,8 @@ container, and findings are sorted by byte span and rule ID.
|
|||
|
||||
`--output human` renders every finding as a plain `miette` diagnostic with the
|
||||
source name, exact excerpt and primary span, severity, stable rule code,
|
||||
message, and remediation help. A final line per document reports the confidence
|
||||
score/band and advice, warning, and error counts. The narratable renderer
|
||||
message, and remediation help. A final line per document reports the low-care
|
||||
evidence score/band and advice, warning, and error counts. The narratable renderer
|
||||
deliberately avoids ANSI escapes.
|
||||
|
||||
## Miette
|
||||
|
|
@ -29,19 +29,19 @@ finding line contains the source identity, half-open UTF-8 byte range, severity,
|
|||
rule ID, and message. Recoverable input errors go to stderr in human and concise
|
||||
modes.
|
||||
|
||||
## JSON Lines schema version 1
|
||||
## JSON Lines schema version 2
|
||||
|
||||
`--output json` emits one compact JSON object per line. An analyzed document has
|
||||
`kind: "analysis"` and this shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"schema_version": 2,
|
||||
"kind": "analysis",
|
||||
"source": "article.md",
|
||||
"summary": {
|
||||
"confidence_score": 17,
|
||||
"confidence_band": "below-threshold",
|
||||
"evidence_score": 17,
|
||||
"evidence_band": "below-threshold",
|
||||
"advice": 1,
|
||||
"warnings": 0,
|
||||
"errors": 0,
|
||||
|
|
@ -66,7 +66,7 @@ modes.
|
|||
}
|
||||
```
|
||||
|
||||
`confidence_band` is `null` when there is no evidence. Spans are zero-based,
|
||||
`evidence_band` is `null` when there is no evidence. Spans are zero-based,
|
||||
half-open UTF-8 byte offsets into the exact source snapshot, not character or
|
||||
UTF-16 positions. `fix` is `null` for findings without a safe configured edit.
|
||||
|
||||
|
|
@ -74,7 +74,7 @@ A recoverable input failure has `kind: "input-error"`:
|
|||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"schema_version": 2,
|
||||
"kind": "input-error",
|
||||
"source": "broken.txt",
|
||||
"message": "input broken.txt is not valid UTF-8"
|
||||
|
|
@ -84,5 +84,7 @@ A recoverable input failure has `kind: "input-error"`:
|
|||
JSON records are the only stdout content in JSON mode and never contain ANSI
|
||||
escapes or logs. Fatal argument/configuration/setup failures can occur before
|
||||
the JSON stream starts; they return status 2, leave stdout empty, and report on
|
||||
stderr. Additive fields may appear within schema version 1; incompatible changes
|
||||
require a new schema version.
|
||||
stderr. Additive fields may appear within schema version 2; incompatible changes
|
||||
require a new schema version. Schema version 2 renamed the version 1 summary
|
||||
fields `confidence_score` and `confidence_band` to `evidence_score` and
|
||||
`evidence_band`; version 1 records are no longer emitted.
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
Run the detector over one or more paths:
|
||||
|
||||
```sh
|
||||
cargo run --bin john-slop -- [--fail-confidence 0..100] <PATH>...
|
||||
cargo run --bin john-slop -- [--fail-score 0..100] <PATH>...
|
||||
cargo run --bin john-slop -- --stdin [--stdin-filename article.md]
|
||||
```
|
||||
|
||||
|
|
@ -28,7 +28,7 @@ annotations when the default `fancy-diagnostics` feature is enabled; it falls
|
|||
back to human mode without that feature. Human mode renders uncolored plain
|
||||
source excerpts, exact labels, remediation help, and a summary. Concise mode
|
||||
emits stable single-line finding and summary records. JSON mode emits one object
|
||||
per analyzed document with `schema_version: 1`, summary fields, exact byte spans,
|
||||
per analyzed document with `schema_version: 2`, summary fields, exact byte spans,
|
||||
and every finding. Machine modes never emit ANSI escapes or logs on stdout.
|
||||
|
||||
Findings cover the embedded or overridden heuristic catalog and retain exact
|
||||
|
|
@ -37,14 +37,15 @@ for the human/concise formats and complete JSON Lines schema.
|
|||
|
||||
Exit statuses are stable:
|
||||
|
||||
- `0`: every analyzed document is below the requested confidence threshold;
|
||||
- `0`: every analyzed document is below the requested evidence-score threshold;
|
||||
- `1`: at least one document reaches the threshold;
|
||||
- `2`: arguments are invalid or an operational/input error prevents completion.
|
||||
|
||||
`--fail-confidence SCORE` selects an exact inclusive score threshold and
|
||||
`--fail-score SCORE` selects an exact inclusive score threshold and
|
||||
defaults to 100. `--fail-level low|medium|high` instead uses the corresponding
|
||||
threshold from the fully resolved configuration. The two options are mutually
|
||||
exclusive.
|
||||
exclusive. The old `--fail-confidence` spelling remains a visible alias for
|
||||
migration.
|
||||
|
||||
Input failures are isolated per document whenever iteration can continue. Valid
|
||||
documents still emit all findings, while any partial input failure makes the
|
||||
|
|
@ -53,8 +54,8 @@ concise input errors go to stderr. JSON mode emits versioned `input-error`
|
|||
records on stdout alongside `analysis` records and leaves stderr empty unless a
|
||||
fatal setup failure prevents the output contract from starting.
|
||||
|
||||
Confidence is a summary of observable configured evidence. It is not an
|
||||
AI-authorship probability. See [Heuristics and confidence](heuristics.md) for
|
||||
The low-care evidence score summarizes observable configured evidence. It is
|
||||
not an AI-authorship probability. See [Heuristics and evidence scoring](heuristics.md) for
|
||||
the formula and [Limitations](limitations.md) before using thresholds in review
|
||||
workflows.
|
||||
|
||||
|
|
@ -77,7 +78,7 @@ formatted output. The executable fixture and protocol tests validate the
|
|||
library, detector, formatter, and LSP against the shared replacement contract.
|
||||
The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase)
|
||||
prints every current prose finding in graphical Miette form and intentionally
|
||||
exits `1` when its low confidence threshold is reached.
|
||||
exits `1` when its low evidence threshold is reached.
|
||||
|
||||
## Supported input policy
|
||||
|
||||
|
|
@ -140,7 +141,7 @@ It stores only currently open UTF-8 snapshots, their URI, language ID, and
|
|||
monotonically increasing version. On open or an accepted newer full-document
|
||||
change it publishes warnings only for enabled Unicode-to-ASCII replacements.
|
||||
Each diagnostic has source `john-slop`, a stable replacement-rule code, the
|
||||
document version, and a UTF-16 range. Heuristic confidence findings are not
|
||||
document version, and a UTF-16 range. Broader heuristic findings are not
|
||||
published by this initial server.
|
||||
|
||||
Quick fixes are recomputed from the current snapshot with the shared replacement
|
||||
|
|
|
|||
2
tasks.md
2
tasks.md
|
|
@ -298,7 +298,7 @@ Optional follow-up, intentionally not required for this milestone:
|
|||
- [x] Rename the public domain model from ambiguous confidence terminology to
|
||||
`EvidenceScore`, `EvidenceBand`, and `EvidenceThresholds`; retain deprecated
|
||||
Rust type aliases and the legacy `--fail-confidence` CLI alias for migration.
|
||||
- [ ] Publish the low-care evidence-score wording across terminal output,
|
||||
- [x] Publish the low-care evidence-score wording across terminal output,
|
||||
examples, and documentation; rename `--fail-confidence` to `--fail-score` and
|
||||
version the JSON field rename as schema version 2.
|
||||
- [ ] Run the complete quality and coverage gates, refresh generated badges, and
|
||||
|
|
|
|||
Loading…
Reference in New Issue