john-slop/docs/heuristics.md

57 lines
3.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Heuristics and evidence scoring
John Slop reports observable low-care signals. It does not classify authorship.
Every contribution to the score is represented by a finding with an exact
source span, stable rule ID, category, message, help, and configured weight.
The detector compiles the resolved rule catalog once. Exact phrases, regular
expressions, repeated-pattern rules, configured Unicode replacements, UTS #39
ASCII lookalikes, and invisible/control characters share the same finding
model. Content kind, extension, path globs, and conventional generated/vendor/
minified paths decide whether each rule applies. Findings are deduplicated by
rule and byte span, then sorted by start offset, end offset, and rule ID so
every adapter observes the same order.
Unicode review rules are deliberately distinct from replacement rules. The
detector can flag a lookalike, invisible scalar, or direction-changing control
at its exact span, but only the explicit ten-entry safe replacement allowlist
can produce a formatter edit or LSP quick fix. A safe replacement finding is
not duplicated as a confusable finding.
## Evidence-score formula
For each rule in deterministic catalog order, only the configured
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
full weight, then later occurrences receive one half, one quarter, and so on.
Findings beyond the cap remain visible but do not increase the score.
Rules are grouped by category as correlated evidence families. The first
matching rule in a category receives its full occurrence-adjusted contribution;
later distinct prompt-leakage rules receive successive factors of `0.90`, while
style, typography, and source-code rules receive successive factors of `0.75`.
Prompt/response markers and copied assistant disclaimers are stronger concrete
artifacts than generic stylistic similarities, but the remaining damping still
prevents a cluster of related phrases from being treated as fully independent
proof.
If the resulting evidence sum is `E`, the displayed score is
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
is explicitly reported as below threshold. Empty input and text with no
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
oversized, and unsupported inputs remain typed input errors; they are never
reported as clean analyses.
Every `Analysis` retains its complete finding list and can produce a compact
summary containing the numeric score, optional named band, and separate advice,
warning, and error counts. Summary generation never filters the evidence used
to explain the score.
Weights and thresholds are review policy, not calibrated probabilities. A
score of 75 does not mean a 75% chance that AI wrote the input.
The diagnostic showcase exercises five distinct prompt-leakage rules, including
two prompt/response markers. Together with its style and typography findings it
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
an estimated probability.