john-slop/docs/heuristics.md

65 lines
3.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Heuristics and evidence scoring
John Slop reports observable low-care signals. It does not classify authorship.
Every contribution to the score is represented by a finding with an exact
source span, stable rule ID, category, message, help, and configured weight.
The detector compiles the resolved rule catalog once. Exact phrases, regular
expressions, repeated-pattern rules, configured Unicode replacements, and
invisible/control characters share the same finding
model. Content kind, extension, path globs, and conventional generated/vendor/
minified paths decide whether each rule applies. Findings are deduplicated by
rule and byte span, then sorted by start offset, end offset, and rule ID so
every adapter observes the same order.
Additional exact-phrase, regex, and repeated-regex rules can be supplied at
runtime through `diagnostics.toml` or repeatable `--diagnostics` arguments.
They receive the same validation, scoping, scoring, ordering, CLI presentation,
and LSP publication as embedded rules. New matcher algorithms and categories
remain code changes rather than executable configuration.
The generated Unicode 16.0 profile maps every non-ASCII scalar with a printable
ASCII UTS #39 skeleton after explicitly reviewed compatibility-width and
punctuation mappings.
The default `replace` policy gives each of these findings a formatter/LSP edit;
`review` keeps the same exact diagnostics without edits, and `off` suppresses
the generated lookalikes. Invisible and direction-changing controls remain
diagnostic-only because deleting them silently is not a generally safe edit.
## Evidence-score formula
For each rule in deterministic catalog order, only the configured
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
full weight, then later occurrences receive one half, one quarter, and so on.
Findings beyond the cap remain visible but do not increase the score.
Rules are grouped by category as correlated evidence families. The first
matching rule in a category receives its full occurrence-adjusted contribution;
later distinct prompt-leakage rules receive successive factors of `0.90`, while
style, typography, and source-code rules receive successive factors of `0.75`.
Prompt/response markers and copied assistant disclaimers are stronger concrete
artifacts than generic stylistic similarities, but the remaining damping still
prevents a cluster of related phrases from being treated as fully independent
proof.
If the resulting evidence sum is `E`, the displayed score is
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
is explicitly reported as below threshold. Empty input and text with no
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
oversized, and unsupported inputs remain typed input errors; they are never
reported as clean analyses.
Every `Analysis` retains its complete finding list and can produce a compact
summary containing the numeric score, optional named band, and separate advice,
warning, and error counts. Summary generation never filters the evidence used
to explain the score.
Weights and thresholds are review policy, not calibrated probabilities. A
score of 75 does not mean a 75% chance that AI wrote the input.
The diagnostic showcase exercises five distinct prompt-leakage rules, including
two prompt/response markers. Together with its style and typography findings it
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
an estimated probability.