john-slop/docs/heuristics.md

3.5 KiB
Raw Permalink Blame History

Heuristics and evidence scoring

John Slop reports observable low-care signals. It does not classify authorship. Every contribution to the score is represented by a finding with an exact source span, stable rule ID, category, message, help, and configured weight.

The detector compiles the resolved rule catalog once. Exact phrases, regular expressions, repeated-pattern rules, configured Unicode replacements, and invisible/control characters share the same finding model. Content kind, extension, path globs, and conventional generated/vendor/ minified paths decide whether each rule applies. Findings are deduplicated by rule and byte span, then sorted by start offset, end offset, and rule ID so every adapter observes the same order.

Additional exact-phrase, regex, and repeated-regex rules can be supplied at runtime through diagnostics.toml or repeatable --diagnostics arguments. They receive the same validation, scoping, scoring, ordering, CLI presentation, and LSP publication as embedded rules. New matcher algorithms and categories remain code changes rather than executable configuration.

The generated Unicode 16.0 profile maps every non-ASCII scalar with a printable ASCII UTS #39 skeleton after explicitly reviewed compatibility-width and punctuation mappings. The default replace policy gives each of these findings a formatter/LSP edit; review keeps the same exact diagnostics without edits, and off suppresses the generated lookalikes. Invisible and direction-changing controls remain diagnostic-only because deleting them silently is not a generally safe edit.

Evidence-score formula

For each rule in deterministic catalog order, only the configured occurrence_cap contributes to the evidence score. Its first occurrence receives the full weight, then later occurrences receive one half, one quarter, and so on. Findings beyond the cap remain visible but do not increase the score.

Rules are grouped by category as correlated evidence families. The first matching rule in a category receives its full occurrence-adjusted contribution; later distinct prompt-leakage rules receive successive factors of 0.90, while style, typography, and source-code rules receive successive factors of 0.75. Prompt/response markers and copied assistant disclaimers are stronger concrete artifacts than generic stylistic similarities, but the remaining damping still prevents a cluster of related phrases from being treated as fully independent proof.

If the resulting evidence sum is E, the displayed score is round(100 × E / (E + 50)), bounded to 0..=100. The configured low, medium, and high thresholds name the resulting band. Evidence below low is explicitly reported as below threshold. Empty input and text with no findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8, oversized, and unsupported inputs remain typed input errors; they are never reported as clean analyses.

Every Analysis retains its complete finding list and can produce a compact summary containing the numeric score, optional named band, and separate advice, warning, and error counts. Summary generation never filters the evidence used to explain the score.

Weights and thresholds are review policy, not calibrated probabilities. A score of 75 does not mean a 75% chance that AI wrote the input.

The diagnostic showcase exercises five distinct prompt-leakage rules, including two prompt/response markers. Together with its style and typography findings it produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not an estimated probability.