65 lines
3.5 KiB
Markdown
65 lines
3.5 KiB
Markdown
# Heuristics and evidence scoring
|
||
|
||
John Slop reports observable low-care signals. It does not classify authorship.
|
||
Every contribution to the score is represented by a finding with an exact
|
||
source span, stable rule ID, category, message, help, and configured weight.
|
||
|
||
The detector compiles the resolved rule catalog once. Exact phrases, regular
|
||
expressions, repeated-pattern rules, configured Unicode replacements, and
|
||
invisible/control characters share the same finding
|
||
model. Content kind, extension, path globs, and conventional generated/vendor/
|
||
minified paths decide whether each rule applies. Findings are deduplicated by
|
||
rule and byte span, then sorted by start offset, end offset, and rule ID so
|
||
every adapter observes the same order.
|
||
|
||
Additional exact-phrase, regex, and repeated-regex rules can be supplied at
|
||
runtime through `diagnostics.toml` or repeatable `--diagnostics` arguments.
|
||
They receive the same validation, scoping, scoring, ordering, CLI presentation,
|
||
and LSP publication as embedded rules. New matcher algorithms and categories
|
||
remain code changes rather than executable configuration.
|
||
|
||
The generated Unicode 16.0 profile maps every non-ASCII scalar with a printable
|
||
ASCII UTS #39 skeleton after explicitly reviewed compatibility-width and
|
||
punctuation mappings.
|
||
The default `replace` policy gives each of these findings a formatter/LSP edit;
|
||
`review` keeps the same exact diagnostics without edits, and `off` suppresses
|
||
the generated lookalikes. Invisible and direction-changing controls remain
|
||
diagnostic-only because deleting them silently is not a generally safe edit.
|
||
|
||
## Evidence-score formula
|
||
|
||
For each rule in deterministic catalog order, only the configured
|
||
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
|
||
full weight, then later occurrences receive one half, one quarter, and so on.
|
||
Findings beyond the cap remain visible but do not increase the score.
|
||
|
||
Rules are grouped by category as correlated evidence families. The first
|
||
matching rule in a category receives its full occurrence-adjusted contribution;
|
||
later distinct prompt-leakage rules receive successive factors of `0.90`, while
|
||
style, typography, and source-code rules receive successive factors of `0.75`.
|
||
Prompt/response markers and copied assistant disclaimers are stronger concrete
|
||
artifacts than generic stylistic similarities, but the remaining damping still
|
||
prevents a cluster of related phrases from being treated as fully independent
|
||
proof.
|
||
|
||
If the resulting evidence sum is `E`, the displayed score is
|
||
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
|
||
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
|
||
is explicitly reported as below threshold. Empty input and text with no
|
||
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
|
||
oversized, and unsupported inputs remain typed input errors; they are never
|
||
reported as clean analyses.
|
||
|
||
Every `Analysis` retains its complete finding list and can produce a compact
|
||
summary containing the numeric score, optional named band, and separate advice,
|
||
warning, and error counts. Summary generation never filters the evidence used
|
||
to explain the score.
|
||
|
||
Weights and thresholds are review policy, not calibrated probabilities. A
|
||
score of 75 does not mean a 75% chance that AI wrote the input.
|
||
|
||
The diagnostic showcase exercises five distinct prompt-leakage rules, including
|
||
two prompt/response markers. Together with its style and typography findings it
|
||
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
|
||
an estimated probability.
|