5.6 KiB
Review signals and evidence methodology
John Slop finds concrete text that may deserve review. It does not determine whether AI wrote a document, identify a model, or assign an authorship probability. A finding names the exact matched text and the configured reason for reviewing it; the evidence score is only a bounded aggregation of those findings.
This boundary matters because general-purpose AI-text detection is not reliable enough to support an authorship claim. Sadasivan et al. show both practical paraphrasing attacks and a theoretical trade-off between detection and false positives as human and model distributions become closer [1]. A comparative evaluation of 14 detection tools likewise found them neither accurate nor reliable, with performance worsening after obfuscation [2]. Liang et al. found that perplexity-based detectors disproportionately classified non-native English writing as AI-generated [3]. John Slop therefore does not use perplexity, burstiness, grammar, spelling, vocabulary richness, dialect, or English fluency as evidence.
Signal tiers
Direct response residue
These are relatively strong, locally explainable signs that conversational material may have been pasted into a document:
- assistant self-identification, capability, or knowledge-cutoff disclaimers;
User:/Assistant:transcript scaffolding;- answer or revision prefaces; and
- follow-up offers such as asking whether the reader wants an expansion.
They still do not prove authorship. A quotation, article about assistants, or intentional transcript can contain the same text, so applicability scopes and near-miss fixtures are part of every rule's review.
Unfinished editing and source residue
Unresolved insertion notes, implementation placeholders, omitted-code comments, tutorial narration, and stray Markdown fences are useful low-care signals regardless of how they were produced. These rules ask whether work is complete; they do not infer who or what produced it.
Subtle population-level clues
Some patterns are meaningful across a large corpus but weak in one document.
Kobak et al. measured abrupt, population-level increases in selected style words
across more than 15 million biomedical abstracts [4]. John Slop represents this
only as excess-vocabulary-cluster: it requires multiple matches, has weight
1.0, is occurrence-capped, and says explicitly that it is not authorship
evidence. A stock closing has similarly low weight. These clues should prompt a
specificity edit at most, never an accusation.
Repeated transitions and templated phrases sit between the tiers. They activate only on an exact conservative pattern; repetition from one rule is damped and cannot alone create a high evidence band.
Verification targets, not offline shape rules
Generated references and package names can be wrong. Walters and Wilder found fabricated and substantially erroneous citations in a controlled set of generated bibliographies [5]. Spracklen et al. measured package hallucinations in a large code-generation study [6]. But a citation's typography or a package name's shape cannot establish that it is fictitious. Reliable checks require an authoritative catalog and contextual validation, while John Slop analysis is local and makes no network calls. These remain human review targets rather than shipped heuristic rules.
How a rule is admitted
A shipped rule must have a stable ID, a narrow pattern, a rationale, review help, an applicability scope, a finite non-negative weight, and an occurrence cap. It also needs:
- a positive fixture showing the intended match;
- a realistic near miss showing what must not match;
- deterministic byte spans and ordering; and
- review of likely harm, especially language-proficiency and demographic proxies.
Exact residue and fixable Unicode receive higher confidence than broad stylistic clues. New research is used to propose testable review signals, not to import a paper's population-level conclusion into an individual-document verdict.
Scoring and editor visibility
Rule weight contributes bounded evidence to the shared score. Per-rule caps and correlation damping prevent repeated copies of one pattern from dominating it. The number is neither accuracy nor probability; see Heuristics and evidence scoring for the formula.
LSP visibility is a separate presentation choice. By default the editor shows
only configured Unicode findings, which are exact and may have safe replacement
edits. --all-diagnostics enables the complete detector catalog for testing or
deliberate review. --minimum-diagnostic-weight <WEIGHT> is the quieter opt-in:
it keeps Unicode findings and adds only non-Unicode findings whose configured
weight meets the threshold. Filtering affects editor diagnostics, not CLI
analysis or scoring.
References
- Sadasivan et al., Can AI-Generated Text be Reliably Detected?
- Weber-Wulff et al., Testing of detection tools for AI-generated text
- Liang et al., GPT detectors are biased against non-native English writers
- Kobak et al., Delving into LLM-assisted writing in biomedical publications through excess vocabulary
- Walters and Wilder, Fabrication and errors in the bibliographic citations generated by ChatGPT
- Spracklen et al., We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs