john-slop/docs/limitations.md

5.0 KiB

Limitations

John Slop is a review aid for observable low-care signals. It is not an AI-authorship detector, plagiarism detector, fact checker, security scanner, or quality oracle. A finding says only that configured text matched at a particular source span. Human context remains decisive.

False positives

Legitimate material can contain every shipped signal. Documentation may quote a chat transcript, an article may discuss knowledge cutoffs, a tutorial may use step-by-step comments deliberately, and a tracked source-code TODO may be valid. Unicode punctuation can be correct typography even when a downstream format requires ASCII. The UTS #39 ASCII-skeleton profile intentionally flags many legitimate Greek, Cyrillic, mathematical, and other non-Latin characters for review because it has no locale or identifier context. Generic transitions are weak style evidence, not defects by themselves.

Use path scopes, rule disabling, and project-specific weights to reflect local policy. Do not automatically reject, shame, or attribute a document based on a score. Review the visible findings and original context.

False negatives and evasion

Exact and regular-expression rules are inherently easy to evade through paraphrasing, case changes, spelling variation, inserted characters, or removal of obvious scaffolding. Careless work need not contain any known signal, while well-reviewed AI-assisted work may contain none. John Slop deliberately avoids opaque statistical authorship guesses, so a clean result is not evidence of human authorship or high quality.

Adversarial inputs can also target applicability assumptions: misleading file extensions, unusual generated-directory names, or stdin without a filename may select a different scope. Archive analysis uses the inner member path but does not infer semantic language from content.

Domain and language shift

The starter prose rules primarily encode English phrases. Other languages, specialized editorial styles, historical corpora, transcripts, generated API documentation, notebooks, and uncommon programming languages are underrepresented. File-extension inference covers a conservative common set and defaults unknown paths to prose. Teams should validate rules against their own reviewed material before changing enforcement thresholds. Disable or narrow set [unicode] confusables = "review" to keep findings without edits, or use "off" when ordinary multilingual text makes character-level review too noisy. Locale-aware suppression is not implemented.

Evidence score is not probability

The score is a deterministic compression of configured weights, occurrence caps, and category damping. Those values are policy choices and have not been calibrated against population-level authorship data. A score of 75 does not mean there is a 75% probability of AI authorship, low care, or an error. Thresholds name review bands only, and changing the catalog changes their meaning.

Every score is traceable to findings, but that explainability does not make the underlying heuristic universally valid. Related signals can remain correlated even after category damping.

Evaluation limits

The checked-in evaluation corpus is small, purpose-built, and intended to catch behavioral regressions. Its current perfect precision and recall reflect fixture construction, not real-world accuracy. It contains only one clean document and must not justify deployment claims or default-weight tuning. Representative, licensed, domain-specific samples and independent human review are required for such decisions.

Input and operational boundaries

Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8, encrypted archives, unsafe archive members, and configured resource-limit violations are errors rather than clean results. Generated, vendor, dependency, and minified paths are skipped by default. The formatter handles strict UTF-8 files and configured ASCII replacements only; it does not rewrite ZIP-based containers, prose, or source structure. Recursive writes require explicit --write --recursive and preflight the complete input set.

The LSP intentionally handles open editor snapshots only and uses full-document synchronization. It publishes the configured heuristic catalog but does not publish a document-level evidence score; editor diagnostics remain individual review signals. It does not read or write a document URI, mutate archives, provide prose rewrites, or implement incremental range changes. Only configured ASCII replacements receive quick fixes or formatting edits. Under the default policy this includes printable-ASCII UTS #39 skeletons, which can alter valid Greek, Cyrillic, mathematical, or other text; use review for non-mutating diagnostics. Invisible and bidirectional-control findings are diagnostic-only, and configuration reload errors retain the last valid configuration and compiled detector.

Analysis is local and performs no network lookup. Consequently, it cannot check citations, current facts, provenance, licensing, or similarity to external sources.