john-slop/docs/limitations.md

95 lines
5.0 KiB
Markdown

# Limitations
John Slop is a review aid for observable low-care signals. It is not an
AI-authorship detector, plagiarism detector, fact checker, security scanner, or
quality oracle. A finding says only that configured text matched at a particular
source span. Human context remains decisive.
## False positives
Legitimate material can contain every shipped signal. Documentation may quote a
chat transcript, an article may discuss knowledge cutoffs, a tutorial may use
step-by-step comments deliberately, and a tracked source-code TODO may be valid.
Unicode punctuation can be correct typography even when a downstream format
requires ASCII. The UTS #39 ASCII-skeleton profile intentionally flags many
legitimate Greek, Cyrillic, mathematical, and other non-Latin characters for
review because it has no locale or identifier context. Generic transitions are
weak style evidence, not defects by themselves.
Use path scopes, rule disabling, and project-specific weights to reflect local
policy. Do not automatically reject, shame, or attribute a document based on a
score. Review the visible findings and original context.
## False negatives and evasion
Exact and regular-expression rules are inherently easy to evade through
paraphrasing, case changes, spelling variation, inserted characters, or removal
of obvious scaffolding. Careless work need not contain any known signal, while
well-reviewed AI-assisted work may contain none. John Slop deliberately avoids
opaque statistical authorship guesses, so a clean result is not evidence of
human authorship or high quality.
Adversarial inputs can also target applicability assumptions: misleading file
extensions, unusual generated-directory names, or stdin without a filename may
select a different scope. Archive analysis uses the inner member path but does
not infer semantic language from content.
## Domain and language shift
The starter prose rules primarily encode English phrases. Other languages,
specialized editorial styles, historical corpora, transcripts, generated API
documentation, notebooks, and uncommon programming languages are
underrepresented. File-extension inference covers a conservative common set and
defaults unknown paths to prose. Teams should validate rules against their own
reviewed material before changing enforcement thresholds. Disable or narrow
set `[unicode] confusables = "review"` to keep findings without edits, or use
`"off"` when ordinary multilingual text makes character-level review too noisy.
Locale-aware suppression is not implemented.
## Evidence score is not probability
The score is a deterministic compression of configured weights, occurrence
caps, and category damping. Those values are policy choices and have not been
calibrated against population-level authorship data. A score of 75 does not mean
there is a 75% probability of AI authorship, low care, or an error. Thresholds
name review bands only, and changing the catalog changes their meaning.
Every score is traceable to findings, but that explainability does not make the
underlying heuristic universally valid. Related signals can remain correlated
even after category damping.
## Evaluation limits
The checked-in evaluation corpus is small, purpose-built, and intended to catch
behavioral regressions. Its current perfect precision and recall reflect fixture
construction, not real-world accuracy. It contains only one clean document and
must not justify deployment claims or default-weight tuning. Representative,
licensed, domain-specific samples and independent human review are required for
such decisions.
## Input and operational boundaries
Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8,
encrypted archives, unsafe archive members, and configured resource-limit
violations are errors rather than clean results. Generated, vendor, dependency,
and minified paths are skipped by default. The formatter handles strict UTF-8
files and configured ASCII replacements only; it does not rewrite ZIP-based
containers, prose, or source structure. Recursive writes require explicit
`--write --recursive` and preflight the complete input set.
The LSP intentionally handles open editor snapshots only and uses full-document
synchronization. It publishes the configured heuristic catalog but does not
publish a document-level evidence score; editor diagnostics remain individual
review signals. It does not read or write a document URI, mutate archives,
provide prose rewrites, or implement incremental range changes. Only configured
ASCII replacements receive quick fixes or formatting edits. Under the default
policy this includes printable-ASCII UTS #39 skeletons, which can alter valid
Greek, Cyrillic, mathematical, or other text; use `review` for non-mutating
diagnostics. Invisible and bidirectional-control findings are diagnostic-only, and
configuration reload errors retain the last valid configuration and compiled
detector.
Analysis is local and performs no network lookup. Consequently, it cannot check
citations, current facts, provenance, licensing, or similarity to external
sources.