95 lines
5.0 KiB
Markdown
95 lines
5.0 KiB
Markdown
# Limitations
|
|
|
|
John Slop is a review aid for observable low-care signals. It is not an
|
|
AI-authorship detector, plagiarism detector, fact checker, security scanner, or
|
|
quality oracle. A finding says only that configured text matched at a particular
|
|
source span. Human context remains decisive.
|
|
|
|
## False positives
|
|
|
|
Legitimate material can contain every shipped signal. Documentation may quote a
|
|
chat transcript, an article may discuss knowledge cutoffs, a tutorial may use
|
|
step-by-step comments deliberately, and a tracked source-code TODO may be valid.
|
|
Unicode punctuation can be correct typography even when a downstream format
|
|
requires ASCII. The UTS #39 ASCII-skeleton profile intentionally flags many
|
|
legitimate Greek, Cyrillic, mathematical, and other non-Latin characters for
|
|
review because it has no locale or identifier context. Generic transitions are
|
|
weak style evidence, not defects by themselves.
|
|
|
|
Use path scopes, rule disabling, and project-specific weights to reflect local
|
|
policy. Do not automatically reject, shame, or attribute a document based on a
|
|
score. Review the visible findings and original context.
|
|
|
|
## False negatives and evasion
|
|
|
|
Exact and regular-expression rules are inherently easy to evade through
|
|
paraphrasing, case changes, spelling variation, inserted characters, or removal
|
|
of obvious scaffolding. Careless work need not contain any known signal, while
|
|
well-reviewed AI-assisted work may contain none. John Slop deliberately avoids
|
|
opaque statistical authorship guesses, so a clean result is not evidence of
|
|
human authorship or high quality.
|
|
|
|
Adversarial inputs can also target applicability assumptions: misleading file
|
|
extensions, unusual generated-directory names, or stdin without a filename may
|
|
select a different scope. Archive analysis uses the inner member path but does
|
|
not infer semantic language from content.
|
|
|
|
## Domain and language shift
|
|
|
|
The starter prose rules primarily encode English phrases. Other languages,
|
|
specialized editorial styles, historical corpora, transcripts, generated API
|
|
documentation, notebooks, and uncommon programming languages are
|
|
underrepresented. File-extension inference covers a conservative common set and
|
|
defaults unknown paths to prose. Teams should validate rules against their own
|
|
reviewed material before changing enforcement thresholds. Disable or narrow
|
|
set `[unicode] confusables = "review"` to keep findings without edits, or use
|
|
`"off"` when ordinary multilingual text makes character-level review too noisy.
|
|
Locale-aware suppression is not implemented.
|
|
|
|
## Evidence score is not probability
|
|
|
|
The score is a deterministic compression of configured weights, occurrence
|
|
caps, and category damping. Those values are policy choices and have not been
|
|
calibrated against population-level authorship data. A score of 75 does not mean
|
|
there is a 75% probability of AI authorship, low care, or an error. Thresholds
|
|
name review bands only, and changing the catalog changes their meaning.
|
|
|
|
Every score is traceable to findings, but that explainability does not make the
|
|
underlying heuristic universally valid. Related signals can remain correlated
|
|
even after category damping.
|
|
|
|
## Evaluation limits
|
|
|
|
The checked-in evaluation corpus is small, purpose-built, and intended to catch
|
|
behavioral regressions. Its current perfect precision and recall reflect fixture
|
|
construction, not real-world accuracy. It contains only one clean document and
|
|
must not justify deployment claims or default-weight tuning. Representative,
|
|
licensed, domain-specific samples and independent human review are required for
|
|
such decisions.
|
|
|
|
## Input and operational boundaries
|
|
|
|
Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8,
|
|
encrypted archives, unsafe archive members, and configured resource-limit
|
|
violations are errors rather than clean results. Generated, vendor, dependency,
|
|
and minified paths are skipped by default. The formatter handles strict UTF-8
|
|
files and configured ASCII replacements only; it does not rewrite ZIP-based
|
|
containers, prose, or source structure. Recursive writes require explicit
|
|
`--write --recursive` and preflight the complete input set.
|
|
|
|
The LSP intentionally handles open editor snapshots only and uses full-document
|
|
synchronization. It publishes the configured heuristic catalog but does not
|
|
publish a document-level evidence score; editor diagnostics remain individual
|
|
review signals. It does not read or write a document URI, mutate archives,
|
|
provide prose rewrites, or implement incremental range changes. Only configured
|
|
ASCII replacements receive quick fixes or formatting edits. Under the default
|
|
policy this includes printable-ASCII UTS #39 skeletons, which can alter valid
|
|
Greek, Cyrillic, mathematical, or other text; use `review` for non-mutating
|
|
diagnostics. Invisible and bidirectional-control findings are diagnostic-only, and
|
|
configuration reload errors retain the last valid configuration and compiled
|
|
detector.
|
|
|
|
Analysis is local and performs no network lookup. Consequently, it cannot check
|
|
citations, current facts, provenance, licensing, or similarity to external
|
|
sources.
|