79 lines
3.8 KiB
Markdown
79 lines
3.8 KiB
Markdown
# Limitations
|
|
|
|
John Slop is a review aid for observable low-care signals. It is not an
|
|
AI-authorship detector, plagiarism detector, fact checker, security scanner, or
|
|
quality oracle. A finding says only that configured text matched at a particular
|
|
source span. Human context remains decisive.
|
|
|
|
## False positives
|
|
|
|
Legitimate material can contain every shipped signal. Documentation may quote a
|
|
chat transcript, an article may discuss knowledge cutoffs, a tutorial may use
|
|
step-by-step comments deliberately, and a tracked source-code TODO may be valid.
|
|
Unicode punctuation can be correct typography even when a downstream format
|
|
requires ASCII. Generic transitions are weak style evidence, not defects by
|
|
themselves.
|
|
|
|
Use path scopes, rule disabling, and project-specific weights to reflect local
|
|
policy. Do not automatically reject, shame, or attribute a document based on a
|
|
score. Review the visible findings and original context.
|
|
|
|
## False negatives and evasion
|
|
|
|
Exact and regular-expression rules are inherently easy to evade through
|
|
paraphrasing, case changes, spelling variation, inserted characters, or removal
|
|
of obvious scaffolding. Careless work need not contain any known signal, while
|
|
well-reviewed AI-assisted work may contain none. John Slop deliberately avoids
|
|
opaque statistical authorship guesses, so a clean result is not evidence of
|
|
human authorship or high quality.
|
|
|
|
Adversarial inputs can also target applicability assumptions: misleading file
|
|
extensions, unusual generated-directory names, or stdin without a filename may
|
|
select a different scope. Archive analysis uses the inner member path but does
|
|
not infer semantic language from content.
|
|
|
|
## Domain and language shift
|
|
|
|
The starter prose rules primarily encode English phrases. Other languages,
|
|
specialized editorial styles, historical corpora, transcripts, generated API
|
|
documentation, notebooks, and uncommon programming languages are
|
|
underrepresented. File-extension inference covers a conservative common set and
|
|
defaults unknown paths to prose. Teams should validate rules against their own
|
|
reviewed material before changing enforcement thresholds.
|
|
|
|
## Confidence is not probability
|
|
|
|
The score is a deterministic compression of configured weights, occurrence
|
|
caps, and category damping. Those values are policy choices and have not been
|
|
calibrated against population-level authorship data. A score of 75 does not mean
|
|
there is a 75% probability of AI authorship, low care, or an error. Thresholds
|
|
name review bands only, and changing the catalog changes their meaning.
|
|
|
|
Every score is traceable to findings, but that explainability does not make the
|
|
underlying heuristic universally valid. Related signals can remain correlated
|
|
even after category damping.
|
|
|
|
## Evaluation limits
|
|
|
|
The checked-in evaluation corpus is small, purpose-built, and intended to catch
|
|
behavioral regressions. Its current perfect precision and recall reflect fixture
|
|
construction, not real-world accuracy. It contains only one clean document and
|
|
must not justify deployment claims or default-weight tuning. Representative,
|
|
licensed, domain-specific samples and independent human review are required for
|
|
such decisions.
|
|
|
|
## Input and operational boundaries
|
|
|
|
Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8,
|
|
encrypted archives, unsafe archive members, and configured resource-limit
|
|
violations are errors rather than clean results. Generated, vendor, dependency,
|
|
and minified paths are skipped by default. The formatter handles strict UTF-8
|
|
files and configured ASCII replacements only; it does not rewrite ZIP-based
|
|
containers, prose, or source structure. Recursive writes require explicit
|
|
`--write --recursive` and preflight the complete input set. The LSP remains a
|
|
scaffold for Milestone 6.
|
|
|
|
Analysis is local and performs no network lookup. Consequently, it cannot check
|
|
citations, current facts, provenance, licensing, or similarity to external
|
|
sources.
|