docs: split detailed guides from readme
This commit is contained in:
parent
239b5198e4
commit
25560fe5d2
99
README.md
99
README.md
|
|
@ -4,90 +4,39 @@
|
|||

|
||||

|
||||
|
||||
John Slop is an early-stage, local-first heuristic linter for concrete signals
|
||||
of low-effort or insufficiently reviewed text and source code. It is intended to
|
||||
highlight evidence for human review, not to prove that content was written by AI
|
||||
or assign an authorship probability.
|
||||
John Slop is an early-stage, local-first linter for concrete signs of low-effort
|
||||
or insufficiently reviewed text and source code. It reports explainable
|
||||
heuristics for human review; it does not prove AI authorship or identify a
|
||||
model.
|
||||
|
||||
## Current prototype
|
||||
Today, the detector reads UTF-8 files, directories, ZIP archives, and common
|
||||
ZIP-based office documents. Its first conservative rule family identifies
|
||||
configured non-ASCII hyphen-like characters and exposes safe ASCII replacement
|
||||
edits. Broader data-driven heuristics, terminal diagnostics, a formatter, and an
|
||||
LSP are planned in [tasks.md](tasks.md).
|
||||
|
||||
The repository currently contains proof-of-concept code that:
|
||||
## Quick start
|
||||
|
||||
- streams UTF-8 text from a file or recursively from a directory;
|
||||
- streams text entries from ZIP-based files, including `.zip`, `.docx`, `.xlsx`,
|
||||
and `.pptx`;
|
||||
- rejects missing paths and invalid UTF-8 with a diagnostic error;
|
||||
- detects a configured-in-code set of hyphen-like Unicode characters; and
|
||||
- returns typed findings, edits, and bounded evidence confidence for those
|
||||
characters.
|
||||
|
||||
The temporary command-line entry point accepts one file, directory, or supported
|
||||
ZIP-based path and prints a summary per document. Its interface will be replaced
|
||||
by the planned Clap-based detector binary during Milestone 1.
|
||||
|
||||
Cargo exposes explicit `john-slop`, `john-slop-fmt`, and `john-slop-lsp` binary
|
||||
targets. The formatter and LSP currently exit with an explanatory unsupported
|
||||
message until their dedicated implementation milestones.
|
||||
|
||||
Run the detector over one or more paths with:
|
||||
Rust 1.98.1 or newer is required.
|
||||
|
||||
```sh
|
||||
cargo run --bin john-slop -- [--fail-confidence 0..100] <PATH>...
|
||||
cargo run --bin john-slop -- path/to/document path/to/source-directory
|
||||
```
|
||||
|
||||
The detector exits `0` below the selected threshold, `1` when any document meets
|
||||
it, and `2` for invalid arguments or operational failures. Confidence describes
|
||||
configured evidence, not an AI-authorship probability.
|
||||
Use `--fail-confidence 0..100` to make the detector exit with status `1` when a
|
||||
document reaches a chosen evidence threshold. Confidence measures configured
|
||||
signals, not the probability that AI wrote the content.
|
||||
|
||||
Reusable consumers should call the library facade: `load_config`,
|
||||
`analyze_source`, `plan_replacements`, and `read_inputs`. Replacement planning is
|
||||
pure and shares the exact resolved configuration used by analysis.
|
||||
Input iteration yields a separate result per document, so a bad directory entry
|
||||
does not discard successfully read siblings.
|
||||
Reads enforce the resource limits in
|
||||
[ADR 0002](docs/decisions/0002-input-policy-and-limits.md), including bounded
|
||||
file/member sizes, archive expansion and compression ratios, entry counts,
|
||||
directory depth, safe member paths, strict UTF-8, and NUL-byte rejection.
|
||||
Library input failures are typed `InputError` variants. Executable boundaries
|
||||
attach operation context and render them with `miette`; library consumers do not
|
||||
need to parse presentation strings.
|
||||
## Documentation
|
||||
|
||||
The implementation is being restructured into a reusable library plus detector,
|
||||
formatter, and LSP binaries. See [tasks.md](tasks.md) for the ordered backlog and
|
||||
[AGENTS.md](AGENTS.md) for the design and contribution rules.
|
||||
Architecture and product choices are recorded under
|
||||
[`docs/decisions`](docs/decisions/), beginning with the
|
||||
[core analysis contracts](docs/decisions/0001-core-analysis-contracts.md).
|
||||
The default traversal, text-recognition, and archive safety policy is specified
|
||||
in [ADR 0002](docs/decisions/0002-input-policy-and-limits.md).
|
||||
- [Documentation index](docs/README.md)
|
||||
- [Command-line usage and current capabilities](docs/usage.md)
|
||||
- [Architecture and public library API](docs/architecture.md)
|
||||
- [Development, tests, features, and coverage](docs/development.md)
|
||||
- [Design decision records](docs/decisions/)
|
||||
- [Background reading](docs/background.md)
|
||||
|
||||
## Development
|
||||
|
||||
Run the current test and quality suite with:
|
||||
|
||||
```sh
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --all-targets --all-features -- -D warnings
|
||||
cargo test --all-targets --all-features
|
||||
cargo doc --no-deps
|
||||
```
|
||||
|
||||
Behavior is intentionally captured with regression tests before the refactor so
|
||||
that later changes are deliberate and reviewable.
|
||||
|
||||
Detector spans use zero-based UTF-8 byte offsets. Presentation layers will
|
||||
convert those canonical offsets to terminal or LSP coordinates at their
|
||||
boundaries.
|
||||
|
||||
The minimum supported Rust version (MSRV) is 1.88. CI checks both the MSRV and
|
||||
the current stable toolchain against the committed dependency lockfile. Linux is
|
||||
the currently exercised platform; support for other platforms will be declared
|
||||
only after CI coverage is added.
|
||||
The ordered implementation plan lives in [tasks.md](tasks.md), and contribution
|
||||
rules live in [AGENTS.md](AGENTS.md).
|
||||
|
||||
John Slop is distributed under the [MIT license](LICENSE).
|
||||
|
||||
## Background reading
|
||||
|
||||
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
|
||||
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
|
||||
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,19 @@
|
|||
# John Slop documentation
|
||||
|
||||
The root [README](../README.md) is the short project overview. These guides hold
|
||||
the details needed to use, embed, and develop John Slop:
|
||||
|
||||
- [Usage](usage.md) documents accepted inputs, detector output, exit statuses,
|
||||
and the current formatter/LSP status.
|
||||
- [Architecture](architecture.md) describes module boundaries, shared domain
|
||||
values, input safety, and the reusable library facade.
|
||||
- [Development](development.md) covers the supported Rust version, feature
|
||||
matrix, snapshots, coverage, badges, and CI quality gates.
|
||||
- [Background](background.md) collects external context on observable signs of
|
||||
low-care AI-assisted writing.
|
||||
- [Decision records](decisions/) preserve durable product and engineering
|
||||
choices, including confidence semantics and input limits.
|
||||
|
||||
The future configuration, heuristic engine, formatter, and LSP contracts remain
|
||||
specified in [tasks.md](../tasks.md) and [AGENTS.md](../AGENTS.md) until their
|
||||
implementation milestones land.
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
# Architecture
|
||||
|
||||
John Slop keeps detection and replacement behavior in a library. Executables are
|
||||
thin adapters responsible for argument parsing, presentation, and process exit
|
||||
statuses.
|
||||
|
||||
The current flow is:
|
||||
|
||||
```text
|
||||
path -> input stream -> Document -> analysis -> Analysis + Finding
|
||||
-> replacement planner -> TextEdit
|
||||
```
|
||||
|
||||
The main modules have deliberately narrow responsibilities:
|
||||
|
||||
- `config` owns the validated embedded replacement catalog and input limits;
|
||||
- `input` streams files, directory entries, and archive members as documents;
|
||||
- `detector` produces explainable findings and bounded confidence;
|
||||
- `replacement` produces non-overlapping edits without performing I/O;
|
||||
- `model` contains presentation-neutral domain values;
|
||||
- `error` exposes typed input failures for library consumers;
|
||||
- `src/bin` contains the detector and the formatter/LSP adapters.
|
||||
|
||||
All source spans are zero-based UTF-8 byte ranges into the exact analyzed source
|
||||
snapshot. Presentation layers will convert those canonical spans to terminal or
|
||||
UTF-16 LSP coordinates at their boundaries. Archive sources retain both their
|
||||
outer path and inner member identity.
|
||||
|
||||
## Public facade
|
||||
|
||||
The crate root exposes four entry points:
|
||||
|
||||
- `load_config` resolves the configuration embedded in the current build;
|
||||
- `read_inputs` returns a streaming iterator of per-document results;
|
||||
- `analyze_source` returns findings and evidence confidence for UTF-8 text;
|
||||
- `plan_replacements` returns safe edits against the same source snapshot.
|
||||
|
||||
Configuration files and precedence layers are a later milestone. Keeping these
|
||||
operations behind the facade lets those layers evolve without duplicating logic
|
||||
across the three binaries.
|
||||
|
||||
See [ADR 0001](decisions/0001-core-analysis-contracts.md) for the foundational
|
||||
contracts and [ADR 0002](decisions/0002-input-policy-and-limits.md) for traversal
|
||||
and archive safety decisions.
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
# Background reading
|
||||
|
||||
John Slop focuses on observable low-care signals and avoids making unsupported
|
||||
authorship claims. These resources provide context for that product boundary and
|
||||
the kinds of review problems the project aims to help surface:
|
||||
|
||||
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
|
||||
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
|
||||
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
|
||||
|
||||
Project-specific decisions take precedence over external guidance. See
|
||||
[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and
|
||||
authorship semantics.
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
# Development
|
||||
|
||||
The project uses Rust edition 2024 and requires Rust 1.98.1 or newer. The
|
||||
committed lockfile makes application builds reproducible. Linux is the currently
|
||||
exercised platform; other platforms are not declared supported until CI covers
|
||||
them.
|
||||
|
||||
## Quality gates
|
||||
|
||||
Run the same core checks required by the contributor guide:
|
||||
|
||||
```sh
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --locked --all-targets --all-features -- -D warnings
|
||||
cargo test --locked --all-targets --all-features
|
||||
cargo test --locked --all-targets --no-default-features
|
||||
cargo doc --locked --no-deps --all-features
|
||||
```
|
||||
|
||||
The default `fancy-diagnostics` feature enables `miette`'s rich terminal
|
||||
renderer. The no-default-features test run ensures the shared library and all
|
||||
binaries remain valid without that optional renderer. Focused unit and
|
||||
integration tests cover domain invariants, input failures, archives, CLI exit
|
||||
behavior, and feature configurations.
|
||||
|
||||
Insta snapshots are reserved for stable user-facing text such as CLI help and
|
||||
usage errors. Structured domain values use direct assertions so failures remain
|
||||
precise and easy to review. Accept snapshot changes only after inspecting the
|
||||
generated diff.
|
||||
|
||||
## Coverage and badges
|
||||
|
||||
Run line coverage with Cargo Tarpaulin:
|
||||
|
||||
```sh
|
||||
cargo tarpaulin --config tarpaulin.toml
|
||||
```
|
||||
|
||||
The configuration enforces a 30% floor. The current suite measured 72.32%
|
||||
(196/271 lines) on 2026-09-29. The low floor is intentional while CLI
|
||||
subprocesses and future protocol adapters are not attributed consistently by
|
||||
every backend; the tracked badge still reports the measured result.
|
||||
|
||||
After a clean test and coverage run, refresh the repository-local badges with:
|
||||
|
||||
```sh
|
||||
rust_autobadges --high-coverage-percent 0.7 --mid-coverage-percent 0.3
|
||||
```
|
||||
|
||||
Commit `badges/*.svg` and `.run_status.toml`; do not commit the generated
|
||||
`tarpaulin-report.html`. Jenkins runs both feature configurations and enforces
|
||||
the Tarpaulin floor before building release binaries.
|
||||
|
||||
## Test scope
|
||||
|
||||
Add behavior tests at the lowest useful layer and an integration test when a
|
||||
change crosses a binary boundary. Important edge cases include malformed input,
|
||||
invalid UTF-8, empty text, Unicode byte spans, deterministic ordering, archive
|
||||
limits, and near misses for detection rules. The detailed expectations and
|
||||
task-level commit discipline are in [AGENTS.md](../AGENTS.md).
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
# Usage
|
||||
|
||||
## Detector
|
||||
|
||||
Run the detector over one or more paths:
|
||||
|
||||
```sh
|
||||
cargo run --bin john-slop -- [--fail-confidence 0..100] <PATH>...
|
||||
```
|
||||
|
||||
Each path may be a UTF-8 text file, a directory, a ZIP archive, or a supported
|
||||
ZIP-based office document (`.docx`, `.xlsx`, or `.pptx`). Directory traversal and
|
||||
archive reading stream one result at a time, so callers do not need to retain an
|
||||
entire input tree in memory.
|
||||
|
||||
The current human-readable output contains the source identity, bounded evidence
|
||||
confidence, and finding count. Findings currently cover configured non-ASCII
|
||||
hyphen-like characters. More heuristic families and source-highlighted `miette`
|
||||
diagnostics are planned.
|
||||
|
||||
Exit statuses are stable:
|
||||
|
||||
- `0`: every analyzed document is below the requested confidence threshold;
|
||||
- `1`: at least one document reaches the threshold;
|
||||
- `2`: arguments are invalid or an operational/input error prevents completion.
|
||||
|
||||
Confidence is a summary of observable configured evidence. It is not an
|
||||
AI-authorship probability.
|
||||
|
||||
## Supported input policy
|
||||
|
||||
Input reads use strict UTF-8 and reject NUL-containing text. They do not follow
|
||||
symlinks by default. Limits on files, walk depth, archive entries, expanded
|
||||
bytes, compression ratio, and member paths protect analysis from unbounded or
|
||||
unsafe input. The exact policy is recorded in
|
||||
[ADR 0002](decisions/0002-input-policy-and-limits.md).
|
||||
|
||||
## Formatter and LSP
|
||||
|
||||
Cargo already exposes `john-slop-fmt` and `john-slop-lsp` so packaging and CI
|
||||
exercise their target names. Both are explicit scaffolds today and return an
|
||||
unsupported diagnostic. Replacement planning is available through the library;
|
||||
file-writing formatter behavior and the stdio language server arrive in later
|
||||
milestones.
|
||||
2
tasks.md
2
tasks.md
|
|
@ -74,7 +74,7 @@ entry do not discard successful documents.
|
|||
coverage, without snapshotting structured domain values unnecessarily.
|
||||
- [x] Configure Tarpaulin with an enforceable coverage floor around 30%, run
|
||||
`rust_autobadges`, and publish local version/test/coverage badges.
|
||||
- [ ] Reduce the root README to a generic project landing page and move detailed
|
||||
- [x] Reduce the root README to a generic project landing page and move detailed
|
||||
usage, architecture, testing, coverage, and background links under `docs/`.
|
||||
|
||||
## Milestone 2 — Typed layered configuration
|
||||
|
|
|
|||
Loading…
Reference in New Issue