From 25560fe5d2a7e26b8e70df458914123f80668cd2 Mon Sep 17 00:00:00 2001 From: Dandaedre Date: Tue, 29 Sep 2026 12:55:53 +0200 Subject: [PATCH] docs: split detailed guides from readme --- README.md | 99 +++++++++++--------------------------------- docs/README.md | 19 +++++++++ docs/architecture.md | 44 ++++++++++++++++++++ docs/background.md | 13 ++++++ docs/development.md | 60 +++++++++++++++++++++++++++ docs/usage.md | 44 ++++++++++++++++++++ tasks.md | 2 +- 7 files changed, 205 insertions(+), 76 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/architecture.md create mode 100644 docs/background.md create mode 100644 docs/development.md create mode 100644 docs/usage.md diff --git a/README.md b/README.md index 913364d..af39712 100644 --- a/README.md +++ b/README.md @@ -4,90 +4,39 @@ ![tests](badges/test.svg) ![test coverage](badges/coverage.svg) -John Slop is an early-stage, local-first heuristic linter for concrete signals -of low-effort or insufficiently reviewed text and source code. It is intended to -highlight evidence for human review, not to prove that content was written by AI -or assign an authorship probability. +John Slop is an early-stage, local-first linter for concrete signs of low-effort +or insufficiently reviewed text and source code. It reports explainable +heuristics for human review; it does not prove AI authorship or identify a +model. -## Current prototype +Today, the detector reads UTF-8 files, directories, ZIP archives, and common +ZIP-based office documents. Its first conservative rule family identifies +configured non-ASCII hyphen-like characters and exposes safe ASCII replacement +edits. Broader data-driven heuristics, terminal diagnostics, a formatter, and an +LSP are planned in [tasks.md](tasks.md). -The repository currently contains proof-of-concept code that: +## Quick start -- streams UTF-8 text from a file or recursively from a directory; -- streams text entries from ZIP-based files, including `.zip`, `.docx`, `.xlsx`, - and `.pptx`; -- rejects missing paths and invalid UTF-8 with a diagnostic error; -- detects a configured-in-code set of hyphen-like Unicode characters; and -- returns typed findings, edits, and bounded evidence confidence for those - characters. - -The temporary command-line entry point accepts one file, directory, or supported -ZIP-based path and prints a summary per document. Its interface will be replaced -by the planned Clap-based detector binary during Milestone 1. - -Cargo exposes explicit `john-slop`, `john-slop-fmt`, and `john-slop-lsp` binary -targets. The formatter and LSP currently exit with an explanatory unsupported -message until their dedicated implementation milestones. - -Run the detector over one or more paths with: +Rust 1.98.1 or newer is required. ```sh -cargo run --bin john-slop -- [--fail-confidence 0..100] ... +cargo run --bin john-slop -- path/to/document path/to/source-directory ``` -The detector exits `0` below the selected threshold, `1` when any document meets -it, and `2` for invalid arguments or operational failures. Confidence describes -configured evidence, not an AI-authorship probability. +Use `--fail-confidence 0..100` to make the detector exit with status `1` when a +document reaches a chosen evidence threshold. Confidence measures configured +signals, not the probability that AI wrote the content. -Reusable consumers should call the library facade: `load_config`, -`analyze_source`, `plan_replacements`, and `read_inputs`. Replacement planning is -pure and shares the exact resolved configuration used by analysis. -Input iteration yields a separate result per document, so a bad directory entry -does not discard successfully read siblings. -Reads enforce the resource limits in -[ADR 0002](docs/decisions/0002-input-policy-and-limits.md), including bounded -file/member sizes, archive expansion and compression ratios, entry counts, -directory depth, safe member paths, strict UTF-8, and NUL-byte rejection. -Library input failures are typed `InputError` variants. Executable boundaries -attach operation context and render them with `miette`; library consumers do not -need to parse presentation strings. +## Documentation -The implementation is being restructured into a reusable library plus detector, -formatter, and LSP binaries. See [tasks.md](tasks.md) for the ordered backlog and -[AGENTS.md](AGENTS.md) for the design and contribution rules. -Architecture and product choices are recorded under -[`docs/decisions`](docs/decisions/), beginning with the -[core analysis contracts](docs/decisions/0001-core-analysis-contracts.md). -The default traversal, text-recognition, and archive safety policy is specified -in [ADR 0002](docs/decisions/0002-input-policy-and-limits.md). +- [Documentation index](docs/README.md) +- [Command-line usage and current capabilities](docs/usage.md) +- [Architecture and public library API](docs/architecture.md) +- [Development, tests, features, and coverage](docs/development.md) +- [Design decision records](docs/decisions/) +- [Background reading](docs/background.md) -## Development - -Run the current test and quality suite with: - -```sh -cargo fmt --all -- --check -cargo clippy --all-targets --all-features -- -D warnings -cargo test --all-targets --all-features -cargo doc --no-deps -``` - -Behavior is intentionally captured with regression tests before the refactor so -that later changes are deliberate and reviewable. - -Detector spans use zero-based UTF-8 byte offsets. Presentation layers will -convert those canonical offsets to terminal or LSP coordinates at their -boundaries. - -The minimum supported Rust version (MSRV) is 1.88. CI checks both the MSRV and -the current stable toolchain against the committed dependency lockfile. Linux is -the currently exercised platform; support for other platforms will be declared -only after CI coverage is added. +The ordered implementation plan lives in [tasks.md](tasks.md), and contribution +rules live in [AGENTS.md](AGENTS.md). John Slop is distributed under the [MIT license](LICENSE). - -## Background reading - -- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing) -- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup) -- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion) diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..5b169aa --- /dev/null +++ b/docs/README.md @@ -0,0 +1,19 @@ +# John Slop documentation + +The root [README](../README.md) is the short project overview. These guides hold +the details needed to use, embed, and develop John Slop: + +- [Usage](usage.md) documents accepted inputs, detector output, exit statuses, + and the current formatter/LSP status. +- [Architecture](architecture.md) describes module boundaries, shared domain + values, input safety, and the reusable library facade. +- [Development](development.md) covers the supported Rust version, feature + matrix, snapshots, coverage, badges, and CI quality gates. +- [Background](background.md) collects external context on observable signs of + low-care AI-assisted writing. +- [Decision records](decisions/) preserve durable product and engineering + choices, including confidence semantics and input limits. + +The future configuration, heuristic engine, formatter, and LSP contracts remain +specified in [tasks.md](../tasks.md) and [AGENTS.md](../AGENTS.md) until their +implementation milestones land. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..e33b579 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,44 @@ +# Architecture + +John Slop keeps detection and replacement behavior in a library. Executables are +thin adapters responsible for argument parsing, presentation, and process exit +statuses. + +The current flow is: + +```text +path -> input stream -> Document -> analysis -> Analysis + Finding + -> replacement planner -> TextEdit +``` + +The main modules have deliberately narrow responsibilities: + +- `config` owns the validated embedded replacement catalog and input limits; +- `input` streams files, directory entries, and archive members as documents; +- `detector` produces explainable findings and bounded confidence; +- `replacement` produces non-overlapping edits without performing I/O; +- `model` contains presentation-neutral domain values; +- `error` exposes typed input failures for library consumers; +- `src/bin` contains the detector and the formatter/LSP adapters. + +All source spans are zero-based UTF-8 byte ranges into the exact analyzed source +snapshot. Presentation layers will convert those canonical spans to terminal or +UTF-16 LSP coordinates at their boundaries. Archive sources retain both their +outer path and inner member identity. + +## Public facade + +The crate root exposes four entry points: + +- `load_config` resolves the configuration embedded in the current build; +- `read_inputs` returns a streaming iterator of per-document results; +- `analyze_source` returns findings and evidence confidence for UTF-8 text; +- `plan_replacements` returns safe edits against the same source snapshot. + +Configuration files and precedence layers are a later milestone. Keeping these +operations behind the facade lets those layers evolve without duplicating logic +across the three binaries. + +See [ADR 0001](decisions/0001-core-analysis-contracts.md) for the foundational +contracts and [ADR 0002](decisions/0002-input-policy-and-limits.md) for traversal +and archive safety decisions. diff --git a/docs/background.md b/docs/background.md new file mode 100644 index 0000000..c0d6c69 --- /dev/null +++ b/docs/background.md @@ -0,0 +1,13 @@ +# Background reading + +John Slop focuses on observable low-care signals and avoids making unsupported +authorship claims. These resources provide context for that product boundary and +the kinds of review problems the project aims to help surface: + +- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing) +- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup) +- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion) + +Project-specific decisions take precedence over external guidance. See +[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and +authorship semantics. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..2bd1b87 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,60 @@ +# Development + +The project uses Rust edition 2024 and requires Rust 1.98.1 or newer. The +committed lockfile makes application builds reproducible. Linux is the currently +exercised platform; other platforms are not declared supported until CI covers +them. + +## Quality gates + +Run the same core checks required by the contributor guide: + +```sh +cargo fmt --all -- --check +cargo clippy --locked --all-targets --all-features -- -D warnings +cargo test --locked --all-targets --all-features +cargo test --locked --all-targets --no-default-features +cargo doc --locked --no-deps --all-features +``` + +The default `fancy-diagnostics` feature enables `miette`'s rich terminal +renderer. The no-default-features test run ensures the shared library and all +binaries remain valid without that optional renderer. Focused unit and +integration tests cover domain invariants, input failures, archives, CLI exit +behavior, and feature configurations. + +Insta snapshots are reserved for stable user-facing text such as CLI help and +usage errors. Structured domain values use direct assertions so failures remain +precise and easy to review. Accept snapshot changes only after inspecting the +generated diff. + +## Coverage and badges + +Run line coverage with Cargo Tarpaulin: + +```sh +cargo tarpaulin --config tarpaulin.toml +``` + +The configuration enforces a 30% floor. The current suite measured 72.32% +(196/271 lines) on 2026-09-29. The low floor is intentional while CLI +subprocesses and future protocol adapters are not attributed consistently by +every backend; the tracked badge still reports the measured result. + +After a clean test and coverage run, refresh the repository-local badges with: + +```sh +rust_autobadges --high-coverage-percent 0.7 --mid-coverage-percent 0.3 +``` + +Commit `badges/*.svg` and `.run_status.toml`; do not commit the generated +`tarpaulin-report.html`. Jenkins runs both feature configurations and enforces +the Tarpaulin floor before building release binaries. + +## Test scope + +Add behavior tests at the lowest useful layer and an integration test when a +change crosses a binary boundary. Important edge cases include malformed input, +invalid UTF-8, empty text, Unicode byte spans, deterministic ordering, archive +limits, and near misses for detection rules. The detailed expectations and +task-level commit discipline are in [AGENTS.md](../AGENTS.md). diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..f305612 --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,44 @@ +# Usage + +## Detector + +Run the detector over one or more paths: + +```sh +cargo run --bin john-slop -- [--fail-confidence 0..100] ... +``` + +Each path may be a UTF-8 text file, a directory, a ZIP archive, or a supported +ZIP-based office document (`.docx`, `.xlsx`, or `.pptx`). Directory traversal and +archive reading stream one result at a time, so callers do not need to retain an +entire input tree in memory. + +The current human-readable output contains the source identity, bounded evidence +confidence, and finding count. Findings currently cover configured non-ASCII +hyphen-like characters. More heuristic families and source-highlighted `miette` +diagnostics are planned. + +Exit statuses are stable: + +- `0`: every analyzed document is below the requested confidence threshold; +- `1`: at least one document reaches the threshold; +- `2`: arguments are invalid or an operational/input error prevents completion. + +Confidence is a summary of observable configured evidence. It is not an +AI-authorship probability. + +## Supported input policy + +Input reads use strict UTF-8 and reject NUL-containing text. They do not follow +symlinks by default. Limits on files, walk depth, archive entries, expanded +bytes, compression ratio, and member paths protect analysis from unbounded or +unsafe input. The exact policy is recorded in +[ADR 0002](decisions/0002-input-policy-and-limits.md). + +## Formatter and LSP + +Cargo already exposes `john-slop-fmt` and `john-slop-lsp` so packaging and CI +exercise their target names. Both are explicit scaffolds today and return an +unsupported diagnostic. Replacement planning is available through the library; +file-writing formatter behavior and the stdio language server arrive in later +milestones. diff --git a/tasks.md b/tasks.md index 03f5132..b1d585f 100644 --- a/tasks.md +++ b/tasks.md @@ -74,7 +74,7 @@ entry do not discard successful documents. coverage, without snapshotting structured domain values unnecessarily. - [x] Configure Tarpaulin with an enforceable coverage floor around 30%, run `rust_autobadges`, and publish local version/test/coverage badges. -- [ ] Reduce the root README to a generic project landing page and move detailed +- [x] Reduce the root README to a generic project landing page and move detailed usage, architecture, testing, coverage, and background links under `docs/`. ## Milestone 2 — Typed layered configuration