diff --git a/README.md b/README.md index d8a72dd..2070df0 100644 --- a/README.md +++ b/README.md @@ -15,8 +15,9 @@ copied assistant scaffolding, generic style residue, source-code placeholders, tutorial narration, and configured non-ASCII characters. Rules retain exact source spans, apply path/content scopes, and feed a bounded, explainable confidence score. The detector renders source-highlighted human diagnostics, -concise records, or versioned JSON Lines. A formatter and LSP are planned in -[tasks.md](tasks.md). +concise records, or versioned JSON Lines. The formatter applies only configured +Unicode-to-ASCII replacements, supports check/diff modes, and requires explicit +opt-in for atomic file writes. The LSP is planned in [tasks.md](tasks.md). ## Quick start @@ -24,12 +25,18 @@ Rust 1.98.1 or newer is required. ```sh cargo run --bin john-slop -- path/to/document path/to/source-directory +cargo run --bin john-slop-fmt -- path/to/document +cargo run --bin john-slop-fmt -- --check path/to/document ``` Use `--fail-confidence 0..100` to make the detector exit with status `1` when a document reaches a chosen evidence threshold. Confidence measures configured signals, not the probability that AI wrote the content. +Formatter file mode prints transformed text without modifying the source. Use +`--write` explicitly for atomic in-place replacement, or pipe editor buffers +through `john-slop-fmt` over stdin/stdout. + ## Documentation - [Runnable Rust examples](examples/README.md) diff --git a/docs/README.md b/docs/README.md index 5a028a0..87ca4f7 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,7 +8,7 @@ the details needed to use, embed, and develop John Slop: - [Usage](usage.md) documents accepted inputs, detector output, exit statuses, and the current formatter/LSP status. - [Editor setup](editors.md) provides Helix (`hx`) and Vim configuration for - the formatter and LSP adapter contracts, with current scaffold caveats. + the formatter and future LSP adapter, with current status caveats. - [Output contracts](output.md) specifies human and concise presentation plus the versioned JSON Lines schema and partial-error records. - [Architecture](architecture.md) describes module boundaries, shared domain @@ -28,6 +28,6 @@ the details needed to use, embed, and develop John Slop: - [Decision records](decisions/) preserve durable product and engineering choices, including confidence semantics and input limits. -The future configuration, heuristic engine, formatter, and LSP contracts remain -specified in [tasks.md](../tasks.md) and [AGENTS.md](../AGENTS.md) until their -implementation milestones land. +Future LSP and release-hardening contracts remain specified in +[tasks.md](../tasks.md) and [AGENTS.md](../AGENTS.md) until their implementation +milestones land. diff --git a/docs/architecture.md b/docs/architecture.md index e7f24e4..2561638 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -9,7 +9,8 @@ The current flow is: ```text path -> input stream -> Document -> analysis -> Analysis + Finding -> diagnostic/output adapters - -> replacement planner -> TextEdit + -> replacement planner -> TextEdit -> applier + -> formatter I/O ``` The main modules have deliberately narrow responsibilities: @@ -20,7 +21,8 @@ The main modules have deliberately narrow responsibilities: explainable findings and bounded confidence; - `diagnostic` binds findings to exact named snapshots as typed `miette` diagnostics; -- `replacement` produces non-overlapping edits without performing I/O; +- `replacement` produces non-overlapping edits and validates/applies arbitrary + edit sets without performing I/O; - `model` contains presentation-neutral domain values; - `error` exposes typed input failures for library consumers; - `src/bin` contains the detector presentation/exit policy and formatter/LSP @@ -40,7 +42,9 @@ The crate root exposes core entry points and presentation-neutral types: - `read_inputs` returns a streaming iterator of per-document results; - `read_stdin` returns one bounded strict-UTF-8 standard-input document; - `analyze_source` returns findings and evidence confidence for UTF-8 text; -- `plan_replacements` returns safe edits against the same source snapshot. +- `plan_replacements` returns safe edits against the same source snapshot; +- `apply_replacements` validates boundaries and overlap, then applies edits in + descending byte order while borrowing no-op input unchanged. `FindingDiagnostic` is the library bridge to `miette`; CLI-specific concise and JSON rendering stays in the detector adapter so it cannot alter analysis. @@ -50,6 +54,11 @@ layers in order. Stable-ID merging and semantic validation happen before a configuration reaches analysis. Keeping these operations behind the facade avoids duplicating policy across the three binaries. +The formatter adapter owns filesystem mutation. It resolves configuration and +preflights every requested document before writing, then uses same-directory +temporary files and preserves permissions. Archive mutation remains outside the +library and adapter contract. + See [ADR 0001](decisions/0001-core-analysis-contracts.md) for the foundational contracts and [ADR 0002](decisions/0002-input-policy-and-limits.md) for traversal and archive safety decisions. diff --git a/docs/configuration.md b/docs/configuration.md index 7e5df37..308b335 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -56,6 +56,12 @@ layer, invalid regular expressions or globs, unsafe matchers, non-ASCII replacement targets, and invalid thresholds are fatal configuration diagnostics. +Replacement `source` values are non-empty Unicode strings containing at least +one non-ASCII byte; `replacement` values are non-empty ASCII strings. Sources +must be unique. When enabled sources share a prefix, the longest matching source +wins at each byte offset, with stable rule ID ordering as the deterministic +tie-break. Edits always refer to the original UTF-8 snapshot. + Each heuristic also configures `severity` as `advice`, `warning`, or `error`. Severity controls diagnostic presentation and summary counts; confidence still comes from the independently configured weight and evidence formula. diff --git a/docs/editors.md b/docs/editors.md index 6cd9dbf..f254d21 100644 --- a/docs/editors.md +++ b/docs/editors.md @@ -1,10 +1,8 @@ # Editor setup -> **Current status:** these configurations are ready for the adapter contracts, -> but `john-slop-fmt` and `john-slop-lsp` are still explicit scaffolds. The -> formatter becomes operational in Milestone 5 and the language server in -> Milestone 6. Until then, both return an unsupported diagnostic when started -> without `--help`. +> **Current status:** `john-slop-fmt` is operational. `john-slop-lsp` remains an +> explicit Milestone 6 scaffold and returns an unsupported diagnostic when +> started without `--help`. Install all three binaries somewhere on `PATH`, then confirm the editor-facing help is available: @@ -37,7 +35,7 @@ auto-format = false This complete custom language entry makes the repository's `example.txt` fixtures useful without depending on a built-in plain-text grammar. Use `:format` to run the formatter. Set `auto-format = true` only if save-time -replacement is wanted after Milestone 5. +replacement is wanted. For an existing language, merge John Slop into that language's entry rather than replacing its useful defaults. For example, Rust should retain @@ -96,10 +94,12 @@ current upstream documentation. Open `examples/fixtures/typography/example.txt` in either editor. Its expected formatter result is the adjacent `formatted.txt`; the shared manifest records -three replacements and one typography rule. The CLI is usable now for a -baseline: +three replacements and one typography rule. Compare detector and formatter +output with: ```sh cargo run --bin john-slop -- --output concise \ examples/fixtures/typography/example.txt +cargo run --bin john-slop-fmt -- \ + examples/fixtures/typography/example.txt ``` diff --git a/docs/limitations.md b/docs/limitations.md index d4878cf..3fca5eb 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -67,8 +67,11 @@ such decisions. Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8, encrypted archives, unsafe archive members, and configured resource-limit violations are errors rather than clean results. Generated, vendor, dependency, -and minified paths are skipped by default. The formatter and LSP binaries remain -scaffolds for later milestones; only the detector currently presents findings. +and minified paths are skipped by default. The formatter handles strict UTF-8 +files and configured ASCII replacements only; it does not rewrite ZIP-based +containers, prose, or source structure. Recursive writes require explicit +`--write --recursive` and preflight the complete input set. The LSP remains a +scaffold for Milestone 6. Analysis is local and performs no network lookup. Consequently, it cannot check citations, current facts, provenance, licensing, or similarity to external diff --git a/docs/usage.md b/docs/usage.md index 04cd2bb..e8a2415 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -72,8 +72,8 @@ planning. Run all example targets as part of the normal test matrix with Its [shared fixture manifest](../examples/fixtures/manifest.toml) pairs several `example.txt` inputs with expected detector rules, replacement counts, and formatted output. The executable fixture test validates both the library and -detector CLI today; formatter and LSP integration tests will consume the same -corpus when those adapters are implemented. +the detector and formatter CLIs today; LSP integration tests will consume the +same corpus when that adapter is implemented. ## Supported input policy @@ -83,12 +83,46 @@ bytes, compression ratio, and member paths protect analysis from unbounded or unsafe input. The exact policy is recorded in [ADR 0002](decisions/0002-input-policy-and-limits.md). -## Formatter and LSP +## Formatter -Cargo already exposes `john-slop-fmt` and `john-slop-lsp` so packaging and CI -exercise their target names. Both are explicit scaffolds today and return an -unsupported diagnostic. Replacement planning is available through the library; -file-writing formatter behavior and the stdio language server arrive in later -milestones. Both scaffolds accept `--help` and show concise Helix and Vim setup; -the complete snippets and validation workflow live in -[Editor setup](editors.md). +With no path, `john-slop-fmt` reads one bounded strict-UTF-8 document from stdin +and writes only formatted bytes to stdout. With a file path, the default remains +read-only and prints the transformed file without changing it: + +```sh +john-slop-fmt < article.txt > article.ascii.txt +john-slop-fmt article.txt +john-slop-fmt --check article.txt +john-slop-fmt --diff article.txt +john-slop-fmt --write article.txt +``` + +The modes are mutually exclusive: + +- default mode emits formatted text and exits `0`; +- `--check` emits nothing and exits `1` when any replacement is needed; +- `--diff` emits a unified diff and exits `1` when any replacement is needed; +- `--write` emits nothing and atomically replaces changed files, exiting `0`; +- argument, configuration, UTF-8, input, or write failures exit `2`. + +All modes use the same embedded/user/named/explicit configuration precedence as +the detector. `--config-name NAME` and `--config FILE` select those layers. +Replacement sources use deterministic longest-match semantics, and formatting +twice is byte-for-byte identical to formatting once. + +Directories require `--recursive`, and recursive operation is deliberately +limited to `--check --recursive` or `--write --recursive`. Before a recursive +write, every input is read, decoded, planned, and edit-validated; a failure +prevents all writes. Changed files use a temporary file in the same directory, +copy original permissions, sync content, and replace by rename where the +platform supports it. Files needing no edit are never rewritten. + +ZIP, DOCX, XLSX, and PPTX mutation is not supported. Extract and format an +individual UTF-8 file instead. + +## LSP + +Cargo exposes `john-slop-lsp` so packaging and CI exercise its target name, but +it remains an explicit Milestone 6 scaffold. Both adapter `--help` pages show +concise Helix and Vim setup; the complete snippets and validation workflow live +in [Editor setup](editors.md). diff --git a/src/bin/editor_help.rs b/src/bin/editor_help.rs index 33ec0c1..05e7275 100644 --- a/src/bin/editor_help.rs +++ b/src/bin/editor_help.rs @@ -1,6 +1,6 @@ //! Shared editor-integration help for the formatter and LSP adapters. -pub const EDITOR_SETUP: &str = r#"EDITOR SETUP (fmt: Milestone 5; LSP: Milestone 6): +pub const EDITOR_SETUP: &str = r#"EDITOR SETUP (fmt available; LSP planned for Milestone 6): Helix ~/.config/helix/languages.toml: [language-server.john-slop] command = "john-slop-lsp" diff --git a/tasks.md b/tasks.md index 9da7677..690b7a6 100644 --- a/tasks.md +++ b/tasks.md @@ -159,27 +159,32 @@ and remediation help; CI can consume stable machine output and exit statuses. ## Milestone 5 — Shared ASCII replacement engine and formatter -- [ ] Parse replacement definitions into a trie or deterministic longest-match +- [x] Parse replacement definitions into a trie or deterministic longest-match planner and emit validated, non-overlapping `TextEdit`s against the original source. -- [ ] Apply edits from highest to lowest byte offset. Preserve line endings and +- [x] Apply edits from highest to lowest byte offset. Preserve line endings and return the original buffer unchanged when there are no edits. -- [ ] Reject non-character-boundary and overlapping edits as typed invariant +- [x] Reject non-character-boundary and overlapping edits as typed invariant errors. Define deterministic behavior when sources share prefixes. -- [ ] Implement `john-slop-fmt` stdin/stdout mode, `--check`, `--diff`, and +- [x] Implement `john-slop-fmt` stdin/stdout mode, `--check`, `--diff`, and explicit `--write`. Require an explicit policy for recursive directory writes. -- [ ] Make writes atomic with a same-directory temporary file plus rename when +- [x] Make writes atomic with a same-directory temporary file plus rename when supported; retain permissions and never rewrite on analysis/config failure. -- [ ] Do not rewrite ZIP/DOCX/XLSX/PPTX members in this milestone; emit a clear +- [x] Do not rewrite ZIP/DOCX/XLSX/PPTX members in this milestone; emit a clear unsupported-operation diagnostic. -- [ ] Add unit/property tests for idempotence, Unicode boundaries, longest match, +- [x] Add unit/property tests for idempotence, Unicode boundaries, longest match, non-overlap, stable output, line endings, empty input, and no-op byte identity. -- [ ] Add integration tests proving `--check` exit status, stdout purity, atomic +- [x] Add integration tests proving `--check` exit status, stdout purity, atomic failure behavior, config overrides, and formatter/detector agreement. Acceptance: formatting twice equals formatting once; default invocation does not modify files; every replacement comes from the resolved typed config. +Decision: `ReplacementRule::source` is a non-empty Unicode `String` rather than +a single `char`, enabling deterministic longest-match behavior for shared +prefixes. This is a deliberate public data-contract change made before the +first formatter milestone tag. + ## Milestone 6 — LSP binary - [ ] Select an actively maintained LSP framework after a short dependency/API