john-slop/tasks.md

14 KiB

John Slop implementation plan

This backlog turns the current proof of concept into a reusable detector library with three thin binaries. Check an item only after its acceptance criteria and quality gates pass. Preserve unrelated user changes while refactoring.

Definition of done

A task is done when its public behavior is documented, errors carry actionable context, automated tests cover success and failure paths, and the following pass:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features
cargo doc --no-deps

For user-visible output, add deterministic snapshots/golden assertions without terminal color. For new dependencies, disable unnecessary default features, explain their role in the commit/PR, and run the available dependency/security audit in CI.

Milestone 0 — Baseline and decisions

  • Capture current behavior with tests before moving code: plain file, recursive directory, ZIP member, invalid UTF-8, missing path, and basic Unicode matching.
  • Fix the misleading Unicode tests/comments (UTF-8 byte lengths and grapheme terminology) so test names reflect what is actually asserted.
  • Record initial decisions in docs/decisions/: evidence confidence is not authorship probability; UTF-8 byte spans are canonical; formatter writes are opt-in; no archive mutation in the first formatter release.
  • Define supported inputs and limits: extensions/MIME policy, maximum file bytes, archive bytes, expanded bytes, entry count, nesting, hidden files, symlinks, and ignore rules.
  • Establish an MSRV in Cargo.toml and CI, add license/repository metadata, and document supported platforms.
  • Move tempfile and other test-only crates to [dev-dependencies]; remove rayon, once_cell, or unicode-segmentation if the final design does not justify them.

Acceptance: the existing behavior is characterized, ambiguous product choices are written down, and a clean baseline passes all quality gates.

Milestone 1 — Library-first restructure

  • Add src/lib.rs and expose a small facade for loading config, analyzing a source, planning replacements, and reading inputs.
  • Replace demo functions and ad hoc errors in main.rs with typed domain values (SourceId, Finding, Analysis, Confidence, TextEdit).
  • Split reader.rs into input/file.rs, input/walk.rs, and input/archive.rs; yield one Result<Document, InputError> at a time rather than loading a directory into a Vec.
  • Preserve outer and inner archive identities. Prevent path traversal and decompression bombs using the limits from Milestone 0.
  • Centralize error enums with thiserror; add miette context only at presentation boundaries. Remove placeholder try_report and sample reports.
  • Add explicit [[bin]] targets for john-slop, john-slop-fmt, and john-slop-lsp; the latter two may initially call stub adapters but all three must compile.
  • Keep binary parsing/orchestration out of the library. Add clap with derive for stable help, validation, and exit-code handling.

Acceptance: integration tests can call the library without spawning a process; all binaries build; no production module is a demo; failures in one directory entry do not discard successful documents.

Requested quality and documentation pass

  • Update the declared Rust toolchain and direct dependencies to their latest stable releases; verify the locked graph with the declared toolchain.
  • Add focused behavioral and snapshot tests, including feature-matrix coverage, without snapshotting structured domain values unnecessarily.
  • Configure Tarpaulin with an enforceable coverage floor around 30%, run rust_autobadges, and publish local version/test/coverage badges.
  • Reduce the root README to a generic project landing page and move detailed usage, architecture, testing, coverage, and background links under docs/.

Milestone 2 — Typed layered configuration

  • Add Serde, figment with TOML support, and a platform-aware configuration directory crate such as directories.
  • Create assets/config.toml as the embedded replacement catalog with schema_version and documented examples.
  • Create assets/heuristics.toml as a separate embedded rule catalog with stable IDs, categories, matcher types, weights, messages, help, occurrence caps, and applicability scopes.
  • Implement exact precedence: embedded defaults < user config.toml < named user <config_name>.toml < explicit --config < explicit CLI flags.
  • Validate config names and refuse traversal, extensions, empty names, and non-ASCII characters. Resolve the user path with platform APIs.
  • Implement merge-by-ID for rule/replacement tables rather than allowing positional array replacement to produce surprising partial configuration.
  • Validate the fully resolved model: schema version, unique IDs/sources, ASCII-only replacement targets, regex compilation, finite non-negative weights, ordered thresholds, safe matchers, and edit conflicts.
  • Preserve enough provider metadata to identify the offending file and key in a miette configuration diagnostic.
  • Add john-slop config show and john-slop config paths so precedence is inspectable; output deterministic TOML or JSON.
  • Test every layer alone and in combination, including missing optional files, malformed TOML, unknown fields, named configs, array semantics, and CLI wins.

Acceptance: installed binaries work without source-tree assets, the resolved configuration is explainable, and all precedence/validation tests pass.

Milestone 3 — Heuristic detection engine

  • Define a Rule/Detector interface and a registry built entirely from the validated heuristic config. Compile reusable matchers once.
  • Implement exact phrase and regex rules with precise byte spans, match text, category, weight, message, help, and stable rule ID.
  • Add conservative first-party rules for copied assistant disclaimers, knowledge-cutoff language, prompt/response leakage, templated filler, repetitive transitions/conclusions, suspicious Unicode, and code-specific placeholder or tutorial residue.
  • Add applicability filters for prose versus source code, extension/language, include/exclude globs, and generated/vendor/minified paths.
  • Design the confidence formula: bounded weighted evidence, per-rule caps, correlated-family damping, normalization to 0..=100, and configurable low/medium/high thresholds.
  • Return zero confidence and no verdict for empty/no-evidence input. Keep unsupported/binary input distinct from a clean analysis.
  • Sort findings deterministically and deduplicate identical rule/span pairs.
  • Add table-driven unit tests for every shipped rule with positive examples and realistic near misses. Add tests for disabled rules, overrides, scope, caps, damping, threshold boundaries, and multilingual text.
  • Build a reviewed evaluation corpus that is legally distributable and labels observable low-care signals, not presumed author identity. Report per-rule precision/recall and aggregate false-positive rates before tuning defaults.
  • Document limitations, adversarial evasion, domain shift, and why the score is not an AI-authorship probability.

Acceptance: every score decomposes into visible findings; no high-confidence result can arise solely from a single weak repeated signal; the shipped rules and formula are pinned by tests.

Milestone 4 — Diagnostics and detector CLI

  • Convert each Finding into a typed miette::Diagnostic with named source, exact SourceSpan, severity, rule ID as diagnostic code, message, and help.
  • Add a summary that reports confidence band/score and counts by severity, while retaining every contributing diagnostic.
  • Build detector CLI inputs for file, directory, and stdin (--stdin-filename for language/path scoping); add ignore and maximum-size controls.
  • Add human, concise, and versioned JSON output modes. Ensure JSON and stdout diagnostics never contain logs or ANSI escapes.
  • Implement documented exit codes: 0 below fail threshold, 1 at/above it, 2 for usage/configuration/I/O failure. Add --fail-level.
  • Report per-document input errors without hiding findings from other documents; define whether partial failure returns 2 and test it.
  • Add CLI integration tests for spans containing multibyte UTF-8, multiple files, stdin, missing paths, malformed configs, stable ordering, JSON schema, and exit codes.

Acceptance: a user can trace the aggregate score to highlighted source ranges and remediation help; CI can consume stable machine output and exit statuses.

Milestone 5 — Shared ASCII replacement engine and formatter

  • Parse replacement definitions into a trie or deterministic longest-match planner and emit validated, non-overlapping TextEdits against the original source.
  • Apply edits from highest to lowest byte offset. Preserve line endings and return the original buffer unchanged when there are no edits.
  • Reject non-character-boundary and overlapping edits as typed invariant errors. Define deterministic behavior when sources share prefixes.
  • Implement john-slop-fmt stdin/stdout mode, --check, --diff, and explicit --write. Require an explicit policy for recursive directory writes.
  • Make writes atomic with a same-directory temporary file plus rename when supported; retain permissions and never rewrite on analysis/config failure.
  • Do not rewrite ZIP/DOCX/XLSX/PPTX members in this milestone; emit a clear unsupported-operation diagnostic.
  • Add unit/property tests for idempotence, Unicode boundaries, longest match, non-overlap, stable output, line endings, empty input, and no-op byte identity.
  • Add integration tests proving --check exit status, stdout purity, atomic failure behavior, config overrides, and formatter/detector agreement.

Acceptance: formatting twice equals formatting once; default invocation does not modify files; every replacement comes from the resolved typed config.

Decision: ReplacementRule::source is a non-empty Unicode String rather than a single char, enabling deterministic longest-match behavior for shared prefixes. This is a deliberate public data-contract change made before the first formatter milestone tag.

Milestone 6 — LSP binary

  • Select an actively maintained LSP framework after a short dependency/API review (evaluate tower-lsp-server); pin the decision and protocol assumptions in a decision record.
  • Implement stdio initialization/shutdown and incremental or full document synchronization with URI, version, language ID, and current text tracked per open document.
  • Build and thoroughly test byte-offset to LSP UTF-16 line/character mapping, including CRLF, emoji, combining marks, astral characters, and EOF positions.
  • Publish diagnostics only for configured ASCII-replacement findings in the initial release. Include stable codes and source john-slop.
  • Expose quick fixes/code actions using the exact edits from the shared replacement engine; reject stale-version requests.
  • Recompute after open/change/configuration updates, clear on close, and avoid publishing obsolete results when analysis overlaps newer edits.
  • Keep all protocol bytes on stdout and diagnostics/logging on stderr. Add opt-in structured logging without document contents.
  • Add in-process protocol tests for lifecycle, versions, Unicode positions, config reload, close cleanup, invalid URI, malformed messages, concurrent documents, and formatter/LSP output equivalence.

Acceptance: a standards-compliant client can see replacement diagnostics and apply a quick fix; the resulting text exactly matches john-slop-fmt under the same config.

Milestone 7 — Hardening, documentation, and release

  • Replace the placeholder README with installation, threat model, examples, configuration precedence/schema, rule authoring, confidence interpretation, CLI/formatter/LSP setup, exit codes, privacy, and limitations.
  • Publish commented sample base and named configs. Document how to disable a rule/replacement and how merge-by-ID works.
  • Add CI on the supported OS/MSRV matrix for formatting, Clippy, tests, docs, audit/deny checks, and fixture/golden consistency.
  • Add fuzz targets for TOML/config merging, regex/rule loading, UTF-8 span mapping, replacement planning, and malicious ZIP metadata. Seed them with regression fixtures.
  • Benchmark large files, many small files, rule scaling, and archive limits. Add parallelism only if measurement shows a benefit and output remains ordered.
  • Review dependency features, licenses, advisories, unsafe code, and release binary size. Generate an SBOM/checksums if releases distribute binaries.
  • Add changelog, contribution guide, code of conduct, security reporting instructions, release profile, and reproducible release procedure.
  • Run a false-positive review against the evaluation corpus and require an explicit sign-off before changing default rule weights or thresholds.

Acceptance: a new user can install and operate all three binaries from the README; CI enforces the documented contracts; release artifacts are reproducible and the detector's limitations are prominent.

Deferred by design

  • Statistical or model-based authorship classification.
  • Automatic prose or source-code rewriting beyond configured ASCII replacements.
  • Locale-aware confusable, invisible, and bidirectional Unicode detection. The researched catalogs and adoption constraints are recorded in docs/unicode-confusables.md; detection must remain separate from automatic replacement.
  • In-place modification of archive members or office documents.
  • Network-backed analysis or telemetry.
  • Editor-specific extensions; the LSP is the integration boundary.

These are not implied follow-ups. Each requires a separate privacy, safety, performance, and product-design review before entering the active milestones.