john-slop/tasks.md

244 lines
14 KiB
Markdown

# John Slop implementation plan
This backlog turns the current proof of concept into a reusable detector library
with three thin binaries. Check an item only after its acceptance criteria and
quality gates pass. Preserve unrelated user changes while refactoring.
## Definition of done
A task is done when its public behavior is documented, errors carry actionable
context, automated tests cover success and failure paths, and the following pass:
```text
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features
cargo doc --no-deps
```
For user-visible output, add deterministic snapshots/golden assertions without
terminal color. For new dependencies, disable unnecessary default features,
explain their role in the commit/PR, and run the available dependency/security
audit in CI.
## Milestone 0 — Baseline and decisions
- [x] Capture current behavior with tests before moving code: plain file,
recursive directory, ZIP member, invalid UTF-8, missing path, and basic Unicode
matching.
- [x] Fix the misleading Unicode tests/comments (UTF-8 byte lengths and grapheme
terminology) so test names reflect what is actually asserted.
- [x] Record initial decisions in `docs/decisions/`: evidence confidence is not
authorship probability; UTF-8 byte spans are canonical; formatter writes are
opt-in; no archive mutation in the first formatter release.
- [x] Define supported inputs and limits: extensions/MIME policy, maximum file
bytes, archive bytes, expanded bytes, entry count, nesting, hidden files,
symlinks, and ignore rules.
- [x] Establish an MSRV in `Cargo.toml` and CI, add license/repository metadata,
and document supported platforms.
- [x] Move `tempfile` and other test-only crates to `[dev-dependencies]`; remove
`rayon`, `once_cell`, or `unicode-segmentation` if the final design does not
justify them.
Acceptance: the existing behavior is characterized, ambiguous product choices
are written down, and a clean baseline passes all quality gates.
## Milestone 1 — Library-first restructure
- [x] Add `src/lib.rs` and expose a small facade for loading config, analyzing a
source, planning replacements, and reading inputs.
- [x] Replace demo functions and ad hoc errors in `main.rs` with typed domain
values (`SourceId`, `Finding`, `Analysis`, `Confidence`, `TextEdit`).
- [x] Split `reader.rs` into `input/file.rs`, `input/walk.rs`, and
`input/archive.rs`; yield one `Result<Document, InputError>` at a time rather
than loading a directory into a `Vec`.
- [x] Preserve outer and inner archive identities. Prevent path traversal and
decompression bombs using the limits from Milestone 0.
- [x] Centralize error enums with `thiserror`; add `miette` context only at
presentation boundaries. Remove placeholder `try_report` and sample reports.
- [x] Add explicit `[[bin]]` targets for `john-slop`, `john-slop-fmt`, and
`john-slop-lsp`; the latter two may initially call stub adapters but all three
must compile.
- [x] Keep binary parsing/orchestration out of the library. Add `clap` with derive
for stable help, validation, and exit-code handling.
Acceptance: integration tests can call the library without spawning a process;
all binaries build; no production module is a demo; failures in one directory
entry do not discard successful documents.
## Requested quality and documentation pass
- [x] Update the declared Rust toolchain and direct dependencies to their latest
stable releases; verify the locked graph with the declared toolchain.
- [x] Add focused behavioral and snapshot tests, including feature-matrix
coverage, without snapshotting structured domain values unnecessarily.
- [x] Configure Tarpaulin with an enforceable coverage floor around 30%, run
`rust_autobadges`, and publish local version/test/coverage badges.
- [x] Reduce the root README to a generic project landing page and move detailed
usage, architecture, testing, coverage, and background links under `docs/`.
## Milestone 2 — Typed layered configuration
- [x] Add Serde, `figment` with TOML support, and a platform-aware configuration
directory crate such as `directories`.
- [x] Create `assets/config.toml` as the embedded replacement catalog with
`schema_version` and documented examples.
- [x] Create `assets/heuristics.toml` as a separate embedded rule catalog with
stable IDs, categories, matcher types, weights, messages, help, occurrence caps,
and applicability scopes.
- [x] Implement exact precedence: embedded defaults < user `config.toml` < named
user `<config_name>.toml` < explicit `--config` < explicit CLI flags.
- [x] Validate config names and refuse traversal, extensions, empty names, and
non-ASCII characters. Resolve the user path with platform APIs.
- [x] Implement merge-by-ID for rule/replacement tables rather than allowing
positional array replacement to produce surprising partial configuration.
- [x] Validate the fully resolved model: schema version, unique IDs/sources,
ASCII-only replacement targets, regex compilation, finite non-negative weights,
ordered thresholds, safe matchers, and edit conflicts.
- [x] Preserve enough provider metadata to identify the offending file and key in
a `miette` configuration diagnostic.
- [x] Add `john-slop config show` and `john-slop config paths` so precedence is
inspectable; output deterministic TOML or JSON.
- [x] Test every layer alone and in combination, including missing optional files,
malformed TOML, unknown fields, named configs, array semantics, and CLI wins.
Acceptance: installed binaries work without source-tree assets, the resolved
configuration is explainable, and all precedence/validation tests pass.
## Milestone 3 — Heuristic detection engine
- [x] Define a `Rule`/`Detector` interface and a registry built entirely from the
validated heuristic config. Compile reusable matchers once.
- [x] Implement exact phrase and regex rules with precise byte spans, match text,
category, weight, message, help, and stable rule ID.
- [x] Add conservative first-party rules for copied assistant disclaimers,
knowledge-cutoff language, prompt/response leakage, templated filler, repetitive
transitions/conclusions, suspicious Unicode, and code-specific placeholder or
tutorial residue.
- [x] Add applicability filters for prose versus source code, extension/language,
include/exclude globs, and generated/vendor/minified paths.
- [x] Design the confidence formula: bounded weighted evidence, per-rule caps,
correlated-family damping, normalization to `0..=100`, and configurable
low/medium/high thresholds.
- [x] Return zero confidence and no verdict for empty/no-evidence input. Keep
unsupported/binary input distinct from a clean analysis.
- [x] Sort findings deterministically and deduplicate identical rule/span pairs.
- [x] Add table-driven unit tests for every shipped rule with positive examples
and realistic near misses. Add tests for disabled rules, overrides, scope,
caps, damping, threshold boundaries, and multilingual text.
- [x] Build a reviewed evaluation corpus that is legally distributable and labels
observable low-care signals, not presumed author identity. Report per-rule
precision/recall and aggregate false-positive rates before tuning defaults.
- [x] Document limitations, adversarial evasion, domain shift, and why the score
is not an AI-authorship probability.
Acceptance: every score decomposes into visible findings; no high-confidence
result can arise solely from a single weak repeated signal; the shipped rules and
formula are pinned by tests.
## Milestone 4 — Diagnostics and detector CLI
- [x] Convert each `Finding` into a typed `miette::Diagnostic` with named source,
exact `SourceSpan`, severity, rule ID as diagnostic code, message, and help.
- [x] Add a summary that reports confidence band/score and counts by severity,
while retaining every contributing diagnostic.
- [x] Build detector CLI inputs for file, directory, and stdin (`--stdin-filename`
for language/path scoping); add ignore and maximum-size controls.
- [x] Add human, concise, and versioned JSON output modes. Ensure JSON and stdout
diagnostics never contain logs or ANSI escapes.
- [x] Implement documented exit codes: `0` below fail threshold, `1` at/above it,
`2` for usage/configuration/I/O failure. Add `--fail-level`.
- [x] Report per-document input errors without hiding findings from other
documents; define whether partial failure returns `2` and test it.
- [x] Add CLI integration tests for spans containing multibyte UTF-8, multiple
files, stdin, missing paths, malformed configs, stable ordering, JSON schema,
and exit codes.
Acceptance: a user can trace the aggregate score to highlighted source ranges
and remediation help; CI can consume stable machine output and exit statuses.
## Milestone 5 — Shared ASCII replacement engine and formatter
- [ ] Parse replacement definitions into a trie or deterministic longest-match
planner and emit validated, non-overlapping `TextEdit`s against the original
source.
- [ ] Apply edits from highest to lowest byte offset. Preserve line endings and
return the original buffer unchanged when there are no edits.
- [ ] Reject non-character-boundary and overlapping edits as typed invariant
errors. Define deterministic behavior when sources share prefixes.
- [ ] Implement `john-slop-fmt` stdin/stdout mode, `--check`, `--diff`, and
explicit `--write`. Require an explicit policy for recursive directory writes.
- [ ] Make writes atomic with a same-directory temporary file plus rename when
supported; retain permissions and never rewrite on analysis/config failure.
- [ ] Do not rewrite ZIP/DOCX/XLSX/PPTX members in this milestone; emit a clear
unsupported-operation diagnostic.
- [ ] Add unit/property tests for idempotence, Unicode boundaries, longest match,
non-overlap, stable output, line endings, empty input, and no-op byte identity.
- [ ] Add integration tests proving `--check` exit status, stdout purity, atomic
failure behavior, config overrides, and formatter/detector agreement.
Acceptance: formatting twice equals formatting once; default invocation does not
modify files; every replacement comes from the resolved typed config.
## Milestone 6 — LSP binary
- [ ] Select an actively maintained LSP framework after a short dependency/API
review (evaluate `tower-lsp-server`); pin the decision and protocol assumptions
in a decision record.
- [ ] Implement stdio initialization/shutdown and incremental or full document
synchronization with URI, version, language ID, and current text tracked per
open document.
- [ ] Build and thoroughly test byte-offset to LSP UTF-16 line/character mapping,
including CRLF, emoji, combining marks, astral characters, and EOF positions.
- [ ] Publish diagnostics only for configured ASCII-replacement findings in the
initial release. Include stable codes and source `john-slop`.
- [ ] Expose quick fixes/code actions using the exact edits from the shared
replacement engine; reject stale-version requests.
- [ ] Recompute after open/change/configuration updates, clear on close, and avoid
publishing obsolete results when analysis overlaps newer edits.
- [ ] Keep all protocol bytes on stdout and diagnostics/logging on stderr. Add
opt-in structured logging without document contents.
- [ ] Add in-process protocol tests for lifecycle, versions, Unicode positions,
config reload, close cleanup, invalid URI, malformed messages, concurrent
documents, and formatter/LSP output equivalence.
Acceptance: a standards-compliant client can see replacement diagnostics and
apply a quick fix; the resulting text exactly matches `john-slop-fmt` under the
same config.
## Milestone 7 — Hardening, documentation, and release
- [ ] Replace the placeholder README with installation, threat model, examples,
configuration precedence/schema, rule authoring, confidence interpretation,
CLI/formatter/LSP setup, exit codes, privacy, and limitations.
- [ ] Publish commented sample base and named configs. Document how to disable a
rule/replacement and how merge-by-ID works.
- [ ] Add CI on the supported OS/MSRV matrix for formatting, Clippy, tests, docs,
audit/deny checks, and fixture/golden consistency.
- [ ] Add fuzz targets for TOML/config merging, regex/rule loading, UTF-8 span
mapping, replacement planning, and malicious ZIP metadata. Seed them with
regression fixtures.
- [ ] Benchmark large files, many small files, rule scaling, and archive limits.
Add parallelism only if measurement shows a benefit and output remains ordered.
- [ ] Review dependency features, licenses, advisories, unsafe code, and release
binary size. Generate an SBOM/checksums if releases distribute binaries.
- [ ] Add changelog, contribution guide, code of conduct, security reporting
instructions, release profile, and reproducible release procedure.
- [ ] Run a false-positive review against the evaluation corpus and require an
explicit sign-off before changing default rule weights or thresholds.
Acceptance: a new user can install and operate all three binaries from the README;
CI enforces the documented contracts; release artifacts are reproducible and the
detector's limitations are prominent.
## Deferred by design
- Statistical or model-based authorship classification.
- Automatic prose or source-code rewriting beyond configured ASCII replacements.
- In-place modification of archive members or office documents.
- Network-backed analysis or telemetry.
- Editor-specific extensions; the LSP is the integration boundary.
These are not implied follow-ups. Each requires a separate privacy, safety,
performance, and product-design review before entering the active milestones.