john-slop/tasks.md

390 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# John Slop implementation plan
This backlog turns the current proof of concept into a reusable detector library
with three thin binaries. Check an item only after its acceptance criteria and
quality gates pass. Preserve unrelated user changes while refactoring.
## Definition of done
A task is done when its public behavior is documented, errors carry actionable
context, automated tests cover success and failure paths, and the following pass:
```text
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features
cargo doc --no-deps
```
For user-visible output, add deterministic snapshots/golden assertions without
terminal color. For new dependencies, disable unnecessary default features,
explain their role in the commit/PR, and run the available dependency/security
audit in CI.
## Milestone 0 — Baseline and decisions
- [x] Capture current behavior with tests before moving code: plain file,
recursive directory, ZIP member, invalid UTF-8, missing path, and basic Unicode
matching.
- [x] Fix the misleading Unicode tests/comments (UTF-8 byte lengths and grapheme
terminology) so test names reflect what is actually asserted.
- [x] Record initial decisions in `docs/decisions/`: evidence confidence is not
authorship probability; UTF-8 byte spans are canonical; formatter writes are
opt-in; no archive mutation in the first formatter release.
- [x] Define supported inputs and limits: extensions/MIME policy, maximum file
bytes, archive bytes, expanded bytes, entry count, nesting, hidden files,
symlinks, and ignore rules.
- [x] Establish an MSRV in `Cargo.toml` and CI, add license/repository metadata,
and document supported platforms.
- [x] Move `tempfile` and other test-only crates to `[dev-dependencies]`; remove
`rayon`, `once_cell`, or `unicode-segmentation` if the final design does not
justify them.
Acceptance: the existing behavior is characterized, ambiguous product choices
are written down, and a clean baseline passes all quality gates.
## Milestone 1 — Library-first restructure
- [x] Add `src/lib.rs` and expose a small facade for loading config, analyzing a
source, planning replacements, and reading inputs.
- [x] Replace demo functions and ad hoc errors in `main.rs` with typed domain
values (`SourceId`, `Finding`, `Analysis`, `Confidence`, `TextEdit`).
- [x] Split `reader.rs` into `input/file.rs`, `input/walk.rs`, and
`input/archive.rs`; yield one `Result<Document, InputError>` at a time rather
than loading a directory into a `Vec`.
- [x] Preserve outer and inner archive identities. Prevent path traversal and
decompression bombs using the limits from Milestone 0.
- [x] Centralize error enums with `thiserror`; add `miette` context only at
presentation boundaries. Remove placeholder `try_report` and sample reports.
- [x] Add explicit `[[bin]]` targets for `john-slop`, `john-slop-fmt`, and
`john-slop-lsp`; the latter two may initially call stub adapters but all three
must compile.
- [x] Keep binary parsing/orchestration out of the library. Add `clap` with derive
for stable help, validation, and exit-code handling.
Acceptance: integration tests can call the library without spawning a process;
all binaries build; no production module is a demo; failures in one directory
entry do not discard successful documents.
## Requested quality and documentation pass
- [x] Update the declared Rust toolchain and direct dependencies to their latest
stable releases; verify the locked graph with the declared toolchain.
- [x] Add focused behavioral and snapshot tests, including feature-matrix
coverage, without snapshotting structured domain values unnecessarily.
- [x] Configure Tarpaulin with an enforceable coverage floor around 30%, run
`rust_autobadges`, and publish local version/test/coverage badges.
- [x] Reduce the root README to a generic project landing page and move detailed
usage, architecture, testing, coverage, and background links under `docs/`.
## Milestone 2 — Typed layered configuration
- [x] Add Serde, `figment` with TOML support, and a platform-aware configuration
directory crate such as `directories`.
- [x] Create `assets/config.toml` as the embedded replacement catalog with
`schema_version` and documented examples.
- [x] Create `assets/heuristics.toml` as a separate embedded rule catalog with
stable IDs, categories, matcher types, weights, messages, help, occurrence caps,
and applicability scopes.
- [x] Implement exact precedence: embedded defaults < user `config.toml` < named
user `<config_name>.toml` < explicit `--config` < explicit CLI flags.
- [x] Validate config names and refuse traversal, extensions, empty names, and
non-ASCII characters. Resolve the user path with platform APIs.
- [x] Implement merge-by-ID for rule/replacement tables rather than allowing
positional array replacement to produce surprising partial configuration.
- [x] Validate the fully resolved model: schema version, unique IDs/sources,
ASCII-only replacement targets, regex compilation, finite non-negative weights,
ordered thresholds, safe matchers, and edit conflicts.
- [x] Preserve enough provider metadata to identify the offending file and key in
a `miette` configuration diagnostic.
- [x] Add `john-slop config show` and `john-slop config paths` so precedence is
inspectable; output deterministic TOML or JSON.
- [x] Test every layer alone and in combination, including missing optional files,
malformed TOML, unknown fields, named configs, array semantics, and CLI wins.
Acceptance: installed binaries work without source-tree assets, the resolved
configuration is explainable, and all precedence/validation tests pass.
## Milestone 3 — Heuristic detection engine
- [x] Define a `Rule`/`Detector` interface and a registry built entirely from the
validated heuristic config. Compile reusable matchers once.
- [x] Implement exact phrase and regex rules with precise byte spans, match text,
category, weight, message, help, and stable rule ID.
- [x] Add conservative first-party rules for copied assistant disclaimers,
knowledge-cutoff language, prompt/response leakage, templated filler, repetitive
transitions/conclusions, suspicious Unicode, and code-specific placeholder or
tutorial residue.
- [x] Add applicability filters for prose versus source code, extension/language,
include/exclude globs, and generated/vendor/minified paths.
- [x] Design the confidence formula: bounded weighted evidence, per-rule caps,
correlated-family damping, normalization to `0..=100`, and configurable
low/medium/high thresholds.
- [x] Return zero confidence and no verdict for empty/no-evidence input. Keep
unsupported/binary input distinct from a clean analysis.
- [x] Sort findings deterministically and deduplicate identical rule/span pairs.
- [x] Add table-driven unit tests for every shipped rule with positive examples
and realistic near misses. Add tests for disabled rules, overrides, scope,
caps, damping, threshold boundaries, and multilingual text.
- [x] Build a reviewed evaluation corpus that is legally distributable and labels
observable low-care signals, not presumed author identity. Report per-rule
precision/recall and aggregate false-positive rates before tuning defaults.
- [x] Document limitations, adversarial evasion, domain shift, and why the score
is not an AI-authorship probability.
Acceptance: every score decomposes into visible findings; no high-confidence
result can arise solely from a single weak repeated signal; the shipped rules and
formula are pinned by tests.
## Milestone 4 — Diagnostics and detector CLI
- [x] Convert each `Finding` into a typed `miette::Diagnostic` with named source,
exact `SourceSpan`, severity, rule ID as diagnostic code, message, and help.
- [x] Add a summary that reports confidence band/score and counts by severity,
while retaining every contributing diagnostic.
- [x] Build detector CLI inputs for file, directory, and stdin (`--stdin-filename`
for language/path scoping); add ignore and maximum-size controls.
- [x] Add human, concise, and versioned JSON output modes. Ensure JSON and stdout
diagnostics never contain logs or ANSI escapes.
- [x] Implement documented exit codes: `0` below fail threshold, `1` at/above it,
`2` for usage/configuration/I/O failure. Add `--fail-level`.
- [x] Report per-document input errors without hiding findings from other
documents; define whether partial failure returns `2` and test it.
- [x] Add CLI integration tests for spans containing multibyte UTF-8, multiple
files, stdin, missing paths, malformed configs, stable ordering, JSON schema,
and exit codes.
Acceptance: a user can trace the aggregate score to highlighted source ranges
and remediation help; CI can consume stable machine output and exit statuses.
## Milestone 5 — Shared ASCII replacement engine and formatter
- [x] Parse replacement definitions into a trie or deterministic longest-match
planner and emit validated, non-overlapping `TextEdit`s against the original
source.
- [x] Apply edits from highest to lowest byte offset. Preserve line endings and
return the original buffer unchanged when there are no edits.
- [x] Reject non-character-boundary and overlapping edits as typed invariant
errors. Define deterministic behavior when sources share prefixes.
- [x] Implement `john-slop-fmt` stdin/stdout mode, `--check`, `--diff`, and
explicit `--write`. Require an explicit policy for recursive directory writes.
- [x] Make writes atomic with a same-directory temporary file plus rename when
supported; retain permissions and never rewrite on analysis/config failure.
- [x] Do not rewrite ZIP/DOCX/XLSX/PPTX members in this milestone; emit a clear
unsupported-operation diagnostic.
- [x] Add unit/property tests for idempotence, Unicode boundaries, longest match,
non-overlap, stable output, line endings, empty input, and no-op byte identity.
- [x] Add integration tests proving `--check` exit status, stdout purity, atomic
failure behavior, config overrides, and formatter/detector agreement.
Acceptance: formatting twice equals formatting once; default invocation does not
modify files; every replacement comes from the resolved typed config.
Decision: `ReplacementRule::source` is a non-empty Unicode `String` rather than
a single `char`, enabling deterministic longest-match behavior for shared
prefixes. This is a deliberate public data-contract change made before the
first formatter milestone tag.
## Milestone 6 — LSP binary
- [x] Select an actively maintained LSP framework after a short dependency/API
review (evaluate `tower-lsp-server`); pin the decision and protocol assumptions
in a decision record.
- [x] Implement stdio initialization/shutdown and incremental or full document
synchronization with URI, version, language ID, and current text tracked per
open document.
- [x] Build and thoroughly test byte-offset to LSP UTF-16 line/character mapping,
including CRLF, emoji, combining marks, astral characters, and EOF positions.
- [x] Publish diagnostics only for configured ASCII-replacement findings in the
initial release. Include stable codes and source `john-slop`.
- [x] Expose quick fixes/code actions using the exact edits from the shared
replacement engine; reject stale-version requests.
- [x] Recompute after open/change/configuration updates, clear on close, and avoid
publishing obsolete results when analysis overlaps newer edits.
- [x] Keep all protocol bytes on stdout and diagnostics/logging on stderr. Add
opt-in structured logging without document contents.
- [x] Add in-process protocol tests for lifecycle, versions, Unicode positions,
config reload, close cleanup, invalid URI, malformed messages, concurrent
documents, and formatter/LSP output equivalence.
Acceptance: a standards-compliant client can see replacement diagnostics and
apply a quick fix; the resulting text exactly matches `john-slop-fmt` under the
same config.
## Milestone 7 — Hardening, documentation, and release
- [x] Replace the placeholder README with installation, threat model, examples,
configuration precedence/schema, rule authoring, confidence interpretation,
CLI/formatter/LSP setup, exit codes, privacy, and limitations.
- [x] Publish commented sample base and named configs. Document how to disable a
rule/replacement and how merge-by-ID works.
- [x] Add CI on the supported OS/MSRV matrix for formatting, Clippy, tests, docs,
audit/deny checks, and fixture/golden consistency.
- [x] Add fuzz targets for TOML/config merging, regex/rule loading, UTF-8 span
mapping, replacement planning, and malicious ZIP metadata. Seed them with
regression fixtures.
- [x] Benchmark large files, many small files, rule scaling, and archive limits.
Add parallelism only if measurement shows a benefit and output remains ordered.
- [x] Review dependency features, licenses, advisories, unsafe code, and release
binary size. Generate an SBOM/checksums if releases distribute binaries.
- [x] Add changelog, contribution guide, code of conduct, security reporting
instructions, release profile, and reproducible release procedure.
- [x] Run a false-positive review against the evaluation corpus and require an
explicit sign-off before changing default rule weights or thresholds.
Acceptance: a new user can install and operate all three binaries from the README;
CI enforces the documented contracts; release artifacts are reproducible and the
detector's limitations are prominent.
Accepted on 2026-09-29 after the full all-feature/minimal-feature quality gate,
dependency policy checks, fuzz-seed runs, reproducible packaging, and an 80.51%
line-coverage run.
## Milestone 8 — Meaningful coverage expansion
- [x] Cover configuration merging, validation failures, domain formatting, and
diagnostic accessors with behavior-focused unit tests.
- [x] Cover archive and input resource-policy branches with generated fixtures,
including unsafe, binary, malformed, and oversized members.
- [x] Cover remaining practical CLI, formatter, and LSP adapter branches without
coupling tests to unreachable platform failures or coverage-tool artifacts.
- [x] Raise measured Tarpaulin line coverage toward 90%, refresh badges and
development documentation, and record any intentionally uncovered remainder.
Acceptance: the suite exercises at least 90% of measurable source lines when
Tarpaulin can attribute them reliably; remaining misses are documented and do
not justify brittle tests or production changes made solely for instrumentation.
Accepted on 2026-09-29 at 92.16% line coverage (1,329/1,442), with the CI floor
raised to 90%. Remaining misses are primarily hard-to-inject OS/I/O failures,
malformed dependency internals, overflow guards, and asynchronous LSP error
plumbing; pursuing them would add test-only seams or backend-sensitive checks.
## Milestone 9 — Terminal-first diagnostics
- [x] Make colored graphical Miette diagnostics the detector default while
retaining explicit plain-human, concise, JSON, and no-feature fallback modes.
Acceptance: a default detector invocation renders graphical Unicode diagnostics
with ANSI severity colors; explicit machine modes remain stable and uncolored,
and builds without `fancy-diagnostics` retain the plain human fallback.
## Milestone 10 — Prompt-leakage scoring priority
- [x] Give distinct prompt-leakage rules lighter correlation damping (`0.90ⁿ`)
while preserving `0.75ⁿ` damping for style, typography, and source-code
families and preserving per-rule occurrence damping/caps.
- [x] Re-run the reviewed corpus, renew the existing default-policy sign-off,
and document the changed formula and showcase score without changing the
sign-off mechanism itself.
Acceptance: multiple independent prompt-leakage artifacts contribute more than
equally weighted correlated style signals; the diagnostic showcase score is
pinned, the evaluation corpus remains free of observed regressions, and all
quality/coverage gates pass.
Accepted on 2026-09-29: the showcase score is 71 (medium); reviewed positive
corpus scores are pinned at 60/33/19/24/17, all five negative documents remain
at zero, and matching metrics remain 11 TP, 0 FP, and 0 FN.
Optional follow-up, intentionally not required for this milestone:
- [ ] After observing real usage, choose one direction for the default-policy
sign-off feature: harden its change boundary beyond the catalog hash, or
remove it if its maintenance cost exceeds its demonstrated value.
## Milestone 11 — Evidence-score terminology
- [x] Rename the public domain model from ambiguous confidence terminology to
`EvidenceScore`, `EvidenceBand`, and `EvidenceThresholds`; retain deprecated
Rust type aliases and the legacy `--fail-confidence` CLI alias for migration.
- [x] Publish the low-care evidence-score wording across terminal output,
examples, and documentation; rename `--fail-confidence` to `--fail-score` and
version the JSON field rename as schema version 2.
- [x] Run the complete quality and coverage gates, refresh generated badges, and
record the compatibility boundary and unchanged scoring formula.
Acceptance: every active user-facing contract describes the metric as a bounded
low-care evidence score rather than a probability or calibrated confidence;
legacy type and CLI names remain usable with clear migration guidance; JSON
consumers can identify the field rename by schema version; scoring results are
unchanged; and all quality/coverage gates pass.
Accepted on 2026-09-29: all-feature and minimal-feature suites, formatting,
Clippy, and documentation pass; deprecated Rust aliases and the legacy CLI alias
are covered; Tarpaulin remains at 92.19% (1,334/1,447 lines) against the 90%
floor; and the generated badges are current.
## Milestone 12 — Experimental crate boundary
- [x] Capture reproducible baseline cold and incremental check times from
`v0.1.11`, identify dependency ownership, and choose the smallest crate
boundary capable of excluding async/LSP dependencies from core builds.
- [x] Extract the reusable detector/configuration/input implementation into a
`john-slop-core` workspace crate and the protocol engine into a
`john-slop-lsp` workspace crate. Keep the root package as the compatibility
facade and binary distribution, preserving public behavior and all three
executable names.
- [x] Run identical clean and incremental measurements on the experiment,
compare compile time, artifact size, dependency isolation, API/release cost,
tests, coverage, and packaging against `v0.1.11`, then record an explicit
merge-or-abandon recommendation.
Acceptance: the branch contains a reviewable implementation commit and a
reproducible before/after report; all existing behavior and quality gates pass;
any claimed build improvement is measured rather than inferred. Do not merge
the experiment when its gains do not justify the added workspace and release
complexity.
Completed on 2026-09-29 with a **do-not-merge-yet** recommendation that was
subsequently superseded by the explicit Milestone 13 adoption decision. Core
and no-LSP checks use roughly 35–55% less CPU/disk, but the full workspace
remains effectively unchanged while package publishing, version
synchronization, asset ownership, feature contracts, CI, and coverage all
become more complex. The complete measurements and adoption rationale are in
`docs/experiments/crate-boundary.md` and ADR 0004.
## Milestone 13 — Supported Cargo workspace
- [x] Promote the experimental three-package layout to the supported project
architecture. Keep the root `john_slop` compatibility facade and binary
package as the workspace root; centralize shared package metadata, dependency
versions, and lint policy with Cargo workspace inheritance while retaining
per-package feature activation and optionality.
- [x] Add automated workspace-contract checks for membership, synchronized
versions, inherited dependency sources, and core/LSP feature isolation. Make
CI, packaging, and publication guidance explicitly workspace-aware and verify
every publishable package in dependency order.
- [x] Replace the experimental recommendation with an adoption decision, update
architecture/dependency/development/release documentation, run the complete
feature, lint, documentation, coverage, and package gates, and refresh the
generated badges and recap.
Acceptance: all three publishable packages inherit one version and one source of
truth for shared dependency requirements; `john-slop-core` remains free of
CLI/async/LSP dependencies; unqualified root commands cover every default
workspace member; each package can be assembled for publication in dependency
order; coverage remains at or above 90%; and the measured costs of supporting
the workspace are recorded rather than hidden.
Accepted on 2026-09-29 after both feature matrices, formatting, Clippy,
documentation, RustSec, cargo-deny, workspace package assembly, and the 90%
coverage floor passed. Tarpaulin remains at 92.19% (1,334/1,447 lines); the
workspace boundary changes dependency ownership, not executable behavior or
measurable source coverage.
## Deferred by design
- Statistical or model-based authorship classification.
- Automatic prose or source-code rewriting beyond configured ASCII replacements.
- Locale-aware confusable, invisible, and bidirectional Unicode detection. The
researched catalogs and adoption constraints are recorded in
`docs/unicode-confusables.md`; detection must remain separate from automatic
replacement.
- In-place modification of archive members or office documents.
- Network-backed analysis or telemetry.
- Editor-specific extensions; the LSP is the integration boundary.
These are not implied follow-ups. Each requires a separate privacy, safety,
performance, and product-design review before entering the active milestones.