390 lines
22 KiB
Markdown
390 lines
22 KiB
Markdown
# John Slop implementation plan
|
||
|
||
This backlog turns the current proof of concept into a reusable detector library
|
||
with three thin binaries. Check an item only after its acceptance criteria and
|
||
quality gates pass. Preserve unrelated user changes while refactoring.
|
||
|
||
## Definition of done
|
||
|
||
A task is done when its public behavior is documented, errors carry actionable
|
||
context, automated tests cover success and failure paths, and the following pass:
|
||
|
||
```text
|
||
cargo fmt --all -- --check
|
||
cargo clippy --all-targets --all-features -- -D warnings
|
||
cargo test --all-targets --all-features
|
||
cargo doc --no-deps
|
||
```
|
||
|
||
For user-visible output, add deterministic snapshots/golden assertions without
|
||
terminal color. For new dependencies, disable unnecessary default features,
|
||
explain their role in the commit/PR, and run the available dependency/security
|
||
audit in CI.
|
||
|
||
## Milestone 0 — Baseline and decisions
|
||
|
||
- [x] Capture current behavior with tests before moving code: plain file,
|
||
recursive directory, ZIP member, invalid UTF-8, missing path, and basic Unicode
|
||
matching.
|
||
- [x] Fix the misleading Unicode tests/comments (UTF-8 byte lengths and grapheme
|
||
terminology) so test names reflect what is actually asserted.
|
||
- [x] Record initial decisions in `docs/decisions/`: evidence confidence is not
|
||
authorship probability; UTF-8 byte spans are canonical; formatter writes are
|
||
opt-in; no archive mutation in the first formatter release.
|
||
- [x] Define supported inputs and limits: extensions/MIME policy, maximum file
|
||
bytes, archive bytes, expanded bytes, entry count, nesting, hidden files,
|
||
symlinks, and ignore rules.
|
||
- [x] Establish an MSRV in `Cargo.toml` and CI, add license/repository metadata,
|
||
and document supported platforms.
|
||
- [x] Move `tempfile` and other test-only crates to `[dev-dependencies]`; remove
|
||
`rayon`, `once_cell`, or `unicode-segmentation` if the final design does not
|
||
justify them.
|
||
|
||
Acceptance: the existing behavior is characterized, ambiguous product choices
|
||
are written down, and a clean baseline passes all quality gates.
|
||
|
||
## Milestone 1 — Library-first restructure
|
||
|
||
- [x] Add `src/lib.rs` and expose a small facade for loading config, analyzing a
|
||
source, planning replacements, and reading inputs.
|
||
- [x] Replace demo functions and ad hoc errors in `main.rs` with typed domain
|
||
values (`SourceId`, `Finding`, `Analysis`, `Confidence`, `TextEdit`).
|
||
- [x] Split `reader.rs` into `input/file.rs`, `input/walk.rs`, and
|
||
`input/archive.rs`; yield one `Result<Document, InputError>` at a time rather
|
||
than loading a directory into a `Vec`.
|
||
- [x] Preserve outer and inner archive identities. Prevent path traversal and
|
||
decompression bombs using the limits from Milestone 0.
|
||
- [x] Centralize error enums with `thiserror`; add `miette` context only at
|
||
presentation boundaries. Remove placeholder `try_report` and sample reports.
|
||
- [x] Add explicit `[[bin]]` targets for `john-slop`, `john-slop-fmt`, and
|
||
`john-slop-lsp`; the latter two may initially call stub adapters but all three
|
||
must compile.
|
||
- [x] Keep binary parsing/orchestration out of the library. Add `clap` with derive
|
||
for stable help, validation, and exit-code handling.
|
||
|
||
Acceptance: integration tests can call the library without spawning a process;
|
||
all binaries build; no production module is a demo; failures in one directory
|
||
entry do not discard successful documents.
|
||
|
||
## Requested quality and documentation pass
|
||
|
||
- [x] Update the declared Rust toolchain and direct dependencies to their latest
|
||
stable releases; verify the locked graph with the declared toolchain.
|
||
- [x] Add focused behavioral and snapshot tests, including feature-matrix
|
||
coverage, without snapshotting structured domain values unnecessarily.
|
||
- [x] Configure Tarpaulin with an enforceable coverage floor around 30%, run
|
||
`rust_autobadges`, and publish local version/test/coverage badges.
|
||
- [x] Reduce the root README to a generic project landing page and move detailed
|
||
usage, architecture, testing, coverage, and background links under `docs/`.
|
||
|
||
## Milestone 2 — Typed layered configuration
|
||
|
||
- [x] Add Serde, `figment` with TOML support, and a platform-aware configuration
|
||
directory crate such as `directories`.
|
||
- [x] Create `assets/config.toml` as the embedded replacement catalog with
|
||
`schema_version` and documented examples.
|
||
- [x] Create `assets/heuristics.toml` as a separate embedded rule catalog with
|
||
stable IDs, categories, matcher types, weights, messages, help, occurrence caps,
|
||
and applicability scopes.
|
||
- [x] Implement exact precedence: embedded defaults < user `config.toml` < named
|
||
user `<config_name>.toml` < explicit `--config` < explicit CLI flags.
|
||
- [x] Validate config names and refuse traversal, extensions, empty names, and
|
||
non-ASCII characters. Resolve the user path with platform APIs.
|
||
- [x] Implement merge-by-ID for rule/replacement tables rather than allowing
|
||
positional array replacement to produce surprising partial configuration.
|
||
- [x] Validate the fully resolved model: schema version, unique IDs/sources,
|
||
ASCII-only replacement targets, regex compilation, finite non-negative weights,
|
||
ordered thresholds, safe matchers, and edit conflicts.
|
||
- [x] Preserve enough provider metadata to identify the offending file and key in
|
||
a `miette` configuration diagnostic.
|
||
- [x] Add `john-slop config show` and `john-slop config paths` so precedence is
|
||
inspectable; output deterministic TOML or JSON.
|
||
- [x] Test every layer alone and in combination, including missing optional files,
|
||
malformed TOML, unknown fields, named configs, array semantics, and CLI wins.
|
||
|
||
Acceptance: installed binaries work without source-tree assets, the resolved
|
||
configuration is explainable, and all precedence/validation tests pass.
|
||
|
||
## Milestone 3 — Heuristic detection engine
|
||
|
||
- [x] Define a `Rule`/`Detector` interface and a registry built entirely from the
|
||
validated heuristic config. Compile reusable matchers once.
|
||
- [x] Implement exact phrase and regex rules with precise byte spans, match text,
|
||
category, weight, message, help, and stable rule ID.
|
||
- [x] Add conservative first-party rules for copied assistant disclaimers,
|
||
knowledge-cutoff language, prompt/response leakage, templated filler, repetitive
|
||
transitions/conclusions, suspicious Unicode, and code-specific placeholder or
|
||
tutorial residue.
|
||
- [x] Add applicability filters for prose versus source code, extension/language,
|
||
include/exclude globs, and generated/vendor/minified paths.
|
||
- [x] Design the confidence formula: bounded weighted evidence, per-rule caps,
|
||
correlated-family damping, normalization to `0..=100`, and configurable
|
||
low/medium/high thresholds.
|
||
- [x] Return zero confidence and no verdict for empty/no-evidence input. Keep
|
||
unsupported/binary input distinct from a clean analysis.
|
||
- [x] Sort findings deterministically and deduplicate identical rule/span pairs.
|
||
- [x] Add table-driven unit tests for every shipped rule with positive examples
|
||
and realistic near misses. Add tests for disabled rules, overrides, scope,
|
||
caps, damping, threshold boundaries, and multilingual text.
|
||
- [x] Build a reviewed evaluation corpus that is legally distributable and labels
|
||
observable low-care signals, not presumed author identity. Report per-rule
|
||
precision/recall and aggregate false-positive rates before tuning defaults.
|
||
- [x] Document limitations, adversarial evasion, domain shift, and why the score
|
||
is not an AI-authorship probability.
|
||
|
||
Acceptance: every score decomposes into visible findings; no high-confidence
|
||
result can arise solely from a single weak repeated signal; the shipped rules and
|
||
formula are pinned by tests.
|
||
|
||
## Milestone 4 — Diagnostics and detector CLI
|
||
|
||
- [x] Convert each `Finding` into a typed `miette::Diagnostic` with named source,
|
||
exact `SourceSpan`, severity, rule ID as diagnostic code, message, and help.
|
||
- [x] Add a summary that reports confidence band/score and counts by severity,
|
||
while retaining every contributing diagnostic.
|
||
- [x] Build detector CLI inputs for file, directory, and stdin (`--stdin-filename`
|
||
for language/path scoping); add ignore and maximum-size controls.
|
||
- [x] Add human, concise, and versioned JSON output modes. Ensure JSON and stdout
|
||
diagnostics never contain logs or ANSI escapes.
|
||
- [x] Implement documented exit codes: `0` below fail threshold, `1` at/above it,
|
||
`2` for usage/configuration/I/O failure. Add `--fail-level`.
|
||
- [x] Report per-document input errors without hiding findings from other
|
||
documents; define whether partial failure returns `2` and test it.
|
||
- [x] Add CLI integration tests for spans containing multibyte UTF-8, multiple
|
||
files, stdin, missing paths, malformed configs, stable ordering, JSON schema,
|
||
and exit codes.
|
||
|
||
Acceptance: a user can trace the aggregate score to highlighted source ranges
|
||
and remediation help; CI can consume stable machine output and exit statuses.
|
||
|
||
## Milestone 5 — Shared ASCII replacement engine and formatter
|
||
|
||
- [x] Parse replacement definitions into a trie or deterministic longest-match
|
||
planner and emit validated, non-overlapping `TextEdit`s against the original
|
||
source.
|
||
- [x] Apply edits from highest to lowest byte offset. Preserve line endings and
|
||
return the original buffer unchanged when there are no edits.
|
||
- [x] Reject non-character-boundary and overlapping edits as typed invariant
|
||
errors. Define deterministic behavior when sources share prefixes.
|
||
- [x] Implement `john-slop-fmt` stdin/stdout mode, `--check`, `--diff`, and
|
||
explicit `--write`. Require an explicit policy for recursive directory writes.
|
||
- [x] Make writes atomic with a same-directory temporary file plus rename when
|
||
supported; retain permissions and never rewrite on analysis/config failure.
|
||
- [x] Do not rewrite ZIP/DOCX/XLSX/PPTX members in this milestone; emit a clear
|
||
unsupported-operation diagnostic.
|
||
- [x] Add unit/property tests for idempotence, Unicode boundaries, longest match,
|
||
non-overlap, stable output, line endings, empty input, and no-op byte identity.
|
||
- [x] Add integration tests proving `--check` exit status, stdout purity, atomic
|
||
failure behavior, config overrides, and formatter/detector agreement.
|
||
|
||
Acceptance: formatting twice equals formatting once; default invocation does not
|
||
modify files; every replacement comes from the resolved typed config.
|
||
|
||
Decision: `ReplacementRule::source` is a non-empty Unicode `String` rather than
|
||
a single `char`, enabling deterministic longest-match behavior for shared
|
||
prefixes. This is a deliberate public data-contract change made before the
|
||
first formatter milestone tag.
|
||
|
||
## Milestone 6 — LSP binary
|
||
|
||
- [x] Select an actively maintained LSP framework after a short dependency/API
|
||
review (evaluate `tower-lsp-server`); pin the decision and protocol assumptions
|
||
in a decision record.
|
||
- [x] Implement stdio initialization/shutdown and incremental or full document
|
||
synchronization with URI, version, language ID, and current text tracked per
|
||
open document.
|
||
- [x] Build and thoroughly test byte-offset to LSP UTF-16 line/character mapping,
|
||
including CRLF, emoji, combining marks, astral characters, and EOF positions.
|
||
- [x] Publish diagnostics only for configured ASCII-replacement findings in the
|
||
initial release. Include stable codes and source `john-slop`.
|
||
- [x] Expose quick fixes/code actions using the exact edits from the shared
|
||
replacement engine; reject stale-version requests.
|
||
- [x] Recompute after open/change/configuration updates, clear on close, and avoid
|
||
publishing obsolete results when analysis overlaps newer edits.
|
||
- [x] Keep all protocol bytes on stdout and diagnostics/logging on stderr. Add
|
||
opt-in structured logging without document contents.
|
||
- [x] Add in-process protocol tests for lifecycle, versions, Unicode positions,
|
||
config reload, close cleanup, invalid URI, malformed messages, concurrent
|
||
documents, and formatter/LSP output equivalence.
|
||
|
||
Acceptance: a standards-compliant client can see replacement diagnostics and
|
||
apply a quick fix; the resulting text exactly matches `john-slop-fmt` under the
|
||
same config.
|
||
|
||
## Milestone 7 — Hardening, documentation, and release
|
||
|
||
- [x] Replace the placeholder README with installation, threat model, examples,
|
||
configuration precedence/schema, rule authoring, confidence interpretation,
|
||
CLI/formatter/LSP setup, exit codes, privacy, and limitations.
|
||
- [x] Publish commented sample base and named configs. Document how to disable a
|
||
rule/replacement and how merge-by-ID works.
|
||
- [x] Add CI on the supported OS/MSRV matrix for formatting, Clippy, tests, docs,
|
||
audit/deny checks, and fixture/golden consistency.
|
||
- [x] Add fuzz targets for TOML/config merging, regex/rule loading, UTF-8 span
|
||
mapping, replacement planning, and malicious ZIP metadata. Seed them with
|
||
regression fixtures.
|
||
- [x] Benchmark large files, many small files, rule scaling, and archive limits.
|
||
Add parallelism only if measurement shows a benefit and output remains ordered.
|
||
- [x] Review dependency features, licenses, advisories, unsafe code, and release
|
||
binary size. Generate an SBOM/checksums if releases distribute binaries.
|
||
- [x] Add changelog, contribution guide, code of conduct, security reporting
|
||
instructions, release profile, and reproducible release procedure.
|
||
- [x] Run a false-positive review against the evaluation corpus and require an
|
||
explicit sign-off before changing default rule weights or thresholds.
|
||
|
||
Acceptance: a new user can install and operate all three binaries from the README;
|
||
CI enforces the documented contracts; release artifacts are reproducible and the
|
||
detector's limitations are prominent.
|
||
|
||
Accepted on 2026-09-29 after the full all-feature/minimal-feature quality gate,
|
||
dependency policy checks, fuzz-seed runs, reproducible packaging, and an 80.51%
|
||
line-coverage run.
|
||
|
||
## Milestone 8 — Meaningful coverage expansion
|
||
|
||
- [x] Cover configuration merging, validation failures, domain formatting, and
|
||
diagnostic accessors with behavior-focused unit tests.
|
||
- [x] Cover archive and input resource-policy branches with generated fixtures,
|
||
including unsafe, binary, malformed, and oversized members.
|
||
- [x] Cover remaining practical CLI, formatter, and LSP adapter branches without
|
||
coupling tests to unreachable platform failures or coverage-tool artifacts.
|
||
- [x] Raise measured Tarpaulin line coverage toward 90%, refresh badges and
|
||
development documentation, and record any intentionally uncovered remainder.
|
||
|
||
Acceptance: the suite exercises at least 90% of measurable source lines when
|
||
Tarpaulin can attribute them reliably; remaining misses are documented and do
|
||
not justify brittle tests or production changes made solely for instrumentation.
|
||
|
||
Accepted on 2026-09-29 at 92.16% line coverage (1,329/1,442), with the CI floor
|
||
raised to 90%. Remaining misses are primarily hard-to-inject OS/I/O failures,
|
||
malformed dependency internals, overflow guards, and asynchronous LSP error
|
||
plumbing; pursuing them would add test-only seams or backend-sensitive checks.
|
||
|
||
## Milestone 9 — Terminal-first diagnostics
|
||
|
||
- [x] Make colored graphical Miette diagnostics the detector default while
|
||
retaining explicit plain-human, concise, JSON, and no-feature fallback modes.
|
||
|
||
Acceptance: a default detector invocation renders graphical Unicode diagnostics
|
||
with ANSI severity colors; explicit machine modes remain stable and uncolored,
|
||
and builds without `fancy-diagnostics` retain the plain human fallback.
|
||
|
||
## Milestone 10 — Prompt-leakage scoring priority
|
||
|
||
- [x] Give distinct prompt-leakage rules lighter correlation damping (`0.90ⁿ`)
|
||
while preserving `0.75ⁿ` damping for style, typography, and source-code
|
||
families and preserving per-rule occurrence damping/caps.
|
||
- [x] Re-run the reviewed corpus, renew the existing default-policy sign-off,
|
||
and document the changed formula and showcase score without changing the
|
||
sign-off mechanism itself.
|
||
|
||
Acceptance: multiple independent prompt-leakage artifacts contribute more than
|
||
equally weighted correlated style signals; the diagnostic showcase score is
|
||
pinned, the evaluation corpus remains free of observed regressions, and all
|
||
quality/coverage gates pass.
|
||
|
||
Accepted on 2026-09-29: the showcase score is 71 (medium); reviewed positive
|
||
corpus scores are pinned at 60/33/19/24/17, all five negative documents remain
|
||
at zero, and matching metrics remain 11 TP, 0 FP, and 0 FN.
|
||
|
||
Optional follow-up, intentionally not required for this milestone:
|
||
|
||
- [ ] After observing real usage, choose one direction for the default-policy
|
||
sign-off feature: harden its change boundary beyond the catalog hash, or
|
||
remove it if its maintenance cost exceeds its demonstrated value.
|
||
|
||
## Milestone 11 — Evidence-score terminology
|
||
|
||
- [x] Rename the public domain model from ambiguous confidence terminology to
|
||
`EvidenceScore`, `EvidenceBand`, and `EvidenceThresholds`; retain deprecated
|
||
Rust type aliases and the legacy `--fail-confidence` CLI alias for migration.
|
||
- [x] Publish the low-care evidence-score wording across terminal output,
|
||
examples, and documentation; rename `--fail-confidence` to `--fail-score` and
|
||
version the JSON field rename as schema version 2.
|
||
- [x] Run the complete quality and coverage gates, refresh generated badges, and
|
||
record the compatibility boundary and unchanged scoring formula.
|
||
|
||
Acceptance: every active user-facing contract describes the metric as a bounded
|
||
low-care evidence score rather than a probability or calibrated confidence;
|
||
legacy type and CLI names remain usable with clear migration guidance; JSON
|
||
consumers can identify the field rename by schema version; scoring results are
|
||
unchanged; and all quality/coverage gates pass.
|
||
|
||
Accepted on 2026-09-29: all-feature and minimal-feature suites, formatting,
|
||
Clippy, and documentation pass; deprecated Rust aliases and the legacy CLI alias
|
||
are covered; Tarpaulin remains at 92.19% (1,334/1,447 lines) against the 90%
|
||
floor; and the generated badges are current.
|
||
|
||
## Milestone 12 — Experimental crate boundary
|
||
|
||
- [x] Capture reproducible baseline cold and incremental check times from
|
||
`v0.1.11`, identify dependency ownership, and choose the smallest crate
|
||
boundary capable of excluding async/LSP dependencies from core builds.
|
||
- [x] Extract the reusable detector/configuration/input implementation into a
|
||
`john-slop-core` workspace crate and the protocol engine into a
|
||
`john-slop-lsp` workspace crate. Keep the root package as the compatibility
|
||
facade and binary distribution, preserving public behavior and all three
|
||
executable names.
|
||
- [x] Run identical clean and incremental measurements on the experiment,
|
||
compare compile time, artifact size, dependency isolation, API/release cost,
|
||
tests, coverage, and packaging against `v0.1.11`, then record an explicit
|
||
merge-or-abandon recommendation.
|
||
|
||
Acceptance: the branch contains a reviewable implementation commit and a
|
||
reproducible before/after report; all existing behavior and quality gates pass;
|
||
any claimed build improvement is measured rather than inferred. Do not merge
|
||
the experiment when its gains do not justify the added workspace and release
|
||
complexity.
|
||
|
||
Completed on 2026-09-29 with a **do-not-merge-yet** recommendation that was
|
||
subsequently superseded by the explicit Milestone 13 adoption decision. Core
|
||
and no-LSP checks use roughly 35–55% less CPU/disk, but the full workspace
|
||
remains effectively unchanged while package publishing, version
|
||
synchronization, asset ownership, feature contracts, CI, and coverage all
|
||
become more complex. The complete measurements and adoption rationale are in
|
||
`docs/experiments/crate-boundary.md` and ADR 0004.
|
||
|
||
## Milestone 13 — Supported Cargo workspace
|
||
|
||
- [x] Promote the experimental three-package layout to the supported project
|
||
architecture. Keep the root `john_slop` compatibility facade and binary
|
||
package as the workspace root; centralize shared package metadata, dependency
|
||
versions, and lint policy with Cargo workspace inheritance while retaining
|
||
per-package feature activation and optionality.
|
||
- [x] Add automated workspace-contract checks for membership, synchronized
|
||
versions, inherited dependency sources, and core/LSP feature isolation. Make
|
||
CI, packaging, and publication guidance explicitly workspace-aware and verify
|
||
every publishable package in dependency order.
|
||
- [x] Replace the experimental recommendation with an adoption decision, update
|
||
architecture/dependency/development/release documentation, run the complete
|
||
feature, lint, documentation, coverage, and package gates, and refresh the
|
||
generated badges and recap.
|
||
|
||
Acceptance: all three publishable packages inherit one version and one source of
|
||
truth for shared dependency requirements; `john-slop-core` remains free of
|
||
CLI/async/LSP dependencies; unqualified root commands cover every default
|
||
workspace member; each package can be assembled for publication in dependency
|
||
order; coverage remains at or above 90%; and the measured costs of supporting
|
||
the workspace are recorded rather than hidden.
|
||
|
||
Accepted on 2026-09-29 after both feature matrices, formatting, Clippy,
|
||
documentation, RustSec, cargo-deny, workspace package assembly, and the 90%
|
||
coverage floor passed. Tarpaulin remains at 92.19% (1,334/1,447 lines); the
|
||
workspace boundary changes dependency ownership, not executable behavior or
|
||
measurable source coverage.
|
||
|
||
## Deferred by design
|
||
|
||
- Statistical or model-based authorship classification.
|
||
- Automatic prose or source-code rewriting beyond configured ASCII replacements.
|
||
- Locale-aware confusable, invisible, and bidirectional Unicode detection. The
|
||
researched catalogs and adoption constraints are recorded in
|
||
`docs/unicode-confusables.md`; detection must remain separate from automatic
|
||
replacement.
|
||
- In-place modification of archive members or office documents.
|
||
- Network-backed analysis or telemetry.
|
||
- Editor-specific extensions; the LSP is the integration boundary.
|
||
|
||
These are not implied follow-ups. Each requires a separate privacy, safety,
|
||
performance, and product-design review before entering the active milestones.
|