Compare commits

..

159 Commits

Author SHA1 Message Date
Dandaedre 3d78c0ca2a
refactor(ci): centralize badge rendering
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-10-04 01:33:47 +02:00
Dandaedre 002b945e9b
fix(ci): escape coverage regex for Groovy
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-10-03 14:43:18 +02:00
Dandaedre d2f6f01fcd
ci: publish current coverage badges
Gitea/john-slop/pipeline/head There was a failure building this commit Details
2026-10-03 03:07:43 +02:00
Dandaedre d672cfdf71
chore: prepare 0.1.27-rc.8
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 22:41:09 +02:00
Dandaedre cba3855ce3
chore: prepare 0.1.27-rc.7
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 22:05:11 +02:00
Dandaedre 7dd69d6973
fix: use repository fallback for releases
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 21:56:12 +02:00
Dandaedre e049842194
docs: add cargo-binstall installation
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 21:27:45 +02:00
Dandaedre fcc29aaefc
fix: publish cargo-binstall archives
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 21:20:38 +02:00
Dandaedre 9c1e5bc083
chore: prepare 0.1.27-rc.5
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 21:11:52 +02:00
Dandaedre ee7e9d00d1
fix: restore ssh origin after release
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 20:39:08 +02:00
Dandaedre 42d49b3e7a
chore: prepare 0.1.27-rc.3
Rust CI JohnSlop passed all CI checks.
Gitea/john-slop/pipeline/head This commit looks good Details
2026-09-30 20:27:41 +02:00
Dandaedre 57781939a3
chore: prepare 0.1.27-rc.2 2026-09-30 20:16:16 +02:00
Dandaedre f9c9a79797
chore: prepare 0.1.27-rc.1
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-30 19:00:23 +02:00
Dandaedre 5c2ff0dca9
build: add Gitea release wrapper
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-30 18:44:23 +02:00
Dandaedre dc21a5bf34
ci: remove obsolete coverage remnants
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-30 16:37:41 +02:00
Dandaedre f52f61f4c4
ci: disable unpublished coverage check 2026-09-30 16:33:25 +02:00
Dandaedre 6e4e27f774
ci: ignore legacy SBOM artifacts 2026-09-30 16:29:39 +02:00
Dandaedre f418f01216
build: replace yanked yoke derive
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-30 16:12:23 +02:00
Dandaedre d59a9dc471
build: remove SBOM generation
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-30 14:57:45 +02:00
Dandaedre 01d72c4600
ci: clean generated release artifacts 2026-09-30 14:51:56 +02:00
Dandaedre 4a0c7c9920
chore: release milestone 26
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-30 02:34:23 +02:00
Dandaedre 590202da05
feat: make LSP diagnostics Unicode-first 2026-09-30 02:26:52 +02:00
Dandaedre aedc130082
feat: add research-backed review signals 2026-09-30 02:23:59 +02:00
Dandaedre 22308219c8
chore: release milestone 25 2026-09-30 02:00:13 +02:00
Dandaedre 861fba5a81
fix: width-fold fullwidth ASCII deterministically 2026-09-30 01:50:49 +02:00
Dandaedre 215a164a04
chore: release milestone 24
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-30 01:13:26 +02:00
Dandaedre 501328d05f
feat: add runtime diagnostic catalog workflow 2026-09-30 01:06:42 +02:00
Dandaedre 70ad6e5cf8
feat: load runtime diagnostic catalogs 2026-09-30 00:59:26 +02:00
Dandaedre 62a64ad816
docs: plan runtime diagnostic catalogs 2026-09-30 00:57:48 +02:00
Dandaedre 964347d041
chore: release milestone 23 2026-09-30 00:56:01 +02:00
Dandaedre ae448ad60c
feat: generate a local default configuration 2026-09-30 00:48:49 +02:00
Dandaedre cd15bb8503
feat: configure diagnostics by category
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-30 00:46:45 +02:00
Dandaedre cdc05af416
docs: plan diagnostic configuration milestone 2026-09-30 00:43:53 +02:00
Dandaedre 62a2b6a608
chore: release milestone 22 2026-09-30 00:42:04 +02:00
Dandaedre f229c1445c
feat: replace printable ascii confusables consistently 2026-09-30 00:27:36 +02:00
Dandaedre 82afdc7a3f
feat: generate maintainable unicode replacement catalog 2026-09-30 00:24:24 +02:00
Dandaedre 893fbfdd67
docs: plan ambiguous unicode replacement catalog 2026-09-30 00:20:44 +02:00
Dandaedre 73e3ecc761
chore: release milestone 21 2026-09-30 00:15:34 +02:00
Dandaedre 98a74e4c61
feat: detect ambiguous unicode for review 2026-09-30 00:10:00 +02:00
Dandaedre b42ac6f221
fix: make lsp formatting match formatter 2026-09-30 00:01:38 +02:00
Dandaedre 62d99e3015
docs: plan lsp formatting and unicode diagnostics 2026-09-29 23:58:26 +02:00
Dandaedre d43850b5a3
chore: synchronize milestone package version 2026-09-29 23:52:36 +02:00
Dandaedre 14e589d52c
docs: plan milestone version synchronization 2026-09-29 23:47:14 +02:00
Dandaedre a3c3946776
feat: publish heuristic diagnostics over LSP 2026-09-29 23:45:07 +02:00
Dandaedre 344889c78f
docs: plan full heuristic LSP diagnostics
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-29 23:36:21 +02:00
Dandaedre 3281bd5a44
example: showcase all printable ASCII replacements 2026-09-29 23:33:59 +02:00
Dandaedre 8d927f1699
docs: plan printable ASCII replacement showcase 2026-09-29 23:30:26 +02:00
Dandaedre bff852ab4b
docs: introduce progressive documentation path 2026-09-29 23:28:37 +02:00
Dandaedre 6aad159fe8
docs: plan progressive documentation entry points 2026-09-29 23:26:03 +02:00
Dandaedre 8efa35c45a
docs: fix local cargo install command
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-29 23:25:14 +02:00
Dandaedre 73ed837812
docs: plan source checkout installation 2026-09-29 23:23:35 +02:00
Dandaedre 8cf91c0507
test: expand diagnostic showcase 2026-09-29 23:22:44 +02:00
Dandaedre dbf4049384
docs: plan comprehensive diagnostic showcase 2026-09-29 23:19:21 +02:00
Dandaedre cf21ff1b9b
feat: improve graphical diagnostic spacing 2026-09-29 23:17:48 +02:00
Dandaedre d49363e80e
docs: plan graphical diagnostic layout 2026-09-29 23:14:45 +02:00
Dandaedre ca8a352a40
docs: adopt supported cargo workspace
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-29 21:45:32 +02:00
Dandaedre e9667a6842
ci: enforce workspace package contracts 2026-09-29 21:41:10 +02:00
Dandaedre f57758d355
build: centralize workspace metadata and dependencies 2026-09-29 21:39:27 +02:00
Dandaedre 184c66325a
docs: plan supported workspace migration 2026-09-29 21:38:30 +02:00
Dandaedre a90e27a27e
docs: adopt v0.1 milestone tags 2026-09-29 21:36:09 +02:00
Dandaedre 933066700f
docs: compare crate boundary tradeoffs 2026-09-29 21:32:16 +02:00
Dandaedre bee9367360
build: make lsp dependency graph optional 2026-09-29 21:24:11 +02:00
Dandaedre 5c133e24c3
refactor: isolate core and lsp crates 2026-09-29 21:23:14 +02:00
Dandaedre 1fbc44e9db
docs: plan crate boundary experiment 2026-09-29 21:20:25 +02:00
Dandaedre c12b6b8551
test: verify evidence score migration 2026-09-29 21:11:43 +02:00
Dandaedre 95d4067dc8
docs: publish evidence score terminology 2026-09-29 21:08:03 +02:00
Dandaedre a4dd4e2c5f
refactor: rename confidence to evidence score 2026-09-29 21:06:05 +02:00
Dandaedre 7d365af5df
test: renew prompt scoring sign-off 2026-09-29 21:00:29 +02:00
Dandaedre d6163076d9
feat: prioritize prompt leakage evidence 2026-09-29 20:57:15 +02:00
Dandaedre 30445d7983
feat: default to colored miette diagnostics
Rust CI JohnSlop failed CI.
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-29 20:02:14 +02:00
Dandaedre 6743a2f373
ci: fix release stage Groovy parsing
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 19:06:06 +02:00
Dandaedre 2001b7070f
test: enforce ninety percent coverage floor 2026-09-29 18:18:43 +02:00
Dandaedre e33f47e285
test: cover practical adapter branches 2026-09-29 18:15:37 +02:00
Dandaedre 2c02cc201b
test: cover archive and input safety limits 2026-09-29 18:12:49 +02:00
Dandaedre 8d1965b4c5
test: cover configuration and diagnostic branches 2026-09-29 18:09:47 +02:00
Dandaedre 5442f3ccec
test: complete hardening and release milestone
Gitea/john_slop/pipeline/head There was a failure building this commit Details
2026-09-29 17:50:43 +02:00
Dandaedre 81a09af61e
test: enforce heuristic false-positive sign-off 2026-09-29 17:47:13 +02:00
Dandaedre c46563845c
build: add reproducible release packaging 2026-09-29 17:44:06 +02:00
Dandaedre 55e438d6ef
build: reduce archive features and document dependency audit 2026-09-29 17:39:17 +02:00
Dandaedre 73a344c0ac
perf: add scaling benchmark suite 2026-09-29 17:36:49 +02:00
Dandaedre d4a2967ac8
test: add fuzz harnesses and regression seeds 2026-09-29 17:33:22 +02:00
Dandaedre ab9f54b22d
ci: enforce release quality and supply-chain gates 2026-09-29 17:30:29 +02:00
Dandaedre a67ee320e7
docs: publish release-ready usage and config samples 2026-09-29 17:26:47 +02:00
Dandaedre e42169da3d
test: refresh diagnostic coverage 2026-09-29 17:24:13 +02:00
Dandaedre 0e4f018a61
docs: record Unicode confusable catalogs
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 17:22:39 +02:00
Dandaedre eeabed9010
test: add expected-failure diagnostic showcase 2026-09-29 17:21:20 +02:00
Dandaedre b40112a724
feat: add graphical Miette diagnostics 2026-09-29 17:20:24 +02:00
Dandaedre 0cb7ec3011
test: complete LSP milestone 2026-09-29 17:16:08 +02:00
Dandaedre 2f12c4e186
docs: publish LSP usage guides 2026-09-29 17:14:04 +02:00
Dandaedre dcae1b1b06
test: cover LSP protocol workflows 2026-09-29 17:11:22 +02:00
Dandaedre 883fb13c12
feat: add opt-in LSP lifecycle logs 2026-09-29 17:07:05 +02:00
Dandaedre 5e99a59f8d
feat: reload LSP configuration 2026-09-29 17:05:29 +02:00
Dandaedre 90c2848d63
feat: add versioned LSP quick fixes 2026-09-29 17:04:44 +02:00
Dandaedre a112b4bcb0
feat: publish LSP replacement diagnostics
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 17:03:04 +02:00
Dandaedre 01be964765
feat: map LSP UTF-16 positions 2026-09-29 17:01:03 +02:00
Dandaedre 453f99f70d
feat: implement LSP document lifecycle 2026-09-29 16:59:34 +02:00
Dandaedre 837b1fce55
docs: select LSP framework 2026-09-29 16:57:21 +02:00
Dandaedre 033be96d79
test: complete formatter milestone
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 16:51:44 +02:00
Dandaedre 709ed232f4
docs: publish formatter contract 2026-09-29 16:49:38 +02:00
Dandaedre 39c6eacdf3
test: exercise replacement properties
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 16:48:07 +02:00
Dandaedre d98db5252d
test: cover formatter workflows 2026-09-29 16:47:13 +02:00
Dandaedre 2ceb6a6c5a
feat: implement safe ASCII formatter 2026-09-29 16:45:01 +02:00
Dandaedre f1daf590f4
feat: harden replacement engine
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 16:41:22 +02:00
Dandaedre 1bacd8a75d
docs: add Helix and Vim adapter setup 2026-09-29 16:36:12 +02:00
Dandaedre 65592760f3
test: add shared adapter fixture corpus 2026-09-29 16:32:38 +02:00
Dandaedre 03d5138475
docs: add runnable library examples 2026-09-29 16:30:56 +02:00
Dandaedre f2360c3d60
test: complete detector CLI milestone
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 16:06:56 +02:00
Dandaedre bc8881132d
docs: publish detector output contracts 2026-09-29 16:05:21 +02:00
Dandaedre 27b4d70962
fix: stabilize directory output order 2026-09-29 16:04:20 +02:00
Dandaedre c763b3e664
fix: identify failed input documents 2026-09-29 16:03:11 +02:00
Dandaedre f3e53b1913
test: complete detector CLI matrix 2026-09-29 16:02:02 +02:00
Dandaedre 593e91a4bc
fix: preserve results across input failures 2026-09-29 16:00:27 +02:00
Dandaedre 7d8a45433a
feat: add configured CLI failure levels 2026-09-29 15:58:44 +02:00
Dandaedre cf4f0be12d
feat: add structured detector output modes 2026-09-29 15:57:40 +02:00
Dandaedre c9b280e9fe
feat: expand detector input controls 2026-09-29 15:52:28 +02:00
Dandaedre 04a3a0b0d7
feat: summarize confidence and severities 2026-09-29 15:49:54 +02:00
Dandaedre 0428cd8a9a
feat: expose typed finding diagnostics 2026-09-29 15:48:58 +02:00
Dandaedre d9e4e378ea
test: complete heuristic detection milestone
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 15:34:30 +02:00
Dandaedre 283a0542b9
docs: define detector limitations 2026-09-29 15:32:39 +02:00
Dandaedre f1fe9f5d8d
test: add reviewed heuristic corpus 2026-09-29 15:31:46 +02:00
Dandaedre f8e2c9ad8e
test: pin shipped heuristic behavior 2026-09-29 15:30:18 +02:00
Dandaedre 0c9c117d5e
fix: normalize detector findings 2026-09-29 15:29:06 +02:00
Dandaedre c62318304a
test: distinguish clean and unsupported inputs 2026-09-29 15:28:32 +02:00
Dandaedre 6fe5f90c17
feat: score bounded heuristic evidence 2026-09-29 15:27:56 +02:00
Dandaedre 536009d32d
feat: scope heuristics by content and path 2026-09-29 15:26:29 +02:00
Dandaedre aad482e8e4
feat: expand conservative heuristic catalog 2026-09-29 15:19:55 +02:00
Dandaedre 90f64fd8ac
feat: execute configured text matchers 2026-09-29 15:17:41 +02:00
Dandaedre 10d607d423
refactor: compile reusable detector registry 2026-09-29 15:15:51 +02:00
Dandaedre 43ede9c717
test: complete configuration layer matrix 2026-09-29 13:33:20 +02:00
Dandaedre 7bb9781064
feat: inspect resolved configuration 2026-09-29 13:30:00 +02:00
Dandaedre 72fe28a94c
feat: diagnose configuration sources 2026-09-29 13:27:13 +02:00
Dandaedre 8226989ce4
feat: validate resolved configuration 2026-09-29 13:24:50 +02:00
Dandaedre 18ec49ceb4
feat: merge configuration records by id 2026-09-29 13:23:02 +02:00
Dandaedre 8b2a02309a
feat: load ordered configuration layers 2026-09-29 13:21:12 +02:00
Dandaedre dcdf958e9c
feat: validate named configuration paths 2026-09-29 13:19:02 +02:00
Dandaedre 177337c0ae
feat: embed typed heuristic catalog 2026-09-29 13:17:48 +02:00
Dandaedre 845fb891f2
feat: embed typed replacement configuration 2026-09-29 13:16:30 +02:00
Dandaedre 4a065d4fc9
build: add layered configuration dependencies 2026-09-29 13:15:17 +02:00
Dandaedre 25560fe5d2
docs: split detailed guides from readme 2026-09-29 12:55:53 +02:00
Dandaedre 239b5198e4
test: enforce coverage and publish badges 2026-09-29 12:53:29 +02:00
Dandaedre 84fefd4b01
test: expand snapshots and feature coverage 2026-09-29 12:49:50 +02:00
Dandaedre 8f3bb6de71
build: update Rust and dependencies 2026-09-29 12:47:15 +02:00
Dandaedre eb1619787d
feat: add detector command-line interface
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-29 12:39:21 +02:00
Dandaedre f1ef56b798
build: define detector formatter and LSP targets 2026-09-29 12:37:13 +02:00
Dandaedre 95e6930a42
refactor: return typed input errors 2026-09-29 12:36:27 +02:00
Dandaedre a9bfca053b
feat: enforce input safety limits 2026-09-29 12:33:26 +02:00
Dandaedre 648a4d50cb
refactor: stream input documents 2026-09-29 12:31:06 +02:00
Dandaedre b54c03744b
refactor: add shared library facade 2026-09-29 12:29:09 +02:00
Dandaedre 4c29830891
refactor: introduce typed analysis model 2026-09-29 12:26:40 +02:00
Dandaedre 1904979405
build: trim prototype dependencies 2026-09-29 12:13:52 +02:00
Dandaedre 5bffeea1fb
build: establish Rust 1.88 MSRV 2026-09-29 12:13:06 +02:00
Dandaedre c88c5b8de2
docs: define input safety limits 2026-09-29 12:11:33 +02:00
Dandaedre 7b5e6b330f
docs: record core analysis contracts 2026-09-29 12:10:54 +02:00
Dandaedre 4d142d60ed
test: correct Unicode offset expectations 2026-09-29 12:10:17 +02:00
Dandaedre acfb775877
test: characterize prototype behavior 2026-09-29 12:09:42 +02:00
Dandaedre 7651667025
chore: ignore local work recap 2026-09-29 12:07:28 +02:00
Dandaedre a41d13444a
docs: define architecture and implementation plan 2026-09-29 12:07:08 +02:00
Dandaedre a144f47452 Add Jenkinsfile
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details
2026-09-13 13:13:14 +02:00
Dandaedre 27a8494dc7 Shitcoded some base functions, too tired to do any useful shit 2025-12-04 17:27:50 +01:00
142 changed files with 32918 additions and 226 deletions

21
.gitignore vendored
View File

@ -4,9 +4,7 @@
debug/
target/
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
Cargo.lock
# Cargo.lock is tracked because this package ships executable binaries.
# These are backup files generated by rustfmt
**/*.rs.bk
@ -19,4 +17,19 @@ Cargo.lock
# Added by cargo
/target
bob/
/dist/
# Local implementation journal (intentionally not committed)
/recap.md
# Generated locally or in CI by Cargo Tarpaulin and the badge scripts
/tarpaulin-report.html
/coverage/
# Legacy cargo-cyclonedx intermediates may remain in reused CI workspaces.
**/sbom.cdx.json
# cargo-fuzz runtime output
/fuzz/artifacts/
/fuzz/coverage/
/fuzz/target/

277
AGENTS.md Normal file
View File

@ -0,0 +1,277 @@
# John Slop contributor guide
## Mission and product boundaries
John Slop is a transparent, local-first heuristic linter for low-effort or
insufficiently reviewed text and source code. It reports concrete signals that
are often associated with careless AI-assisted work. It must not claim to prove
that a person used AI, identify a model, or infer authorship from writing style.
Treat the **low-care evidence score** as the bounded strength of configured
signals, not as the probability that content is AI-generated. User-facing language must say
"suspicious", "low-care signal", or "needs review" rather than making an
accusation. A result must always be explainable from the findings that produced
it.
The formatter scope is deliberately narrow: configured Unicode characters with
ASCII replacements. The generated UTS #39 profile is broad and therefore must
retain `replace`, non-mutating `review`, and `off` policies. The LSP publishes
every applicable detector finding but exposes edits only under replacement
policy, reusing the same library
implementations as the CLI and formatter. Do not add prose rewriting, generative
inference, or destructive archive editing without a separate design decision.
## Required architecture
Keep domain logic in a library and binaries as thin adapters. The intended
layout is:
```text
src/
lib.rs compatibility facade over workspace libraries
bin/
john-slop.rs command-line detector
john-slop-fmt.rs replacement formatter
john-slop-lsp.rs stdio language server
crates/
john-slop-core/
assets/ embedded replacement and heuristic catalogs
src/ config, input, detection, diagnostics, and edits
john-slop-lsp/src/ protocol state, UTF-16 mapping, and code actions
tests/
fixtures/ small, reviewed input/output fixtures
```
Names can evolve when Rust conventions demand it, but preserve these boundaries:
- `input` acquires text and never scores or changes it.
- `detection` is pure over text plus validated rules.
- `replacement` returns non-overlapping edits and does no I/O.
- `diagnostic` renders findings but does not discover them.
- the CLI, formatter, and LSP call the same config, detector, and replacement APIs.
- filesystem writes are confined to explicit adapter methods.
Do not retain the current all-files-in-memory iterator design as the core API.
Prefer a streaming iterator of per-file results so one unreadable or binary file
does not erase findings from unrelated files. Preserve the outer archive path and
inner member name as separate fields. Reject unsafe archive paths and apply size,
entry-count, and nesting limits before extraction or decompression.
## Core data contracts
Use typed, presentation-neutral domain values. The exact names may change, but
the model must express at least:
```rust
struct Finding {
rule_id: RuleId,
category: Category,
message: String,
help: Option<String>,
byte_span: Range<usize>,
matched_text: String,
weight: f64,
fix: Option<TextEdit>,
}
struct Analysis {
findings: Vec<Finding>,
evidence_score: EvidenceScore,
}
struct TextEdit {
byte_span: Range<usize>,
replacement: String,
}
```
All internal spans are zero-based UTF-8 byte ranges into the exact source
snapshot that was analyzed. Validate boundaries before slicing. Convert spans at
the edge: `miette::SourceSpan` for terminal output and UTF-16 line/column ranges
for LSP. Keep an indexed line map per document so LSP conversions are tested and
not repeatedly O(n).
Rule IDs are stable, unique, lowercase kebab-case identifiers. Findings are
sorted by source identity, span start, then rule ID so output is deterministic.
Never use a hash-map iteration order as output order.
## Heuristics and evidence scoring
Store data-driven heuristic definitions in
`crates/john-slop-core/assets/heuristics.toml`, separately
from replacement configuration. Compile and validate rules once at startup.
Invalid regular expressions, duplicate IDs, negative/non-finite weights, invalid
thresholds, and unsafe empty matches are configuration errors—not ignored rules.
Each rule needs an ID, category, rationale/message, help text, matcher type,
weight, and optional applicability filters such as prose/source-code extensions.
Start with high-precision, independently testable signals:
- copied assistant disclaimers or knowledge-cutoff phrases;
- prompt/response scaffolding accidentally left in a document;
- templated filler and repeated conclusion/transition phrases;
- suspicious Unicode/confusable punctuation when the configured corpus expects
ASCII;
- source-code placeholders, tutorial residue, fabricated references, or comments
that merely narrate obvious code, using conservative language-specific rules.
Avoid weak demographic, dialect, spelling, grammar, vocabulary, or
English-fluency proxies. Repetition alone must not trigger a high evidence-band
verdict. Give users category and per-rule enablement controls, per-rule weight
overrides, and path/language scopes. A rule is active only when both its
category gate and individual `enabled` field are active.
Scoring must be documented and deterministic. Use bounded weighted evidence,
apply a per-rule occurrence cap, and damp correlated findings from the same rule
or family. Map the aggregate onto `0..=100` and named bands such as `low`,
`medium`, and `high`; keep thresholds configurable. Tests must pin the formula,
caps, threshold edges, and empty-input result. Never label the numeric result an
accuracy or probability unless it has been independently calibrated against a
published, representative corpus.
Every terminal finding must become a `miette::Diagnostic` with named source,
precise label, stable diagnostic code (the rule ID), severity, message, and help.
A summary score supplements findings; it never replaces their evidence. Machine
output must offer a versioned JSON schema and must not include ANSI escapes.
## Configuration contract
Use `figment` with its TOML support and Serde for typed layered configuration.
Its providers allow embedded serialized defaults and later TOML providers to
override individual values while retaining source metadata for useful errors.
Use `directories` (or an equivalent platform-aware crate) to resolve the user
configuration directory; do not hand-build it from `HOME`.
The precedence from lowest to highest is:
1. embedded `crates/john-slop-core/assets/config.toml` and
`crates/john-slop-core/assets/heuristics.toml`;
2. runtime diagnostic catalogs: the optional user-wide `diagnostics.toml`, then
explicitly selected catalogs in command-line order;
3. user base config at `$XDG_CONFIG_HOME/john-slop/config.toml`, conventionally
`~/.config/john-slop/config.toml` on Linux;
4. named user config at `$XDG_CONFIG_HOME/john-slop/<config_name>.toml` when a
validated config name is selected;
5. an explicit `--config <path>` file;
6. explicit CLI flags for the current invocation.
Diagnostic catalog files may contain only `schema_version` and `[[rules]]`.
They can add or patch data-driven rules but cannot define executable matcher
algorithms, categories, replacements, thresholds, or input policy. Normal
configuration layers intentionally load afterward so they can disable or tune
catalog rules by stable ID.
Missing optional files are allowed; malformed files are fatal and identify their
origin. A config name accepts only ASCII letters, digits, `_`, and `-`; it cannot
contain separators, `.` segments, or an extension. Decide array behavior per
field: replacement/rule tables merge by stable ID, while ignore globs and file
extensions replace as complete arrays unless the schema explicitly says
otherwise. Do not rely on a library's default array-merge semantics implicitly.
`crates/john-slop-core/assets/unicode.toml` is the canonical generated Unicode
replacement and diagnostic catalog; `config.toml` owns other defaults. Each
replacement entry has an ID, Unicode source string/code point, ASCII target,
rationale, and enabled flag. Reject duplicate sources, empty sources, non-ASCII
targets, and conflicting edits. Longer sources win matching precedence. Provide a command
that prints the resolved configuration with secrets/redacted fields if secrets
are ever introduced.
Configuration schema changes require a `schema_version`, migration notes, and
tests for precedence. Embed assets with `include_str!` so installed binaries do
not depend on the source tree or current working directory.
## CLI, formatter, and LSP behavior
Use explicit Cargo binaries named `john-slop`, `john-slop-fmt`, and
`john-slop-lsp`.
- The detector reads files/directories/stdin, reports findings, and exits `0`
below the selected threshold, `1` when findings cross it, and `2` for usage or
operational/configuration errors.
- The formatter supports stdin/stdout, `--check`, and an explicit `--write`.
Default behavior must not rewrite a file. Writes must be atomic where the
platform permits, preserve original bytes when no edit applies, preserve line
endings, and be idempotent.
- The LSP communicates only over stdin/stdout; logs go to stderr. It publishes
applicable configured heuristic diagnostics and exposes quick-fix/code-action
edits only for safe replacements. It must honor document versions, clear
diagnostics on close, handle invalid URIs without panicking, and never write
editor-owned files.
Replacement edits are generated from the original snapshot, sorted, checked for
overlap, then applied from the end of the buffer toward the beginning. The
formatter and LSP must produce byte-for-byte equivalent results for the same
source and resolved config.
## Errors, safety, and observability
- Libraries return typed errors with `thiserror`; application boundaries attach
path/rule context and render with `miette`.
- Do not use `unwrap`, `expect`, `panic!`, `todo!`, or `unreachable!` on
user-controlled input. An `expect` is acceptable only for an invariant proved
during construction and explained in a comment.
- Treat invalid UTF-8 and binary content as explicit skipped-input diagnostics
according to policy; do not silently truncate or lossily rewrite it.
- Do not follow symlinks by default. Respect ignore configuration and establish
file-size, archive-size, archive-entry, and recursion limits.
- Do not make network calls during analysis. Results should be reproducible and
private by default.
- Structured logs go to stderr and are disabled or quiet by default. Never mix
logs with JSON, formatter stdout, or the LSP protocol stream.
## Testing and quality gates
Each behavior change requires tests at the lowest useful layer plus an
integration test when it crosses a binary boundary. Include positive,
near-miss/negative, Unicode, empty, large-input, malformed-config, and ordering
cases. Regression fixtures must state why they should or should not match.
Before considering a task complete, run:
```text
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features
cargo test --all-targets --no-default-features
cargo doc --no-deps
```
Run `cargo tarpaulin --config tarpaulin.toml` when tests or executable Rust code
change. It enforces the 90% line-coverage floor. Jenkins renders the version,
test, and coverage badges from package metadata and CI results; generated badge
files and status snapshots do not belong in the repository.
Also run focused CLI golden tests and LSP protocol tests once those binaries
exist. Favor property tests for replacement idempotence/non-overlap and fuzz
targets for config parsing, span conversion, archives, and replacement planning.
Benchmarks should cover a representative large document and directory; optimize
only from measured results.
## Change discipline
Keep commits and patches small enough to review. Do not combine a structural
move with new detection behavior unless the new behavior requires it. Preserve
public behavior during refactors or document the deliberate break. Update sample
configuration, schema/version notes, CLI help, and tests together.
When implementing `tasks.md`, work in dependency order, mark only verified items
complete, and add newly discovered work under the relevant milestone. Record a
short decision note in the task file when changing configuration precedence,
the score formula, a public data contract, or safety limits.
After completing and verifying each individual task, create a dedicated commit
before starting the next task. Stage only the files that belong to that task;
never include unrelated or pre-existing user changes. Use a concise conventional
commit message that describes the completed behavior. If a task cannot be
verified or committed, leave it unchecked and report the blocker instead of
combining it with later work.
After every milestone's acceptance criteria and quality gates pass, create an
annotated Git tag named `v0.1.N`, where `N` is the milestone number, on its
completion commit. Before that completion commit, set the workspace package
version and internal registry dependency requirements to `0.1.N`, refresh
`Cargo.lock` and cut the matching changelog entry. Jenkins derives the version
badge from that metadata. Verify the manifests, lockfile, changelog, and binary
`--version` output agree before tagging. Never move an existing milestone tag;
add a corrective commit and a documented successor tag if a published milestone
needs amendment.

230
CHANGELOG.md Normal file
View File

@ -0,0 +1,230 @@
# Changelog
All notable changes to John Slop are recorded here. The format follows
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and releases use
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.1.27-rc.8] - 2026-09-30
### Fixed
- Selected current Cross target images so all Linux and Windows release
artifacts compile with the declared Rust 1.98.1-or-newer toolchain.
## [0.1.27-rc.7] - 2026-09-30
### Fixed
- Used the publisher's repository fallback mapping so releases reach the HTTPS
proxy while the developer SSH origin remains unchanged.
## [0.1.27-rc.6] - 2026-09-30
### Fixed
- Published cargo-binstall `.tgz` archives alongside raw binaries so the
package metadata added in RC5 resolves to release assets.
## [0.1.27-rc.5] - 2026-09-30
### Added
- Added `cargo-binstall` package metadata for downloading versioned target
archives from Gitea releases.
## [0.1.27-rc.4] - 2026-09-30
### Fixed
- Made the release wrapper temporarily select the canonical HTTPS origin for
publication and restore the developer SSH origin whether publication succeeds
or fails.
## [0.1.27-rc.3] - 2026-09-30
### Changed
- Unified the root Cargo package, repository slug, release bundles, runtime
configuration directory, documentation, and Git remote around the canonical
`john-slop` spelling. Rust source continues to import the package as
`john_slop`, because Rust identifiers cannot contain hyphens.
## [0.1.27-rc.2] - 2026-09-30
### Fixed
- Switched canonical repository and release metadata to the TLS-valid
`proxy.dandaedre.com/gitea` endpoint so authenticated Gitea API publication
does not fail during connection setup.
## [0.1.27-rc.1] - 2026-09-30
### Added
- Added a token-file wrapper around `rust_realease` for publishing tagged raw
binary release candidates without placing Gitea credentials in arguments.
### Changed
- Removed unused SBOM generation and unpublished Tarpaulin execution from the
Jenkins pipeline while retaining local coverage validation.
- Made reused Jenkins workspaces tolerate legacy generated release artifacts.
### Fixed
- Replaced the yanked transitive `yoke-derive 0.8.3` lockfile entry with
`0.8.4`.
## [0.1.26] - 2026-09-30
### Added
- Added conservative response-residue, editorial-placeholder, source-residue,
stock-closing, and clustered excess-vocabulary rules with positive and
realistic near-miss characterization fixtures.
- Added a research-backed review-signal methodology that separates directly
reviewable residue, low-weight population clues, verification targets, and
deliberately excluded language-proficiency proxies.
- Added LSP `--all-diagnostics` and `--minimum-diagnostic-weight WEIGHT`
switches for explicit full-catalog and weighted editor review.
### Changed
- The LSP now publishes only configured Unicode diagnostics by default. The
visibility policy does not alter shared detector results, evidence scores,
replacement planning, or formatting.
## [0.1.25] - 2026-09-30
### Fixed
- Fullwidth forms U+FF01 through U+FF5E now width-fold one-to-one to ASCII
`!` through `~`, instead of using visually motivated UTS #39 skeletons or
remaining unchanged. Ideographic space now maps to ASCII space.
- LSP formatting, quick fixes, the standalone formatter, and detector fixes
share the corrected mappings while retaining previously shipped stable IDs.
## [0.1.24] - 2026-09-30
### Added
- Added restricted runtime `diagnostics.toml` catalogs for defining
exact-phrase, regex, and repeated-regex rules without recompilation.
- Added repeatable `--diagnostics <FILE>` support across the detector,
formatter, and LSP, including LSP configuration reloads.
- Added `john-slop config init-diagnostics`, a commented starter catalog, and a
complete project example.
### Changed
- User and explicit diagnostic catalogs load before normal configuration so
later base, named, and explicit layers can tune or disable their stable IDs.
- Runtime catalogs reject code-backed matcher engines and unrelated settings;
executable plugin behavior remains outside the declarative configuration
boundary.
## [0.1.23] - 2026-09-30
### Added
- Added typed category-level diagnostic switches for `typography`, `style`,
`prompt-leakage`, and `source-code`, composed with the existing individual
rule switches through every configuration layer.
- Added `john-slop config init` and its `config generate` alias to write the
commented embedded base configuration into the current directory.
### Changed
- Configuration generation protects an existing `config.toml` unless
`--force` is explicitly supplied.
## [0.1.22] - 2026-09-30
### Added
- Added a generated, versioned Unicode policy containing 1,829 mappings and
their shared diagnostic metadata, plus an exhaustive drift test and
deterministic regeneration example.
- Added `replace`, `review`, and `off` bulk policies for printable-ASCII UTS #39
mappings.
### Changed
- Printable-ASCII Unicode 16.0 UTS #39 lookalikes now receive consistent fixes
from CLI diagnostics, the standalone formatter, LSP quick fixes, and LSP
document formatting under the default `replace` policy.
- Invisible and bidirectional controls remain diagnostic-only.
## [0.1.21] - 2026-09-30
### Added
- Added real LSP document formatting backed by the same resolved replacement
plan as `john-slop-fmt`.
- Added non-mutating exact-span diagnostics for Unicode 16.0 UTS #39
ASCII-lookalikes and pinned Gitea invisible/bidirectional controls.
### Changed
- Expanded the shared fixtures and expected-failure diagnostic showcase with
deliberately non-fixable Unicode review cases.
## [0.1.20] - 2026-09-29
### Changed
- Synchronized workspace package versions, internal dependency requirements,
lockfile metadata, binary version output, milestone tags, and the generated
version badge, with a regression contract to prevent future drift.
- Made graphical Miette output easier to scan with marked rule-code titles,
attached diagnostic bodies, and spacing between complete findings.
- Expanded the expected-failure diagnostic showcase to include prose and source
rules plus the complete embedded replacement-character set.
- Documented Cargo's required `--path .` syntax when installing all binaries
from a local source checkout.
- Restructured the README and documentation index as a progressive path from
quickstart to usage guides, with engineering detail in a dedicated advanced
guide.
- Added an opt-in Rust example that maps fullwidth Unicode forms to all 95
printable ASCII targets while leaving production defaults unchanged.
- Extended the LSP from replacement-only warnings to every applicable heuristic
finding while retaining quick fixes only for safe ASCII replacements.
- Adopted the measured three-package Cargo workspace. Shared package metadata,
dependency requirements, and lint policy now use workspace inheritance while
the reusable core remains isolated from CLI, async, and LSP dependencies.
- Renamed the ambiguous confidence metric to low-care evidence score throughout
the Rust API, CLI, terminal output, and JSON schema version 2. Deprecated Rust
type aliases and `--fail-confidence` remain available for migration; the
scoring formula and values are unchanged.
## [0.1.0] - 2026-09-29
### Added
- Local, explainable heuristic analysis with a bounded low-care evidence score.
- Layered embedded, user, named, explicit, and CLI configuration.
- File, directory, ZIP, office-container, and stdin input adapters with limits.
- Human, graphical Miette, concise, and versioned JSON detector output.
- Conservative Unicode-to-ASCII formatter with check, diff, and atomic write.
- Stdio LSP replacement diagnostics and versioned quick fixes.
- Runnable examples, reviewed fixtures, fuzz targets, benchmarks, and CI gates.
[Unreleased]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.8...HEAD
[0.1.27-rc.8]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.7...v0.1.27-rc.8
[0.1.27-rc.7]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.6...v0.1.27-rc.7
[0.1.27-rc.6]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.5...v0.1.27-rc.6
[0.1.27-rc.5]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.4...v0.1.27-rc.5
[0.1.27-rc.4]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.3...v0.1.27-rc.4
[0.1.27-rc.3]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.2...v0.1.27-rc.3
[0.1.27-rc.2]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.1...v0.1.27-rc.2
[0.1.27-rc.1]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.26...v0.1.27-rc.1
[0.1.26]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.25...v0.1.26
[0.1.25]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.24...v0.1.25
[0.1.24]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.23...v0.1.24
[0.1.23]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.22...v0.1.23
[0.1.22]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.21...v0.1.22
[0.1.21]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.20...v0.1.21
[0.1.20]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.0...v0.1.20
[0.1.0]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/releases/tag/v0.1.0

15
CODE_OF_CONDUCT.md Normal file
View File

@ -0,0 +1,15 @@
# Code of Conduct
John Slop uses the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
Be respectful, specific, and evidence-led. Harassment, discriminatory language,
personal attacks, doxxing, sexualized conduct, and sustained disruption are not
acceptable in project spaces. Maintainers may edit, remove, or reject comments,
commits, issues, and other contributions that violate these expectations, and
may temporarily or permanently restrict participation.
Report conduct concerns privately to the repository owner through the hosting
service profile. Reports will be reviewed promptly and handled with discretion;
the maintainer will disclose information only when necessary to investigate or
protect participants. This project does not tolerate retaliation against a
reporter acting in good faith.

22
CONTRIBUTING.md Normal file
View File

@ -0,0 +1,22 @@
# Contributing
Contributions should preserve John Slop's explainable, local-first product
boundary. Findings must describe observable signals without asserting AI
authorship, model identity, or personal characteristics.
1. Open an issue for behavior or public-contract changes before investing in a
large patch.
2. Create a focused branch and add tests at the lowest useful layer.
3. Update user documentation, samples, and stable output snapshots together.
4. Run the quality, supply-chain, fuzz-smoke, or benchmark commands relevant to
the change as documented in [Development](docs/development.md).
5. Submit small conventional commits and explain rule false-positive tradeoffs.
New heuristic rules require a stable lowercase kebab-case ID, conservative
positive and realistic near-miss fixtures, scopes, a rationale and help text,
and evaluation-corpus review. Default weights or thresholds require the
documented sign-off process in [Evaluation](docs/evaluation.md).
By participating, contributors agree to follow the
[Code of Conduct](CODE_OF_CONDUCT.md). Report vulnerabilities through
[SECURITY.md](SECURITY.md), not a public issue.

1775
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

View File

@ -1,13 +1,109 @@
[package]
name = "john_slop"
version = "0.1.0"
[workspace]
members = ["crates/john-slop-core", "crates/john-slop-lsp"]
default-members = [".", "crates/john-slop-core", "crates/john-slop-lsp"]
resolver = "3"
[workspace.package]
version = "0.1.27-rc.8"
edition = "2024"
rust-version = "1.98.1"
license = "MIT"
repository = "https://proxy.dandaedre.com/gitea/idle-org/john-slop"
[workspace.dependencies]
clap = { version = "4.6.7", default-features = false, features = ["derive", "error-context", "help", "std", "suggestions", "usage"] }
criterion = { version = "0.8.2", default-features = false }
directories = "6.0.0"
figment = { version = "0.10.19", default-features = false, features = ["toml"] }
futures = { version = "0.3.34", default-features = false, features = ["std"] }
globset = { version = "0.4.20", default-features = false }
insta = { version = "1.48.0", default-features = false }
john-slop-core = { version = "0.1.27-rc.8", path = "crates/john-slop-core" }
john-slop-lsp-engine = { package = "john-slop-lsp", version = "0.1.27-rc.8", path = "crates/john-slop-lsp" }
miette = { version = "7.6.0", default-features = false, features = ["derive"] }
regex = "1.13.1"
serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.151"
sha2 = "0.11.0"
similar = "2.7.0"
tempfile = "3.27.0"
thiserror = "2.0.21"
tokio = { version = "1.53.1", default-features = false }
toml = "1.1.6"
tower = { version = "0.5.3", default-features = false, features = ["util"] }
tower-lsp-server = { version = "0.23.0", default-features = false, features = ["runtime-tokio"] }
unicode-security = "0.1.2"
walkdir = "2.5.0"
zip = { version = "8.6.0", default-features = false, features = ["deflate-flate2-zlib-rs"] }
[workspace.lints.rust]
unsafe_code = "forbid"
[package]
name = "john-slop"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
description = "Inspect files, directories, or ZIP-based documents for low-care signals"
autobins = false
[package.metadata.binstall]
pkg-url = "{ repo }/releases/download/v{ version }/{ name }-{ target }-v{ version }.tgz"
pkg-fmt = "tgz"
bin-dir = "{ name }-{ target }-v{ version }/{ bin }{ binary-ext }"
[[bin]]
name = "john-slop"
path = "src/bin/john-slop.rs"
[[bin]]
name = "john-slop-fmt"
path = "src/bin/john-slop-fmt.rs"
[[bin]]
name = "john-slop-lsp"
path = "src/bin/john-slop-lsp.rs"
required-features = ["lsp"]
[features]
default = ["fancy-diagnostics", "lsp"]
fancy-diagnostics = ["miette/fancy"]
lsp = ["dep:john-slop-lsp-engine", "dep:tokio", "dep:tower-lsp-server"]
[dependencies]
anyhow = "1.0.100"
clap = { version = "4.5.53", features = ["derive"] }
rayon = "1.11.0"
regex = "1.12.2"
serde = { version = "1.0.228", features = ["derive"] }
toml = "0.9.8"
walkdir = "2.5.0"
clap.workspace = true
globset.workspace = true
john-slop-core.workspace = true
john-slop-lsp-engine = { workspace = true, optional = true }
miette.workspace = true
serde.workspace = true
serde_json.workspace = true
similar.workspace = true
tempfile.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["io-std", "macros", "rt-multi-thread", "sync"], optional = true }
toml.workspace = true
tower-lsp-server = { workspace = true, optional = true }
[dev-dependencies]
criterion.workspace = true
futures.workspace = true
insta.workspace = true
sha2.workspace = true
tower.workspace = true
zip.workspace = true
[lints]
workspace = true
[[bench]]
name = "core"
harness = false
[profile.release]
codegen-units = 1
lto = "thin"
panic = "abort"
strip = "symbols"

13
Cross.toml Normal file
View File

@ -0,0 +1,13 @@
[build]
[build.env]
passthrough = ["RUSTUP_TOOLCHAIN"]
[target.x86_64-unknown-linux-musl]
image = "ghcr.io/cross-rs/x86_64-unknown-linux-musl:main"
[target.aarch64-unknown-linux-musl]
image = "ghcr.io/cross-rs/aarch64-unknown-linux-musl:main"
[target.x86_64-pc-windows-gnu]
image = "ghcr.io/cross-rs/x86_64-pc-windows-gnu:main"

363
Jenkinsfile vendored Normal file
View File

@ -0,0 +1,363 @@
// Add this webhook in Gitea:
// http://192.168.1.37:8080/gitea-webhook/post
pipeline {
agent { label 'rust_no_docker' }
environment {
CARGO_TERM_COLOR = 'always'
RUST_BACKTRACE = '1'
MSRV = '1.98.1'
CARGO_TARPAULIN_VERSION = '0.37.5'
CARGO_AUDIT_VERSION = '0.22.2'
CARGO_DENY_VERSION = '0.20.2'
BADGE_PUBLISH_ROOT = '/mnt/assets/coverage'
}
stages {
stage('Checkout') {
steps {
checkout scm
sh '''
set -eu
git submodule sync --recursive
git submodule update --init --recursive
'''
}
}
stage('Setup Rust') {
steps {
sh '''
set -eu
if ! command -v rustup >/dev/null 2>&1; then
echo "ERROR: rustup is not installed on this worker." >&2
exit 1
fi
echo "Ensuring the stable Rust toolchain is available..."
rustup toolchain install stable \
--profile minimal \
--component rustfmt \
--component clippy
rustup toolchain install "${MSRV}" \
--profile minimal
rustup default stable
echo
echo "Using:"
rustc --version
cargo --version
rustfmt --version
cargo clippy --version
INSTALLED_AUDIT="$(cargo audit --version 2>/dev/null || true)"
if [ "${INSTALLED_AUDIT}" != "cargo-audit ${CARGO_AUDIT_VERSION}" ]; then
cargo install cargo-audit --locked --version "${CARGO_AUDIT_VERSION}"
fi
INSTALLED_DENY="$(cargo deny --version 2>/dev/null || true)"
if [ "${INSTALLED_DENY}" != "cargo-deny ${CARGO_DENY_VERSION}" ]; then
cargo install cargo-deny --locked --version "${CARGO_DENY_VERSION}"
fi
INSTALLED_TARPAULIN="$(cargo tarpaulin --version 2>/dev/null || true)"
if [ "${INSTALLED_TARPAULIN}" != "cargo-tarpaulin-tarpaulin ${CARGO_TARPAULIN_VERSION}" ]; then
cargo install cargo-tarpaulin --locked --version "${CARGO_TARPAULIN_VERSION}"
fi
'''
}
}
stage('MSRV') {
steps {
sh '''
set -eu
echo "Checking all targets with Rust ${MSRV}..."
cargo "+${MSRV}" check \
--locked \
--all-targets \
--all-features
'''
}
}
stage('Format') {
steps {
sh '''
set -eu
echo "Checking Rust formatting..."
cargo fmt --all -- --check
'''
}
}
stage('Check') {
steps {
sh '''
set -eu
echo "Checking all targets..."
cargo check \
--locked \
--all-targets \
--all-features
'''
}
}
stage('Test') {
steps {
sh '''
set -eu
echo "Running tests..."
cargo test \
--locked \
--all-targets \
--all-features
'''
}
}
stage('Test Minimal Features') {
steps {
sh '''
set -eu
echo "Running tests without default features..."
cargo test \
--locked \
--all-targets \
--no-default-features
'''
}
}
stage('Fixtures and Goldens') {
steps {
sh '''
set -eu
cargo test --locked --test example_fixtures --all-features
cargo test --locked --test cli --all-features
mkdir -p coverage
: > "coverage/.tests-passed-${BUILD_NUMBER}"
'''
}
}
stage('Coverage') {
steps {
sh '''
set -eu
cargo tarpaulin \
--config tarpaulin.toml \
--out Json \
--output-dir coverage
mv coverage/tarpaulin-report.json coverage/summary.json
coverage_percent=$(sed -n \
's/^.*],"coverage":\\([0-9][0-9.]*\\),"covered":[0-9][0-9]*,"coverable":[0-9][0-9]*}$/\\1/p' \
coverage/summary.json)
scripts/render-coverage-badge.sh \
"${coverage_percent}" \
coverage/coverage.svg
rm coverage/summary.json
: > "coverage/.coverage-ready-${BUILD_NUMBER}"
'''
}
}
stage('Clippy') {
steps {
sh '''
set -eu
echo "Running Clippy..."
cargo clippy \
--locked \
--all-targets \
--all-features \
-- \
-D warnings
'''
}
}
stage('Documentation') {
steps {
sh '''
set -eu
cargo doc --locked --no-deps --all-features
'''
}
}
stage('Supply Chain') {
steps {
sh '''
set -eu
cargo audit --deny warnings
cargo deny --locked check
'''
}
}
stage('Workspace Packages') {
steps {
sh '''
set -eu
echo "Assembling all publishable workspace packages..."
scripts/package-workspace.sh --no-verify
'''
}
}
stage('Build Release') {
steps {
sh '''
set -eu
echo "Building John Slop release binaries..."
# Jenkins reuses workspaces. Remove only ignored release
# output so rerunning the same commit stays deterministic.
rm -rf -- dist
# The packaging script reads and validates Cargo.toml itself.
scripts/package-release.sh
echo
echo "Release build completed."
ls -lh \
target/release/john-slop \
target/release/john-slop-fmt \
target/release/john-slop-lsp \
dist/*.tar.gz \
dist/*.sha256
'''
}
}
}
post {
success {
publishChecks(
conclusion: 'SUCCESS',
name: 'Rust CI',
summary: 'JohnSlop passed all CI checks.',
text: 'The Linux MSRV/stable matrix, formatting, feature tests, fixtures, coverage and badge generation, docs, Clippy, supply-chain policy, and release build completed successfully.',
title: 'Build Successful'
)
echo '✅ JohnSlop passed all CI checks.'
}
failure {
publishChecks(
conclusion: 'FAILURE',
name: 'Rust CI',
summary: 'JohnSlop failed CI.',
text: 'A toolchain, formatting, test, fixture, coverage, badge generation, documentation, Clippy, supply-chain, or release check failed. Check the Jenkins log.',
title: 'Build Failed'
)
echo '❌ JohnSlop CI failed.'
}
unstable {
publishChecks(
conclusion: 'ACTION_REQUIRED',
name: 'Rust CI',
summary: 'JohnSlop CI is unstable.',
text: 'The Jenkins build completed with an unstable status.',
title: 'Build Unstable'
)
echo '⚠️ JohnSlop CI is unstable.'
}
cleanup {
sh '''
set -eu
mkdir -p coverage
test_status=failing
if [ -f "coverage/.tests-passed-${BUILD_NUMBER}" ]; then
test_status=passing
fi
scripts/render-test-badge.sh \
"${test_status}" \
coverage/test.svg
if [ ! -f "coverage/.coverage-ready-${BUILD_NUMBER}" ]; then
scripts/render-coverage-badge.sh - coverage/coverage.svg
fi
package_version=$(awk '
$0 == "[workspace.package]" { in_package = 1; next }
substr($0, 1, 1) == "[" { in_package = 0 }
in_package && $1 == "version" && $2 == "=" {
gsub(/^"|"$/, "", $3)
print $3
exit
}
' Cargo.toml)
if [ -z "${package_version}" ]; then
echo "could not determine workspace package version" >&2
exit 2
fi
scripts/render-version-badge.sh \
"${package_version}" \
coverage/version.svg
rm -f \
coverage/summary.json \
coverage/tarpaulin-report.json \
"coverage/.tests-passed-${BUILD_NUMBER}" \
"coverage/.coverage-ready-${BUILD_NUMBER}"
'''
archiveArtifacts(
allowEmptyArchive: false,
artifacts: 'coverage/coverage.svg,coverage/test.svg,coverage/version.svg',
fingerprint: true
)
sh '''
set -eu
if [ "${BRANCH_NAME:-}" = "main" ]; then
scripts/publish-badges.sh \
"${BADGE_PUBLISH_ROOT}" \
coverage \
test \
version
fi
'''
}
}
}

128
README.md
View File

@ -1,9 +1,125 @@
# JohnSlop
# John Slop
Detect AI slop markers
![version](https://assets.dandaedre.com/coverage/idle-org/john-slop/main/version.svg)
![tests](https://assets.dandaedre.com/coverage/idle-org/john-slop/main/test.svg)
![test coverage](https://assets.dandaedre.com/coverage/idle-org/john-slop/main/coverage.svg)
John Slop is a local-first linter for concrete signs of low-effort or
insufficiently reviewed text and source code. It reports explainable evidence
for human review; it does not prove AI authorship or identify a model.
# Links
* [https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing]
* [https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup]
* [https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion]
## Quickstart
With `cargo-binstall` available, install all three prebuilt binaries from the
latest release candidate:
```sh
cargo binstall --git https://proxy.dandaedre.com/gitea/idle-org/john-slop \
john-slop
```
To build from source, Rust 1.98.1 or newer is required. Install from the
upstream Git repository:
```sh
cargo install --git https://proxy.dandaedre.com/gitea/idle-org/john-slop \
--locked --all-features
```
Or install a checkout from its repository root:
```sh
cargo install --path . --locked --all-features
```
Scan a document or source directory:
```sh
john-slop path/to/document path/to/source-directory
```
Generate a commented `config.toml` in the current directory, then edit category
or individual-rule switches as needed:
```sh
john-slop config init
john-slop config init-diagnostics
```
The second command creates a runtime rule catalog: exact-phrase and regex
diagnostics can then be added without changing or recompiling the program.
Diagnostics are colored Miette reports by default. Make the command fail when a
document reaches a chosen low-care evidence score with `--fail-score 0..100`:
```sh
john-slop --fail-score 40 path/to/document
```
Preview configured Unicode-to-ASCII replacements, or check them in automation:
```sh
john-slop-fmt path/to/document
john-slop-fmt --check path/to/document
```
Formatting is read-only unless `--write` is explicit. Start the editor server
with `john-slop-lsp`; the [editor guide](docs/editors.md) has ready-to-use Helix
and Vim configuration.
## Overview
John Slop provides three binaries backed by the same configuration and source
spans:
- `john-slop` detects copied assistant scaffolding, generic style residue,
source placeholders, tutorial narration, configured replacements, ASCII-like
Unicode, and invisible or bidirectional controls.
- `john-slop-fmt` previews, checks, or explicitly applies configured ASCII
replacements, including the default printable-ASCII UTS #39 profile.
- `john-slop-lsp` publishes Unicode diagnostics over stdio by default, offers
explicit weighted or all-diagnostic opt-ins, provides quick fixes for safe
replacements, and formats without writing editor-owned files.
The detector accepts UTF-8 files, directories, ZIP archives, and common
ZIP-based office documents. Each finding includes its source span, stable rule
ID, severity, message, and remediation help. A bounded evidence score aggregates
configured signals; it is not the probability that AI wrote the content.
Analysis is local and deterministic, with no network requests or telemetry.
False positives remain possible—especially in transcripts, tutorials,
multilingual text, and intentional typography—so findings are prompts for
review, not verdicts. Projects with multilingual text should consider the
Unicode profile's non-mutating `review` mode before writing formatted output.
## Learn progressively
Start with the guide closest to what you want to do:
1. [Use the CLI and understand exit statuses](docs/usage.md).
2. [Read or integrate the available output formats](docs/output.md).
3. [Configure rules, replacements, and layered overrides](docs/configuration.md),
with [commented configuration examples](examples/config/).
4. [Run the Rust and expected-failure examples](examples/README.md).
5. [Connect the formatter and LSP to Helix or Vim](docs/editors.md).
6. [Understand the review-signal methodology](docs/review-signals.md), the
[heuristics and evidence score](docs/heuristics.md), and then the
[interpretation limits](docs/limitations.md).
The [documentation index](docs/README.md) offers the same guided path with a
little more context.
## Advanced topics
Architecture, the public library API, evaluation results, performance,
dependency policy, design decisions, Unicode research, development, and release
engineering are collected in the [advanced guide](docs/advanced.md).
## Project
See [CONTRIBUTING.md](CONTRIBUTING.md) to work on the project and
[SECURITY.md](SECURITY.md) to report a vulnerability. The ordered implementation
plan lives in [tasks.md](tasks.md), and agent contribution rules live in
[AGENTS.md](AGENTS.md).
John Slop is distributed under the [MIT license](LICENSE).

22
SECURITY.md Normal file
View File

@ -0,0 +1,22 @@
# Security policy
The current supported line is `0.1.x`. Older snapshots and untagged forks do not
receive security fixes.
Do not open a public issue for a suspected vulnerability. Use the repository
host's private security-advisory/reporting interface when available. Otherwise,
contact the repository owner privately through the hosting profile and request
a secure channel. Include the affected version/commit, operating system, impact,
minimal reproduction, and whether details have been shared elsewhere. Do not
include private documents processed by John Slop.
The maintainer will acknowledge a report within seven days, assess scope and
severity, coordinate a fix and disclosure date when confirmed, and credit the
reporter if requested. This is a best-effort volunteer response target, not a
service-level guarantee.
John Slop processes untrusted text and archives but is not a sandbox. Archive
limits and path validation reduce exposure; reports that bypass those limits,
cause unintended writes, leak document contents, corrupt protocol output, or
execute code are in scope. False-positive disagreements and requests for new
heuristics belong in the normal issue tracker.

119
benches/core.rs Normal file
View File

@ -0,0 +1,119 @@
use std::{
fs::{self, File},
io::Write,
};
use criterion::{Criterion, Throughput, criterion_group, criterion_main};
use john_slop::{AnalysisContext, Config, Detector, read_inputs};
use tempfile::TempDir;
use zip::write::FileOptions;
fn benchmark_large_document(criterion: &mut Criterion) {
let config = Config::embedded();
let detector = Detector::new(&config).expect("embedded detector");
let source = "Reviewed prose without configured signals.\n".repeat(25_000);
let context = AnalysisContext::for_path("large.txt");
let mut group = criterion.benchmark_group("analysis");
group.throughput(Throughput::Bytes(source.len() as u64));
group.bench_function("one-megabyte-document", |bencher| {
bencher.iter(|| detector.analyze_with(&source, &context));
});
group.finish();
}
fn benchmark_many_small_files(criterion: &mut Criterion) {
let directory = small_file_tree(250);
let config = Config::embedded();
let mut group = criterion.benchmark_group("input");
group.throughput(Throughput::Elements(250));
group.bench_function("250-small-files", |bencher| {
bencher.iter(|| {
read_inputs(&config, directory.path())
.expect("open fixture directory")
.collect::<Result<Vec<_>, _>>()
.expect("read fixture directory")
});
});
group.finish();
}
fn benchmark_rule_scaling(criterion: &mut Criterion) {
let mut config = Config::embedded();
let original = config.rules.clone();
for generation in 1..10 {
config
.rules
.extend(original.iter().cloned().map(|mut rule| {
rule.id = format!("{}-{generation}", rule.id);
rule
}));
}
config.validate().expect("scaled config remains valid");
let source = "Assistant: Certainly! Here's a result—review it.\n".repeat(100);
let context = AnalysisContext::for_path("scaled.txt");
criterion.bench_function("rules/build-and-analyze-110-rules", |bencher| {
bencher.iter(|| {
Detector::new(&config)
.expect("build scaled detector")
.analyze_with(&source, &context)
});
});
}
fn benchmark_archive_limits(criterion: &mut Criterion) {
let directory = archive_with_members(100);
let path = directory.path().join("many.zip");
let config = Config::embedded();
let mut group = criterion.benchmark_group("input");
group.throughput(Throughput::Elements(100));
group.bench_function("100-member-archive", |bencher| {
bencher.iter(|| {
read_inputs(&config, &path)
.expect("open fixture archive")
.collect::<Result<Vec<_>, _>>()
.expect("read fixture archive")
});
});
group.finish();
}
fn small_file_tree(count: usize) -> TempDir {
let directory = tempfile::tempdir().expect("create benchmark directory");
for index in 0..count {
fs::write(
directory.path().join(format!("document-{index:04}.txt")),
"Reviewed prose without configured signals.\n",
)
.expect("write benchmark document");
}
directory
}
fn archive_with_members(count: usize) -> TempDir {
let directory = tempfile::tempdir().expect("create benchmark directory");
let file = File::create(directory.path().join("many.zip")).expect("create benchmark archive");
let mut archive = zip::ZipWriter::new(file);
let options: FileOptions<()> = FileOptions::default()
.compression_method(zip::CompressionMethod::Stored)
.unix_permissions(0o644);
for index in 0..count {
archive
.start_file(format!("document-{index:04}.txt"), options)
.expect("start benchmark member");
archive
.write_all(b"Reviewed prose without configured signals.\n")
.expect("write benchmark member");
}
archive.finish().expect("finish benchmark archive");
directory
}
criterion_group!(
benches,
benchmark_large_document,
benchmark_many_small_files,
benchmark_rule_scaling,
benchmark_archive_limits
);
criterion_main!(benches);

View File

@ -1,7 +0,0 @@
[[patterns]]
regex = "\\p{Pd}"
replacement = "-"
[[patterns]]
regex = "[\u2018\u2019\u201A\u201B\u02BC\u02BB\u00B4\uFF40\uFF07]"
replacement = "'"

View File

@ -0,0 +1,27 @@
[package]
name = "john-slop-core"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
description = "Reusable heuristic analysis, configuration, input, and replacement engine for John Slop"
[dependencies]
directories.workspace = true
figment.workspace = true
globset.workspace = true
miette.workspace = true
regex.workspace = true
serde.workspace = true
thiserror.workspace = true
unicode-security.workspace = true
walkdir.workspace = true
zip.workspace = true
[dev-dependencies]
tempfile.workspace = true
toml.workspace = true
[lints]
workspace = true

View File

@ -0,0 +1,27 @@
# Embedded input defaults. Unicode replacements live in the generated
# `unicode.toml` policy asset so detection, diagnostics, and fixes stay aligned.
schema_version = 1
replacements = []
# Diagnostic category gates are applied in addition to each rule's `enabled`
# field. Set a category to false to silence that family without copying every
# embedded rule into this file.
[diagnostics.categories]
typography = true
style = true
prompt-leakage = true
source-code = true
# Disable or re-enable one diagnostic by its stable ID. Uncomment and change
# the ID as needed; `john-slop config show` lists the complete resolved rules.
# [[rules]]
# id = "templated-conclusion"
# enabled = false
[input]
max_document_bytes = 8388608
max_archive_bytes = 67108864
max_expanded_bytes = 268435456
max_archive_entries = 10000
max_compression_ratio = 100
max_walk_depth = 64

View File

@ -0,0 +1,27 @@
# Runtime diagnostic catalog. Copy or generate this as `diagnostics.toml` and
# add data-driven rules without rebuilding John Slop.
schema_version = 1
# This starter rule is disabled until its phrase and metadata fit your policy.
# Duplicate this complete table for additional diagnostics; every ID must be
# unique lowercase kebab-case.
[[rules]]
id = "custom-review-marker"
category = "style"
severity = "warning"
matcher = "exact-phrase"
pattern = "REVIEW BEFORE PUBLISHING"
weight = 8.0
message = "a project review marker remains in the source"
help = "Resolve or remove the marker before publishing."
occurrence_cap = 1
enabled = false
[rules.scope]
content = ["prose", "source-code"]
extensions = []
include = []
exclude = []
skip_generated = true
skip_vendor = true
skip_minified = true

View File

@ -0,0 +1,280 @@
# Embedded detection-rule metadata. Milestone 3 compiles and executes these
# matchers; Milestone 2 makes their configuration typed, layered, and validated.
schema_version = 1
[thresholds]
low = 20
medium = 50
high = 75
[[rules]]
id = "assistant-self-identification"
category = "prompt-leakage"
severity = "warning"
matcher = "exact-phrase"
pattern = "As an AI language model"
weight = 30.0
message = "assistant self-identification appears in the source"
help = "Review whether an assistant response was copied into the document."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "knowledge-cutoff-disclaimer"
category = "prompt-leakage"
severity = "warning"
matcher = "exact-phrase"
pattern = "knowledge cutoff"
weight = 20.0
message = "a knowledge-cutoff disclaimer appears in the source"
help = "Verify the statement and remove assistant boilerplate if it is not relevant."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "assistant-capability-disclaimer"
category = "prompt-leakage"
severity = "warning"
matcher = "exact-phrase"
pattern = "I cannot browse the internet"
weight = 20.0
message = "an assistant capability disclaimer appears in the source"
help = "Replace generic capability boilerplate with a source-specific limitation if needed."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "assistant-answer-preface"
category = "prompt-leakage"
severity = "warning"
matcher = "exact-phrase"
pattern = "Certainly! Here's"
weight = 15.0
message = "a stock assistant answer preface appears in the source"
help = "Review whether conversational response scaffolding was copied into the document."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "prompt-response-scaffolding"
category = "prompt-leakage"
severity = "warning"
matcher = "regex"
pattern = "(?m)^(User|Assistant):[ \\t]"
weight = 25.0
message = "prompt/response scaffolding appears in the source"
help = "Confirm that chat transcript markers are intentional."
occurrence_cap = 2
enabled = true
[rules.scope]
content = ["prose", "source-code"]
extensions = []
[[rules]]
id = "assistant-follow-up-offer"
category = "prompt-leakage"
severity = "warning"
matcher = "regex"
pattern = "(?i)\\b(?:let me know|tell me) if you(?:'d| would) like me to\\b"
weight = 12.0
message = "a conversational assistant follow-up offer appears in the source"
help = "Review whether a response-closing offer was copied into the document."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "assistant-revision-preface"
category = "prompt-leakage"
severity = "warning"
matcher = "regex"
pattern = "(?i)\\bhere(?:'s| is) (?:a|the) (?:revised|polished|rewritten) (?:version|draft)\\b"
weight = 12.0
message = "an assistant-style revision preface appears in the source"
help = "Remove conversational revision scaffolding if it is not part of the document."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "templated-conclusion"
category = "style"
severity = "advice"
matcher = "regex"
pattern = "(?i)\\bin conclusion,? (it is|we can)"
weight = 5.0
message = "a templated conclusion phrase appears in the source"
help = "Review whether the conclusion adds specific information."
occurrence_cap = 2
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "templated-filler"
category = "style"
severity = "advice"
matcher = "regex"
pattern = "(?i)\\bit is (?:important|worth) (?:to note|noting) that\\b"
weight = 4.0
message = "a generic filler transition appears in the source"
help = "Prefer a direct, source-specific statement when the transition adds no meaning."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "repeated-transition"
category = "style"
severity = "advice"
matcher = "repeated-regex"
pattern = "(?im)^(?:Moreover|Furthermore|Additionally),[ \\t]"
weight = 4.0
message = "the same family of stock transitions is repeated"
help = "Review repeated transitions and keep only those that clarify the argument."
occurrence_cap = 3
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "editorial-insertion-placeholder"
category = "style"
severity = "warning"
matcher = "regex"
pattern = "(?i)\\[(?:insert|add) (?:citation|reference|source|example|details?)(?: here)?\\]"
weight = 10.0
message = "an unresolved editorial insertion placeholder appears in the source"
help = "Supply the requested material or remove the placeholder before publishing."
occurrence_cap = 2
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "stock-helpful-closing"
category = "style"
severity = "advice"
matcher = "regex"
pattern = "(?im)^[ \\t]*I hope this helps[.!][ \\t]*$"
weight = 2.0
message = "a stock conversational closing appears in the source"
help = "Keep the closing only when the document is intentionally conversational."
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "excess-vocabulary-cluster"
category = "style"
severity = "advice"
matcher = "repeated-regex"
pattern = "(?i)\\b(?:delv(?:e|es|ed|ing)|intricate|multifaceted|pivotal|underscore[sd]?|realm|landscape)\\b"
weight = 1.0
message = "several terms from a population-level excess-vocabulary cluster appear"
help = "This weak corpus-level clue is not authorship evidence; review only for fit and specificity."
occurrence_cap = 3
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["md", "txt", "rst", "adoc"]
[[rules]]
id = "source-placeholder-todo"
category = "source-code"
severity = "warning"
matcher = "regex"
pattern = "(?i)\\bTODO:[ \\t]+(implement|add implementation)\\b"
weight = 10.0
message = "an implementation placeholder remains in source code"
help = "Implement the missing behavior or link the placeholder to tracked work."
occurrence_cap = 3
enabled = true
[rules.scope]
content = ["source-code"]
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
[[rules]]
id = "source-tutorial-narration"
category = "source-code"
severity = "advice"
matcher = "regex"
pattern = "(?im)^[ \\t]*//[ \\t]*(?:first|next|finally),?[ \\t]+we\\b"
weight = 8.0
message = "a tutorial-style narration comment appears in source code"
help = "Describe why the code exists rather than narrating the next obvious operation."
occurrence_cap = 2
enabled = true
[rules.scope]
content = ["source-code"]
extensions = ["c", "cpp", "go", "java", "js", "rs", "ts"]
[[rules]]
id = "source-omitted-code-placeholder"
category = "source-code"
severity = "warning"
matcher = "regex"
pattern = "(?im)^[ \\t]*(?://|#)[ \\t]*(?:\\.\\.\\.[ \\t]*)?(?:existing|remaining|rest of (?:the )?)[ \\t]+code(?:[ \\t]+(?:here|remains?|unchanged))?[ \\t]*$"
weight = 10.0
message = "an omitted-code placeholder remains in source code"
help = "Restore the omitted implementation or remove the placeholder."
occurrence_cap = 2
enabled = true
[rules.scope]
content = ["source-code"]
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
[[rules]]
id = "source-markdown-fence"
category = "source-code"
severity = "advice"
matcher = "regex"
pattern = "(?m)^[ \\t]*```(?:[A-Za-z0-9_+-]+)?[ \\t]*$"
weight = 6.0
message = "a Markdown code fence appears in a source-code file"
help = "Remove response-formatting fences unless they are intentionally embedded data."
occurrence_cap = 2
enabled = true
[rules.scope]
content = ["source-code"]
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,192 @@
//! Regenerate `assets/unicode.toml` from reviewed mappings and Unicode Security.
use std::collections::BTreeSet;
use serde::Serialize;
const CURATED: &[(&str, &str, &str, &str)] = &[
("hyphen", "‐", "-", "Unicode hyphen"),
("non-breaking-hyphen", "‑", "-", "non-breaking hyphen"),
("figure-dash", "‒", "-", "figure dash"),
("en-dash", "–", "-", "en dash"),
("em-dash", "—", "-", "em dash"),
("horizontal-bar", "―", "-", "horizontal bar"),
("minus-sign", "−", "-", "mathematical minus sign"),
("small-em-dash", "﹘", "-", "small em dash"),
("small-hyphen-minus", "﹣", "-", "small hyphen-minus"),
];
#[derive(Serialize)]
struct Catalog {
schema_version: u32,
unicode: UnicodePolicy,
replacements: Vec<Replacement>,
rules: Vec<Rule>,
}
#[derive(Serialize)]
struct UnicodePolicy {
confusables: &'static str,
}
#[derive(Serialize)]
struct Replacement {
id: String,
source: String,
replacement: String,
rationale: String,
enabled: bool,
}
#[derive(Serialize)]
struct Rule {
id: &'static str,
category: &'static str,
severity: &'static str,
matcher: &'static str,
pattern: &'static str,
weight: f64,
message: &'static str,
help: &'static str,
occurrence_cap: usize,
enabled: bool,
scope: Scope,
}
#[derive(Serialize)]
struct Scope {
content: [&'static str; 2],
extensions: [String; 0],
}
fn main() {
let mut replacements = CURATED
.iter()
.map(|(id, source, replacement, name)| Replacement {
id: (*id).to_owned(),
source: (*source).to_owned(),
replacement: (*replacement).to_owned(),
rationale: format!("Replace the {name} with reviewed ASCII `{replacement}`."),
enabled: true,
})
.collect::<Vec<_>>();
replacements.push(Replacement {
id: "ideographic-space".to_owned(),
source: '\u{3000}'.to_string(),
replacement: " ".to_owned(),
rationale: "Width-fold the ideographic space to ASCII space.".to_owned(),
enabled: true,
});
for ascii in b'!'..=b'~' {
// U+FF01..=U+FF5E is a contiguous block corresponding to ASCII !..=~.
let value = u32::from(ascii) + 0xfee0;
let Some(fullwidth) = char::from_u32(value) else {
continue;
};
let id = if ascii == b'-' {
// Preserve the public ID shipped before the complete width-folding set.
"fullwidth-hyphen-minus".to_owned()
} else if printable_ascii_skeleton(fullwidth).is_some() {
// Preserve IDs already shipped as generated UTS #39 mappings.
format!("uts39-u{value:04x}")
} else {
format!("fullwidth-u{value:04x}")
};
let replacement = char::from(ascii).to_string();
replacements.push(Replacement {
id,
source: fullwidth.to_string(),
rationale: format!(
"Width-fold Unicode fullwidth form U+{value:04X} to ASCII {replacement:?}."
),
replacement,
enabled: true,
});
}
let reviewed_sources = replacements
.iter()
.filter_map(|rule| rule.source.chars().next())
.collect::<BTreeSet<_>>();
for value in 0..=char::MAX as u32 {
let Some(character) = char::from_u32(value) else {
continue;
};
if character.is_ascii() || reviewed_sources.contains(&character) {
continue;
}
let source = character.to_string();
let Some(skeleton) = printable_ascii_skeleton(character) else {
continue;
};
replacements.push(Replacement {
id: format!("uts39-u{value:04x}"),
source,
replacement: skeleton.clone(),
rationale: format!(
"Replace Unicode 16.0 UTS #39 lookalike U+{value:04X} with its printable ASCII skeleton {skeleton:?}."
),
enabled: true,
});
}
let catalog = Catalog {
schema_version: 1,
unicode: UnicodePolicy {
confusables: "replace",
},
replacements,
rules: vec![
Rule {
id: "unicode-ascii-replacement",
category: "typography",
severity: "advice",
matcher: "replacement",
pattern: "configured-replacements",
weight: 10.0,
message: "a Unicode character or sequence has a configured ASCII replacement",
help: "Review the character or apply the configured ASCII replacement.",
occurrence_cap: 5,
enabled: true,
scope: Scope {
content: ["prose", "source-code"],
extensions: [],
},
},
Rule {
id: "unicode-invisible-control",
category: "typography",
severity: "warning",
matcher: "invisible",
pattern: "gitea-invisible-9c715c0",
weight: 12.0,
message: "an invisible or bidirectional Unicode control appears in the source",
help: "Review the hidden or direction-changing character; it is not removed automatically.",
occurrence_cap: 5,
enabled: true,
scope: Scope {
content: ["prose", "source-code"],
extensions: [],
},
},
],
};
println!(
"# Generated by `cargo run -p john-slop-core --example generate_unicode_catalog`.\n\
# Unicode Security 0.1.2 / Unicode 16.0.0 UTS #39, plus reviewed punctuation and width folding.\n\
# Do not edit generated mappings by hand; edit this generator and regenerate.\n{}",
toml::to_string_pretty(&catalog).expect("the catalog model must serialize")
);
}
fn printable_ascii_skeleton(character: char) -> Option<String> {
let source = character.to_string();
let skeleton = unicode_security::skeleton(&source).collect::<String>();
(!skeleton.is_empty()
&& skeleton.bytes().all(|byte| (b' '..=b'~').contains(&byte))
&& skeleton != source)
.then_some(skeleton)
}

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,943 @@
//! Compiled, reusable registry of configured detection rules.
use std::{
collections::BTreeMap,
path::{Path, PathBuf},
};
use globset::{Glob, GlobSet, GlobSetBuilder};
use regex::Regex;
use crate::{
Analysis, Category, Config, ConfigValidationError, ContentKind, EvidenceBand, EvidenceScore,
Finding, HeuristicRule, MatcherType, RuleId, SourceId, replacement, unicode_review,
};
/// A validated heuristic definition paired with its reusable matcher.
#[derive(Debug, Clone)]
pub struct Rule {
definition: HeuristicRule,
matcher: CompiledMatcher,
scope: CompiledScope,
}
impl Rule {
/// Return this rule's stable identifier.
pub fn id(&self) -> &str {
&self.definition.id
}
/// Return this rule's evidence family.
pub const fn category(&self) -> Category {
self.definition.category
}
/// Return whether this rule is enabled in the resolved configuration.
pub const fn is_enabled(&self) -> bool {
self.definition.enabled
}
/// Return the literal, regex source, or replacement-catalog sentinel.
pub fn pattern(&self) -> &str {
match &self.matcher {
CompiledMatcher::ExactPhrase(pattern) => pattern,
CompiledMatcher::Regex(regex) => regex.as_str(),
CompiledMatcher::RepeatedRegex(regex) => regex.as_str(),
CompiledMatcher::Replacement => "configured-replacements",
CompiledMatcher::Invisible => "gitea-invisible-9c715c0",
}
}
}
/// Input metadata used to select applicable rules without reading the source twice.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AnalysisContext {
path: Option<PathBuf>,
content_kind: ContentKind,
}
impl AnalysisContext {
/// Describe prose without a known filesystem path, such as standard input.
pub const fn prose() -> Self {
Self {
path: None,
content_kind: ContentKind::Prose,
}
}
/// Describe source code without a known filesystem path.
pub const fn source_code() -> Self {
Self {
path: None,
content_kind: ContentKind::SourceCode,
}
}
/// Infer the broad content kind from a path's extension.
pub fn for_path(path: impl Into<PathBuf>) -> Self {
let path = path.into();
let content_kind = infer_content_kind(&path);
Self {
path: Some(path),
content_kind,
}
}
/// Describe a loaded document using its exact source identity.
pub fn for_source(source: &SourceId) -> Self {
match source {
SourceId::Stdin => Self::prose(),
SourceId::File(path) => Self::for_path(path),
SourceId::ArchiveMember { member, .. } => Self::for_path(member),
}
}
/// Return the path used by extension and glob filters, when known.
pub fn path(&self) -> Option<&Path> {
self.path.as_deref()
}
/// Return the broad content kind selected for this input.
pub const fn content_kind(&self) -> ContentKind {
self.content_kind
}
}
impl Default for AnalysisContext {
fn default() -> Self {
Self::prose()
}
}
/// Matcher compiled once when the detector registry is built.
#[derive(Debug, Clone)]
enum CompiledMatcher {
ExactPhrase(String),
Regex(Regex),
RepeatedRegex(Regex),
Replacement,
Invisible,
}
#[derive(Debug, Clone)]
struct CompiledScope {
include: Option<GlobSet>,
exclude: GlobSet,
}
/// Reusable detection engine built from one resolved configuration snapshot.
#[derive(Debug, Clone)]
pub struct Detector {
config: Config,
rules: Vec<Rule>,
}
impl Detector {
/// Validate a resolved configuration and compile every reusable matcher.
pub fn new(config: &Config) -> Result<Self, DetectorBuildError> {
config
.validate()
.map_err(|error| DetectorBuildError::InvalidConfig(Box::new(error)))?;
let rules = config
.rules
.iter()
.cloned()
.map(Rule::compile)
.collect::<Result<Vec<_>, _>>()?;
Ok(Self {
config: config.clone(),
rules,
})
}
/// Return the compiled registry in deterministic configuration order.
pub fn rules(&self) -> &[Rule] {
&self.rules
}
/// Analyze text with the compiled registry.
pub fn analyze(&self, source: &str) -> Analysis {
self.analyze_with(source, &AnalysisContext::default())
}
/// Analyze text with explicit content and path applicability metadata.
pub fn analyze_with(&self, source: &str, context: &AnalysisContext) -> Analysis {
let mut findings = self
.rules
.iter()
.filter(|rule| {
rule.is_enabled()
&& self.config.diagnostics.category_enabled(rule.category())
&& rule.applies(context)
})
.flat_map(|rule| rule.findings(&self.config, source))
.collect::<Vec<_>>();
normalize_findings(&mut findings);
let evidence_score = aggregate_evidence_score(&self.config, &findings);
let evidence_band = (!findings.is_empty())
.then(|| band_for_score(self.config.thresholds, evidence_score.score()));
Analysis {
findings,
evidence_score,
evidence_band,
}
}
}
fn normalize_findings(findings: &mut Vec<Finding>) {
findings.sort_by(|left, right| {
(left.byte_span.start, left.byte_span.end, &left.rule_id).cmp(&(
right.byte_span.start,
right.byte_span.end,
&right.rule_id,
))
});
findings
.dedup_by(|right, left| right.rule_id == left.rule_id && right.byte_span == left.byte_span);
}
fn aggregate_evidence_score(config: &Config, findings: &[Finding]) -> EvidenceScore {
let mut family_rule_counts = BTreeMap::<Category, usize>::new();
let mut evidence = 0.0;
for rule in &config.rules {
let occurrences = findings
.iter()
.filter(|finding| finding.rule_id.as_str() == rule.id)
.count()
.min(rule.occurrence_cap);
if occurrences == 0 {
continue;
}
let occurrence_factor = (0..occurrences)
.map(|index| 0.5_f64.powi(i32::try_from(index).unwrap_or(i32::MAX)))
.sum::<f64>();
let family_index = family_rule_counts.entry(rule.category).or_default();
let family_factor =
family_decay(rule.category).powi(i32::try_from(*family_index).unwrap_or(i32::MAX));
evidence += rule.weight * occurrence_factor * family_factor;
*family_index += 1;
}
let normalized = if evidence == 0.0 {
0.0
} else {
100.0 * evidence / (evidence + 50.0)
};
EvidenceScore::from_score(normalized.round() as u8)
}
const fn family_decay(category: Category) -> f64 {
match category {
Category::PromptLeakage => 0.90,
Category::Typography | Category::Style | Category::SourceCode => 0.75,
}
}
fn band_for_score(thresholds: crate::EvidenceThresholds, score: u8) -> EvidenceBand {
if score >= thresholds.high {
EvidenceBand::High
} else if score >= thresholds.medium {
EvidenceBand::Medium
} else if score >= thresholds.low {
EvidenceBand::Low
} else {
EvidenceBand::BelowThreshold
}
}
impl Rule {
fn compile(definition: HeuristicRule) -> Result<Self, DetectorBuildError> {
let matcher = match definition.matcher {
MatcherType::ExactPhrase => CompiledMatcher::ExactPhrase(definition.pattern.clone()),
MatcherType::Regex => {
let regex = Regex::new(&definition.pattern).map_err(|source| {
DetectorBuildError::InvalidRegex {
id: definition.id.clone(),
source,
}
})?;
CompiledMatcher::Regex(regex)
}
MatcherType::RepeatedRegex => {
let regex = Regex::new(&definition.pattern).map_err(|source| {
DetectorBuildError::InvalidRegex {
id: definition.id.clone(),
source,
}
})?;
CompiledMatcher::RepeatedRegex(regex)
}
MatcherType::Replacement => CompiledMatcher::Replacement,
MatcherType::Invisible => CompiledMatcher::Invisible,
};
let scope = CompiledScope::compile(&definition)?;
Ok(Self {
definition,
matcher,
scope,
})
}
fn applies(&self, context: &AnalysisContext) -> bool {
if !self
.definition
.scope
.content
.contains(&context.content_kind)
{
return false;
}
let Some(path) = context.path() else {
return self.scope.include.is_none();
};
if !self.definition.scope.extensions.is_empty()
&& !path
.extension()
.and_then(|extension| extension.to_str())
.is_some_and(|extension| {
self.definition
.scope
.extensions
.iter()
.any(|allowed| extension.eq_ignore_ascii_case(allowed))
})
{
return false;
}
if self
.scope
.include
.as_ref()
.is_some_and(|include| !include.is_match(path))
|| self.scope.exclude.is_match(path)
{
return false;
}
!(self.definition.scope.skip_generated && is_generated(path)
|| self.definition.scope.skip_vendor && is_vendor(path)
|| self.definition.scope.skip_minified && is_minified(path))
}
fn findings(&self, config: &Config, source: &str) -> Vec<Finding> {
match &self.matcher {
CompiledMatcher::ExactPhrase(pattern) => source
.match_indices(pattern)
.map(|(start, matched)| self.finding(source, start..start + matched.len(), None))
.collect(),
CompiledMatcher::Regex(regex) => regex
.find_iter(source)
.map(|matched| self.finding(source, matched.range(), None))
.collect(),
CompiledMatcher::RepeatedRegex(regex) => {
let matches = regex.find_iter(source).collect::<Vec<_>>();
if matches.len() < 2 {
Vec::new()
} else {
matches
.into_iter()
.map(|matched| self.finding(source, matched.range(), None))
.collect()
}
}
CompiledMatcher::Replacement => {
let mut findings = replacement::plan(config, source)
.into_iter()
.map(|edit| self.finding(source, edit.byte_span.clone(), Some(edit)))
.collect::<Vec<_>>();
findings.extend(
replacement::plan_confusables_for_review(config, source)
.into_iter()
.map(|edit| self.finding(source, edit.byte_span, None)),
);
findings
}
CompiledMatcher::Invisible => source
.char_indices()
.filter(|(start, character)| {
unicode_review::is_gitea_invisible(*character)
&& !(*character == '\r'
&& source.as_bytes().get(start + 1) == Some(&b'\n'))
})
.map(|(start, character)| {
let mut finding =
self.finding(source, start..start + character.len_utf8(), None);
finding.help = Some(format!(
"Review the hidden or direction-changing character U+{:04X}. It is never removed automatically.",
character as u32
));
finding
})
.collect(),
}
}
fn finding(
&self,
source: &str,
byte_span: std::ops::Range<usize>,
fix: Option<crate::TextEdit>,
) -> Finding {
Finding {
rule_id: RuleId::new(self.id()),
category: self.category(),
severity: self.definition.severity,
message: self.definition.message.clone(),
help: Some(self.definition.help.clone()),
matched_text: source[byte_span.clone()].to_owned(),
byte_span,
weight: self.definition.weight,
fix,
}
}
}
impl CompiledScope {
fn compile(rule: &HeuristicRule) -> Result<Self, DetectorBuildError> {
Ok(Self {
include: if rule.scope.include.is_empty() {
None
} else {
Some(compile_globs(rule, "include", &rule.scope.include)?)
},
exclude: compile_globs(rule, "exclude", &rule.scope.exclude)?,
})
}
}
fn compile_globs(
rule: &HeuristicRule,
field: &'static str,
patterns: &[String],
) -> Result<GlobSet, DetectorBuildError> {
let mut builder = GlobSetBuilder::new();
for pattern in patterns {
builder.add(
Glob::new(pattern).map_err(|source| DetectorBuildError::InvalidGlob {
id: rule.id.clone(),
field,
pattern: pattern.clone(),
source,
})?,
);
}
builder
.build()
.map_err(|source| DetectorBuildError::InvalidGlob {
id: rule.id.clone(),
field,
pattern: patterns.join(", "),
source,
})
}
fn infer_content_kind(path: &Path) -> ContentKind {
const SOURCE_EXTENSIONS: &[&str] = &[
"c", "cc", "cpp", "cs", "go", "h", "hpp", "java", "js", "jsx", "kt", "php", "py", "rb",
"rs", "swift", "ts", "tsx",
];
if path
.extension()
.and_then(|extension| extension.to_str())
.is_some_and(|extension| {
SOURCE_EXTENSIONS
.iter()
.any(|candidate| extension.eq_ignore_ascii_case(candidate))
})
{
ContentKind::SourceCode
} else {
ContentKind::Prose
}
}
fn has_component(path: &Path, candidates: &[&str]) -> bool {
path.components().any(|component| {
component.as_os_str().to_str().is_some_and(|value| {
candidates
.iter()
.any(|candidate| value.eq_ignore_ascii_case(candidate))
})
})
}
fn is_generated(path: &Path) -> bool {
has_component(path, &["generated", "gen", "dist", "build", "target"])
}
fn is_vendor(path: &Path) -> bool {
has_component(
path,
&[
"vendor",
"vendors",
"node_modules",
"third_party",
"third-party",
],
)
}
fn is_minified(path: &Path) -> bool {
path.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| {
let name = name.to_ascii_lowercase();
name.contains(".min.") || name.ends_with("-min.js") || name.ends_with("-min.css")
})
}
/// Failure while constructing a reusable detector registry.
#[derive(Debug, thiserror::Error)]
pub enum DetectorBuildError {
/// The resolved configuration was not semantically valid.
#[error("cannot build detector from invalid configuration")]
InvalidConfig(#[source] Box<ConfigValidationError>),
/// A configured regular expression could not be compiled.
#[error("failed to compile matcher for rule `{id}`")]
InvalidRegex {
/// Stable rule identifier.
id: String,
/// Regex compiler failure.
#[source]
source: regex::Error,
},
/// A configured path applicability glob could not be compiled.
#[error("failed to compile {field} glob `{pattern}` for rule `{id}`")]
InvalidGlob {
/// Stable rule identifier.
id: String,
/// Scope field containing the glob.
field: &'static str,
/// Invalid glob source.
pattern: String,
/// Glob compiler failure.
#[source]
source: globset::Error,
},
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn registry_is_built_from_the_validated_configuration() {
let config = Config::embedded();
let detector = Detector::new(&config).unwrap();
assert_eq!(detector.rules().len(), config.rules.len());
assert_eq!(detector.rules()[0].id(), config.rules[0].id);
assert_eq!(detector.rules()[0].pattern(), config.rules[0].pattern);
assert_eq!(
detector
.rules()
.iter()
.filter(|rule| rule.is_enabled())
.count(),
config.rules.iter().filter(|rule| rule.enabled).count()
);
}
#[test]
fn replacement_rule_preserves_explainable_findings() {
let detector = Detector::new(&Config::embedded()).unwrap();
let analysis = detector.analyze("ordinary-word and suspicious—dash");
assert_eq!(analysis.findings.len(), 1);
assert_eq!(
analysis.findings[0].rule_id.as_str(),
"unicode-ascii-replacement"
);
assert_eq!(analysis.findings[0].matched_text, "—");
assert_eq!(analysis.findings[0].fix.as_ref().unwrap().replacement, "-");
assert_eq!(analysis.evidence_score.score(), 17);
assert_eq!(analysis.evidence_band, Some(EvidenceBand::BelowThreshold));
}
#[test]
fn category_and_individual_switches_are_both_required() {
let source = "In conclusion, it is complete.";
let mut config = Config::embedded();
config.diagnostics.categories.style = false;
assert!(
Detector::new(&config)
.unwrap()
.analyze(source)
.findings
.is_empty()
);
config.diagnostics.categories.style = true;
config
.rules
.iter_mut()
.find(|rule| rule.id == "templated-conclusion")
.unwrap()
.enabled = false;
assert!(
Detector::new(&config)
.unwrap()
.analyze(source)
.findings
.is_empty()
);
config
.rules
.iter_mut()
.find(|rule| rule.id == "templated-conclusion")
.unwrap()
.enabled = true;
assert_eq!(
Detector::new(&config)
.unwrap()
.analyze(source)
.findings
.len(),
1
);
}
#[test]
fn unicode_review_findings_have_exact_spans_and_safe_fixes() {
let detector = Detector::new(&Config::embedded()).unwrap();
let source = "é pаss\u{200b}word\u{202e}\r\nnext";
let findings = detector.analyze(source).findings;
assert_eq!(findings.len(), 3);
assert_eq!(findings[0].rule_id.as_str(), "unicode-ascii-replacement");
assert_eq!(findings[0].matched_text, "а");
assert_eq!(&source[findings[0].byte_span.clone()], "а");
assert_eq!(findings[0].fix.as_ref().unwrap().replacement, "a");
assert_eq!(
findings
.iter()
.filter(|finding| finding.rule_id.as_str() == "unicode-invisible-control")
.count(),
2
);
assert!(findings[1..].iter().all(|finding| finding.fix.is_none()));
assert!(findings.iter().all(|finding| finding.matched_text != "\r"));
}
#[test]
fn confusable_policy_switches_between_fix_review_and_off() {
let mut config = Config::embedded();
config.unicode.confusables = crate::ConfusablePolicy::Review;
let review = Detector::new(&config).unwrap().analyze("pаss");
assert_eq!(review.findings.len(), 1);
assert!(review.findings[0].fix.is_none());
config.unicode.confusables = crate::ConfusablePolicy::Off;
assert!(
Detector::new(&config)
.unwrap()
.analyze("pаss")
.findings
.is_empty()
);
}
#[test]
fn safe_replacement_characters_are_not_duplicated_as_confusable_findings() {
let findings = Detector::new(&Config::embedded())
.unwrap()
.analyze("one—two")
.findings;
assert_eq!(findings.len(), 1);
assert_eq!(findings[0].rule_id.as_str(), "unicode-ascii-replacement");
}
#[test]
fn compiled_regex_uses_utf8_byte_offsets() {
let regex = Regex::new(r"\u{1D306}").unwrap();
let matched = regex.find("a𝌆b").unwrap();
assert_eq!(matched.range(), 1..5);
}
#[test]
fn exact_phrase_findings_use_configured_metadata_and_byte_spans() {
let detector = Detector::new(&Config::embedded()).unwrap();
let source = "préface: As an AI language model, I cannot verify this.";
let finding = detector
.analyze(source)
.findings
.into_iter()
.find(|finding| finding.rule_id.as_str() == "assistant-self-identification")
.unwrap();
let start = source.find("As an AI language model").unwrap();
assert_eq!(finding.byte_span, start..start + finding.matched_text.len());
assert_eq!(finding.matched_text, "As an AI language model");
assert_eq!(finding.category, Category::PromptLeakage);
assert_eq!(finding.weight, 30.0);
assert!(finding.message.contains("self-identification"));
assert!(finding.help.unwrap().contains("copied"));
assert!(finding.fix.is_none());
}
#[test]
fn regex_findings_cover_the_exact_utf8_source_slice() {
let detector = Detector::new(&Config::embedded()).unwrap();
let source = "préface\nAssistant: generated response";
let finding = detector
.analyze(source)
.findings
.into_iter()
.find(|finding| finding.rule_id.as_str() == "prompt-response-scaffolding")
.unwrap();
assert_eq!(&source[finding.byte_span.clone()], finding.matched_text);
assert_eq!(finding.matched_text, "Assistant: ");
assert_eq!(finding.category, Category::PromptLeakage);
assert_eq!(finding.weight, 25.0);
}
#[test]
fn embedded_catalog_covers_each_conservative_signal_family() {
let detector = Detector::new(&Config::embedded()).unwrap();
let ids = detector
.rules()
.iter()
.map(Rule::id)
.collect::<std::collections::BTreeSet<_>>();
for expected in [
"assistant-self-identification",
"assistant-capability-disclaimer",
"assistant-answer-preface",
"knowledge-cutoff-disclaimer",
"prompt-response-scaffolding",
"templated-filler",
"templated-conclusion",
"repeated-transition",
"source-placeholder-todo",
"source-tutorial-narration",
"unicode-ascii-replacement",
] {
assert!(ids.contains(expected), "missing {expected}");
}
}
#[test]
fn repeated_transition_requires_multiple_occurrences() {
let detector = Detector::new(&Config::embedded()).unwrap();
let one = detector.analyze("Moreover, one transition is ordinary.");
let repeated =
detector.analyze("Moreover, the first claim.\nAdditionally, the second claim.");
assert!(
!one.findings
.iter()
.any(|finding| finding.rule_id.as_str() == "repeated-transition")
);
assert_eq!(
repeated
.findings
.iter()
.filter(|finding| finding.rule_id.as_str() == "repeated-transition")
.count(),
2
);
}
#[test]
fn content_and_extension_scopes_select_the_expected_rules() {
let detector = Detector::new(&Config::embedded()).unwrap();
let text = "TODO: implement the parser";
assert!(
detector
.analyze_with(text, &AnalysisContext::prose())
.findings
.is_empty()
);
assert_eq!(
detector
.analyze_with(text, &AnalysisContext::for_path("src/parser.rs"))
.findings
.len(),
1
);
assert!(
detector
.analyze_with(text, &AnalysisContext::for_path("src/parser.md"))
.findings
.is_empty()
);
}
#[test]
fn include_and_exclude_globs_are_compiled_and_applied() {
let mut config = Config::embedded();
let rule = config
.rules
.iter_mut()
.find(|rule| rule.id == "assistant-self-identification")
.unwrap();
rule.scope.include = vec!["docs/**".to_owned()];
rule.scope.exclude = vec!["docs/archive/**".to_owned()];
let detector = Detector::new(&config).unwrap();
let text = "As an AI language model, I can help.";
assert_eq!(
detector
.analyze_with(text, &AnalysisContext::for_path("docs/guide.md"))
.findings
.len(),
1
);
assert!(
detector
.analyze_with(text, &AnalysisContext::for_path("notes/guide.md"))
.findings
.is_empty()
);
assert!(
detector
.analyze_with(text, &AnalysisContext::for_path("docs/archive/guide.md"))
.findings
.is_empty()
);
assert!(detector.analyze(text).findings.is_empty());
}
#[test]
fn generated_vendor_and_minified_paths_are_skipped_by_default() {
let detector = Detector::new(&Config::embedded()).unwrap();
let text = "TODO: implement the parser";
for path in [
"generated/parser.rs",
"vendor/parser.rs",
"node_modules/parser.js",
"src/parser.min.js",
] {
assert!(
detector
.analyze_with(text, &AnalysisContext::for_path(path))
.findings
.is_empty(),
"unexpected finding for {path}"
);
}
}
#[test]
fn archive_members_use_the_inner_path_for_applicability() {
let context = AnalysisContext::for_source(&SourceId::ArchiveMember {
archive: PathBuf::from("bundle.zip"),
member: PathBuf::from("src/lib.rs"),
});
assert_eq!(context.path(), Some(Path::new("src/lib.rs")));
assert_eq!(context.content_kind(), ContentKind::SourceCode);
}
#[test]
fn evidence_score_caps_occurrences_and_applies_diminishing_returns() {
let detector = Detector::new(&Config::embedded()).unwrap();
let five = detector.analyze(&"—".repeat(5));
let twenty = detector.analyze(&"—".repeat(20));
assert_eq!(five.evidence_score, twenty.evidence_score);
assert_eq!(five.evidence_score.score(), 28);
assert_eq!(twenty.findings.len(), 20);
}
#[test]
fn evidence_score_damps_distinct_rules_in_the_same_family() {
let source = "As an AI language model, I cannot browse the internet.";
let correlated = Detector::new(&Config::embedded()).unwrap().analyze(source);
let mut independent_config = Config::embedded();
independent_config
.rules
.iter_mut()
.find(|rule| rule.id == "assistant-capability-disclaimer")
.unwrap()
.category = Category::Style;
let independent = Detector::new(&independent_config).unwrap().analyze(source);
assert!(correlated.evidence_score < independent.evidence_score);
assert_eq!(correlated.findings.len(), independent.findings.len());
}
#[test]
fn prompt_leakage_uses_lighter_family_damping_than_other_categories() {
assert_eq!(family_decay(Category::PromptLeakage), 0.90);
assert_eq!(family_decay(Category::Style), 0.75);
assert_eq!(family_decay(Category::Typography), 0.75);
assert_eq!(family_decay(Category::SourceCode), 0.75);
let source = concat!(
"User: request\n",
"Assistant: Certainly! Here's a response.\n",
"As an AI language model, I cannot browse the internet. ",
"My knowledge cutoff applies."
);
let analysis = Detector::new(&Config::embedded()).unwrap().analyze(source);
assert_eq!(analysis.findings.len(), 6);
assert_eq!(analysis.evidence_score.score(), 67);
assert_eq!(analysis.evidence_band, Some(EvidenceBand::Medium));
}
#[test]
fn configured_thresholds_define_named_band_boundaries() {
let thresholds = crate::EvidenceThresholds {
low: 20,
medium: 50,
high: 75,
};
assert_eq!(band_for_score(thresholds, 19), EvidenceBand::BelowThreshold);
assert_eq!(band_for_score(thresholds, 20), EvidenceBand::Low);
assert_eq!(band_for_score(thresholds, 50), EvidenceBand::Medium);
assert_eq!(band_for_score(thresholds, 75), EvidenceBand::High);
}
#[test]
fn one_weak_repeated_signal_cannot_reach_high_evidence() {
let detector = Detector::new(&Config::embedded()).unwrap();
let source = "Moreover, first.\nAdditionally, second.\nFurthermore, third.";
let analysis = detector.analyze(source);
assert_eq!(analysis.findings.len(), 3);
assert_ne!(analysis.evidence_band, Some(EvidenceBand::High));
}
#[test]
fn empty_and_clean_text_have_zero_evidence_and_no_verdict() {
let detector = Detector::new(&Config::embedded()).unwrap();
for source in ["", "A specific, carefully sourced statement."] {
let analysis = detector.analyze(source);
assert!(analysis.findings.is_empty());
assert_eq!(analysis.evidence_score.score(), 0);
assert_eq!(analysis.evidence_band, None);
}
}
#[test]
fn findings_are_sorted_and_identical_rule_spans_are_deduplicated() {
let detector = Detector::new(&Config::embedded()).unwrap();
let mut findings = detector.analyze("As an AI language model—").findings;
findings.reverse();
findings.push(findings[0].clone());
normalize_findings(&mut findings);
assert_eq!(findings.len(), 2);
assert!(findings[0].byte_span.start < findings[1].byte_span.start);
}
}

View File

@ -0,0 +1,156 @@
//! Rich source diagnostics derived from presentation-neutral findings.
use std::{error::Error, fmt};
use miette::{Diagnostic, LabeledSpan, NamedSource, Severity, SourceCode, SourceSpan};
use crate::{Finding, FindingSeverity, RuleId, SourceId};
/// Owned `miette` diagnostic for one finding against an exact source snapshot.
#[derive(Debug)]
pub struct FindingDiagnostic {
rule_id: RuleId,
severity: FindingSeverity,
message: String,
help: Option<String>,
matched_text: String,
source: NamedSource<String>,
span: SourceSpan,
}
impl FindingDiagnostic {
/// Create a diagnostic by binding a finding to its named source snapshot.
pub fn new(source_id: &SourceId, source: &str, finding: &Finding) -> Self {
Self {
rule_id: finding.rule_id.clone(),
severity: finding.severity,
message: finding.message.clone(),
help: finding.help.clone(),
matched_text: finding.matched_text.clone(),
source: NamedSource::new(source_id.to_string(), source.to_owned()),
span: finding.byte_span.clone().into(),
}
}
/// Return the stable rule identifier used as the diagnostic code.
pub fn rule_id(&self) -> &RuleId {
&self.rule_id
}
/// Return the configured finding severity.
pub const fn finding_severity(&self) -> FindingSeverity {
self.severity
}
/// Return the exact source span labeled by this diagnostic.
pub const fn span(&self) -> SourceSpan {
self.span
}
/// Return the named source snapshot used for rendering.
pub fn named_source(&self) -> &NamedSource<String> {
&self.source
}
}
impl fmt::Display for FindingDiagnostic {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
self.message.fmt(formatter)
}
}
impl Error for FindingDiagnostic {}
impl Diagnostic for FindingDiagnostic {
fn code(&self) -> Option<Box<dyn fmt::Display + '_>> {
Some(Box::new(&self.rule_id))
}
fn severity(&self) -> Option<Severity> {
Some(match self.severity {
FindingSeverity::Advice => Severity::Advice,
FindingSeverity::Warning => Severity::Warning,
FindingSeverity::Error => Severity::Error,
})
}
fn help(&self) -> Option<Box<dyn fmt::Display + '_>> {
self.help
.as_deref()
.map(|help| Box::new(help) as Box<dyn fmt::Display>)
}
fn source_code(&self) -> Option<&dyn SourceCode> {
Some(&self.source)
}
fn labels(&self) -> Option<Box<dyn Iterator<Item = LabeledSpan> + '_>> {
Some(Box::new(std::iter::once(
LabeledSpan::new_primary_with_span(Some(self.matched_text.clone()), self.span),
)))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{AnalysisContext, Config, Detector};
#[test]
fn finding_diagnostic_preserves_code_severity_source_span_and_help() {
let source = "préface: As an AI language model, I can help.";
let finding = Detector::new(&Config::embedded())
.unwrap()
.analyze_with(source, &AnalysisContext::for_path("article.md"))
.findings
.into_iter()
.find(|finding| finding.rule_id.as_str() == "assistant-self-identification")
.unwrap();
let diagnostic =
FindingDiagnostic::new(&SourceId::File("article.md".into()), source, &finding);
assert_eq!(
diagnostic.code().unwrap().to_string(),
finding.rule_id.as_str()
);
assert_eq!(diagnostic.severity(), Some(Severity::Warning));
assert!(diagnostic.help().unwrap().to_string().contains("copied"));
assert_eq!(diagnostic.named_source().name(), "article.md");
let label = diagnostic.labels().unwrap().next().unwrap();
assert_eq!(label.offset(), finding.byte_span.start);
assert_eq!(label.len(), finding.byte_span.len());
assert_eq!(
&source[label.offset()..label.offset() + label.len()],
finding.matched_text
);
assert_eq!(diagnostic.rule_id(), &finding.rule_id);
assert_eq!(diagnostic.finding_severity(), FindingSeverity::Warning);
assert_eq!(diagnostic.span(), finding.byte_span.clone().into());
assert_eq!(diagnostic.to_string(), finding.message);
assert!(diagnostic.source_code().is_some());
}
#[test]
fn every_finding_severity_maps_to_miette_and_help_remains_optional() {
for (finding_severity, miette_severity) in [
(FindingSeverity::Advice, Severity::Advice),
(FindingSeverity::Warning, Severity::Warning),
(FindingSeverity::Error, Severity::Error),
] {
let finding = Finding {
rule_id: RuleId::from_static("severity-test"),
category: crate::Category::Style,
severity: finding_severity,
message: "review this".into(),
help: None,
byte_span: 0..1,
matched_text: "x".into(),
weight: 1.0,
fix: None,
};
let diagnostic = FindingDiagnostic::new(&SourceId::Stdin, "x", &finding);
assert_eq!(diagnostic.severity(), Some(miette_severity));
assert!(diagnostic.help().is_none());
}
}
}

View File

@ -0,0 +1,171 @@
//! Typed failures returned by the input layer.
use std::{io, path::PathBuf, string::FromUtf8Error};
use thiserror::Error;
use crate::SourceId;
/// Failure while discovering or reading one input document.
#[derive(Debug, Error)]
pub enum InputError {
/// Reading standard input failed.
#[error("failed to read standard input")]
StdinIo {
/// Underlying operating-system error.
#[source]
source: io::Error,
},
/// The requested path does not identify a file or directory.
#[error("path `{path}` is neither a file nor a directory")]
MissingPath {
/// Requested path.
path: PathBuf,
},
/// A filesystem operation failed.
#[error("failed to {action} `{path}`")]
Io {
/// Operation being attempted.
action: &'static str,
/// Affected filesystem path.
path: PathBuf,
/// Underlying operating-system error.
#[source]
source: io::Error,
},
/// Recursive directory traversal failed for one entry.
#[error("failed while walking an input directory")]
Walk {
/// Underlying traversal error.
#[source]
source: walkdir::Error,
},
/// A ZIP container or member could not be decoded.
#[error("failed to read ZIP data from `{archive}`")]
Zip {
/// Outer archive path.
archive: PathBuf,
/// Underlying ZIP format error.
#[source]
source: zip::result::ZipError,
},
/// A source exceeded a configured byte or count limit.
#[error("{source_id} uses {actual} {resource}, exceeding the limit of {limit}")]
LimitExceeded {
/// Source that violated the limit.
source_id: SourceId,
/// Human-readable resource unit.
resource: &'static str,
/// Observed value, or `u64::MAX` after arithmetic overflow.
actual: u64,
/// Configured maximum.
limit: u64,
},
/// Archive expansion is disproportionate to compressed size.
#[error("archive member {source_id} exceeds the configured {limit}:1 compression ratio")]
CompressionRatio {
/// Archive member identity.
source_id: SourceId,
/// Maximum allowed ratio.
limit: u64,
},
/// An archive member name is unsafe to treat as a relative identity.
#[error("unsafe archive member path `{member}` in `{archive}`")]
UnsafeArchiveMember {
/// Outer archive path.
archive: PathBuf,
/// Untrusted member name from the archive.
member: String,
},
/// Encrypted members are unsupported.
#[error("archive member {source_id} is encrypted")]
EncryptedMember {
/// Encrypted archive member identity.
source_id: SourceId,
},
/// Input contained a NUL byte and is treated as binary.
#[error("input {source_id} contains a NUL byte")]
BinaryContent {
/// Source treated as binary.
source_id: SourceId,
},
/// Input was not strict UTF-8.
#[error("input {source_id} is not valid UTF-8")]
InvalidUtf8 {
/// Source that failed decoding.
source_id: SourceId,
/// Underlying UTF-8 conversion error.
#[source]
source: FromUtf8Error,
},
}
impl InputError {
/// Return the exact document identity when the failure occurred after discovery.
pub const fn source_id(&self) -> Option<&SourceId> {
match self {
Self::LimitExceeded { source_id, .. }
| Self::CompressionRatio { source_id, .. }
| Self::EncryptedMember { source_id }
| Self::BinaryContent { source_id }
| Self::InvalidUtf8 { source_id, .. } => Some(source_id),
Self::StdinIo { .. }
| Self::MissingPath { .. }
| Self::Io { .. }
| Self::Walk { .. }
| Self::Zip { .. }
| Self::UnsafeArchiveMember { .. } => None,
}
}
}
/// Result type used by the input layer.
pub type InputResult<T> = Result<T, InputError>;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn source_identity_is_available_only_for_document_specific_failures() {
let source_id = SourceId::Stdin;
for error in [
InputError::LimitExceeded {
source_id: source_id.clone(),
resource: "bytes",
actual: 2,
limit: 1,
},
InputError::CompressionRatio {
source_id: source_id.clone(),
limit: 2,
},
InputError::EncryptedMember {
source_id: source_id.clone(),
},
InputError::BinaryContent {
source_id: source_id.clone(),
},
InputError::InvalidUtf8 {
source_id: source_id.clone(),
source: String::from_utf8(vec![0xff]).unwrap_err(),
},
] {
assert_eq!(error.source_id(), Some(&source_id));
}
let error = InputError::MissingPath {
path: PathBuf::from("missing"),
};
assert_eq!(error.source_id(), None);
}
}

View File

@ -0,0 +1,426 @@
//! Streaming ZIP-based archive input.
use std::{
fs::{self, File},
io::Read,
path::{Component, Path, PathBuf},
};
use zip::ZipArchive;
use crate::{
SourceId,
config::InputLimits,
error::{InputError, InputResult},
input::Document,
};
/// Iterator over text members in one ZIP archive.
pub(super) struct ArchiveDocuments {
/// Outer archive identity.
outer: PathBuf,
/// Open ZIP reader retained between iterator calls.
archive: ZipArchive<File>,
/// Index of the next member to inspect.
index: usize,
/// Cumulative declared expanded bytes accepted so far.
expanded_bytes: u64,
/// Resource policy copied from resolved configuration.
limits: InputLimits,
}
impl std::fmt::Debug for ArchiveDocuments {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("ArchiveDocuments")
.field("outer", &self.outer)
.field("members", &self.archive.len())
.field("index", &self.index)
.finish()
}
}
impl ArchiveDocuments {
/// Open an archive for streaming member reads.
pub(super) fn open(outer: PathBuf, limits: &InputLimits) -> InputResult<Self> {
let archive_id = SourceId::File(outer.clone());
let archive_size = fs::metadata(&outer)
.map_err(|source| InputError::Io {
action: "inspect",
path: outer.clone(),
source,
})?
.len();
if archive_size > limits.max_archive_bytes {
return Err(InputError::LimitExceeded {
source_id: archive_id.clone(),
resource: "bytes",
actual: archive_size,
limit: limits.max_archive_bytes,
});
}
let file = File::open(&outer).map_err(|source| InputError::Io {
action: "open",
path: outer.clone(),
source,
})?;
let archive = ZipArchive::new(file).map_err(|source| InputError::Zip {
archive: outer.clone(),
source,
})?;
if archive.len() > limits.max_archive_entries {
return Err(InputError::LimitExceeded {
source_id: archive_id,
resource: "entries",
actual: u64::try_from(archive.len()).unwrap_or(u64::MAX),
limit: u64::try_from(limits.max_archive_entries).unwrap_or(u64::MAX),
});
}
Ok(Self {
outer,
archive,
index: 0,
expanded_bytes: 0,
limits: limits.clone(),
})
}
}
impl Iterator for ArchiveDocuments {
type Item = InputResult<Document>;
fn next(&mut self) -> Option<Self::Item> {
while self.index < self.archive.len() {
let index = self.index;
self.index += 1;
let mut member = match self
.archive
.by_index(index)
.map_err(|source| InputError::Zip {
archive: self.outer.clone(),
source,
}) {
Ok(member) => member,
Err(error) => return Some(Err(error)),
};
if member.is_dir() {
continue;
}
let name = match safe_member_path(&self.outer, member.name()) {
Ok(name) => name,
Err(error) => return Some(Err(error)),
};
let source_id = SourceId::ArchiveMember {
archive: self.outer.clone(),
member: name.clone(),
};
if member.encrypted() {
return Some(Err(InputError::EncryptedMember { source_id }));
}
let expanded_size = member.size();
if expanded_size > self.limits.max_document_bytes {
return Some(Err(InputError::LimitExceeded {
source_id,
resource: "bytes",
actual: expanded_size,
limit: self.limits.max_document_bytes,
}));
}
let compressed_size = member.compressed_size();
let allowed_expanded =
compressed_size.saturating_mul(self.limits.max_compression_ratio);
if expanded_size > 0 && expanded_size > allowed_expanded {
return Some(Err(InputError::CompressionRatio {
source_id,
limit: self.limits.max_compression_ratio,
}));
}
self.expanded_bytes = match self.expanded_bytes.checked_add(expanded_size) {
Some(total) if total <= self.limits.max_expanded_bytes => total,
_ => {
return Some(Err(InputError::LimitExceeded {
source_id: SourceId::File(self.outer.clone()),
resource: "expanded bytes",
actual: u64::MAX,
limit: self.limits.max_expanded_bytes,
}));
}
};
let read_limit = self.limits.max_document_bytes.saturating_add(1);
let mut bytes = Vec::new();
if let Err(source) = member.by_ref().take(read_limit).read_to_end(&mut bytes) {
return Some(Err(InputError::Io {
action: "read archive member from",
path: self.outer.clone(),
source,
}));
}
if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > self.limits.max_document_bytes {
return Some(Err(InputError::LimitExceeded {
source_id,
resource: "bytes",
actual: u64::try_from(bytes.len()).unwrap_or(u64::MAX),
limit: self.limits.max_document_bytes,
}));
}
if bytes.contains(&0) {
return Some(Err(InputError::BinaryContent { source_id }));
}
let content = match String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 {
source_id: source_id.clone(),
source,
}) {
Ok(content) => content,
Err(error) => return Some(Err(error)),
};
return Some(Ok(Document {
source: SourceId::ArchiveMember {
archive: self.outer.clone(),
member: name,
},
content,
}));
}
None
}
}
/// Parse an archive name into a safe relative member identity.
fn safe_member_path(archive: &Path, name: &str) -> InputResult<PathBuf> {
let normalized = name.replace('\\', "/");
let path = Path::new(&normalized);
if normalized.is_empty()
|| path
.components()
.any(|component| !matches!(component, Component::Normal(_)))
{
return Err(InputError::UnsafeArchiveMember {
archive: archive.to_path_buf(),
member: name.to_owned(),
});
}
Ok(path.to_path_buf())
}
#[cfg(test)]
mod tests {
use std::io::Write;
use tempfile::tempdir;
use zip::write::FileOptions;
use super::*;
fn create_archive(path: &Path, entries: &[(&str, &[u8])], compression: zip::CompressionMethod) {
let file = File::create(path).unwrap();
let mut writer = zip::ZipWriter::new(file);
let options: FileOptions<()> = FileOptions::default().compression_method(compression);
for (name, bytes) in entries {
writer.start_file(*name, options).unwrap();
writer.write_all(bytes).unwrap();
}
writer.finish().unwrap();
}
#[test]
fn rejects_absolute_and_traversing_member_paths() {
for path in [
"../escape.txt",
"/absolute.txt",
"a/../../escape",
r"a\..\escape",
] {
assert!(
safe_member_path(Path::new("archive.zip"), path).is_err(),
"accepted {path}"
);
}
assert_eq!(
safe_member_path(Path::new("archive.zip"), "safe/member.txt").unwrap(),
Path::new("safe/member.txt")
);
assert!(safe_member_path(Path::new("archive.zip"), "").is_err());
}
#[test]
fn opening_archives_reports_filesystem_format_size_and_entry_limits() {
let directory = tempdir().unwrap();
let missing = directory.path().join("missing.zip");
assert!(matches!(
ArchiveDocuments::open(missing.clone(), &InputLimits::default()),
Err(InputError::Io { action: "inspect", path, .. }) if path == missing
));
let malformed = directory.path().join("malformed.zip");
fs::write(&malformed, b"not a zip").unwrap();
assert!(matches!(
ArchiveDocuments::open(malformed.clone(), &InputLimits::default()),
Err(InputError::Zip { archive, .. }) if archive == malformed
));
let path = directory.path().join("limits.zip");
create_archive(
&path,
&[("one.txt", b"one"), ("two.txt", b"two")],
zip::CompressionMethod::Stored,
);
let limits = InputLimits {
max_archive_bytes: 1,
..InputLimits::default()
};
assert!(matches!(
ArchiveDocuments::open(path.clone(), &limits),
Err(InputError::LimitExceeded {
resource: "bytes",
..
})
));
let limits = InputLimits {
max_archive_bytes: u64::MAX,
max_archive_entries: 1,
..InputLimits::default()
};
assert!(matches!(
ArchiveDocuments::open(path, &limits),
Err(InputError::LimitExceeded {
resource: "entries",
actual: 2,
limit: 1,
..
})
));
}
#[test]
fn iterator_skips_directories_and_preserves_safe_member_identity() {
let directory = tempdir().unwrap();
let path = directory.path().join("documents.zip");
let file = File::create(&path).unwrap();
let mut writer = zip::ZipWriter::new(file);
let options: FileOptions<()> = FileOptions::default();
writer.add_directory("docs/", options).unwrap();
writer.start_file("docs/article.txt", options).unwrap();
writer.write_all("héllo".as_bytes()).unwrap();
writer.finish().unwrap();
let mut archive = ArchiveDocuments::open(path.clone(), &InputLimits::default()).unwrap();
let debug = format!("{archive:?}");
assert!(debug.contains("members: 2"));
let document = archive.next().unwrap().unwrap();
assert_eq!(document.content, "héllo");
assert_eq!(
document.source,
SourceId::ArchiveMember {
archive: path,
member: PathBuf::from("docs/article.txt"),
}
);
assert!(archive.next().is_none());
}
#[test]
fn iterator_rejects_unsafe_binary_and_non_utf8_members_independently() {
for (name, bytes, expected) in [
("../escape.txt", b"text".as_slice(), "unsafe"),
("binary.txt", b"a\0b".as_slice(), "binary"),
("invalid.txt", [0xff].as_slice(), "utf8"),
] {
let directory = tempdir().unwrap();
let path = directory.path().join("invalid-member.zip");
create_archive(&path, &[(name, bytes)], zip::CompressionMethod::Stored);
let error = ArchiveDocuments::open(path, &InputLimits::default())
.unwrap()
.next()
.unwrap()
.unwrap_err();
assert!(
matches!(
(expected, error),
("unsafe", InputError::UnsafeArchiveMember { .. })
| ("binary", InputError::BinaryContent { .. })
| ("utf8", InputError::InvalidUtf8 { .. })
),
"wrong error for {name}"
);
}
}
#[test]
fn iterator_enforces_member_expansion_and_compression_limits() {
let directory = tempdir().unwrap();
let oversized = directory.path().join("oversized.zip");
create_archive(
&oversized,
&[("large.txt", b"12345")],
zip::CompressionMethod::Stored,
);
let limits = InputLimits {
max_document_bytes: 4,
..InputLimits::default()
};
let error = ArchiveDocuments::open(oversized, &limits)
.unwrap()
.next()
.unwrap()
.unwrap_err();
assert!(matches!(
error,
InputError::LimitExceeded {
resource: "bytes",
actual: 5,
limit: 4,
..
}
));
let compressed = directory.path().join("compressed.zip");
create_archive(
&compressed,
&[("repeated.txt", &[b'a'; 4096])],
zip::CompressionMethod::Deflated,
);
let limits = InputLimits {
max_compression_ratio: 1,
..InputLimits::default()
};
assert!(matches!(
ArchiveDocuments::open(compressed, &limits)
.unwrap()
.next()
.unwrap(),
Err(InputError::CompressionRatio { limit: 1, .. })
));
let cumulative = directory.path().join("cumulative.zip");
create_archive(
&cumulative,
&[("one.txt", b"1234"), ("two.txt", b"5678")],
zip::CompressionMethod::Stored,
);
let limits = InputLimits {
max_expanded_bytes: 6,
..InputLimits::default()
};
let mut archive = ArchiveDocuments::open(cumulative, &limits).unwrap();
assert!(archive.next().unwrap().is_ok());
assert!(matches!(
archive.next().unwrap(),
Err(InputError::LimitExceeded {
resource: "expanded bytes",
actual: u64::MAX,
limit: 6,
..
})
));
}
}

View File

@ -0,0 +1,37 @@
//! Regular-file input support.
use std::{fs, path::Path};
use crate::{
SourceId,
config::InputLimits,
error::{InputError, InputResult},
};
/// Read a regular file as strict UTF-8 text.
pub(super) fn read_text(path: &Path, limits: &InputLimits) -> InputResult<String> {
let source_id = SourceId::File(path.to_path_buf());
let metadata = fs::metadata(path).map_err(|source| InputError::Io {
action: "inspect",
path: path.to_path_buf(),
source,
})?;
if metadata.len() > limits.max_document_bytes {
return Err(InputError::LimitExceeded {
source_id,
resource: "bytes",
actual: metadata.len(),
limit: limits.max_document_bytes,
});
}
let bytes = fs::read(path).map_err(|source| InputError::Io {
action: "read",
path: path.to_path_buf(),
source,
})?;
if bytes.contains(&0) {
return Err(InputError::BinaryContent { source_id });
}
String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 { source_id, source })
}

View File

@ -0,0 +1,127 @@
//! Streaming input acquisition for files, directories, and ZIP containers.
#[cfg(test)]
mod tests;
mod archive;
mod file;
mod walk;
use std::{io::Read, path::Path, vec};
use crate::{
SourceId,
config::InputLimits,
error::{InputError, InputResult},
};
use self::{archive::ArchiveDocuments, walk::WalkDocuments};
/// One strict UTF-8 source document.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Document {
/// Stable identity of the file or archive member.
pub source: SourceId,
/// Complete UTF-8 content of this document.
pub content: String,
}
/// Streaming documents produced from one requested path.
#[derive(Debug)]
pub struct Documents {
/// Input-specific iterator implementation.
inner: DocumentsInner,
}
/// Private iterator variants hidden behind the stable public wrapper.
#[derive(Debug)]
enum DocumentsInner {
/// One regular-file result.
File(vec::IntoIter<InputResult<Document>>),
/// Recursive directory results.
Directory(WalkDocuments),
/// ZIP member results.
Archive(ArchiveDocuments),
}
impl Iterator for Documents {
type Item = InputResult<Document>;
fn next(&mut self) -> Option<Self::Item> {
match &mut self.inner {
DocumentsInner::File(document) => document.next(),
DocumentsInner::Directory(documents) => documents.next(),
DocumentsInner::Archive(documents) => documents.next(),
}
}
}
/// Open a file, directory, or recognized ZIP-based container for streaming.
pub fn read(path: &Path, limits: &InputLimits) -> InputResult<Documents> {
if path.is_file() {
if has_zip_extension(path) {
return ArchiveDocuments::open(path.to_path_buf(), limits).map(|archive| Documents {
inner: DocumentsInner::Archive(archive),
});
}
let result = file::read_text(path, limits).map(|content| Document {
source: SourceId::File(path.to_path_buf()),
content,
});
return Ok(Documents {
inner: DocumentsInner::File(vec![result].into_iter()),
});
}
if path.is_dir() {
return Ok(Documents {
inner: DocumentsInner::Directory(WalkDocuments::new(path, limits)),
});
}
Err(InputError::MissingPath {
path: path.to_path_buf(),
})
}
/// Read standard input as one bounded strict UTF-8 document.
pub fn read_stdin(reader: impl Read, limits: &InputLimits) -> InputResult<Document> {
let source_id = SourceId::Stdin;
let mut bytes = Vec::new();
reader
.take(limits.max_document_bytes.saturating_add(1))
.read_to_end(&mut bytes)
.map_err(|source| InputError::StdinIo { source })?;
if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > limits.max_document_bytes {
return Err(InputError::LimitExceeded {
source_id,
resource: "bytes",
actual: u64::try_from(bytes.len()).unwrap_or(u64::MAX),
limit: limits.max_document_bytes,
});
}
if bytes.contains(&0) {
return Err(InputError::BinaryContent { source_id });
}
let content = String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 {
source_id: source_id.clone(),
source,
})?;
Ok(Document {
source: source_id,
content,
})
}
/// Whether a path has an extension treated as a ZIP container.
fn has_zip_extension(path: &Path) -> bool {
path.extension()
.and_then(|extension| extension.to_str())
.is_some_and(|extension| {
matches!(
extension.to_ascii_lowercase().as_str(),
"zip" | "docx" | "xlsx" | "pptx"
)
})
}

View File

@ -0,0 +1,197 @@
use std::{
fs::{self, File},
io::Write,
path::Path,
};
use tempfile::tempdir;
use zip::write::FileOptions;
use super::*;
struct FailingReader;
impl std::io::Read for FailingReader {
fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {
Err(std::io::Error::other("fixture read failure"))
}
}
fn limits() -> InputLimits {
InputLimits::default()
}
fn write_file(path: &Path, content: &str) {
let mut file = File::create(path).expect("create fixture");
file.write_all(content.as_bytes()).expect("write fixture");
}
fn create_zip(path: &Path) -> std::io::Result<()> {
let file = File::create(path)?;
let mut zip = zip::ZipWriter::new(file);
let options: FileOptions<()> = FileOptions::default()
.compression_method(zip::CompressionMethod::Deflated)
.unix_permissions(0o644);
zip.start_file("foo.txt", options)?;
zip.write_all(b"hello foo")?;
zip.start_file("bar.txt", options)?;
zip.write_all(b"hello bar")?;
zip.finish()?;
Ok(())
}
#[test]
fn reads_one_file() {
let directory = tempdir().expect("temp dir");
let path = directory.path().join("test.txt");
write_file(&path, "line1\nline2");
let documents = read(&path, &limits())
.expect("open input")
.collect::<InputResult<Vec<_>>>()
.expect("read input");
assert_eq!(documents.len(), 1);
assert_eq!(documents[0].source, SourceId::File(path));
assert_eq!(documents[0].content, "line1\nline2");
}
#[test]
fn yields_a_directory_error_without_discarding_other_files() {
let directory = tempdir().expect("temp dir");
write_file(&directory.path().join("a.txt"), "a");
fs::write(directory.path().join("b.txt"), [0xff]).expect("write invalid fixture");
write_file(&directory.path().join("c.txt"), "c");
let results = read(directory.path(), &limits())
.expect("open directory")
.collect::<Vec<_>>();
assert_eq!(results.len(), 3);
assert_eq!(results.iter().filter(|result| result.is_ok()).count(), 2);
assert_eq!(results.iter().filter(|result| result.is_err()).count(), 1);
}
#[test]
fn streams_archive_members_with_outer_identity() {
let directory = tempdir().expect("temp dir");
let path = directory.path().join("archive.zip");
create_zip(&path).expect("create zip");
let documents = read(&path, &limits())
.expect("open archive")
.collect::<InputResult<Vec<_>>>()
.expect("read archive");
assert_eq!(documents.len(), 2);
assert!(documents.iter().all(|document| matches!(
&document.source,
SourceId::ArchiveMember { archive, .. } if archive == &path
)));
}
#[test]
fn invalid_utf8_is_a_per_document_error() {
let directory = tempdir().expect("temp dir");
let path = directory.path().join("invalid.txt");
fs::write(&path, [0xff, 0xfe]).expect("write fixture");
let mut documents = read(&path, &limits()).expect("open input");
assert!(documents.next().expect("one result").is_err());
assert!(documents.next().is_none());
}
#[test]
fn binary_content_is_an_input_error_not_a_clean_document() {
let directory = tempdir().expect("temp dir");
let path = directory.path().join("binary.txt");
fs::write(&path, b"clean prefix\0binary suffix").expect("write fixture");
let mut documents = read(&path, &limits()).expect("open input");
let error = documents.next().expect("one result").unwrap_err();
assert!(matches!(
error,
InputError::BinaryContent {
source_id: SourceId::File(error_path)
} if error_path == path
));
}
#[test]
fn missing_path_is_rejected() {
let directory = tempdir().expect("temp dir");
let path = directory.path().join("missing.txt");
let error = read(&path, &limits()).expect_err("missing input should fail");
assert!(matches!(error, InputError::MissingPath { path: error_path } if error_path == path));
}
#[test]
fn enforces_document_size_limit_before_reading() {
let directory = tempdir().expect("temp dir");
let path = directory.path().join("large.txt");
write_file(&path, "too large");
let mut limits = limits();
limits.max_document_bytes = 3;
let mut documents = read(&path, &limits).expect("open input");
let error = documents
.next()
.expect("one result")
.expect_err("oversized input");
assert!(matches!(
error,
InputError::LimitExceeded {
actual: 9,
limit: 3,
..
}
));
}
#[test]
fn stdin_is_bounded_and_decoded_as_strict_utf8() {
let document = read_stdin("hello".as_bytes(), &limits()).unwrap();
assert_eq!(document.source, SourceId::Stdin);
assert_eq!(document.content, "hello");
let mut small = limits();
small.max_document_bytes = 3;
assert!(matches!(
read_stdin("hello".as_bytes(), &small),
Err(InputError::LimitExceeded {
source_id: SourceId::Stdin,
..
})
));
assert!(matches!(
read_stdin([0xff].as_slice(), &limits()),
Err(InputError::InvalidUtf8 {
source_id: SourceId::Stdin,
..
})
));
assert!(matches!(
read_stdin(b"nul\0byte".as_slice(), &limits()),
Err(InputError::BinaryContent {
source_id: SourceId::Stdin
})
));
assert!(matches!(
read_stdin(FailingReader, &limits()),
Err(InputError::StdinIo { .. })
));
}
#[test]
fn regular_file_metadata_failures_retain_operation_context() {
let directory = tempdir().expect("temp dir");
let missing = directory.path().join("missing.txt");
assert!(matches!(
file::read_text(&missing, &limits()),
Err(InputError::Io { action: "inspect", path, .. }) if path == missing
));
}

View File

@ -0,0 +1,62 @@
//! Streaming directory traversal.
use std::path::Path;
use walkdir::{IntoIter, WalkDir};
use crate::{
SourceId,
config::InputLimits,
error::{InputError, InputResult},
input::Document,
};
/// Iterator over readable regular files below a directory.
#[derive(Debug)]
pub(super) struct WalkDocuments {
/// Underlying streaming directory iterator.
entries: IntoIter,
/// Limits applied to every yielded file.
limits: InputLimits,
}
impl WalkDocuments {
/// Begin a recursive walk without following symbolic links.
pub(super) fn new(root: &Path, limits: &InputLimits) -> Self {
Self {
entries: WalkDir::new(root)
.follow_links(false)
.max_depth(limits.max_walk_depth)
.sort_by_file_name()
.into_iter(),
limits: limits.clone(),
}
}
}
impl Iterator for WalkDocuments {
type Item = InputResult<Document>;
fn next(&mut self) -> Option<Self::Item> {
loop {
let entry = match self.entries.next()? {
Ok(entry) => entry,
Err(source) => return Some(Err(InputError::Walk { source })),
};
if !entry.file_type().is_file() {
continue;
}
let path = entry.into_path();
let content = match super::file::read_text(&path, &self.limits) {
Ok(content) => content,
Err(error) => return Some(Err(error)),
};
return Some(Ok(Document {
source: SourceId::File(path),
content,
}));
}
}
}

View File

@ -0,0 +1,106 @@
//! Shared analysis library for John Slop's command-line, formatter, and LSP
//! adapters.
#![warn(missing_docs)]
use std::path::Path;
mod config;
mod detector;
mod diagnostic;
mod error;
mod input;
mod model;
mod replacement;
mod unicode_review;
pub use config::{
Config, ConfigError, ConfigLayerError, ConfigLoader, ConfigName, ConfigNameError,
ConfigOverrides, ConfigPaths, ConfigValidationError, ConfusablePolicy, ContentKind,
DiagnosticCategories, DiagnosticPolicy, EvidenceThresholds, HeuristicRule, InputLimits,
InputOverrides, MatcherType, ReplacementRule, RuleScope, ThresholdOverrides, UnicodePolicy,
default_config_template, default_diagnostics_template,
};
pub use detector::{AnalysisContext, Detector, DetectorBuildError, Rule};
pub use diagnostic::FindingDiagnostic;
pub use error::{InputError, InputResult};
pub use input::{Document, Documents, read_stdin};
pub use model::{
Analysis, AnalysisSummary, Category, EvidenceBand, EvidenceScore, Finding, FindingSeverity,
RuleId, SeverityCounts, SourceId, TextEdit,
};
/// Deprecated name for [`EvidenceScore`].
#[deprecated(since = "0.1.0", note = "use EvidenceScore")]
pub type Confidence = EvidenceScore;
/// Deprecated name for [`EvidenceBand`].
#[deprecated(since = "0.1.0", note = "use EvidenceBand")]
pub type ConfidenceBand = EvidenceBand;
/// Deprecated name for [`EvidenceThresholds`].
#[deprecated(since = "0.1.0", note = "use EvidenceThresholds")]
pub type ConfidenceThresholds = EvidenceThresholds;
pub use replacement::ReplacementError;
/// Load embedded defaults plus optional platform user configuration.
pub fn load_config() -> Result<Config, ConfigError> {
ConfigLoader::discover(None, None)?.load()
}
/// Analyze a UTF-8 source snapshot with the resolved configuration.
pub fn analyze_source(config: &Config, source: &str) -> Analysis {
Detector::new(config)
.expect("validated configuration must build a detector")
.analyze(source)
}
/// Plan safe ASCII replacements without modifying the source.
pub fn plan_replacements(config: &Config, source: &str) -> Vec<TextEdit> {
replacement::plan(config, source)
}
/// Validate and apply text edits against their original UTF-8 source snapshot.
pub fn apply_replacements<'source>(
source: &'source str,
edits: &[TextEdit],
) -> Result<std::borrow::Cow<'source, str>, ReplacementError> {
replacement::apply(source, edits)
}
/// Read documents from a file, directory, or supported ZIP-based container.
pub fn read_inputs(config: &Config, path: &Path) -> InputResult<Documents> {
input::read(path, &config.input)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn facade_uses_one_configuration_for_analysis_and_edits() {
let config = load_config().unwrap();
let source = "review—this";
let analysis = analyze_source(&config, source);
let edits = plan_replacements(&config, source);
assert_eq!(analysis.findings.len(), 1);
assert_eq!(analysis.findings[0].fix.as_ref(), edits.first());
}
#[test]
#[allow(deprecated)]
fn legacy_score_type_aliases_remain_source_compatible() {
let score: Confidence = EvidenceScore::from_score(42);
let band: ConfidenceBand = EvidenceBand::Medium;
let thresholds: ConfidenceThresholds = EvidenceThresholds {
low: 20,
medium: 50,
high: 75,
};
assert_eq!(score.score(), 42);
assert_eq!(band, EvidenceBand::Medium);
assert_eq!(thresholds.low, 20);
}
}

View File

@ -0,0 +1,377 @@
//! Presentation-neutral domain values shared by analysis adapters.
use std::{fmt, ops::Range, path::PathBuf};
use serde::{Deserialize, Serialize};
/// Stable identifier for a detection rule.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct RuleId(String);
impl RuleId {
/// Construct a rule identifier from validated configuration text.
pub fn new(value: impl Into<String>) -> Self {
Self(value.into())
}
/// Construct a rule identifier from a static, trusted value.
pub fn from_static(value: &'static str) -> Self {
Self(value.to_owned())
}
/// Return the identifier as text.
pub fn as_str(&self) -> &str {
&self.0
}
}
impl fmt::Display for RuleId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
self.0.fmt(formatter)
}
}
/// Broad family of evidence reported by a rule.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum Category {
/// Suspicious typography or character usage.
Typography,
/// Repeated or templated writing style.
Style,
/// Prompt, response, or assistant scaffolding left in the source.
PromptLeakage,
/// Low-care patterns specific to source code.
SourceCode,
}
impl Category {
/// Return the stable kebab-case presentation name.
pub const fn as_str(self) -> &'static str {
match self {
Self::Typography => "typography",
Self::Style => "style",
Self::PromptLeakage => "prompt-leakage",
Self::SourceCode => "source-code",
}
}
}
impl fmt::Display for Category {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.as_str())
}
}
/// Review severity attached to an individual configured finding.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum FindingSeverity {
/// Informational signal worth considering during review.
Advice,
/// Stronger signal that should be reviewed before acceptance.
Warning,
/// Definite policy violation under the selected configuration.
Error,
}
impl FindingSeverity {
/// Return the stable lowercase presentation name.
pub const fn as_str(self) -> &'static str {
match self {
Self::Advice => "advice",
Self::Warning => "warning",
Self::Error => "error",
}
}
}
impl fmt::Display for FindingSeverity {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.as_str())
}
}
/// Bounded strength of the configured low-care evidence found in a document.
///
/// This is deliberately not an AI-authorship probability.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord)]
pub struct EvidenceScore(u8);
impl EvidenceScore {
/// Construct an evidence score, clamping values above 100.
pub const fn from_score(score: u8) -> Self {
Self(if score > 100 { 100 } else { score })
}
/// Return the normalized score in the inclusive range `0..=100`.
pub const fn score(self) -> u8 {
self.0
}
}
/// Named interpretation of a non-empty analysis score.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum EvidenceBand {
/// Evidence exists but remains below the configured low threshold.
BelowThreshold,
/// Score meets the configured low threshold.
Low,
/// Score meets the configured medium threshold.
Medium,
/// Score meets the configured high threshold.
High,
}
impl EvidenceBand {
/// Return the stable kebab-case presentation name.
pub const fn as_str(self) -> &'static str {
match self {
Self::BelowThreshold => "below-threshold",
Self::Low => "low",
Self::Medium => "medium",
Self::High => "high",
}
}
}
impl fmt::Display for EvidenceBand {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter.write_str(self.as_str())
}
}
/// Identity of the exact source snapshot being analyzed.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SourceId {
/// Content supplied through standard input.
Stdin,
/// A regular file on disk.
File(PathBuf),
/// A member contained in an archive file.
ArchiveMember {
/// Path of the outer archive.
archive: PathBuf,
/// Path-like member name stored in the archive.
member: PathBuf,
},
}
impl fmt::Display for SourceId {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Stdin => formatter.write_str("<stdin>"),
Self::File(path) => path.display().fmt(formatter),
Self::ArchiveMember { archive, member } => {
write!(formatter, "{}::{}", archive.display(), member.display())
}
}
}
}
/// A non-overlapping replacement against an analyzed source snapshot.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TextEdit {
/// Zero-based, half-open UTF-8 byte range to replace.
pub byte_span: Range<usize>,
/// Text to insert in place of the range.
pub replacement: String,
}
/// One explainable piece of evidence produced by a detection rule.
#[derive(Debug, Clone, PartialEq)]
pub struct Finding {
/// Stable rule identifier.
pub rule_id: RuleId,
/// Evidence family.
pub category: Category,
/// Configured review severity.
pub severity: FindingSeverity,
/// Concise description of the observed signal.
pub message: String,
/// Optional remediation or review guidance.
pub help: Option<String>,
/// Zero-based, half-open UTF-8 byte range in the analyzed source.
pub byte_span: Range<usize>,
/// Exact source text covered by `byte_span`.
pub matched_text: String,
/// Configured evidence weight before aggregate normalization.
pub weight: f64,
/// Optional safe replacement for this finding.
pub fix: Option<TextEdit>,
}
/// Complete deterministic analysis of one source snapshot.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct Analysis {
/// Findings sorted by source offset and rule identifier.
pub findings: Vec<Finding>,
/// Aggregate strength of the configured evidence found.
pub evidence_score: EvidenceScore,
/// Named score band, or no verdict when no evidence was found.
pub evidence_band: Option<EvidenceBand>,
}
impl Analysis {
/// Summarize score, named band, and severity counts without consuming findings.
pub fn summary(&self) -> AnalysisSummary {
AnalysisSummary {
evidence_score: self.evidence_score.score(),
evidence_band: self.evidence_band,
findings: SeverityCounts::from_findings(&self.findings),
}
}
}
/// Compact deterministic summary of one completed analysis.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct AnalysisSummary {
/// Normalized evidence score in `0..=100`.
pub evidence_score: u8,
/// Named band, or no verdict when the source has no evidence.
pub evidence_band: Option<EvidenceBand>,
/// Finding totals grouped by configured severity.
pub findings: SeverityCounts,
}
/// Finding counts grouped by configured presentation severity.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct SeverityCounts {
/// Informational findings.
pub advice: usize,
/// Warning findings.
pub warnings: usize,
/// Error findings.
pub errors: usize,
}
impl SeverityCounts {
/// Return the total number of findings across severities.
pub const fn total(self) -> usize {
self.advice + self.warnings + self.errors
}
fn from_findings(findings: &[Finding]) -> Self {
let mut counts = Self::default();
for finding in findings {
match finding.severity {
FindingSeverity::Advice => counts.advice += 1,
FindingSeverity::Warning => counts.warnings += 1,
FindingSeverity::Error => counts.errors += 1,
}
}
counts
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn evidence_score_is_bounded() {
assert_eq!(EvidenceScore::from_score(0).score(), 0);
assert_eq!(EvidenceScore::from_score(73).score(), 73);
assert_eq!(EvidenceScore::from_score(255).score(), 100);
}
#[test]
fn source_id_display_distinguishes_all_source_kinds() {
assert_eq!(SourceId::Stdin.to_string(), "<stdin>");
assert_eq!(
SourceId::File(PathBuf::from("article.md")).to_string(),
"article.md"
);
assert_eq!(
SourceId::ArchiveMember {
archive: PathBuf::from("bundle.zip"),
member: PathBuf::from("docs/article.md"),
}
.to_string(),
"bundle.zip::docs/article.md"
);
}
#[test]
fn rule_id_has_stable_text_and_ordering() {
let first = RuleId::new("alpha-rule");
let second = RuleId::from_static("beta-rule");
assert_eq!(first.as_str(), "alpha-rule");
assert_eq!(first.to_string(), "alpha-rule");
assert!(first < second);
}
#[test]
fn analysis_summary_counts_severity_without_removing_findings() {
let analysis = Analysis {
findings: vec![
Finding {
rule_id: RuleId::from_static("advice-rule"),
category: Category::Style,
severity: FindingSeverity::Advice,
message: "advice".to_owned(),
help: None,
byte_span: 0..1,
matched_text: "a".to_owned(),
weight: 1.0,
fix: None,
},
Finding {
rule_id: RuleId::from_static("warning-rule"),
category: Category::PromptLeakage,
severity: FindingSeverity::Warning,
message: "warning".to_owned(),
help: None,
byte_span: 2..3,
matched_text: "b".to_owned(),
weight: 10.0,
fix: None,
},
],
evidence_score: EvidenceScore::from_score(32),
evidence_band: Some(EvidenceBand::Low),
};
let summary = analysis.summary();
assert_eq!(analysis.findings.len(), 2);
assert_eq!(summary.evidence_score, 32);
assert_eq!(summary.evidence_band, Some(EvidenceBand::Low));
assert_eq!(summary.findings.advice, 1);
assert_eq!(summary.findings.warnings, 1);
assert_eq!(summary.findings.errors, 0);
assert_eq!(summary.findings.total(), analysis.findings.len());
}
#[test]
fn enum_names_are_stable_for_display_and_serialization() {
for (category, expected) in [
(Category::Typography, "typography"),
(Category::Style, "style"),
(Category::PromptLeakage, "prompt-leakage"),
(Category::SourceCode, "source-code"),
] {
assert_eq!(category.as_str(), expected);
assert_eq!(category.to_string(), expected);
}
for (severity, expected) in [
(FindingSeverity::Advice, "advice"),
(FindingSeverity::Warning, "warning"),
(FindingSeverity::Error, "error"),
] {
assert_eq!(severity.as_str(), expected);
assert_eq!(severity.to_string(), expected);
}
for (band, expected) in [
(EvidenceBand::BelowThreshold, "below-threshold"),
(EvidenceBand::Low, "low"),
(EvidenceBand::Medium, "medium"),
(EvidenceBand::High, "high"),
] {
assert_eq!(band.as_str(), expected);
assert_eq!(band.to_string(), expected);
}
}
}

View File

@ -0,0 +1,367 @@
//! Pure planning and application of configured text replacements.
use std::{borrow::Cow, collections::BTreeMap, ops::Range};
use miette::Diagnostic;
use crate::{Config, ConfusablePolicy, ReplacementRule, TextEdit};
/// An edit set cannot be applied safely to its source snapshot.
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error, Diagnostic)]
#[diagnostic(code(john_slop::replacement::invalid_edit))]
pub enum ReplacementError {
/// A range is empty, reversed, or extends beyond the source.
#[error("replacement byte range {start}..{end} is invalid for a {source_len}-byte source")]
InvalidRange {
/// Inclusive start byte supplied by the edit.
start: usize,
/// Exclusive end byte supplied by the edit.
end: usize,
/// Source snapshot length in bytes.
source_len: usize,
},
/// A range endpoint does not lie on a UTF-8 character boundary.
#[error("replacement byte offset {offset} is not a UTF-8 character boundary")]
InvalidCharacterBoundary {
/// Invalid endpoint.
offset: usize,
},
/// Two edits cover at least one common source byte.
#[error("replacement ranges overlap at byte {offset}")]
OverlappingEdits {
/// Start of the later edit inside an earlier edit.
offset: usize,
},
}
/// Plan deterministic non-overlapping edits without modifying the source.
pub fn plan(config: &Config, source: &str) -> Vec<TextEdit> {
plan_matching(config, source, |rule| {
rule.enabled
&& (!is_bulk_confusable(rule)
|| config.unicode.confusables == ConfusablePolicy::Replace)
})
}
pub(crate) fn plan_confusables_for_review(config: &Config, source: &str) -> Vec<TextEdit> {
if config.unicode.confusables != ConfusablePolicy::Review {
return Vec::new();
}
plan_matching(config, source, |rule| {
rule.enabled && is_bulk_confusable(rule)
})
}
fn is_bulk_confusable(rule: &ReplacementRule) -> bool {
let mut characters = rule.source.chars();
let is_fullwidth_scalar =
matches!(characters.next(), Some('\u{ff01}'..='\u{ff5e}')) && characters.next().is_none();
rule.id.starts_with("uts39-") && !is_fullwidth_scalar
}
fn plan_matching(
config: &Config,
source: &str,
include: impl Fn(&ReplacementRule) -> bool,
) -> Vec<TextEdit> {
let mut single_character_rules = BTreeMap::new();
let mut sequence_rules = Vec::new();
for rule in config.replacements.iter().filter(|rule| include(rule)) {
let mut characters = rule.source.chars();
let first = characters.next();
if let (Some(character), None) = (first, characters.next()) {
single_character_rules.insert(character, rule);
} else {
sequence_rules.push(rule);
}
}
sequence_rules.sort_by(|left, right| {
right
.source
.len()
.cmp(&left.source.len())
.then_with(|| left.id.cmp(&right.id))
});
let mut edits = Vec::new();
let mut offset = 0;
while offset < source.len() {
if let Some(rule) = sequence_rules
.iter()
.find(|rule| source[offset..].starts_with(&rule.source))
{
let end = offset + rule.source.len();
edits.push(TextEdit {
byte_span: offset..end,
replacement: rule.replacement.clone(),
});
offset = end;
} else {
let Some(character) = source[offset..].chars().next() else {
break;
};
let end = offset + character.len_utf8();
if let Some(rule) = single_character_rules.get(&character) {
edits.push(TextEdit {
byte_span: offset..end,
replacement: rule.replacement.clone(),
});
}
offset = end;
}
}
edits
}
/// Validate and apply edits from the end of the source toward the beginning.
///
/// With no edits, the returned [`Cow`] borrows the original source without an
/// allocation. All ranges refer to the original source snapshot.
pub fn apply<'source>(
source: &'source str,
edits: &[TextEdit],
) -> Result<Cow<'source, str>, ReplacementError> {
if edits.is_empty() {
return Ok(Cow::Borrowed(source));
}
let mut ordered = edits.iter().collect::<Vec<_>>();
ordered.sort_by(|left, right| range_order(&left.byte_span, &right.byte_span));
let mut previous_end = 0;
for edit in &ordered {
let range = &edit.byte_span;
if range.start >= range.end || range.end > source.len() {
return Err(ReplacementError::InvalidRange {
start: range.start,
end: range.end,
source_len: source.len(),
});
}
for offset in [range.start, range.end] {
if !source.is_char_boundary(offset) {
return Err(ReplacementError::InvalidCharacterBoundary { offset });
}
}
if range.start < previous_end {
return Err(ReplacementError::OverlappingEdits {
offset: range.start,
});
}
previous_end = range.end;
}
let mut output = source.to_owned();
for edit in ordered.into_iter().rev() {
output.replace_range(edit.byte_span.clone(), &edit.replacement);
}
Ok(Cow::Owned(output))
}
fn range_order(left: &Range<usize>, right: &Range<usize>) -> std::cmp::Ordering {
(left.start, left.end).cmp(&(right.start, right.end))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn plans_only_configured_non_ascii_characters() {
let edits = plan(&Config::embedded(), "plain-word—fixed");
assert_eq!(edits.len(), 1);
assert_eq!(edits[0].byte_span, 10..13);
assert_eq!(edits[0].replacement, "-");
}
#[test]
fn plans_multiple_edits_in_source_order_with_utf8_ranges() {
let edits = plan(&Config::embedded(), "a—b−c");
assert_eq!(
edits,
vec![
TextEdit {
byte_span: 1..4,
replacement: "-".to_owned(),
},
TextEdit {
byte_span: 5..8,
replacement: "-".to_owned(),
},
]
);
}
#[test]
fn generated_catalog_replaces_letters_digits_punctuation_and_multi_byte_skeletons() {
let source = "а𝐀!𝟘ⅿ🯹";
let edits = plan(&Config::embedded(), source);
assert_eq!(edits.len(), 6);
assert_eq!(apply(source, &edits).unwrap(), "aA!Orn9");
assert_eq!(edits[4].replacement, "rn");
}
#[test]
fn generated_catalog_width_folds_every_printable_fullwidth_ascii_character() {
let source = std::iter::once('\u{3000}')
.chain((b'!'..=b'~').filter_map(|ascii| char::from_u32(u32::from(ascii) + 0xfee0)))
.collect::<String>();
let expected = (b' '..=b'~').map(char::from).collect::<String>();
let edits = plan(&Config::embedded(), &source);
assert_eq!(edits.len(), 95);
assert_eq!(apply(&source, &edits).unwrap(), expected);
}
#[test]
fn bulk_confusable_policy_preserves_curated_replacements() {
let mut config = Config::embedded();
config.unicode.confusables = ConfusablePolicy::Review;
assert_eq!(apply("а—I ", &plan(&config, "а—I ")).unwrap(), "а-I ");
assert_eq!(plan_confusables_for_review(&config, "а—I ").len(), 1);
config.unicode.confusables = ConfusablePolicy::Off;
assert_eq!(apply("а—I ", &plan(&config, "а—I ")).unwrap(), "а-I ");
assert!(plan_confusables_for_review(&config, "а—I ").is_empty());
}
#[test]
fn empty_or_ascii_sources_need_no_edits() {
let config = Config::embedded();
assert!(plan(&config, "").is_empty());
assert!(plan(&config, "ordinary-ascii").is_empty());
}
#[test]
fn longest_source_wins_independently_of_configuration_order() {
let mut config = Config::embedded();
config.replacements.insert(
0,
crate::ReplacementRule {
id: "double-em-dash".to_owned(),
source: "——".to_owned(),
replacement: "--".to_owned(),
rationale: "Preserve a doubled dash.".to_owned(),
enabled: true,
},
);
config.validate().unwrap();
let edits = plan(&config, "a——b—c");
assert_eq!(edits.len(), 2);
assert_eq!(edits[0].byte_span, 1..7);
assert_eq!(edits[0].replacement, "--");
assert_eq!(edits[1].byte_span, 8..11);
}
#[test]
fn applies_edits_in_reverse_and_preserves_line_endings() {
let source = "a—b\r\nc−d\n";
let edits = plan(&Config::embedded(), source);
assert_eq!(apply(source, &edits).unwrap(), "a-b\r\nc-d\n");
assert_eq!(
apply(&apply(source, &edits).unwrap(), &[]).unwrap(),
"a-b\r\nc-d\n"
);
}
#[test]
fn no_edits_borrow_the_original_source() {
let source = "ordinary ASCII";
assert!(matches!(apply(source, &[]).unwrap(), Cow::Borrowed(value) if value == source));
}
#[test]
fn rejects_invalid_boundaries_ranges_and_overlaps() {
let invalid_boundary = [TextEdit {
byte_span: 1..2,
replacement: "-".to_owned(),
}];
assert!(matches!(
apply("é", &invalid_boundary),
Err(ReplacementError::InvalidCharacterBoundary { offset: 1 })
));
let invalid_range = [TextEdit {
byte_span: 0..3,
replacement: "-".to_owned(),
}];
assert!(matches!(
apply("ab", &invalid_range),
Err(ReplacementError::InvalidRange { .. })
));
let overlapping = [
TextEdit {
byte_span: 0..2,
replacement: "x".to_owned(),
},
TextEdit {
byte_span: 1..3,
replacement: "y".to_owned(),
},
];
assert!(matches!(
apply("abc", &overlapping),
Err(ReplacementError::OverlappingEdits { offset: 1 })
));
}
#[test]
fn generated_sequences_are_stable_non_overlapping_and_idempotent() {
let config = Config::embedded();
let alphabet = ["a", "é", "—", "−", "\r", "\n"];
for first in alphabet {
for second in alphabet {
for third in alphabet {
for fourth in alphabet {
let source = [first, second, third, fourth].concat();
let edits = plan(&config, &source);
assert_eq!(edits, plan(&config, &source));
for edit in &edits {
assert!(source.is_char_boundary(edit.byte_span.start));
assert!(source.is_char_boundary(edit.byte_span.end));
}
assert!(
edits
.windows(2)
.all(|pair| { pair[0].byte_span.end <= pair[1].byte_span.start })
);
let formatted = apply(&source, &edits).unwrap();
assert_eq!(
source
.chars()
.filter(|character| *character == '\r')
.count(),
formatted
.chars()
.filter(|character| *character == '\r')
.count()
);
assert_eq!(
source
.chars()
.filter(|character| *character == '\n')
.count(),
formatted
.chars()
.filter(|character| *character == '\n')
.count()
);
let second_edits = plan(&config, &formatted);
assert!(second_edits.is_empty());
assert_eq!(apply(&formatted, &second_edits).unwrap(), formatted);
}
}
}
}
}
}

View File

@ -0,0 +1,55 @@
//! Pinned invisible and bidirectional-control review profile.
/// Match the invisible/control profile generated by Gitea at commit 9c715c0.
///
/// Gitea removes ordinary space, tab, and line feed before generating its
/// table. Callers remain responsible for treating CRLF as one ordinary line
/// ending rather than a suspicious standalone carriage return.
pub(crate) const fn is_gitea_invisible(character: char) -> bool {
matches!(
character as u32,
11..=13
| 127
| 160
| 173
| 847
| 1564
| 4447
| 4448
| 6068
| 6069
| 6155..=6158
| 7355..=7356
| 8192..=8207
| 8234..=8239
| 8287..=8303
| 10240
| 12288
| 12644
| 65024..=65039
| 65279
| 65440
| 65520..=65528
| 65532
| 78844
| 119155..=119162
| 917504..=917631
| 917760..=917999
)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn gitea_profile_covers_invisible_and_bidi_ranges_without_normal_whitespace() {
assert!(is_gitea_invisible('\u{200b}'));
assert!(is_gitea_invisible('\u{202e}'));
assert!(is_gitea_invisible('\u{2066}'));
assert!(is_gitea_invisible('\u{e0001}'));
assert!(!is_gitea_invisible(' '));
assert!(!is_gitea_invisible('\t'));
assert!(!is_gitea_invisible('\n'));
}
}

View File

@ -0,0 +1,21 @@
[package]
name = "john-slop-lsp"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
description = "Language Server Protocol engine for John Slop"
[lib]
name = "john_slop_lsp_engine"
[dependencies]
john-slop-core.workspace = true
serde_json.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["sync"] }
tower-lsp-server.workspace = true
[lints]
workspace = true

File diff suppressed because it is too large Load Diff

32
deny.toml Normal file
View File

@ -0,0 +1,32 @@
[graph]
all-features = true
[advisories]
version = 2
yanked = "deny"
[licenses]
version = 2
confidence-threshold = 0.8
allow = [
"0BSD",
"Apache-2.0",
"Apache-2.0 WITH LLVM-exception",
"ISC",
"MIT",
"MIT-0",
"MPL-2.0",
"Unicode-3.0",
"Unlicense",
"Zlib",
]
[bans]
multiple-versions = "warn"
wildcards = "deny"
highlight = "all"
[sources]
unknown-registry = "deny"
unknown-git = "deny"
allow-registry = ["https://github.com/rust-lang/crates.io-index"]

39
docs/README.md Normal file
View File

@ -0,0 +1,39 @@
# John Slop documentation
The root [README](../README.md) gets you from installation to a first scan. Move
through these guides as your needs grow; you do not need to understand the
internals to use the tools.
## Start using it
1. [Usage](usage.md) covers accepted inputs, detector options, stable exit
statuses, formatter modes, and the LSP command.
2. [Output](output.md) explains graphical Miette reports, plain human output,
concise records, versioned JSON Lines, and partial input errors.
3. [Configuration](configuration.md) documents source precedence, named
profiles, stable-ID merges, validation, and inspection commands. Start from
the [commented examples](../examples/config/).
## Put it into practice
4. [Runnable examples](../examples/README.md) demonstrate in-memory analysis,
streaming path input, shared replacement planning, and an expected-failure
diagnostic showcase.
5. [Editor setup](editors.md) provides working Helix (`hx`) and Vim formatter
and LSP configurations with validation steps.
6. [Review signals and evidence methodology](review-signals.md) explains the
research boundary, evidence tiers, and deliberately excluded proxies.
7. [Heuristics and evidence scoring](heuristics.md) explains what each rule
contributes and how the bounded score is calculated.
8. [Limitations](limitations.md) covers false positives, evasion, domain shift,
score semantics, and safe interpretation.
## Go deeper
The [advanced guide](advanced.md) is the single entry point for architecture,
the public library API, evaluation, benchmarks, dependencies, design records,
research references, development, security, and releases.
Project work is tracked in [tasks.md](../tasks.md), while contribution policy is
defined in [CONTRIBUTING.md](../CONTRIBUTING.md) and
[AGENTS.md](../AGENTS.md).

58
docs/advanced.md Normal file
View File

@ -0,0 +1,58 @@
# Advanced concepts and project internals
This guide collects material that is useful after the normal CLI,
configuration, output, and editor workflows are familiar. The root
[README](../README.md) and [documentation path](README.md) remain the best entry
points for day-to-day use.
## Detection model and evaluation
- [Heuristics and evidence scoring](heuristics.md) describes applicability,
occurrence damping, normalization, and why the score is not authorship
confidence.
- [Review signals and evidence methodology](review-signals.md) records the
research basis, evidence tiers, and signals deliberately kept out.
- [Evaluation](evaluation.md) records corpus provenance, per-rule measurements,
aggregate false positives, and the limits of the current metrics.
- [Limitations](limitations.md) covers evasion, domain shift, interpretation,
and operational boundaries.
## Architecture and embedding
- [Architecture and public API](architecture.md) describes the Cargo workspace,
module boundaries, shared domain values, streaming input, and reusable facade.
- [Output contracts](output.md) specifies human presentation and the versioned
JSON Lines schema for downstream integrations.
- [Decision records](decisions/) preserve the reasoning behind core contracts,
input limits, the LSP protocol, and the supported workspace.
- [Crate-boundary experiment](experiments/crate-boundary.md) contains the
measurements that preceded the workspace decision.
## Performance, dependencies, and safety
- [Performance benchmarks](benchmarks.md) defines representative workloads and
the evidence required before adding parallel execution.
- [Dependency and binary review](dependencies.md) records feature choices,
advisory and license policy, unsafe boundaries, and release-size measurements.
- [Security policy](../SECURITY.md) explains supported versions and private
vulnerability reporting.
## Research references
- [Background](background.md) collects external context on observable signs of
low-care AI-assisted writing.
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
Gitea's locale-aware catalog, its upstream source, and safe adoption
boundaries.
## Development and release engineering
- [Development](development.md) covers the Rust toolchain, feature matrix,
tests, snapshots, coverage, badges, and CI gates.
- [Contributing](../CONTRIBUTING.md) explains the patch workflow and review
expectations; [AGENTS.md](../AGENTS.md) defines repository-specific rules for
coding agents.
- [Release procedure](releasing.md) defines tagged reproducible packaging,
checksums, and provenance metadata.
- [Implementation plan](../tasks.md) records completed milestones and deferred
product work.

109
docs/architecture.md Normal file
View File

@ -0,0 +1,109 @@
# Architecture
John Slop is a workspace with a compatibility facade and two reusable libraries.
Executables remain thin adapters responsible for argument parsing, presentation,
and process exit statuses.
The current flow is:
```text
path -> input stream -> Document -> analysis -> Analysis + Finding
-> diagnostic/output adapters
-> replacement planner -> TextEdit -> applier
-> formatter I/O
-> LSP UTF-16 edit
```
The crate boundary separates dependency families:
- `john-slop-core` owns configuration, input, detection, diagnostics,
replacement planning, and presentation-neutral domain values;
- `john-slop-lsp` owns asynchronous editor state and protocol conversion while
depending only on `john-slop-core`;
- the root `john-slop` package re-exports both libraries to preserve its public
API and distributes the detector, formatter, and LSP binaries.
The root manifest is intentionally both a package and the workspace root. A
virtual manifest would relocate the compatibility facade, binaries, examples,
and fixtures without improving dependency isolation. `[workspace.package]`,
`[workspace.dependencies]`, and `[workspace.lints]` provide one source of truth
for shared release metadata, dependency requirements, and first-party unsafe
code policy. Member manifests retain only package-specific descriptions,
dependency selections, feature activation, and optionality. In particular, the
workspace Tokio requirement enables no features: the LSP library selects
`sync`, and the root LSP adapter selects its runtime and stdio features.
The modules within those crates retain deliberately narrow responsibilities:
- `config` owns validated embedded defaults, restricted runtime diagnostic
catalogs, the generated Unicode policy, and input limits;
- `input` streams files, directory entries, and archive members as documents;
- `detector` compiles configured matchers and path scopes once, then produces
explainable findings and a bounded low-care evidence score;
- `diagnostic` binds findings to exact named snapshots as typed `miette`
diagnostics;
- `replacement` produces non-overlapping edits and validates/applies arbitrary
edit sets without performing I/O;
- `unicode_review` pins the UTS #39 ASCII-skeleton version and Gitea-compatible
invisible/control ranges; generated printable-ASCII mappings feed the shared
replacement planner while controls remain non-mutating;
- `lsp` indexes open snapshots, runs the compiled detector, converts canonical
finding and replacement spans to UTF-16 coordinates, and implements
versioned quick fixes plus whole-document formatting;
- `model` contains presentation-neutral domain values;
- `error` exposes typed input failures for library consumers;
- `src/bin` contains the detector presentation/exit policy and formatter/LSP
adapters.
All source spans are zero-based UTF-8 byte ranges into the exact analyzed source
snapshot. Presentation layers will convert those canonical spans to terminal or
UTF-16 LSP coordinates at their boundaries. Archive sources retain both their
outer path and inner member identity. Rule applicability uses the member path
so language and path filters behave the same inside and outside archives.
## Public facade
The root compatibility facade and `john-slop-core` expose the same core entry
points and presentation-neutral types:
- `load_config` resolves the configuration embedded in the current build;
- `read_inputs` returns a streaming iterator of per-document results;
- `read_stdin` returns one bounded strict-UTF-8 standard-input document;
- `analyze_source` returns findings and an evidence score for UTF-8 text;
- `plan_replacements` returns safe edits against the same source snapshot;
- `apply_replacements` validates boundaries and overlap, then applies edits in
descending byte order while borrowing no-op input unchanged.
`FindingDiagnostic` is the library bridge to `miette`; CLI-specific concise and
JSON rendering stays in the detector adapter so it cannot alter analysis.
`ConfigLoader` applies embedded catalogs, runtime diagnostic catalogs, user
base, named user, explicit file, and CLI layers in order. Stable-ID merging and
semantic validation happen before a configuration reaches analysis. Keeping
these operations behind the facade avoids duplicating policy across the three
binaries. Runtime catalogs are declarative: they can add rules using existing
matcher implementations but cannot load executable code.
The formatter adapter owns filesystem mutation. It resolves configuration and
preflights every requested document before writing, then uses same-directory
temporary files and preserves permissions. Archive mutation remains outside the
library and adapter contract.
The LSP adapter owns no filesystem mutation. `tower-lsp-server` supplies stdio
JSON-RPC framing while `LspBackend` tracks authoritative open-document versions
behind asynchronous locks. Every publication is checked against the current
version after analysis so a replaced snapshot cannot overwrite newer results.
Configuration reload swaps only a fully validated model and republishes open
documents in deterministic URI order. The detector is compiled once per
resolved configuration rather than once per edit. File URI paths drive normal
scope applicability; language IDs supply source/prose context for untitled or
extensionless buffers. The default publication filter retains only Unicode
findings; explicit all-diagnostic and minimum-weight modes expose broader
findings, while only findings with core `TextEdit`s become code actions. Document formatting calls the same
replacement planner directly and never derives mutations from diagnostics. See
[ADR 0003](decisions/0003-lsp-framework-and-protocol.md) for framework and
protocol choices.
See [ADR 0001](decisions/0001-core-analysis-contracts.md) for the foundational
contracts and [ADR 0002](decisions/0002-input-policy-and-limits.md) for traversal
and archive safety decisions.

14
docs/background.md Normal file
View File

@ -0,0 +1,14 @@
# Background reading
John Slop focuses on observable low-care signals and avoids making unsupported
authorship claims. These resources provide context for that product boundary and
the kinds of review problems the project aims to help surface:
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
Project-specific decisions take precedence over external guidance. See
[ADR 0001](decisions/0001-core-analysis-contracts.md) for evidence scoring and
authorship semantics.

41
docs/benchmarks.md Normal file
View File

@ -0,0 +1,41 @@
# Performance benchmarks
Criterion benchmarks cover the scaling boundaries most likely to change
architectural decisions:
- analysis of an approximately 1 MiB clean prose document;
- streaming and decoding 250 small files in deterministic path order;
- compiling and analyzing with 110 rules (10× the embedded catalog); and
- opening, validating, and reading a 100-member stored ZIP archive.
Run them on an otherwise idle machine:
```sh
cargo bench --locked --bench core
```
Use `-- --quick` for a smoke measurement. Criterion output under `target/` is
not a portable performance artifact: record the CPU, OS, Rust version, commit,
and exact command when comparing results.
The Milestone 7 smoke baseline was measured on 2026-09-29 with
`cargo bench --locked --bench core -- --quick`, Rust 1.100.0-nightly, Linux
7.2.7, and an AMD Ryzen 9 9950X3D:
| Workload | Criterion estimate |
| --- | ---: |
| approximately 1 MiB document | 15.686 ms / 65.36 MiB/s |
| 250 small files | 286.54 µs / 872.48 Kfiles/s |
| build and analyze 110 rules | 9.474 ms |
| 100-member stored archive | 116.37 µs / 859.34 Kmembers/s |
These quick-mode figures are orientation data, not a cross-machine performance
guarantee. A release decision should use Criterion's normal sampling mode.
Milestone 7 retains the ordered single-threaded implementation. The baseline
workloads complete in milliseconds on the development machine, and directory
and archive results already stream rather than accumulating the entire input
tree before analysis. Parallel traversal would add scheduling and output-order
coordination without an established user-visible bottleneck. Reconsider it only
with a representative workload, a reproducible before/after benchmark, and
tests proving byte-for-byte output order remains unchanged.

213
docs/configuration.md Normal file
View File

@ -0,0 +1,213 @@
# Configuration
John Slop embeds schema-versioned base, heuristic, and Unicode catalogs in every
binary. Installed binaries therefore work without a source checkout or external
default files.
## Precedence
Configuration layers are applied from lowest to highest priority:
1. embedded `assets/config.toml` and `assets/heuristics.toml`;
2. the optional platform user diagnostic catalog `diagnostics.toml`, followed
by explicitly selected diagnostic catalogs in declaration order;
3. the platform user file `config.toml`;
4. a selected user `<config_name>.toml`;
5. an explicit `--config <FILE>`;
6. configuration flags for the current invocation.
Diagnostic catalogs load before normal configuration so a base, named, or
explicit configuration can tune or disable a catalog rule by stable ID. A
catalog accepts only `schema_version` and `[[rules]]`; unrelated settings are a
configuration error rather than being silently applied from the wrong file.
On Linux, the user directory conventionally resolves to
`~/.config/john-slop`. John Slop uses operating-system APIs rather than reading
`HOME` directly, so Windows and macOS receive their native application config
directories. Run this to inspect the exact paths on the current machine:
The RC3 naming migration changes the former `john_slop` application directory
to `john-slop`. Move existing `config.toml`, named configurations, and
`diagnostics.toml` into the new directory before upgrading; the old directory
is no longer loaded automatically.
```sh
john-slop config paths
```
Named configurations accept only ASCII letters, digits, `_`, and `-`. Do not
include a directory, dot component, or `.toml` extension. Missing user and named
files are optional; a selected explicit file must exist.
## Generate a project configuration
Create the embedded, commented base configuration as `./config.toml`:
```sh
john-slop config init
```
The command refuses to overwrite an existing file. Use
`john-slop config init --force` only when replacing that file is intentional.
`config generate` is an alias for `config init`. Load a project-local file with
`john-slop --config config.toml path/to/input` or inspect it with the command
shown below.
Generate a dedicated runtime diagnostic catalog separately:
```sh
john-slop config init-diagnostics
```
Place the result at the `user_diagnostics` path printed by `config paths` for
automatic loading, or select one or more project/team catalogs explicitly:
```sh
john-slop --diagnostics team.toml --diagnostics project.toml article.md
john-slop-lsp --diagnostics project.toml
```
Explicit catalogs are applied in command-line order. All catalogs load before
`config.toml`, named configuration, and `--config`, allowing those normal
layers to override their rules.
## Schema
The current `schema_version` is `1`. The complete defaults live in
[`config.toml`](../crates/john-slop-core/assets/config.toml),
[`heuristics.toml`](../crates/john-slop-core/assets/heuristics.toml), and the
generated [`unicode.toml`](../crates/john-slop-core/assets/unicode.toml).
Ready-to-copy, commented partial layers live in
[`examples/config`](../examples/config/): `config.toml` demonstrates a user-wide
base and `editorial.toml` demonstrates a named overlay.
Replacement and heuristic arrays merge by stable `id`. A partial record changes
only the supplied fields:
```toml
[diagnostics.categories]
# Disable every style diagnostic as one family.
style = false
[[replacements]]
id = "em-dash"
enabled = false
[[rules]]
id = "templated-conclusion"
# An individual rule switch is independent of the category switch.
enabled = false
```
The category keys are `typography`, `style`, `prompt-leakage`, and
`source-code`. A diagnostic runs only when its category switch and its
individual `[[rules]]` `enabled` field are both `true`. Consequently, enabling
a category does not silently re-enable a rule that another layer disabled
individually; set that rule's `enabled = true` explicitly when that is intended.
Later files override earlier category values using the normal precedence above.
A new ID must provide every field. Scope arrays replace the prior array as a
whole. A rule scope can select `prose` or `source-code`, restrict lowercase
extensions, and use `include`/`exclude` path globs. The `skip_generated`,
`skip_vendor`, and `skip_minified` switches default to `true`; set one to
`false` only when those paths are intentionally in scope. Archive rules match
the inner member path. Unknown keys, malformed TOML, duplicate IDs within one
layer, invalid regular expressions or globs, unsafe matchers, non-ASCII
replacement targets, and invalid thresholds are fatal configuration
diagnostics.
The built-in matcher names are `exact-phrase`, `regex`, `repeated-regex`,
`replacement`, and `invisible`. The last two use validated sentinel patterns
because their implementations are data-backed rather than free-form strings:
`configured-replacements` and `gitea-invisible-9c715c0`. They remain ordinary
rule records, so later layers
can disable them or adjust severity, weight, occurrence cap, and scope without
changing code. Invisible findings never authorize edits; generated confusable
mappings follow the bulk policy below.
## Runtime diagnostic catalog
`diagnostics.toml` supports new data-driven diagnostics without recompilation.
Every new rule must provide the complete rule shape:
```toml
schema_version = 1
[[rules]]
id = "custom-review-marker"
category = "style"
severity = "warning"
matcher = "exact-phrase"
pattern = "REVIEW BEFORE PUBLISHING"
weight = 8.0
message = "a project review marker remains"
help = "Resolve or remove the marker before publishing."
occurrence_cap = 1
enabled = true
scope = { content = ["prose", "source-code"], extensions = [] }
```
Runtime rules may select `exact-phrase`, `regex`, or `repeated-regex`.
`replacement` and `invisible` are code-backed catalog sentinels and cannot
define new matching behavior. Categories remain the four stable values listed
above. Unknown fields, invalid or empty-match regular expressions, duplicate
IDs within one catalog, incomplete new rules, and missing explicit catalogs are
fatal. These constraints keep startup deterministic and prevent a configuration
file from executing code.
The generated UTS #39 mappings have one bulk policy:
```toml
[unicode]
confusables = "review" # replace (default), review, or off
```
`replace` enables diagnostics and edits, `review` retains diagnostics without
formatter/LSP edits, and `off` suppresses generated lookalikes. The reviewed
punctuation and width-folding replacements and invisible/control diagnostics
are unaffected.
Set `enabled = false` on a known rule or replacement ID to disable it. Because
records merge by ID, do not copy the full embedded table merely to change one
field; a later partial record preserves every unspecified value.
Replacement `source` values are non-empty Unicode strings containing at least
one non-ASCII byte; `replacement` values are non-empty ASCII strings. Sources
must be unique. When enabled sources share a prefix, the longest matching source
wins at each byte offset, with stable rule ID ordering as the deterministic
tie-break. Edits always refer to the original UTF-8 snapshot.
Generated confusable IDs use `uts39-u<lowercase-codepoint>`. New fullwidth
mappings use `fullwidth-u<lowercase-codepoint>`; mappings shipped previously
retain their `uts39-*` or `fullwidth-hyphen-minus` IDs for compatibility, while
ideographic space uses `ideographic-space`. Every entry can be disabled or
overridden through normal merge-by-ID behavior.
Maintain the embedded list by editing
`crates/john-slop-core/examples/generate_unicode_catalog.rs` and running:
```sh
cargo run -p john-slop-core --example generate_unicode_catalog \
> crates/john-slop-core/assets/unicode.toml
```
The exhaustive catalog test walks every Unicode scalar and rejects drift from
the pinned Unicode 16.0 tables.
Each heuristic also configures `severity` as `advice`, `warning`, or `error`.
Severity controls CLI presentation, LSP severity, and summary counts; the
evidence score still comes from the independently configured weight and
aggregation formula. In the LSP, advice is information while warnings and errors
retain their corresponding levels.
## Inspection
Print the fully layered and validated model as deterministic TOML:
```sh
john-slop config show
john-slop config show --config project.toml
john-slop config show --config-name strict
```
The output contains no secrets in schema version 1. If secret-bearing fields are
introduced later, inspection output must redact them by default.

View File

@ -0,0 +1,55 @@
# ADR 0001: Core analysis contracts
- Status: accepted
- Date: 2026-09-29
## Context
John Slop needs stable semantics before its proof-of-concept modules are split
into a library, CLI, formatter, and language server. In particular, consumers
need to know what the aggregate evidence score means, how source locations are represented,
whether formatting is destructive by default, and whether archive members can
be rewritten.
## Decision
### The low-care evidence score represents evidence, not authorship
The detector reports the bounded strength of configured, observable low-care
signals. It does not report the probability that content is AI-generated and
must not present a score as an authorship determination. Every score must be
decomposable into source-labelled findings from stable rule IDs.
### UTF-8 byte spans are canonical
Internal source spans are zero-based, half-open UTF-8 byte ranges into the exact
source snapshot analyzed. This matches Rust string and regex offsets and avoids
lossy conversions inside the detection engine. Presentation adapters validate
boundaries and convert to their required representation: `miette::SourceSpan`
for terminal diagnostics and UTF-16 line/character positions for LSP.
### Formatter writes are opt-in
The formatter's default file behavior is read-only. It may print transformed
text or a diff, while an explicit `--write` is required to mutate files.
`--check` only reports whether an edit would be made. A failed analysis,
configuration load, or edit validation must leave the original file untouched.
### Archive mutation is deferred
The first formatter release may inspect text members in supported ZIP-based
containers, but it will not rewrite archives, DOCX, XLSX, or PPTX files. In-place
archive editing has additional integrity, metadata-preservation, atomicity, and
decompression-safety concerns. It requires a separate decision record and test
plan before implementation.
## Consequences
- User-facing terminology must describe evidence and request review rather than
accuse an author.
- The domain model stores byte ranges, while the LSP owns a tested UTF-16 mapping
layer tied to document versions.
- Replacement planning remains a pure operation; filesystem mutation lives in
an explicit formatter adapter.
- Archive inputs can produce findings but an attempted archive write produces a
clear unsupported-operation diagnostic.

View File

@ -0,0 +1,65 @@
# ADR 0002: Input policy and resource limits
- Status: accepted
- Date: 2026-09-29
## Context
The detector accepts individual files, directory trees, and ZIP-based
containers. Unbounded traversal or decompression can exhaust memory, CPU, file
descriptors, or disk. Extension-only detection can also misclassify binary files
as text. The input layer needs deterministic defaults before it is redesigned as
a streaming API.
## Decision
Limits apply per invocation and can be lowered by configuration or CLI flags.
Raising them requires an explicit option; there is no unlimited sentinel.
| Resource | Default limit |
| --- | ---: |
| Plain file or archive member | 8 MiB |
| Input archive file | 64 MiB |
| Total expanded archive bytes | 256 MiB |
| Archive entries inspected | 10,000 |
| Expanded/compressed ratio per member | 100:1 |
| Directory traversal depth | 64 levels |
| Nested archive traversal | disabled (depth 0) |
An exact file passed by the user is considered regardless of extension. During
directory traversal, configurable extension allowlists select likely prose,
markup, configuration, and source-code files. Content still has to decode as
strict UTF-8 and must not contain a NUL byte. Unsupported/binary content produces
a skipped-input result rather than a clean analysis or a lossy conversion.
ZIP-based containers are recognized from both a supported extension (`zip`,
`docx`, `xlsx`, or `pptx`) and ZIP magic bytes. An extension/magic mismatch is a
typed input error. Members are streamed and counted before allocation. Directory
members and encrypted members are skipped with a reason. Absolute paths,
platform prefixes, and any path containing a parent (`..`) component are
rejected. Archive member names are identities only; members are never extracted
to the filesystem by the analysis path.
Directory walking does not follow symbolic links. Hidden paths are skipped by
default. The default ignore set includes `.git`, `.hg`, `.svn`, `target`,
`node_modules`, `vendor`, common generated directories, source maps, and minified
assets. Repository ignore files are respected when the eventual walker supports
them. An explicitly named file can bypass path/extension ignores, but cannot
bypass content or resource-safety limits.
One unreadable, invalid, or oversized path yields a per-document input error and
does not discard successful documents from the same directory or archive.
Results remain ordered by normalized source identity even if reading is later
parallelized.
## Consequences
- The input API must stream per-document results and track cumulative archive
counters.
- `Document` needs distinct outer-path and optional archive-member identities.
- Size checks must use checked arithmetic and occur before allocating from
metadata supplied by an archive.
- Tests need boundary cases immediately below, at, and above each configurable
limit, plus malicious archive names and misleading extensions.
- The README and CLI help must distinguish skipped inputs, operational errors,
and clean analyzed documents.

View File

@ -0,0 +1,101 @@
# ADR 0003: LSP framework and protocol contract
- Status: accepted, amended by Milestone 19
- Date: 2026-09-29
## Context
John Slop needs a small stdio language server that reuses the validated
replacement configuration and exact `TextEdit`s from the library. Implementing
JSON-RPC framing and the Language Server Protocol state machine locally would
add protocol risk without improving detection. The original `tower-lsp` crate's
latest release is 0.20.0, while its maintained community continuation publishes
the same Tower-oriented API as `tower-lsp-server`.
The dependency review compared the crates' published metadata, documentation,
runtime features, and minimum Rust versions. `tower-lsp-server` 0.23.0 is the
latest stable release; 0.24.0-rc.1 is a prerelease. Version 0.23.0 requires Rust
1.85, below this project's Rust 1.98.1 baseline, and provides the Tokio stdio
transport, generated LSP types, client notifications, and in-process service
interface needed by the milestone. Its MIT OR Apache-2.0 licensing is compatible
with this project.
## Decision
Use `tower-lsp-server` 0.23 and its Tokio runtime feature. Use Tokio directly
only for the stdio runtime and synchronized in-memory document state. Do not
enable proposed LSP extensions or an alternate URI implementation.
The initial server targets stable LSP 3.17 behavior with these assumptions:
- transport is stdio only, with protocol bytes reserved for stdout;
- document positions are zero-based UTF-16 code-unit offsets;
- clients send full-document synchronization, simplifying authoritative version
handling while replacement behavior is established;
- the server owns snapshots of open documents only and never writes their URIs;
- diagnostics and quick fixes cover configured ASCII replacements, not heuristic
authorship or aggregate heuristic findings;
- quick fixes use versioned workspace edits and are rejected when the request
refers to a stale or unknown document version;
- `workspace/didChangeConfiguration` reloads the same layered configuration used
by the CLI and formatter, then republishes diagnostics for open snapshots;
- malformed messages and invalid URIs are protocol/input errors, never panics.
## Consequences
- The replacement engine remains the single source of edit ranges and text.
- A tested line index converts canonical UTF-8 byte offsets to LSP UTF-16
positions at the adapter boundary.
- Protocol tests can drive `LspService` in process without launching a child
process, while a small stdio smoke test protects stream purity.
- Incremental synchronization may be added later, but must preserve version and
stale-result guarantees before it is advertised.
- Upgrading to the 0.24 line waits for a stable release and a separate API and
minimum-Rust review.
## Amendment: full heuristic diagnostics
On 2026-09-29, Milestone 19 superseded the initial replacement-only diagnostic
limit after the core detector and configuration contracts had stabilized. The
server now compiles the same detector as the CLI once per resolved configuration
and publishes every applicable finding for each open snapshot.
This amendment does not broaden mutation behavior:
- safe configured replacements remain the only findings with quick fixes;
- style, prompt-leakage, and source-code review findings are diagnostic-only;
- stable heuristic IDs replace replacement-entry IDs as published diagnostic
codes, avoiding duplicate typography diagnostics;
- configured advice, warning, and error severities map to LSP information,
warning, and error respectively;
- file URI paths retain extension and path-scope behavior, while editor language
IDs determine prose/source applicability for untitled or extensionless
buffers; and
- configuration reload atomically swaps the validated configuration and compiled
detector before open documents are republished.
The original full-snapshot synchronization, UTF-16 conversion, version checks,
stdio purity, no-write boundary, and stale-action guarantees remain unchanged.
## Amendment: quiet diagnostic publication
On 2026-09-30, Milestone 26 changed the presentation default without changing
the shared detector. The LSP now publishes only Unicode findings unless it is
started with `--all-diagnostics` or `--minimum-diagnostic-weight WEIGHT`.
Minimum-weight mode always retains Unicode findings and filters only the other
categories. This makes exact, fixable problems the quiet editor default while
keeping the full catalog easy to exercise in tests. Filtering occurs after
analysis, so CLI results, evidence scores, formatting, configuration reload,
and the core finding contract remain unchanged.
## Amendment: document formatting parity
Milestone 21 adds the standard `textDocument/formatting` capability. The server
plans edits from its current open-document snapshot with the same resolved
configuration and core replacement planner used by `john-slop-fmt`, then
converts only the edit coordinates from UTF-8 bytes to LSP UTF-16 positions.
Formatting remains limited to configured safe ASCII replacements; the server
does not write files, infer formatting from diagnostics, or apply broader
Unicode review findings. Protocol tests compare the result of an actual
formatting request with the standalone formatter contract, including Unicode,
CRLF, no-op, and configuration-reload cases.

View File

@ -0,0 +1,55 @@
# ADR 0004: Supported Cargo workspace
- Status: accepted
- Date: 2026-09-29
## Context
The crate-boundary experiment extracted reusable analysis/configuration/input
code into `john-slop-core` and LSP protocol state into `john-slop-lsp`, leaving
the root `john-slop` package as a compatibility facade and binary distribution.
Cold core and no-LSP builds used roughly 35–55% less CPU and disk, while LSP and
whole-project builds were effectively unchanged. The experiment initially
recommended waiting because three publishable packages require synchronized
versions, ordered publication, workspace-aware CI, and clear asset ownership.
The project has now chosen to offer the reusable package boundary and accepts
that release overhead. Repeating package metadata and dependency versions in
three manifests would make that overhead unnecessarily error-prone.
## Decision
Adopt the three-package layout as the supported architecture:
- `john-slop-core` owns embedded catalogs and reusable synchronous domain logic;
- `john-slop-lsp` depends on core and owns protocol state/conversion;
- root `john-slop` remains the compatibility facade and owns all three binaries.
Keep the root manifest as both a package and the workspace root. Moving its
sources under another directory solely to create a virtual manifest adds path,
fixture, and packaging churn without changing the dependency graph.
Centralize shared version, edition, MSRV, license, repository, dependency
requirements, and lint policy in the root workspace tables. Every member
inherits them. Dependency feature activation and optionality remain in member
manifests: workspace inheritance must not cause the core package to compile
Tokio/LSP dependencies or make the facade's LSP stack unconditional.
Keep all three packages as default members so existing root quality commands
cover the whole workspace. Enforce membership, synchronized resolved versions,
inheritance, and feature isolation with an integration test. Assemble packages
with Cargo's workspace packaging command and publish in dependency order: core,
LSP, then facade.
## Consequences
- Core/no-LSP consumers retain the measured lean dependency graph.
- Existing `john_slop` API paths and binary names remain compatible by default.
- Shared requirements have one update point, while member feature choices stay
reviewable at their point of use.
- Full builds gain no material speedup from the boundary.
- A release coordinates three matching package versions. Before matching
dependency versions exist in the registry, package assembly uses
`--no-verify`; registry-form verification follows publication.
- Embedded catalog changes belong to `john-slop-core`, and workspace coverage,
CI, dependency review, and packaging remain permanent maintenance costs.

62
docs/dependencies.md Normal file
View File

@ -0,0 +1,62 @@
# Dependency and binary review
Milestone 7 reviewed the locked graph on 2026-09-29 with Rust 1.100.0-nightly
on x86-64 Linux.
## Features and scope
The normal dependency graph contains 135 distinct package names. Direct crates
serve narrow boundaries: Clap parses adapters, Figment/Serde/TOML load typed
configuration, Regex/Globset compile rules and scopes, Miette renders
diagnostics, Walkdir/Zip acquire inputs, Tokio/tower-lsp-server implement the
stdio protocol, Unicode Security supplies the pinned UTS #39 skeleton data, and
Tempfile/Similar support atomic writes and diffs.
Default features are disabled where they would add unrelated behavior. In
particular, Zip previously enabled AES, Bzip2, Deflate64, LZMA, PPMd, Zstandard,
time, and multiple encoders. John Slop reads unencrypted office/ZIP inputs and
now enables only the zlib-rs Deflate path required by common containers. A
Deflate archive regression test pins that support. This removed 394 lockfile
lines. Regex retains Unicode and performance defaults because configured rules
use Unicode text and case matching; Miette's graphical stack remains isolated
behind `fancy-diagnostics`.
All publishable packages inherit dependency requirements from the root
`[workspace.dependencies]` table. This centralizes versions and baseline feature
policy without globally enabling member-specific features: `john-slop-core`
does not select Clap, Tokio, or the LSP stack; `john-slop-lsp` selects only
Tokio synchronization; and the root adapter enables runtime/stdio support only
behind its optional `lsp` feature. `tests/workspace_contract.rs` pins this
boundary and the synchronized package metadata.
## Security, licenses, and unsafe code
`cargo audit --deny warnings` scanned 193 locked packages against 1,277 RustSec
advisories without a finding. `cargo deny --locked check` passed advisories,
licenses, and sources; it reports seven transitive duplicate-version families
as warnings for ongoing review. [`deny.toml`](../deny.toml) contains the reviewed
license allow-list and rejects wildcard versions, yanked packages, and unknown
registries or Git sources.
A workspace lint forbids first-party Rust `unsafe` code, and the source scan
found no existing blocks. Dependencies may use
unsafe code internally, notably for compression, operating-system integration,
and async/runtime primitives; this review does not claim otherwise. Lockfile,
advisory, source, and license checks remain CI gates because this result changes
when dependencies change.
## Release size baseline
Before release-profile stripping, the all-feature Linux binaries measured 6.21,
5.84, and 9.38 MB. The finalized thin-LTO, single-codegen-unit,
abort-on-panic, symbol-stripped profile measures:
| Binary | Bytes |
| --- | ---: |
| `john-slop` | 3,542,544 |
| `john-slop-fmt` | 3,322,944 |
| `john-slop-lsp` | 4,756,688 |
These are build-specific engineering measurements, not promised download sizes.
The release procedure records final sizes and emits checksums beside any
distributed binaries.

119
docs/development.md Normal file
View File

@ -0,0 +1,119 @@
# Development
The project uses Rust edition 2024 and requires Rust 1.98.1 or newer. The
committed lockfile makes application builds reproducible. Linux is the supported
and exercised platform; other platforms are not declared supported until CI
covers them. Jenkins checks Linux on both Rust 1.98.1 (the MSRV) and current
stable, with all features and without default features.
## Quality gates
Run the same core checks required by the contributor guide:
```sh
cargo fmt --all -- --check
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo test --locked --all-targets --all-features
cargo test --locked --all-targets --no-default-features
cargo doc --locked --no-deps --all-features
cargo audit --deny warnings
cargo deny --locked check
```
The root package is also the workspace root, and all three packages are default
members. These unqualified commands therefore cover the compatibility/binary
package, `john-slop-core`, and `john-slop-lsp`. Use `-p john-slop-core` for the
isolated reusable dependency graph or `-p john-slop-lsp` for protocol-only work.
The workspace contract test validates membership, version synchronization,
manifest inheritance, and feature isolation. Before a release, assemble every
publishable crate with `scripts/package-workspace.sh --no-verify`; see the
[release procedure](releasing.md) for the registry verification boundary.
To smoke-test installation from the current source checkout, identify the local
package explicitly (recent Cargo versions no longer infer it from an omitted
package argument):
```sh
cargo install --path . --locked --all-features
```
Use `--root <temporary-directory>` when verifying this command without changing
the normal Cargo installation root.
The default `fancy-diagnostics` feature enables the detector's colored Miette
renderer and default output mode. The default `lsp` feature exposes the
compatibility facade's protocol module and builds the LSP binary. The
no-default-features test run exercises the core, detector, and formatter without
the async protocol dependency graph and with the plain diagnostic fallback. Focused
unit and integration tests cover domain invariants, input failures, archives,
CLI exit behavior, and feature configurations.
CI runs the executable fixture manifest and Insta CLI snapshots explicitly in
addition to the full suite. `cargo-audit` rejects RustSec advisories and yanked
packages. [`deny.toml`](../deny.toml) rejects unknown dependency sources,
wildcard versions, and licenses outside the reviewed allow-list; duplicate
versions are reported for review while upstream compatibility is assessed.
Insta snapshots are reserved for stable user-facing text such as CLI help and
usage errors. Structured domain values use direct assertions so failures remain
precise and easy to review. Accept snapshot changes only after inspecting the
generated diff.
## Coverage and badges
Run the same line-coverage command used by Jenkins:
```sh
cargo tarpaulin --config tarpaulin.toml --out Json --output-dir coverage
```
The configuration enforces a 90% floor. The current suite measured 92.72%
(1,604/1,730 lines) on 2026-09-30. The remaining lines are dominated by
platform/I/O failure injection, malformed archive internals, coordinate
overflow, LSP logging/reload failures, and subprocess attribution. Do not add
brittle tests or production seams solely to make Tarpaulin count those paths.
`Cargo.toml` is the authoritative version source. Jenkins temporarily renames
the Tarpaulin JSON output to `coverage/summary.json`, extracts and validates its
overall percentage, renders the badges locally, and then removes the summary.
It archives only `coverage/version.svg`, `coverage/test.svg`, and
`coverage/coverage.svg`, with fingerprinting. Successful `main` builds publish
those files atomically under
`/mnt/assets/coverage/idle-org/john-slop/main/`, which serves these mutable URLs:
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/version.svg`
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/test.svg`
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/coverage.svg`
The asset server is deployment-owned: it must mount assets read-only, disable
directory listings, serve SVG with a restrictive content security policy, and
revalidate these URLs or apply a short cache lifetime.
## Test scope
Add behavior tests at the lowest useful layer and an integration test when a
change crosses a binary boundary. Important edge cases include malformed input,
invalid UTF-8, empty text, Unicode byte spans, deterministic ordering, archive
limits, and near misses for detection rules. The detailed expectations and
task-level commit discipline are in [AGENTS.md](../AGENTS.md).
Each completed milestone receives an immutable annotated `v0.1.N` Git tag,
where `N` is the milestone number, after its acceptance criteria and quality
gates pass. Its completion commit must also use package version `0.1.N` in the
workspace manifest, internal registry requirements, lockfile, changelog, binary
version output. Jenkins derives the current version badge from that metadata.
## Fuzzing
The standalone [`fuzz`](../fuzz/) package contains five `cargo-fuzz` targets and
reviewed seeds for configuration merging, rule validation/compilation, UTF-8
span conversion, replacement invariants, and hostile archive bytes. See its
README for invocation. Fuzz artifacts remain ignored until a failure has been
minimized and promoted to a normal regression test.
## Benchmarks
Run `cargo bench --locked --bench core` for the large-document, many-file,
rule-scaling, and archive-limit workloads described in
[Performance benchmarks](benchmarks.md). Benchmark reports are local artifacts;
only methodology and contextualized measurements belong in version control.

141
docs/editors.md Normal file
View File

@ -0,0 +1,141 @@
# Editor setup
Both `john-slop-fmt` and `john-slop-lsp` are operational. The LSP publishes only
configured Unicode replacement and control findings by default, keeping noisy
or error-prone heuristics out of the normal editor loop. Safe ASCII-replacement
findings additionally provide versioned quick fixes, and the server implements
whole-document formatting with the exact same replacement planner as
`john-slop-fmt`.
Install all three binaries somewhere on `PATH`, then confirm the editor-facing
help is available:
```sh
john-slop-fmt --help
john-slop-lsp --help
```
## Helix (`hx` / `helix-term`)
Helix configures formatters and language servers in `languages.toml`. Put this
in `~/.config/helix/languages.toml` on Linux, or in `.helix/languages.toml` for
project-local configuration:
```toml
[language-server.john-slop]
command = "john-slop-lsp"
[[language]]
name = "john-slop-text"
language-id = "plaintext"
scope = "text.john-slop"
file-types = ["txt"]
formatter = { command = "john-slop-fmt" }
language-servers = ["john-slop"]
auto-format = false
```
This complete custom language entry makes the repository's `example.txt`
fixtures useful without depending on a built-in plain-text grammar. It selects
the external `john-slop-fmt` adapter. To exercise formatting through the LSP
instead, omit the `formatter = ...` line; `:format` then requests
`textDocument/formatting` from `john-slop-lsp`. Both routes resolve the same
configuration and produce the same text. Set `auto-format = true` only if
save-time replacement is wanted.
For an existing language, merge John Slop into that language's entry rather
than replacing its useful defaults. For example, Rust should retain
`rust-analyzer`:
```toml
[[language]]
name = "rust"
formatter = { command = "john-slop-fmt" }
language-servers = ["rust-analyzer", "john-slop"]
```
When several language servers advertise formatting, retain an explicit Helix
`formatter` or select the intended server interactively; John Slop must not
silently replace language-specific formatting from `rust-analyzer` or another
primary server.
Run `hx --health rust` (or the selected language name) to confirm both binaries
are visible. Helix documents that a formatter reads the buffer from stdin and
writes it to stdout, while an LSP command must be on `PATH`; those are the
contracts the two John Slop milestones target. See the official
[Helix language configuration reference](https://docs.helix-editor.com/languages.html).
## Vim
Vim's native `formatprg` option is enough for formatting. Add this to
`~/.vimrc`:
```vim
augroup john-slop-format
autocmd!
autocmd BufRead,BufNewFile *.txt setfiletype text
autocmd FileType text,markdown,rust setlocal formatprg=john-slop-fmt
augroup END
```
Use `gggqG` to filter the whole buffer through the formatter, or select a range
and press `gq`. Vim requires `formatprg` programs to read stdin and write stdout;
see the official [`formatprg` and `gq` help](https://vimhelp.org/options.txt.html#%27formatprg%27).
Classic Vim needs an LSP client plugin. With
[`prabirshrestha/vim-lsp`](https://github.com/prabirshrestha/vim-lsp), register
John Slop in `~/.vimrc`:
```vim
if executable('john-slop-lsp')
au User lsp_setup call lsp#register_server({
\ 'name': 'john-slop',
\ 'cmd': {server_info->['john-slop-lsp']},
\ 'allowlist': ['text', 'markdown', 'rust'],
\ })
endif
```
Use the plugin's `:LspStatus`, `:LspNextDiagnostic`, and `:LspCodeAction`
commands to inspect server state, move through heuristic diagnostics, and apply
available ASCII quick fixes. The registration shape and `allowlist` follow the
plugin's current upstream documentation.
Clients that expose LSP document formatting can request it directly from John
Slop instead of configuring `formatprg`; both paths use the same replacement
catalog. Keep `formatprg=john-slop-fmt` when the Vim client cannot select one
formatter among multiple attached language servers.
The server uses the same configuration layers as the other binaries. Add
`args = ["--config-name", "editorial"]` to the Helix language-server table, or
append those arguments to the Vim `cmd` list, when an editor-specific named
configuration is wanted. Use `args = ["--diagnostics", "diagnostics.toml"]` (and
the equivalent Vim command argument) for a project runtime diagnostic catalog.
Add `--all-diagnostics` to exercise every configured rule during testing, or
`--minimum-diagnostic-weight 10` to retain Unicode findings and opt into only
non-Unicode findings with weight 10 or greater. This visibility filter does not
change detector scoring or formatting.
`--log-json` is available for troubleshooting and writes lifecycle metadata to
stderr without document contents.
## Try the shared fixtures
Open `examples/fixtures/typography/example.txt` in either editor. Its expected
formatter result is the adjacent `formatted.txt`; the shared manifest records
three replacements and one typography rule. Compare detector and formatter
output with:
```sh
cargo run --bin john-slop -- --output concise \
examples/fixtures/typography/example.txt
cargo run --bin john-slop-fmt -- \
examples/fixtures/typography/example.txt
```
After starting the server with `--all-diagnostics`, open
`examples/fixtures/style/example.txt` to see `repeated-transition` and other
style findings, or `examples/fixtures/source/example.rs` to see source-only
placeholder and narration findings alongside typography. Advice maps to LSP
information, warnings to warnings, and configured errors to errors. Every
diagnostic carries its stable heuristic rule ID as the code and includes the
configured remediation text.

83
docs/evaluation.md Normal file
View File

@ -0,0 +1,83 @@
# Evaluation corpus
Milestone 3 ships a small, reviewed characterization corpus under
`tests/fixtures/evaluation`. Its text was authored for this repository and is
covered by the MIT license. Labels name observable configured signals, never a
presumed author or generation method.
With the embedded schema-version 1 catalog, the corpus currently reports:
| Rule | True positive | False positive | False negative | Precision | Recall |
| --- | ---: | ---: | ---: | ---: | ---: |
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
| assistant-follow-up-offer | 1 | 0 | 0 | 100% | 100% |
| assistant-revision-preface | 1 | 0 | 0 | 100% | 100% |
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
| templated-filler | 1 | 0 | 0 | 100% | 100% |
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
| editorial-insertion-placeholder | 1 | 0 | 0 | 100% | 100% |
| stock-helpful-closing | 1 | 0 | 0 | 100% | 100% |
| excess-vocabulary-cluster | 1 | 0 | 0 | 100% | 100% |
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
| source-omitted-code-placeholder | 1 | 0 | 0 | 100% | 100% |
| source-markdown-fence | 1 | 0 | 0 | 100% | 100% |
| unicode-ascii-replacement | 2 | 0 | 0 | 100% | 100% |
| unicode-invisible-control | 1 | 0 | 0 | 100% | 100% |
Aggregate labeled-rule counts are 20 true positives, 0 false positives, and 0
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
suite recomputes these values from the manifest and fails on drift.
The same manifest pins evidence scores for regression characterization:
| Corpus case | Score |
| --- | ---: |
| six assistant disclaimer/preface rules | 64 |
| one prompt/response marker | 33 |
| style signals | 25 |
| source-code signals | 33 |
| typography signal | 17 |
| one ASCII lookalike and one invisible character | 28 |
| each of five negative documents | 0 |
These perfect values describe a tiny purpose-built regression set, not expected
performance on real documents. They must not be used as an accuracy or
authorship claim. Broader, independently sourced data and domain-specific review
are required before tuning default weights or thresholds.
## Default-policy sign-off
Each shipped rule has at least one explicitly labeled realistic near miss. The
initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds
unchanged. The prompt-leakage review accepted lighter `0.90ⁿ` correlation
damping for distinct prompt-leakage rules while retaining `0.75ⁿ` for other
categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings
replaceable by default, retains a non-mutating `review` policy, and keeps the
Gitea-compatible invisible/control profile diagnostic-only, with explicit
multilingual limitations. The Milestone 25 review gives one-to-one fullwidth
ASCII compatibility folds precedence over visual skeletons under every policy.
The Milestone 26 review adds narrow response and unfinished-editing residue,
keeps population-level vocabulary and stock closings deliberately low-weight,
and excludes language-proficiency proxies.
The checked-in `signoff.toml` records those decisions
and contains the SHA-256 digest of `config.toml`, `heuristics.toml`, and
`unicode.toml`. This binds the reviewed scoring behavior to both rule catalogs
and the default category policy.
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
change invalidates the sign-off test. Renewal requires reviewing positive and
near-miss cases, reporting changed per-rule/aggregate metrics, explaining the
false-positive tradeoff, updating the digest and decision text, and obtaining
explicit maintainer approval in the change review. A green test alone is not
approval to tune defaults.
Runtime diagnostic catalogs are deployment policy and are not part of the
embedded-catalog digest. Their rules use the same validation and scoring
formula, but teams must evaluate their own added patterns and near misses for
their corpus rather than treating the embedded sign-off as validation of custom
rules.

View File

@ -0,0 +1,99 @@
# Crate-boundary experiment
Status: **adopted for Milestone 13**. The experiment originally recommended
waiting because full builds do not improve and three publishable packages add
release coordination. The project subsequently chose to support the reusable
package boundary and accepted those measured costs; the durable policy is in
[ADR 0004](../decisions/0004-supported-cargo-workspace.md).
## Candidate design
The experiment keeps the root `john-slop` package as a compatibility facade and
binary distribution. It moves reusable analysis into `john-slop-core`, moves
protocol state into `john-slop-lsp`, and makes the facade's LSP dependency
optional behind the default-enabled `lsp` feature. Existing default API paths
and executable names remain intact.
The embedded catalogs live with `john-slop-core`, which makes that package
self-contained. Workspace default members include all three packages so the
existing unqualified quality commands continue to cover them.
## Method
Baseline commit `c12b6b8` (`v0.1.11`) and the experimental branch were
checked in separate worktrees. Each cold command used an empty, dedicated
`CARGO_TARGET_DIR`; dependency sources and the operating-system file cache were
warm. Timings are one directional sample rather than a statistical benchmark.
Artifact size is the resulting target directory size reported by `du -sh`.
Environment: Rust/Cargo 1.100.0-nightly from 2026-09-15, x86-64 Linux, 32 logical
CPUs. Commands used `--locked`; the full checks used all targets and features.
## Results
| Scenario | Baseline wall / CPU | Candidate wall / CPU | Baseline / candidate artifacts |
|---|---:|---:|---:|
| Reusable core | 7.51s / 29.96s¹ | 3.82s / 13.56s | 256 / 116 MiB¹ |
| Root library, no defaults | 7.51s / 29.96s | 4.09s / 19.04s | 256 / 158 MiB |
| Formatter, no defaults | 6.98s / 29.50s | 4.28s / 19.19s | 244 / 160 MiB |
| LSP | 7.92s / 30.13s | 6.97s / 29.66s | 243 / 244 MiB |
| Whole project | 7.90s / 49.68s | 7.73s / 50.69s | 415 / 416 MiB |
| No-change library check | 0.043s wall | 0.036s wall | not material |
¹ The single-crate baseline has no isolated core target, so its library result
is the nearest comparison.
The useful changes are therefore concentrated in non-LSP builds:
- the directly consumable core uses about 55% less CPU and disk and 49% less
wall time than the former monolithic library check;
- the compatibility library without default features uses about 36% less CPU,
38% less disk, and 46% less wall time;
- the formatter without default features uses about 35% less CPU and disk and
39% less wall time;
- LSP, full-project, and no-change incremental costs are effectively unchanged.
Tarpaulin still reports 92.19% (1,334/1,447 lines) with workspace coverage, so
the boundary neither gains nor loses measured test coverage.
## Gains
- Async protocol dependencies no longer enter core or no-LSP builds.
- The module ownership matches the conceptual detector/LSP boundary.
- A future library consumer can depend on `john-slop-core` without CLI, JSON,
diff, Tokio, or LSP dependencies.
- The root facade avoids a forced source migration for default-feature users.
## Costs and risks
- One package and manifest become three, with synchronized versions and the
publish order core → LSP → facade.
- The embedded catalogs must move into the core package, changing repository
paths and release ownership.
- Coverage and CI must remain workspace-aware.
- `john_slop::lsp` is unavailable when default features are disabled; this is a
feature-contract change even though default builds are compatible.
- Ordinary default and full-project builds receive no material speedup, and
runtime behavior and performance do not improve.
## Original recommendation and adoption
Do not merge this branch while John Slop is primarily distributed as one set of
three binaries and has no demonstrated external core-library consumers. The
measured savings are real but apply only when deliberately excluding LSP or
depending directly on the new core package; the normal full build is unchanged.
Reconsider the branch if `john-slop-core` becomes a supported published API, if
the adapters gain independent release schedules, or if the full build grows
enough that the package boundaries enable additional parallelism. Until then,
optionalizing the LSP module in the existing single crate would capture most of
the no-default-feature compile benefit with substantially less release overhead.
Milestone 13 deliberately chose the first reconsideration condition: treat
`john-slop-core` and `john-slop-lsp` as supported reusable packages. The measured
full-build result has not changed. Adoption instead values the enforceable
dependency boundary, lean core consumers, and clearer ownership enough to
accept synchronized releases and workspace-aware CI/package checks. Shared
metadata and dependency requirements now use Cargo workspace inheritance, while
member-specific feature activation remains local so centralization does not
erase the isolation this experiment measured.

64
docs/heuristics.md Normal file
View File

@ -0,0 +1,64 @@
# Heuristics and evidence scoring
John Slop reports observable low-care signals. It does not classify authorship.
Every contribution to the score is represented by a finding with an exact
source span, stable rule ID, category, message, help, and configured weight.
The detector compiles the resolved rule catalog once. Exact phrases, regular
expressions, repeated-pattern rules, configured Unicode replacements, and
invisible/control characters share the same finding
model. Content kind, extension, path globs, and conventional generated/vendor/
minified paths decide whether each rule applies. Findings are deduplicated by
rule and byte span, then sorted by start offset, end offset, and rule ID so
every adapter observes the same order.
Additional exact-phrase, regex, and repeated-regex rules can be supplied at
runtime through `diagnostics.toml` or repeatable `--diagnostics` arguments.
They receive the same validation, scoping, scoring, ordering, CLI presentation,
and LSP publication as embedded rules. New matcher algorithms and categories
remain code changes rather than executable configuration.
The generated Unicode 16.0 profile maps every non-ASCII scalar with a printable
ASCII UTS #39 skeleton after explicitly reviewed compatibility-width and
punctuation mappings.
The default `replace` policy gives each of these findings a formatter/LSP edit;
`review` keeps the same exact diagnostics without edits, and `off` suppresses
the generated lookalikes. Invisible and direction-changing controls remain
diagnostic-only because deleting them silently is not a generally safe edit.
## Evidence-score formula
For each rule in deterministic catalog order, only the configured
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
full weight, then later occurrences receive one half, one quarter, and so on.
Findings beyond the cap remain visible but do not increase the score.
Rules are grouped by category as correlated evidence families. The first
matching rule in a category receives its full occurrence-adjusted contribution;
later distinct prompt-leakage rules receive successive factors of `0.90`, while
style, typography, and source-code rules receive successive factors of `0.75`.
Prompt/response markers and copied assistant disclaimers are stronger concrete
artifacts than generic stylistic similarities, but the remaining damping still
prevents a cluster of related phrases from being treated as fully independent
proof.
If the resulting evidence sum is `E`, the displayed score is
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
is explicitly reported as below threshold. Empty input and text with no
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
oversized, and unsupported inputs remain typed input errors; they are never
reported as clean analyses.
Every `Analysis` retains its complete finding list and can produce a compact
summary containing the numeric score, optional named band, and separate advice,
warning, and error counts. Summary generation never filters the evidence used
to explain the score.
Weights and thresholds are review policy, not calibrated probabilities. A
score of 75 does not mean a 75% chance that AI wrote the input.
The diagnostic showcase exercises five distinct prompt-leakage rules, including
two prompt/response markers. Together with its style and typography findings it
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
an estimated probability.

94
docs/limitations.md Normal file
View File

@ -0,0 +1,94 @@
# Limitations
John Slop is a review aid for observable low-care signals. It is not an
AI-authorship detector, plagiarism detector, fact checker, security scanner, or
quality oracle. A finding says only that configured text matched at a particular
source span. Human context remains decisive.
## False positives
Legitimate material can contain every shipped signal. Documentation may quote a
chat transcript, an article may discuss knowledge cutoffs, a tutorial may use
step-by-step comments deliberately, and a tracked source-code TODO may be valid.
Unicode punctuation can be correct typography even when a downstream format
requires ASCII. The UTS #39 ASCII-skeleton profile intentionally flags many
legitimate Greek, Cyrillic, mathematical, and other non-Latin characters for
review because it has no locale or identifier context. Generic transitions are
weak style evidence, not defects by themselves.
Use path scopes, rule disabling, and project-specific weights to reflect local
policy. Do not automatically reject, shame, or attribute a document based on a
score. Review the visible findings and original context.
## False negatives and evasion
Exact and regular-expression rules are inherently easy to evade through
paraphrasing, case changes, spelling variation, inserted characters, or removal
of obvious scaffolding. Careless work need not contain any known signal, while
well-reviewed AI-assisted work may contain none. John Slop deliberately avoids
opaque statistical authorship guesses, so a clean result is not evidence of
human authorship or high quality.
Adversarial inputs can also target applicability assumptions: misleading file
extensions, unusual generated-directory names, or stdin without a filename may
select a different scope. Archive analysis uses the inner member path but does
not infer semantic language from content.
## Domain and language shift
The starter prose rules primarily encode English phrases. Other languages,
specialized editorial styles, historical corpora, transcripts, generated API
documentation, notebooks, and uncommon programming languages are
underrepresented. File-extension inference covers a conservative common set and
defaults unknown paths to prose. Teams should validate rules against their own
reviewed material before changing enforcement thresholds. Disable or narrow
set `[unicode] confusables = "review"` to keep findings without edits, or use
`"off"` when ordinary multilingual text makes character-level review too noisy.
Locale-aware suppression is not implemented.
## Evidence score is not probability
The score is a deterministic compression of configured weights, occurrence
caps, and category damping. Those values are policy choices and have not been
calibrated against population-level authorship data. A score of 75 does not mean
there is a 75% probability of AI authorship, low care, or an error. Thresholds
name review bands only, and changing the catalog changes their meaning.
Every score is traceable to findings, but that explainability does not make the
underlying heuristic universally valid. Related signals can remain correlated
even after category damping.
## Evaluation limits
The checked-in evaluation corpus is small, purpose-built, and intended to catch
behavioral regressions. Its current perfect precision and recall reflect fixture
construction, not real-world accuracy. It contains only one clean document and
must not justify deployment claims or default-weight tuning. Representative,
licensed, domain-specific samples and independent human review are required for
such decisions.
## Input and operational boundaries
Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8,
encrypted archives, unsafe archive members, and configured resource-limit
violations are errors rather than clean results. Generated, vendor, dependency,
and minified paths are skipped by default. The formatter handles strict UTF-8
files and configured ASCII replacements only; it does not rewrite ZIP-based
containers, prose, or source structure. Recursive writes require explicit
`--write --recursive` and preflight the complete input set.
The LSP intentionally handles open editor snapshots only and uses full-document
synchronization. It publishes the configured heuristic catalog but does not
publish a document-level evidence score; editor diagnostics remain individual
review signals. It does not read or write a document URI, mutate archives,
provide prose rewrites, or implement incremental range changes. Only configured
ASCII replacements receive quick fixes or formatting edits. Under the default
policy this includes printable-ASCII UTS #39 skeletons, which can alter valid
Greek, Cyrillic, mathematical, or other text; use `review` for non-mutating
diagnostics. Invisible and bidirectional-control findings are diagnostic-only, and
configuration reload errors retain the last valid configuration and compiled
detector.
Analysis is local and performs no network lookup. Consequently, it cannot check
citations, current facts, provenance, licensing, or similarity to external
sources.

93
docs/output.md Normal file
View File

@ -0,0 +1,93 @@
# Detector output contracts
The detector exposes four formats through `--output`. All formats preserve the
same deterministic document and finding order. Requested paths retain command
line order, directory entries are sorted by filename, archive order follows the
container, and findings are sorted by byte span and rule ID.
## Human
`--output human` renders every finding as a plain `miette` diagnostic with the
source name, exact excerpt and primary span, severity, stable rule code,
message, and remediation help. A final line per document reports the low-care
evidence score/band and advice, warning, and error counts. The narratable renderer
deliberately avoids ANSI escapes.
## Miette
`--output miette` is the default. It renders the same typed diagnostics with
Miette's graphical Unicode source annotations and ANSI severity colors,
followed by the same summary. Each diagnostic starts with a `◆`-marked rule-code
title immediately followed by its severity/message, and ends with an empty line
that separates it from the next diagnostic or summary. This mode requires the
default `fancy-diagnostics` Cargo feature; builds made with
`--no-default-features`
accept the option but fall back to the plain human renderer. Select `human`,
`concise`, or `json` when ANSI escapes are not appropriate.
## Concise
`--output concise` emits one line per finding followed by one summary line. A
finding line contains the source identity, half-open UTF-8 byte range, severity,
rule ID, and message. Recoverable input errors go to stderr in human and concise
modes.
## JSON Lines schema version 2
`--output json` emits one compact JSON object per line. An analyzed document has
`kind: "analysis"` and this shape:
```json
{
"schema_version": 2,
"kind": "analysis",
"source": "article.md",
"summary": {
"evidence_score": 17,
"evidence_band": "below-threshold",
"advice": 1,
"warnings": 0,
"errors": 0,
"total": 1
},
"findings": [
{
"rule_id": "unicode-ascii-replacement",
"category": "typography",
"severity": "advice",
"message": "a Unicode character or sequence has a configured ASCII replacement",
"help": "Review the character or apply the configured ASCII replacement.",
"byte_span": { "start": 8, "end": 11 },
"matched_text": "—",
"weight": 10.0,
"fix": {
"byte_span": { "start": 8, "end": 11 },
"replacement": "-"
}
}
]
}
```
`evidence_band` is `null` when there is no evidence. Spans are zero-based,
half-open UTF-8 byte offsets into the exact source snapshot, not character or
UTF-16 positions. `fix` is `null` for findings without a safe configured edit.
A recoverable input failure has `kind: "input-error"`:
```json
{
"schema_version": 2,
"kind": "input-error",
"source": "broken.txt",
"message": "input broken.txt is not valid UTF-8"
}
```
JSON records are the only stdout content in JSON mode and never contain ANSI
escapes or logs. Fatal argument/configuration/setup failures can occur before
the JSON stream starts; they return status 2, leave stdout empty, and report on
stderr. Additive fields may appear within schema version 2; incompatible changes
require a new schema version. Schema version 2 renamed the version 1 summary
fields `confidence_score` and `confidence_band` to `evidence_score` and
`evidence_band`; version 1 records are no longer emitted.

63
docs/releasing.md Normal file
View File

@ -0,0 +1,63 @@
# Release procedure
Releases are built from a clean, reviewed Linux checkout with the exact Rust
toolchain recorded in `BUILD-INFO`.
If publishing Rust packages, publish and verify `john-slop-core` first,
`john-slop-lsp` second, and the root `john-slop` facade last. All three versions
must remain synchronized; path dependencies carry matching registry versions.
Before publication, assemble all three crates in Cargo's dependency order with
`scripts/package-workspace.sh --no-verify`. The flag is required until the
matching core and LSP versions exist in the registry; after publication, rerun
the script without it to verify the registry-form dependency graph.
1. Ensure `CHANGELOG.md`, `Cargo.toml`, `Cargo.lock`, documentation, and the
version badge agree; run every command in `docs/development.md`. Milestone
`v0.1.N` completion commits use package version `0.1.N`; regenerate the badge
only after updating the manifest and lockfile.
2. Run normal Criterion sampling and the fuzz targets appropriate to changes.
3. Commit the release preparation and create signed or annotated tag `vX.Y.Z`.
4. Check out that tag and build the release archive:
```sh
scripts/package-release.sh X.Y.Z
```
5. Run the packaging command a second time in a fresh checkout with the same
Rust version and compare the archive SHA-256 digest.
6. Inspect `BUILD-INFO`, `SHA256SUMS`, binary help, archive contents, and recorded
sizes. Sign the archive and checksum if a signing identity is available.
7. Publish the archive, archive checksum, changelog excerpt, and signature
together. Never publish an untagged or dirty build as a release.
`SOURCE_DATE_EPOCH` comes from the tagged commit. The release profile uses thin
LTO, one codegen unit, abort-on-panic, and symbol stripping; GNU tar normalizes
entry order, timestamps, ownership, and gzip metadata. Reproducibility still
depends on the same target, Rust toolchain, linker, and system libraries, all of
which must be recorded or controlled by the release environment.
The packaging smoke test produced byte-identical archives in two consecutive
runs on the Milestone 7 development host and validated both archive and internal
checksums.
## Publish with rust_realease
Install [`rust_realease`](https://proxy.dandaedre.com/gitea/idle-org/rust_realease),
then pass the path of a readable Gitea token file to the repository wrapper:
```sh
./release.sh /path/to/token.txt --dry-run
./release.sh /path/to/token.txt
```
The wrapper keeps the token out of command-line arguments, exports it as
`GITEA_TOKEN` only to `rust_realease`, selects the `john-slop` workspace package,
and publishes cargo-binstall archives alongside raw binary artifacts. The
project-local `rust-release.toml` maps the SSH development origin to the
canonical HTTPS proxy for release API and download operations without modifying
the Git remote. A hyphenated Cargo version such as
`0.1.27-rc.8` automatically marks the Gitea release as a prerelease. Additional
arguments are forwarded, so use `--dry-run` before publication and
`--replace-assets` only after reviewing an existing release collision. The
checkout must be clean and its local and remote `vX.Y.Z` tag must match the
package version as required by `rust_realease`.

106
docs/review-signals.md Normal file
View File

@ -0,0 +1,106 @@
# Review signals and evidence methodology
John Slop finds concrete text that may deserve review. It does **not** determine
whether AI wrote a document, identify a model, or assign an authorship
probability. A finding names the exact matched text and the configured reason
for reviewing it; the evidence score is only a bounded aggregation of those
findings.
This boundary matters because general-purpose AI-text detection is not reliable
enough to support an authorship claim. Sadasivan et al. show both practical
paraphrasing attacks and a theoretical trade-off between detection and false
positives as human and model distributions become closer [1]. A comparative
evaluation of 14 detection tools likewise found them neither accurate nor
reliable, with performance worsening after obfuscation [2]. Liang et al. found
that perplexity-based detectors disproportionately classified non-native English
writing as AI-generated [3]. John Slop therefore does not use perplexity,
burstiness, grammar, spelling, vocabulary richness, dialect, or English fluency
as evidence.
## Signal tiers
### Direct response residue
These are relatively strong, locally explainable signs that conversational
material may have been pasted into a document:
- assistant self-identification, capability, or knowledge-cutoff disclaimers;
- `User:` / `Assistant:` transcript scaffolding;
- answer or revision prefaces; and
- follow-up offers such as asking whether the reader wants an expansion.
They still do not prove authorship. A quotation, article about assistants, or
intentional transcript can contain the same text, so applicability scopes and
near-miss fixtures are part of every rule's review.
### Unfinished editing and source residue
Unresolved insertion notes, implementation placeholders, omitted-code comments,
tutorial narration, and stray Markdown fences are useful low-care signals
regardless of how they were produced. These rules ask whether work is complete;
they do not infer who or what produced it.
### Subtle population-level clues
Some patterns are meaningful across a large corpus but weak in one document.
Kobak et al. measured abrupt, population-level increases in selected style words
across more than 15 million biomedical abstracts [4]. John Slop represents this
only as `excess-vocabulary-cluster`: it requires multiple matches, has weight
`1.0`, is occurrence-capped, and says explicitly that it is not authorship
evidence. A stock closing has similarly low weight. These clues should prompt a
specificity edit at most, never an accusation.
Repeated transitions and templated phrases sit between the tiers. They activate
only on an exact conservative pattern; repetition from one rule is damped and
cannot alone create a high evidence band.
### Verification targets, not offline shape rules
Generated references and package names can be wrong. Walters and Wilder found
fabricated and substantially erroneous citations in a controlled set of
generated bibliographies [5]. Spracklen et al. measured package hallucinations
in a large code-generation study [6]. But a citation's typography or a package
name's shape cannot establish that it is fictitious. Reliable checks require an
authoritative catalog and contextual validation, while John Slop analysis is
local and makes no network calls. These remain human review targets rather than
shipped heuristic rules.
## How a rule is admitted
A shipped rule must have a stable ID, a narrow pattern, a rationale, review
help, an applicability scope, a finite non-negative weight, and an occurrence
cap. It also needs:
1. a positive fixture showing the intended match;
2. a realistic near miss showing what must not match;
3. deterministic byte spans and ordering; and
4. review of likely harm, especially language-proficiency and demographic
proxies.
Exact residue and fixable Unicode receive higher confidence than broad stylistic
clues. New research is used to propose testable review signals, not to import a
paper's population-level conclusion into an individual-document verdict.
## Scoring and editor visibility
Rule weight contributes bounded evidence to the shared score. Per-rule caps and
correlation damping prevent repeated copies of one pattern from dominating it.
The number is neither accuracy nor probability; see
[Heuristics and evidence scoring](heuristics.md) for the formula.
LSP visibility is a separate presentation choice. By default the editor shows
only configured Unicode findings, which are exact and may have safe replacement
edits. `--all-diagnostics` enables the complete detector catalog for testing or
deliberate review. `--minimum-diagnostic-weight <WEIGHT>` is the quieter opt-in:
it keeps Unicode findings and adds only non-Unicode findings whose configured
weight meets the threshold. Filtering affects editor diagnostics, not CLI
analysis or scoring.
## References
1. Sadasivan et al., [Can AI-Generated Text be Reliably Detected?](https://arxiv.org/abs/2303.11156)
2. Weber-Wulff et al., [Testing of detection tools for AI-generated text](https://link.springer.com/article/10.1007/s40979-023-00146-z)
3. Liang et al., [GPT detectors are biased against non-native English writers](https://pmc.ncbi.nlm.nih.gov/articles/PMC10382961/)
4. Kobak et al., [Delving into LLM-assisted writing in biomedical publications through excess vocabulary](https://arxiv.org/abs/2406.07016)
5. Walters and Wilder, [Fabrication and errors in the bibliographic citations generated by ChatGPT](https://pmc.ncbi.nlm.nih.gov/articles/PMC10484980/)
6. Spracklen et al., [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen)

View File

@ -0,0 +1,72 @@
# Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested
"ambiguous Unicode characters" catalog:
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
is the generator input;
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
are easier to browse as code;
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
combines a locale-specific table with `_common` at runtime; and
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
records the upstream source and a separate invisible-rune catalog.
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
1,538 pairs; `_default` and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
## Implemented John Slop profile
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
scalar whose single-character skeleton is non-empty printable ASCII, plus nine
reviewed punctuation mappings and 95 compatibility width folds. Width folding
takes precedence when a UTS #39 visual skeleton would produce a different
target, such as fullwidth `I` mapping to `I` rather than `l`. That is 1,884
mappings in the current profile.
Each entry has a stable ID, source, target, rationale, and enabled switch, and
the same asset owns the diagnostic metadata.
The default `[unicode] confusables = "replace"` policy emits
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
`review` for identical findings without edits, or `off` to suppress generated
lookalikes. The reviewed punctuation and width-folding mappings remain active
under all three settings and can still be disabled individually.
This broad mapping is intentionally more aggressive than Gitea's locale-aware
display warning. Greek, Cyrillic, mathematical, and other characters may be
legitimate text, so projects handling multilingual content should prefer
`review` unless ASCII normalization is their explicit policy.
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
pinned commit above, including bidi embedding/override and isolate controls.
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
reported. Each other match receives an exact UTF-8 source span that adapters
convert normally. Controls never feed the formatter or LSP quick fixes because
silent deletion has no generally safe meaning.
Regenerate the checked-in policy deterministically with:
```sh
cargo run -p john-slop-core --example generate_unicode_catalog \
> crates/john-slop-core/assets/unicode.toml
```
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
Future profile updates must pin their upstream Unicode/data version, retain
provenance and licensing, test ordinary multilingual near misses, and renew the
hash-bound default-policy sign-off.

184
docs/usage.md Normal file
View File

@ -0,0 +1,184 @@
# Usage
## Detector
Run the detector over one or more paths:
```sh
cargo run --bin john-slop -- [--fail-score 0..100] <PATH>...
cargo run --bin john-slop -- --stdin [--stdin-filename article.md]
```
Each path may be a UTF-8 text file, a directory, a ZIP archive, or a supported
ZIP-based office document (`.docx`, `.xlsx`, or `.pptx`). Directory traversal and
archive reading stream one result at a time, so callers do not need to retain an
entire input tree in memory.
`--stdin` reads one bounded strict UTF-8 document from standard input. Supply
`--stdin-filename PATH` when extension, language, and path scopes should use a
virtual filename. Standard input can be analyzed alongside filesystem paths.
Repeat `--ignore GLOB` to skip matching file or archive-member paths. Resource
limits can be overridden with `--max-document-bytes`, `--max-archive-bytes`,
`--max-expanded-bytes`, `--max-archive-entries`,
`--max-compression-ratio`, and `--max-walk-depth`.
Select output with `--output human`, `--output miette`, `--output concise`, or
`--output json`. Miette is the default and uses colored Unicode graphical source
annotations when the default `fancy-diagnostics` feature is enabled. Its marked
rule-code titles stay attached to their messages, while an empty line separates
complete diagnostics; it falls
back to human mode without that feature. Human mode renders uncolored plain
source excerpts, exact labels, remediation help, and a summary. Concise mode
emits stable single-line finding and summary records. JSON mode emits one object
per analyzed document with `schema_version: 2`, summary fields, exact byte spans,
and every finding. Machine modes never emit ANSI escapes or logs on stdout.
Findings cover the embedded or overridden heuristic catalog and retain exact
UTF-8 byte spans through every output mode. See [Output contracts](output.md)
for the human/concise formats and complete JSON Lines schema.
Exit statuses are stable:
- `0`: every analyzed document is below the requested evidence-score threshold;
- `1`: at least one document reaches the threshold;
- `2`: arguments are invalid or an operational/input error prevents completion.
`--fail-score SCORE` selects an exact inclusive score threshold and
defaults to 100. `--fail-level low|medium|high` instead uses the corresponding
threshold from the fully resolved configuration. The two options are mutually
exclusive. The old `--fail-confidence` spelling remains a visible alias for
migration.
Input failures are isolated per document whenever iteration can continue. Valid
documents still emit all findings, while any partial input failure makes the
final status `2` (taking precedence over threshold status `1`). Human and
concise input errors go to stderr. JSON mode emits versioned `input-error`
records on stdout alongside `analysis` records and leaves stderr empty unless a
fatal setup failure prevents the output contract from starting.
The low-care evidence score summarizes observable configured evidence. It is
not an AI-authorship probability. See [Heuristics and evidence scoring](heuristics.md) for
the formula and [Limitations](limitations.md) before using thresholds in review
workflows.
Configuration layers can be selected with `--config-name NAME` and `--config
FILE`; repeat `--diagnostics FILE` to load runtime rule catalogs without
recompilation. Command-line resource limits are the highest-precedence input overrides.
Use `john-slop config show` to print the resolved TOML and `john-slop config
paths` to inspect source precedence. See [Configuration](configuration.md) for
the schema and merge rules.
The LSP publishes Unicode findings by default. Start `john-slop-lsp` with
`--all-diagnostics` to publish the complete configured detector catalog, or use
`--minimum-diagnostic-weight WEIGHT` to retain Unicode findings and add only
non-Unicode findings at or above that weight. This editor-only filter does not
change CLI analysis, evidence scores, replacement planning, or formatting.
## Rust examples
The [`examples` directory](../examples/README.md) contains runnable Cargo
examples for direct text analysis, streaming path input, and replacement edit
planning. Run all example targets as part of the normal test matrix with
`cargo test --examples --all-features`.
Its [shared fixture manifest](../examples/fixtures/manifest.toml) pairs several
`example.txt` inputs with expected detector rules, replacement counts, and
formatted output. The executable fixture and protocol tests validate the
library, detector, formatter, and LSP against the shared replacement contract.
The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase)
prints every current prose finding in graphical Miette form and intentionally
exits `1` when its low evidence threshold is reached.
## Supported input policy
Input reads use strict UTF-8 and reject NUL-containing text. They do not follow
symlinks by default. Limits on files, walk depth, archive entries, expanded
bytes, compression ratio, and member paths protect analysis from unbounded or
unsafe input. The exact policy is recorded in
[ADR 0002](decisions/0002-input-policy-and-limits.md).
## Formatter
With no path, `john-slop-fmt` reads one bounded strict-UTF-8 document from stdin
and writes only formatted bytes to stdout. With a file path, the default remains
read-only and prints the transformed file without changing it:
```sh
john-slop-fmt < article.txt > article.ascii.txt
john-slop-fmt article.txt
john-slop-fmt --check article.txt
john-slop-fmt --diff article.txt
john-slop-fmt --write article.txt
```
The modes are mutually exclusive:
- default mode emits formatted text and exits `0`;
- `--check` emits nothing and exits `1` when any replacement is needed;
- `--diff` emits a unified diff and exits `1` when any replacement is needed;
- `--write` emits nothing and atomically replaces changed files, exiting `0`;
- argument, configuration, UTF-8, input, or write failures exit `2`.
All modes use the same embedded/user/named/explicit configuration precedence as
the detector. `--config-name NAME`, `--config FILE`, and repeatable
`--diagnostics FILE` select those layers. Diagnostic rules do not change the
formatter's replacement plan, but catalog validation stays consistent across
all adapters.
Replacement sources use deterministic longest-match semantics, and formatting
twice is byte-for-byte identical to formatting once.
Directories require `--recursive`, and recursive operation is deliberately
limited to `--check --recursive` or `--write --recursive`. Before a recursive
write, every input is read, decoded, planned, and edit-validated; a failure
prevents all writes. Changed files use a temporary file in the same directory,
copy original permissions, sync content, and replace by rename where the
platform supports it. Files needing no edit are never rewritten.
ZIP, DOCX, XLSX, and PPTX mutation is not supported. Extract and format an
individual UTF-8 file instead.
## LSP
`john-slop-lsp` is a stdio Language Server Protocol 3.17 adapter. Editors launch
it without positional arguments:
```sh
john-slop-lsp
john-slop-lsp --config-name editorial
john-slop-lsp --config ./john-slop.toml
john-slop-lsp --diagnostics ./diagnostics.toml
john-slop-lsp --all-diagnostics
john-slop-lsp --minimum-diagnostic-weight 10
```
The server advertises full-document synchronization and quick-fix code actions.
It stores only currently open UTF-8 snapshots, their URI, language ID, and
monotonically increasing version. On open or an accepted newer full-document
change it runs the same configured detector as the CLI and publishes every
applicable Unicode finding. `--all-diagnostics` publishes the complete result;
`--minimum-diagnostic-weight WEIGHT` adds only non-Unicode findings meeting the
threshold. File URI extensions select prose/source and path scopes; editor
language IDs provide the prose/source fallback for untitled or extensionless
buffers.
Each diagnostic has source `john-slop`, the stable heuristic rule ID as its
code, the configured severity and remediation text, document-version metadata,
and an exact UTF-16 range. Advice maps to LSP information, while warning and
error retain their corresponding LSP severities.
Runtime catalog rules are reloaded by
`workspace/didChangeConfiguration` together with normal configuration. Quick
fixes remain intentionally narrower than diagnostics: they are recomputed
from the current snapshot with the shared replacement planner and returned only
for findings carrying safe configured ASCII edits. Style, prompt-leakage, and
source-review findings are diagnostic-only. A request carrying stale John Slop
diagnostic metadata is rejected. Closing a document removes its snapshot and
publishes an empty diagnostic set. `workspace/didChangeConfiguration` reloads
the normal embedded, user, named, and explicit layers, recompiles the detector,
and republishes every open snapshot under the unchanged visibility filter; a failed reload keeps the last valid
configuration and detector.
Stdout is reserved for JSON-RPC framing. The server is quiet by default; pass
`--log-json` to emit lifecycle-only JSON Lines to stderr. Those records contain
event metadata, URI, and version but never document contents. Complete Helix and
Vim setup and validation steps live in [Editor setup](editors.md).

100
examples/README.md Normal file
View File

@ -0,0 +1,100 @@
# Runnable Rust examples
These Cargo examples exercise the same library boundaries used by the three
John Slop binaries:
```sh
cargo run --example analyze_text
cargo run --example analyze_path -- examples/fixtures
cargo run --example plan_replacements
cargo run --example ascii_replacements
```
- `analyze_text` analyzes an in-memory prose snapshot and prints its explainable
findings.
- `analyze_path` streams a file, directory, or supported ZIP container. Its
optional path defaults to `examples/fixtures`.
- `plan_replacements` prints the exact UTF-8 byte edits shared by the formatter
and LSP quick fixes.
- `ascii_replacements` demonstrates the embedded width-folding catalog across
all 95 printable ASCII targets.
These examples use embedded defaults so their behavior does not depend on a
developer's user configuration. Applications that want normal configuration
layering should call `john_slop::load_config()` instead.
## Shared fixture corpus
[`fixtures/manifest.toml`](fixtures/manifest.toml) describes small inputs shared
by detector CLI, formatter, and LSP testing. Several directories deliberately
contain an `example.txt`, covering clean prose, copied assistant residue, stock
style, safe Unicode punctuation replacements, and diagnostic-only Unicode
lookalikes/invisibles. A Rust source example covers source-only rules. Each
input has a checked-in expected formatted result; the Unicode review fixture's
expected result is intentionally byte-identical because those findings do not
authorize rewriting.
Exercise the currently implemented boundaries with:
```sh
cargo run --bin john-slop -- --output concise examples/fixtures/assistant/example.txt
cargo run --bin john-slop-fmt -- examples/fixtures/typography/example.txt
cargo run --bin john-slop-fmt -- --check examples/fixtures/clean/example.txt
cargo run --example analyze_path -- examples/fixtures
cargo test --test example_fixtures
cargo test --test lsp
```
The formatter integration test executes every case and compares stdout with its
expected ASCII file. The manifest also names `lsp` as a consumer, and the
in-process protocol suite proves that applying LSP edits produces the same bytes
as the formatter.
## Printable ASCII replacement showcase
[`ascii_replacements.rs`](ascii_replacements.rs) applies the embedded fullwidth
mappings for ASCII `!` through `~` and maps ideographic space to ASCII space.
Its [Unicode input](ascii-replacement-showcase/fullwidth.txt)
and [expected output](ascii-replacement-showcase/ascii.txt) display all 95
printable ASCII targets in one place:
```sh
cargo run --example ascii_replacements
```
The example reports 95 planned edits on stderr and writes only the transformed
fixture to stdout. These compatibility width folds are production defaults;
broader script-confusable replacements remain controlled separately by the
`[unicode] confusables` policy.
## Expected-failure diagnostic showcase
[`diagnostic-showcase.txt`](diagnostic-showcase.txt) contains every currently
shipped prose signal, while
[`diagnostic-showcase/source.rs`](diagnostic-showcase/source.rs) adds the
source-placeholder and tutorial-narration rules plus the reviewed
punctuation replacements, a fixable Cyrillic ASCII lookalike, a zero-width space, and a
right-to-left override. The adjacent example configuration promotes assistant
self-identification from warning to error and lifts the typography occurrence
cap for this display, making one run show Miette advice, warning, error,
fixable, and deliberately non-fixable control diagnostics together:
```sh
cargo run --bin john-slop -- \
--config examples/diagnostic-showcase.toml \
--output miette --fail-level low \
examples/diagnostic-showcase.txt \
examples/diagnostic-showcase/source.rs
```
That command is expected to exit with status `1`: reaching the configured
threshold is the behavior under demonstration. The repository test asserts the
nonzero result and representative prose/source categories rather than requiring
exact equality with an evolving catalog. Keeping new diagnostic kinds visible
here is nevertheless part of the intended maintenance workflow. Run the test
with capture disabled to print the complete report:
```sh
cargo test --test example_fixtures \
diagnostic_showcase_spans_prose_source_and_severities -- --nocapture
```

31
examples/analyze_path.rs Normal file
View File

@ -0,0 +1,31 @@
//! Stream and analyze every supported document under a path.
use std::{env, error::Error, path::PathBuf};
use john_slop::{AnalysisContext, Config, Detector, read_inputs};
fn main() -> Result<(), Box<dyn Error>> {
let path = env::args_os()
.nth(1)
.map_or_else(|| PathBuf::from("examples/fixtures"), PathBuf::from);
let config = Config::embedded();
let detector = Detector::new(&config)?;
for document in read_inputs(&config, &path)? {
match document {
Ok(document) => {
let context = AnalysisContext::for_source(&document.source);
let analysis = detector.analyze_with(&document.content, &context);
println!(
"{}: evidence_score={}, findings={}",
document.source,
analysis.evidence_score.score(),
analysis.findings.len()
);
}
Err(error) => eprintln!("input error: {error}"),
}
}
Ok(())
}

30
examples/analyze_text.rs Normal file
View File

@ -0,0 +1,30 @@
//! Analyze an in-memory document with the embedded heuristic catalog.
use std::error::Error;
use john_slop::{AnalysisContext, Config, Detector};
fn main() -> Result<(), Box<dyn Error>> {
let source = "As an AI language model, I can help—but this draft needs review.";
let detector = Detector::new(&Config::embedded())?;
let analysis = detector.analyze_with(source, &AnalysisContext::for_path("article.md"));
let summary = analysis.summary();
println!(
"evidence_score={} band={} findings={}",
summary.evidence_score,
summary
.evidence_band
.map_or("none", john_slop::EvidenceBand::as_str),
summary.findings.total()
);
for finding in analysis.findings {
println!(
"{} {}..{} {:?}",
finding.rule_id, finding.byte_span.start, finding.byte_span.end, finding.matched_text
);
}
Ok(())
}

View File

@ -0,0 +1,2 @@
Fullwidth ASCII showcase:
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

View File

@ -0,0 +1,2 @@
Fullwidth ASCII showcase:
 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~

View File

@ -0,0 +1,50 @@
//! Showcase embedded width folding for every printable ASCII target.
use std::io::Write;
use john_slop::{Config, apply_replacements, plan_replacements};
const SOURCE: &str = include_str!("ascii-replacement-showcase/fullwidth.txt");
fn showcase_config() -> Config {
Config::embedded()
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let config = showcase_config();
let edits = plan_replacements(&config, SOURCE);
let formatted = apply_replacements(SOURCE, &edits)?;
eprintln!(
"planned {} embedded replacements covering ASCII space through tilde",
edits.len()
);
std::io::stdout().write_all(formatted.as_bytes())?;
Ok(())
}
#[cfg(test)]
mod tests {
use std::collections::BTreeSet;
use super::*;
const EXPECTED: &str = include_str!("ascii-replacement-showcase/ascii.txt");
#[test]
fn showcase_maps_each_printable_ascii_target_once() {
let config = showcase_config();
let edits = plan_replacements(&config, SOURCE);
let targets = edits
.iter()
.map(|edit| edit.replacement.as_bytes()[0])
.collect::<BTreeSet<_>>();
assert_eq!(edits.len(), 95);
assert_eq!(targets, (b' '..=b'~').collect());
assert_eq!(
apply_replacements(SOURCE, &edits).expect("apply non-overlapping showcase edits"),
EXPECTED
);
}
}

35
examples/config/README.md Normal file
View File

@ -0,0 +1,35 @@
# Commented configuration examples
Copy [`config.toml`](config.toml) to the platform path printed by
`john-slop config paths` for user-wide policy. Copy [`editorial.toml`](editorial.toml)
beside it and select it with `--config-name editorial` for a named overlay.
Both are deliberately partial. Embedded defaults remain active for fields and
stable IDs they do not mention. Inspect the exact merged result before enforcing
it:
```sh
john-slop config show --config-name editorial
```
Project-local policy can use either file with `--config path/to/file.toml`.
The CLI can also create the embedded commented base in the current directory:
```sh
john-slop config init
```
It preserves an existing `config.toml` unless `--force` is explicit.
[`diagnostics.toml`](diagnostics.toml) is a complete runtime rule definition.
Place it at the `user_diagnostics` path reported by `john-slop config paths` for
automatic loading, or keep it with a project and select it explicitly:
```sh
john-slop --diagnostics examples/config/diagnostics.toml article.md
john-slop-lsp --diagnostics examples/config/diagnostics.toml
```
Generate the commented starter with `john-slop config init-diagnostics`. The
starter rule is disabled until its phrase and metadata are intentionally
customized.

View File

@ -0,0 +1,36 @@
# User-wide base layer. `schema_version` is optional in overlays, but spelling
# it out makes compatibility expectations explicit.
schema_version = 1
[diagnostics.categories]
# Category gates can silence a complete diagnostic family. All four categories
# (`typography`, `style`, `prompt-leakage`, and `source-code`) default to true.
style = true
prompt-leakage = true
[input]
# Bound each decoded document to 4 MiB for this installation.
max_document_bytes = 4194304
[thresholds]
# Evidence bands affect labels and `--fail-level`; they are not probabilities.
low = 20
medium = 50
high = 75
# Arrays merge by stable ID. A known ID needs only the fields being changed.
[[replacements]]
id = "em-dash"
# Keep typographic em dashes instead of offering an ASCII replacement.
enabled = false
[[rules]]
id = "templated-conclusion"
# Disable one rule even while its `style` category remains enabled.
enabled = false
[[rules]]
id = "source-placeholder-todo"
# Severity changes presentation; weight changes evidence-score contribution.
severity = "error"
weight = 15.0

View File

@ -0,0 +1,24 @@
# Runtime diagnostic catalog. Add data-driven rules here without rebuilding
# John Slop, then pass this file with `--diagnostics`.
schema_version = 1
[[rules]]
id = "custom-review-marker"
category = "style"
severity = "warning"
matcher = "exact-phrase"
pattern = "REVIEW BEFORE PUBLISHING"
weight = 8.0
message = "a project review marker remains in the source"
help = "Resolve or remove the marker before publishing."
occurrence_cap = 1
enabled = false
[rules.scope]
content = ["prose", "source-code"]
extensions = []
include = []
exclude = []
skip_generated = true
skip_vendor = true
skip_minified = true

View File

@ -0,0 +1,28 @@
# Named overlay selected with `--config-name editorial`. It is applied after the
# user-wide config.toml and before an explicit `--config` file.
[thresholds]
low = 15
medium = 40
high = 70
[[rules]]
id = "templated-filler"
weight = 2.0
# Scope is one value, so supplied arrays replace the embedded scope arrays.
[[rules]]
id = "assistant-self-identification"
[rules.scope]
content = ["prose"]
extensions = ["md", "txt"]
include = ["articles/**", "docs/**"]
exclude = ["docs/quoted-transcripts/**"]
skip_generated = true
skip_vendor = true
skip_minified = true
[[replacements]]
id = "en-dash"
enabled = false

View File

@ -0,0 +1,11 @@
# This example-only override demonstrates Miette's error presentation. The
# embedded catalog otherwise classifies this finding as a warning.
[[rules]]
id = "assistant-self-identification"
severity = "error"
# Let the companion source fixture display every embedded replacement character
# instead of stopping at the production catalog's normal per-document cap.
[[rules]]
id = "unicode-ascii-replacement"
occurrence_cap = 10

View File

@ -0,0 +1,12 @@
User: Write a polished summary without checking any sources.
Assistant: Certainly! Here's a comprehensive answer.
As an AI language model, I cannot browse the internet. My knowledge cutoff may
therefore make this inaccurate. It is important to note that the result is
seamless—production-ready, even.
Moreover, the design is robust.
Furthermore, the implementation is scalable.
Additionally, the solution is intuitive.
In conclusion, it is important to use this result as-is.

View File

@ -0,0 +1,7 @@
fn main() {
// First, we declare an intentionally incomplete showcase value.
// Next, we leave the implementation placeholder visible for review.
// TODO: implement
let _replacement_characters = "‐ ‑ ‒ – — ― − ﹘ ﹣ -";
let _unicode_review_only = "pаss​word‮";
}

View File

@ -0,0 +1,2 @@
Certainly! Here's a polished summary—ready to publish.
As an AI language model, I cannot browse the internet.

View File

@ -0,0 +1,2 @@
Certainly! Here's a polished summary-ready to publish.
As an AI language model, I cannot browse the internet.

View File

@ -0,0 +1 @@
The release checklist names an owner, a deadline, and a verification command.

View File

@ -0,0 +1 @@
The release checklist names an owner, a deadline, and a verification command.

View File

@ -0,0 +1,51 @@
schema_version = 1
targets = ["cli", "fmt", "lsp"]
[[cases]]
path = "clean/example.txt"
formatted = "clean/formatted.txt"
expected_rules = []
expected_replacements = 0
[[cases]]
path = "assistant/example.txt"
formatted = "assistant/formatted.txt"
expected_rules = [
"assistant-answer-preface",
"assistant-capability-disclaimer",
"assistant-self-identification",
"unicode-ascii-replacement",
]
expected_replacements = 1
[[cases]]
path = "style/example.txt"
formatted = "style/formatted.txt"
expected_rules = [
"repeated-transition",
"templated-conclusion",
"templated-filler",
]
expected_replacements = 0
[[cases]]
path = "typography/example.txt"
formatted = "typography/formatted.txt"
expected_rules = ["unicode-ascii-replacement"]
expected_replacements = 3
[[cases]]
path = "unicode-review/example.txt"
formatted = "unicode-review/formatted.txt"
expected_rules = ["unicode-ascii-replacement", "unicode-invisible-control"]
expected_replacements = 7
[[cases]]
path = "source/example.rs"
formatted = "source/formatted.rs"
expected_rules = [
"source-placeholder-todo",
"source-tutorial-narration",
"unicode-ascii-replacement",
]
expected_replacements = 1

View File

@ -0,0 +1,5 @@
fn parse() {
// First, we create the parser.
// TODO: implement the parser
let status = "draft—only";
}

View File

@ -0,0 +1,5 @@
fn parse() {
// First, we create the parser.
// TODO: implement the parser
let status = "draft-only";
}

View File

@ -0,0 +1,4 @@
Moreover, the draft repeats a transition.
Furthermore, it repeats the same pattern.
It is important to note that the claim still needs a source.
In conclusion, we can publish it without adding specifics.

View File

@ -0,0 +1,4 @@
Moreover, the draft repeats a transition.
Furthermore, it repeats the same pattern.
It is important to note that the claim still needs a source.
In conclusion, we can publish it without adding specifics.

View File

@ -0,0 +1 @@
Résumé — coût 5−2; portée – limitée.

View File

@ -0,0 +1 @@
Résumé - coût 5-2; portée - limitée.

View File

@ -0,0 +1,2 @@
The pаss​word contains lookalike and invisible characters.
Catalog sample: а𝐀!𝟘ⅿ🯹.

View File

@ -0,0 +1,2 @@
The pass​word contains lookalike and invisible characters.
Catalog sample: aA!Orn9.

View File

@ -0,0 +1,18 @@
//! Inspect the edits shared by detector, formatter, and LSP adapters.
use john_slop::{Config, apply_replacements, plan_replacements};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let source = "Review—then subtract 5−2.";
let edits = plan_replacements(&Config::embedded(), source);
println!("source: {source}");
for edit in &edits {
println!(
"replace bytes {}..{} with {:?}",
edit.byte_span.start, edit.byte_span.end, edit.replacement
);
}
println!("result: {}", apply_replacements(source, &edits)?);
Ok(())
}

1160
fuzz/Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

52
fuzz/Cargo.toml Normal file
View File

@ -0,0 +1,52 @@
[package]
name = "john-slop-fuzz"
version = "0.0.0"
publish = false
edition = "2024"
[package.metadata]
cargo-fuzz = true
[dependencies]
john-slop = { path = "..", default-features = false, features = ["lsp"] }
libfuzzer-sys = "0.4.13"
tempfile = "3.27.0"
toml = "1.1.6"
[[bin]]
name = "config-merge"
path = "fuzz_targets/config_merge.rs"
test = false
doc = false
bench = false
[[bin]]
name = "rule-loading"
path = "fuzz_targets/rule_loading.rs"
test = false
doc = false
bench = false
[[bin]]
name = "utf8-span-mapping"
path = "fuzz_targets/utf8_span_mapping.rs"
test = false
doc = false
bench = false
[[bin]]
name = "replacement-planning"
path = "fuzz_targets/replacement_planning.rs"
test = false
doc = false
bench = false
[[bin]]
name = "archive-metadata"
path = "fuzz_targets/archive_metadata.rs"
test = false
doc = false
bench = false
[workspace]
members = ["."]

21
fuzz/README.md Normal file
View File

@ -0,0 +1,21 @@
# Fuzzing
The five harnesses exercise configuration layering, complete rule loading,
UTF-8-to-UTF-16 span conversion, replacement planning/application, and hostile
ZIP bytes under deliberately small resource limits.
Install `cargo-fuzz` and run one target from the repository root:
```sh
cargo install cargo-fuzz --locked
cargo fuzz run config-merge
cargo fuzz run rule-loading
cargo fuzz run utf8-span-mapping
cargo fuzz run replacement-planning
cargo fuzz run archive-metadata
```
Checked-in seeds cover a valid partial override, a complete rule catalog,
multibyte/CRLF text, overlapping replacement opportunities, and malformed ZIP
metadata. Crashes and minimized artifacts are local build output and must not be
committed until converted into a reviewed regression test.

View File

@ -0,0 +1 @@
PKtruncated-central-directory

View File

@ -0,0 +1,3 @@
[[rules]]
id = "templated-filler"
enabled = false

View File

@ -0,0 +1 @@
one—two–three−four‐five

View File

@ -0,0 +1,31 @@
schema_version = 1
replacements = []
[thresholds]
low = 20
medium = 50
high = 75
[input]
max_document_bytes = 1024
max_archive_bytes = 2048
max_expanded_bytes = 4096
max_archive_entries = 4
max_compression_ratio = 10
max_walk_depth = 4
[[rules]]
id = "seed-rule"
category = "style"
severity = "advice"
matcher = "regex"
pattern = "seed+"
weight = 1.0
message = "seed match"
help = "review the seed"
occurrence_cap = 1
enabled = true
[rules.scope]
content = ["prose"]
extensions = ["txt"]

View File

@ -0,0 +1,2 @@
alpha
é😀é

View File

@ -0,0 +1,27 @@
#![no_main]
use std::fs;
use john_slop::{Config, read_inputs};
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
let Ok(directory) = tempfile::tempdir() else {
return;
};
let path = directory.path().join("input.zip");
if fs::write(&path, data).is_err() {
return;
}
let mut config = Config::embedded();
config.input.max_archive_bytes = 64 * 1024;
config.input.max_expanded_bytes = 128 * 1024;
config.input.max_document_bytes = 16 * 1024;
config.input.max_archive_entries = 32;
config.input.max_compression_ratio = 20;
if let Ok(documents) = read_inputs(&config, &path) {
for document in documents.take(33) {
let _ = document;
}
}
});

View File

@ -0,0 +1,23 @@
#![no_main]
use std::fs;
use john_slop::{ConfigLoader, ConfigPaths};
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
let Ok(directory) = tempfile::tempdir() else {
return;
};
let explicit = directory.path().join("fuzz.toml");
if fs::write(&explicit, data).is_err() {
return;
}
let loader = ConfigLoader::from_paths(ConfigPaths {
directory: directory.path().to_path_buf(),
base: directory.path().join("absent-base.toml"),
named: Some(directory.path().join("absent-named.toml")),
explicit: Some(explicit),
});
let _ = loader.load();
});

View File

@ -0,0 +1,17 @@
#![no_main]
use john_slop::{Config, apply_replacements, plan_replacements};
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
let Ok(source) = std::str::from_utf8(data) else {
return;
};
let config = Config::embedded();
let edits = plan_replacements(&config, source);
let Ok(formatted) = apply_replacements(source, &edits) else {
return;
};
let second = plan_replacements(&config, &formatted);
let _ = apply_replacements(&formatted, &second);
});

View File

@ -0,0 +1,16 @@
#![no_main]
use john_slop::{Config, Detector};
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
let Ok(source) = std::str::from_utf8(data) else {
return;
};
let Ok(config) = toml::from_str::<Config>(source) else {
return;
};
if config.validate().is_ok() {
let _ = Detector::new(&config);
}
});

View File

@ -0,0 +1,19 @@
#![no_main]
use john_slop::lsp::LineIndex;
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
let Ok(source) = std::str::from_utf8(data) else {
return;
};
let index = LineIndex::new(source);
for offset in 0..=source.len().saturating_add(1) {
let _ = index.position(offset);
}
if !source.is_empty() {
let midpoint = source.len() / 2;
let _ = index.range(midpoint..source.len());
let _ = index.range(source.len()..midpoint);
}
});

43
release.sh Executable file
View File

@ -0,0 +1,43 @@
#!/bin/sh
set -eu
usage() {
echo "usage: $0 <gitea-token-file> [rust_realease options...]" >&2
exit 2
}
[ "$#" -ge 1 ] || usage
token_file=$1
shift
if [ ! -f "$token_file" ] || [ ! -r "$token_file" ]; then
echo "token file is not a readable regular file: $token_file" >&2
exit 2
fi
if ! command -v rust_realease >/dev/null 2>&1; then
echo "rust_realease is not installed or not available in PATH" >&2
exit 2
fi
GITEA_TOKEN=
IFS= read -r GITEA_TOKEN <"$token_file" || :
if [ -z "$GITEA_TOKEN" ]; then
echo "token file is empty: $token_file" >&2
exit 2
fi
export GITEA_TOKEN
repository_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
manifest_version=$(sed -n 's/^version = "\([^"]*\)"/\1/p' "$repository_dir/Cargo.toml" | head -n 1)
set -- \
"$repository_dir" \
--package john-slop \
--artifact-format both \
"$@"
case "$manifest_version" in
*-*) set -- "$@" --prerelease ;;
esac
exec rust_realease "$@"

3
rust-release.toml Normal file
View File

@ -0,0 +1,3 @@
[[repository.fallbacks]]
origin = "git@git.dandaedre.com:idle-org/john-slop.git"
fallback-url = "https://proxy.dandaedre.com/gitea/idle-org/john-slop"

270
scripts/make-badge.sh Executable file
View File

@ -0,0 +1,270 @@
#!/bin/sh
set -eu
usage() {
cat >&2 <<'EOF'
usage: $0 <version|test|coverage> [options]
Render one deterministic SVG badge. Defaults:
version --text version --value unknown --color #007ec6
test --text tests --value passing --color #4c1
coverage --text coverage --value - --color #9f9f9f --suffix %
Options:
--text TEXT left-hand label
--value VALUE right-hand value
--color COLOR value background (#RGB or #RRGGBB)
--size PX font size, from 6 through 24 (default: 11)
--height PX badge height, from 14 through 64 (default: 20)
--label-width PX fixed label width instead of automatic sizing
--value-width PX fixed value width instead of automatic sizing
--suffix TEXT text appended to the displayed value
--output PATH output file (default: coverage/<kind>.svg)
-h, --help show this help
Examples:
./ci-coverage-badges.sh version --value 1.2.3
./ci-coverage-badges.sh test --value failing
./ci-coverage-badges.sh coverage --value 87.4
./ci-coverage-badges.sh version --text release --value rc.12 --size 12 --color '#8a2be2'
EOF
exit "${1:-2}"
}
[ "$#" -ge 1 ] || usage
kind=$1
shift
case "$kind" in
version)
text=version
value=unknown
color='#007ec6'
suffix=
output=coverage/version.svg
;;
test)
text=tests
value=passing
color=
suffix=
output=coverage/test.svg
;;
coverage)
text=coverage
value=-
color=
suffix=%
output=coverage/coverage.svg
;;
-h|--help) usage 0 ;;
*)
echo "unsupported badge kind: $kind" >&2
usage
;;
esac
font_size=11
height=20
label_width=
value_width=
color_set=false
while [ "$#" -gt 0 ]; do
option=$1
case "$option" in
--text|--value|--color|--size|--height|--label-width|--value-width|--suffix|--output)
[ "$#" -ge 2 ] || {
echo "missing value for $option" >&2
exit 2
}
option_value=$2
shift 2
case "$option" in
--text) text=$option_value ;;
--value) value=$option_value ;;
--color) color=$option_value; color_set=true ;;
--size) font_size=$option_value ;;
--height) height=$option_value ;;
--label-width) label_width=$option_value ;;
--value-width) value_width=$option_value ;;
--suffix) suffix=$option_value ;;
--output) output=$option_value ;;
esac
;;
-h|--help) usage 0 ;;
*)
echo "unknown option: $option" >&2
usage
;;
esac
done
validate_text() {
field_name=$1
field_value=$2
if ! printf '%sX\n' "$field_value" | LC_ALL=C awk '
BEGIN { valid = 1 }
NR != 1 || $0 !~ /^[ -~]+X$/ || length($0) <= 1 { valid = 0 }
END { exit(valid == 0 ? 1 : 0) }
'; then
echo "$field_name must contain printable ASCII characters: $field_value" >&2
exit 2
fi
}
validate_integer() {
field_name=$1
field_value=$2
minimum=$3
maximum=$4
case "$field_value" in
''|*[!0-9]*)
echo "$field_name must be an integer from $minimum through $maximum: $field_value" >&2
exit 2
;;
esac
if [ "$field_value" -lt "$minimum" ] || [ "$field_value" -gt "$maximum" ]; then
echo "$field_name must be an integer from $minimum through $maximum: $field_value" >&2
exit 2
fi
}
validate_text text "$text"
validate_text value "$value"
if [ -n "$suffix" ]; then
validate_text suffix "$suffix"
fi
validate_integer size "$font_size" 6 24
validate_integer height "$height" 14 64
if [ -n "$label_width" ]; then
validate_integer label-width "$label_width" 20 1000
fi
if [ -n "$value_width" ]; then
validate_integer value-width "$value_width" 20 1000
fi
case "$output" in
''|*/) echo "output must name an SVG file: $output" >&2; exit 2 ;;
esac
case "$kind" in
test)
if [ "$color_set" = false ]; then
case "$value" in
passing) color='#4c1' ;;
failing) color='#e05d44' ;;
*) color='#9f9f9f' ;;
esac
fi
;;
coverage)
if [ "$value" = '-' ]; then
[ "$color_set" = true ] || color='#9f9f9f'
else
if ! awk -v candidate="$value" 'BEGIN {
valid = candidate ~ /^[0-9]+([.][0-9]+)?$/ && candidate >= 0 && candidate <= 100
exit(valid ? 0 : 1)
}'; then
echo "coverage value must be - or a number from 0 through 100: $value" >&2
exit 2
fi
value=$(awk -v candidate="$value" 'BEGIN { printf "%.0f", candidate }')
if [ "$color_set" = false ]; then
if [ "$value" -ge 90 ]; then
color='#4c1'
elif [ "$value" -ge 70 ]; then
color='#dfb317'
else
color='#e05d44'
fi
fi
fi
;;
esac
case "$color" in
\#[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]|\#[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]) ;;
*) echo "color must be #RGB or #RRGGBB: $color" >&2; exit 2 ;;
esac
display_value=$value$suffix
if [ -z "$label_width" ]; then
label_width=$(printf '%s\\n' "$text" | awk -v size="$font_size" '{
width = int(length($0) * size * 0.64 + 14)
print (width < 35 ? 35 : width)
}')
fi
if [ -z "$value_width" ]; then
value_width=$(printf '%s\\n' "$display_value" | awk -v size="$font_size" '{
width = int(length($0) * size * 0.64 + 14)
print (width < 35 ? 35 : width)
}')
fi
badge_width=$((label_width + value_width))
label_center=$(awk -v width="$label_width" 'BEGIN { printf "%.1f", width / 2 }')
value_center=$(awk -v left="$label_width" -v width="$value_width" 'BEGIN {
printf "%.1f", left + width / 2
}')
vertical_center=$(awk -v badge_height="$height" 'BEGIN { printf "%.1f", badge_height / 2 }')
xml_escape() {
sed \
-e 's/&/\&amp;/g' \
-e 's/</\&lt;/g' \
-e 's/>/\&gt;/g' \
-e 's/"/\&quot;/g'
}
sed_replacement_escape() {
sed -e 's/[\\&|]/\\&/g'
}
escaped_text=$(printf '%s' "$text" | xml_escape | sed_replacement_escape)
escaped_value=$(printf '%s' "$display_value" | xml_escape | sed_replacement_escape)
escaped_title=$(printf '%s: %s' "$text" "$display_value" | xml_escape | sed_replacement_escape)
output_dir=${output%/*}
if [ "$output_dir" != "$output" ]; then
mkdir -p "$output_dir"
fi
temporary="${output}.tmp.$$"
trap 'rm -f "$temporary"' EXIT HUP INT TERM
sed \
-e "s/@BADGE_WIDTH@/$badge_width/g" \
-e "s/@HEIGHT@/$height/g" \
-e "s/@LABEL_WIDTH@/$label_width/g" \
-e "s/@VALUE_WIDTH@/$value_width/g" \
-e "s/@LABEL_CENTER@/$label_center/g" \
-e "s/@VALUE_CENTER@/$value_center/g" \
-e "s/@VERTICAL_CENTER@/$vertical_center/g" \
-e "s/@FONT_SIZE@/$font_size/g" \
-e "s/@COLOR@/$color/g" \
-e "s|@TITLE@|$escaped_title|g" \
-e "s|@TEXT@|$escaped_text|g" \
-e "s|@VALUE@|$escaped_value|g" \
>"$temporary" <<'EOF'
<svg xmlns="http://www.w3.org/2000/svg" width="@BADGE_WIDTH@" height="@HEIGHT@" role="img" aria-label="@TITLE@">
<title>@TITLE@</title>
<linearGradient id="s" x2="0" y2="100%">
<stop offset="0" stop-color="#fff" stop-opacity=".7"/>
<stop offset=".1" stop-color="#aaa" stop-opacity=".1"/>
<stop offset=".9" stop-opacity=".3"/>
<stop offset="1" stop-opacity=".5"/>
</linearGradient>
<clipPath id="r"><rect width="@BADGE_WIDTH@" height="@HEIGHT@" rx="3"/></clipPath>
<g clip-path="url(#r)">
<rect width="@LABEL_WIDTH@" height="@HEIGHT@" fill="#555"/>
<rect x="@LABEL_WIDTH@" width="@VALUE_WIDTH@" height="@HEIGHT@" fill="@COLOR@"/>
<rect width="@BADGE_WIDTH@" height="@HEIGHT@" fill="url(#s)"/>
</g>
<g fill="#fff" text-anchor="middle" dominant-baseline="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="@FONT_SIZE@">
<text x="@LABEL_CENTER@" y="@VERTICAL_CENTER@">@TEXT@</text>
<text x="@VALUE_CENTER@" y="@VERTICAL_CENTER@">@VALUE@</text>
</g>
</svg>
EOF
mv "$temporary" "$output"
trap - EXIT HUP INT TERM

Some files were not shown because too many files have changed in this diff Show More