Compare commits
159 Commits
poc-replac
...
main
| Author | SHA1 | Date |
|---|---|---|
|
|
3d78c0ca2a | |
|
|
002b945e9b | |
|
|
d2f6f01fcd | |
|
|
d672cfdf71 | |
|
|
cba3855ce3 | |
|
|
7dd69d6973 | |
|
|
e049842194 | |
|
|
fcc29aaefc | |
|
|
9c1e5bc083 | |
|
|
ee7e9d00d1 | |
|
|
42d49b3e7a | |
|
|
57781939a3 | |
|
|
f9c9a79797 | |
|
|
5c2ff0dca9 | |
|
|
dc21a5bf34 | |
|
|
f52f61f4c4 | |
|
|
6e4e27f774 | |
|
|
f418f01216 | |
|
|
d59a9dc471 | |
|
|
01d72c4600 | |
|
|
4a0c7c9920 | |
|
|
590202da05 | |
|
|
aedc130082 | |
|
|
22308219c8 | |
|
|
861fba5a81 | |
|
|
215a164a04 | |
|
|
501328d05f | |
|
|
70ad6e5cf8 | |
|
|
62a64ad816 | |
|
|
964347d041 | |
|
|
ae448ad60c | |
|
|
cd15bb8503 | |
|
|
cdc05af416 | |
|
|
62a2b6a608 | |
|
|
f229c1445c | |
|
|
82afdc7a3f | |
|
|
893fbfdd67 | |
|
|
73e3ecc761 | |
|
|
98a74e4c61 | |
|
|
b42ac6f221 | |
|
|
62d99e3015 | |
|
|
d43850b5a3 | |
|
|
14e589d52c | |
|
|
a3c3946776 | |
|
|
344889c78f | |
|
|
3281bd5a44 | |
|
|
8d927f1699 | |
|
|
bff852ab4b | |
|
|
6aad159fe8 | |
|
|
8efa35c45a | |
|
|
73ed837812 | |
|
|
8cf91c0507 | |
|
|
dbf4049384 | |
|
|
cf21ff1b9b | |
|
|
d49363e80e | |
|
|
ca8a352a40 | |
|
|
e9667a6842 | |
|
|
f57758d355 | |
|
|
184c66325a | |
|
|
a90e27a27e | |
|
|
933066700f | |
|
|
bee9367360 | |
|
|
5c133e24c3 | |
|
|
1fbc44e9db | |
|
|
c12b6b8551 | |
|
|
95d4067dc8 | |
|
|
a4dd4e2c5f | |
|
|
7d365af5df | |
|
|
d6163076d9 | |
|
|
30445d7983 | |
|
|
6743a2f373 | |
|
|
2001b7070f | |
|
|
e33f47e285 | |
|
|
2c02cc201b | |
|
|
8d1965b4c5 | |
|
|
5442f3ccec | |
|
|
81a09af61e | |
|
|
c46563845c | |
|
|
55e438d6ef | |
|
|
73a344c0ac | |
|
|
d4a2967ac8 | |
|
|
ab9f54b22d | |
|
|
a67ee320e7 | |
|
|
e42169da3d | |
|
|
0e4f018a61 | |
|
|
eeabed9010 | |
|
|
b40112a724 | |
|
|
0cb7ec3011 | |
|
|
2f12c4e186 | |
|
|
dcae1b1b06 | |
|
|
883fb13c12 | |
|
|
5e99a59f8d | |
|
|
90c2848d63 | |
|
|
a112b4bcb0 | |
|
|
01be964765 | |
|
|
453f99f70d | |
|
|
837b1fce55 | |
|
|
033be96d79 | |
|
|
709ed232f4 | |
|
|
39c6eacdf3 | |
|
|
d98db5252d | |
|
|
2ceb6a6c5a | |
|
|
f1daf590f4 | |
|
|
1bacd8a75d | |
|
|
65592760f3 | |
|
|
03d5138475 | |
|
|
f2360c3d60 | |
|
|
bc8881132d | |
|
|
27b4d70962 | |
|
|
c763b3e664 | |
|
|
f3e53b1913 | |
|
|
593e91a4bc | |
|
|
7d8a45433a | |
|
|
cf4f0be12d | |
|
|
c9b280e9fe | |
|
|
04a3a0b0d7 | |
|
|
0428cd8a9a | |
|
|
d9e4e378ea | |
|
|
283a0542b9 | |
|
|
f1fe9f5d8d | |
|
|
f8e2c9ad8e | |
|
|
0c9c117d5e | |
|
|
c62318304a | |
|
|
6fe5f90c17 | |
|
|
536009d32d | |
|
|
aad482e8e4 | |
|
|
90f64fd8ac | |
|
|
10d607d423 | |
|
|
43ede9c717 | |
|
|
7bb9781064 | |
|
|
72fe28a94c | |
|
|
8226989ce4 | |
|
|
18ec49ceb4 | |
|
|
8b2a02309a | |
|
|
dcdf958e9c | |
|
|
177337c0ae | |
|
|
845fb891f2 | |
|
|
4a065d4fc9 | |
|
|
25560fe5d2 | |
|
|
239b5198e4 | |
|
|
84fefd4b01 | |
|
|
8f3bb6de71 | |
|
|
eb1619787d | |
|
|
f1ef56b798 | |
|
|
95e6930a42 | |
|
|
a9bfca053b | |
|
|
648a4d50cb | |
|
|
b54c03744b | |
|
|
4c29830891 | |
|
|
1904979405 | |
|
|
5bffeea1fb | |
|
|
c88c5b8de2 | |
|
|
7b5e6b330f | |
|
|
4d142d60ed | |
|
|
acfb775877 | |
|
|
7651667025 | |
|
|
a41d13444a | |
|
|
a144f47452 | |
|
|
27a8494dc7 |
|
|
@ -4,9 +4,7 @@
|
||||||
debug/
|
debug/
|
||||||
target/
|
target/
|
||||||
|
|
||||||
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
|
# Cargo.lock is tracked because this package ships executable binaries.
|
||||||
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
|
|
||||||
Cargo.lock
|
|
||||||
|
|
||||||
# These are backup files generated by rustfmt
|
# These are backup files generated by rustfmt
|
||||||
**/*.rs.bk
|
**/*.rs.bk
|
||||||
|
|
@ -19,3 +17,19 @@ Cargo.lock
|
||||||
# Added by cargo
|
# Added by cargo
|
||||||
|
|
||||||
/target
|
/target
|
||||||
|
/dist/
|
||||||
|
|
||||||
|
# Local implementation journal (intentionally not committed)
|
||||||
|
/recap.md
|
||||||
|
|
||||||
|
# Generated locally or in CI by Cargo Tarpaulin and the badge scripts
|
||||||
|
/tarpaulin-report.html
|
||||||
|
/coverage/
|
||||||
|
|
||||||
|
# Legacy cargo-cyclonedx intermediates may remain in reused CI workspaces.
|
||||||
|
**/sbom.cdx.json
|
||||||
|
|
||||||
|
# cargo-fuzz runtime output
|
||||||
|
/fuzz/artifacts/
|
||||||
|
/fuzz/coverage/
|
||||||
|
/fuzz/target/
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,277 @@
|
||||||
|
# John Slop contributor guide
|
||||||
|
|
||||||
|
## Mission and product boundaries
|
||||||
|
|
||||||
|
John Slop is a transparent, local-first heuristic linter for low-effort or
|
||||||
|
insufficiently reviewed text and source code. It reports concrete signals that
|
||||||
|
are often associated with careless AI-assisted work. It must not claim to prove
|
||||||
|
that a person used AI, identify a model, or infer authorship from writing style.
|
||||||
|
|
||||||
|
Treat the **low-care evidence score** as the bounded strength of configured
|
||||||
|
signals, not as the probability that content is AI-generated. User-facing language must say
|
||||||
|
"suspicious", "low-care signal", or "needs review" rather than making an
|
||||||
|
accusation. A result must always be explainable from the findings that produced
|
||||||
|
it.
|
||||||
|
|
||||||
|
The formatter scope is deliberately narrow: configured Unicode characters with
|
||||||
|
ASCII replacements. The generated UTS #39 profile is broad and therefore must
|
||||||
|
retain `replace`, non-mutating `review`, and `off` policies. The LSP publishes
|
||||||
|
every applicable detector finding but exposes edits only under replacement
|
||||||
|
policy, reusing the same library
|
||||||
|
implementations as the CLI and formatter. Do not add prose rewriting, generative
|
||||||
|
inference, or destructive archive editing without a separate design decision.
|
||||||
|
|
||||||
|
## Required architecture
|
||||||
|
|
||||||
|
Keep domain logic in a library and binaries as thin adapters. The intended
|
||||||
|
layout is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
src/
|
||||||
|
lib.rs compatibility facade over workspace libraries
|
||||||
|
bin/
|
||||||
|
john-slop.rs command-line detector
|
||||||
|
john-slop-fmt.rs replacement formatter
|
||||||
|
john-slop-lsp.rs stdio language server
|
||||||
|
crates/
|
||||||
|
john-slop-core/
|
||||||
|
assets/ embedded replacement and heuristic catalogs
|
||||||
|
src/ config, input, detection, diagnostics, and edits
|
||||||
|
john-slop-lsp/src/ protocol state, UTF-16 mapping, and code actions
|
||||||
|
tests/
|
||||||
|
fixtures/ small, reviewed input/output fixtures
|
||||||
|
```
|
||||||
|
|
||||||
|
Names can evolve when Rust conventions demand it, but preserve these boundaries:
|
||||||
|
|
||||||
|
- `input` acquires text and never scores or changes it.
|
||||||
|
- `detection` is pure over text plus validated rules.
|
||||||
|
- `replacement` returns non-overlapping edits and does no I/O.
|
||||||
|
- `diagnostic` renders findings but does not discover them.
|
||||||
|
- the CLI, formatter, and LSP call the same config, detector, and replacement APIs.
|
||||||
|
- filesystem writes are confined to explicit adapter methods.
|
||||||
|
|
||||||
|
Do not retain the current all-files-in-memory iterator design as the core API.
|
||||||
|
Prefer a streaming iterator of per-file results so one unreadable or binary file
|
||||||
|
does not erase findings from unrelated files. Preserve the outer archive path and
|
||||||
|
inner member name as separate fields. Reject unsafe archive paths and apply size,
|
||||||
|
entry-count, and nesting limits before extraction or decompression.
|
||||||
|
|
||||||
|
## Core data contracts
|
||||||
|
|
||||||
|
Use typed, presentation-neutral domain values. The exact names may change, but
|
||||||
|
the model must express at least:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
struct Finding {
|
||||||
|
rule_id: RuleId,
|
||||||
|
category: Category,
|
||||||
|
message: String,
|
||||||
|
help: Option<String>,
|
||||||
|
byte_span: Range<usize>,
|
||||||
|
matched_text: String,
|
||||||
|
weight: f64,
|
||||||
|
fix: Option<TextEdit>,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct Analysis {
|
||||||
|
findings: Vec<Finding>,
|
||||||
|
evidence_score: EvidenceScore,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct TextEdit {
|
||||||
|
byte_span: Range<usize>,
|
||||||
|
replacement: String,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
All internal spans are zero-based UTF-8 byte ranges into the exact source
|
||||||
|
snapshot that was analyzed. Validate boundaries before slicing. Convert spans at
|
||||||
|
the edge: `miette::SourceSpan` for terminal output and UTF-16 line/column ranges
|
||||||
|
for LSP. Keep an indexed line map per document so LSP conversions are tested and
|
||||||
|
not repeatedly O(n).
|
||||||
|
|
||||||
|
Rule IDs are stable, unique, lowercase kebab-case identifiers. Findings are
|
||||||
|
sorted by source identity, span start, then rule ID so output is deterministic.
|
||||||
|
Never use a hash-map iteration order as output order.
|
||||||
|
|
||||||
|
## Heuristics and evidence scoring
|
||||||
|
|
||||||
|
Store data-driven heuristic definitions in
|
||||||
|
`crates/john-slop-core/assets/heuristics.toml`, separately
|
||||||
|
from replacement configuration. Compile and validate rules once at startup.
|
||||||
|
Invalid regular expressions, duplicate IDs, negative/non-finite weights, invalid
|
||||||
|
thresholds, and unsafe empty matches are configuration errors—not ignored rules.
|
||||||
|
|
||||||
|
Each rule needs an ID, category, rationale/message, help text, matcher type,
|
||||||
|
weight, and optional applicability filters such as prose/source-code extensions.
|
||||||
|
Start with high-precision, independently testable signals:
|
||||||
|
|
||||||
|
- copied assistant disclaimers or knowledge-cutoff phrases;
|
||||||
|
- prompt/response scaffolding accidentally left in a document;
|
||||||
|
- templated filler and repeated conclusion/transition phrases;
|
||||||
|
- suspicious Unicode/confusable punctuation when the configured corpus expects
|
||||||
|
ASCII;
|
||||||
|
- source-code placeholders, tutorial residue, fabricated references, or comments
|
||||||
|
that merely narrate obvious code, using conservative language-specific rules.
|
||||||
|
|
||||||
|
Avoid weak demographic, dialect, spelling, grammar, vocabulary, or
|
||||||
|
English-fluency proxies. Repetition alone must not trigger a high evidence-band
|
||||||
|
verdict. Give users category and per-rule enablement controls, per-rule weight
|
||||||
|
overrides, and path/language scopes. A rule is active only when both its
|
||||||
|
category gate and individual `enabled` field are active.
|
||||||
|
|
||||||
|
Scoring must be documented and deterministic. Use bounded weighted evidence,
|
||||||
|
apply a per-rule occurrence cap, and damp correlated findings from the same rule
|
||||||
|
or family. Map the aggregate onto `0..=100` and named bands such as `low`,
|
||||||
|
`medium`, and `high`; keep thresholds configurable. Tests must pin the formula,
|
||||||
|
caps, threshold edges, and empty-input result. Never label the numeric result an
|
||||||
|
accuracy or probability unless it has been independently calibrated against a
|
||||||
|
published, representative corpus.
|
||||||
|
|
||||||
|
Every terminal finding must become a `miette::Diagnostic` with named source,
|
||||||
|
precise label, stable diagnostic code (the rule ID), severity, message, and help.
|
||||||
|
A summary score supplements findings; it never replaces their evidence. Machine
|
||||||
|
output must offer a versioned JSON schema and must not include ANSI escapes.
|
||||||
|
|
||||||
|
## Configuration contract
|
||||||
|
|
||||||
|
Use `figment` with its TOML support and Serde for typed layered configuration.
|
||||||
|
Its providers allow embedded serialized defaults and later TOML providers to
|
||||||
|
override individual values while retaining source metadata for useful errors.
|
||||||
|
Use `directories` (or an equivalent platform-aware crate) to resolve the user
|
||||||
|
configuration directory; do not hand-build it from `HOME`.
|
||||||
|
|
||||||
|
The precedence from lowest to highest is:
|
||||||
|
|
||||||
|
1. embedded `crates/john-slop-core/assets/config.toml` and
|
||||||
|
`crates/john-slop-core/assets/heuristics.toml`;
|
||||||
|
2. runtime diagnostic catalogs: the optional user-wide `diagnostics.toml`, then
|
||||||
|
explicitly selected catalogs in command-line order;
|
||||||
|
3. user base config at `$XDG_CONFIG_HOME/john-slop/config.toml`, conventionally
|
||||||
|
`~/.config/john-slop/config.toml` on Linux;
|
||||||
|
4. named user config at `$XDG_CONFIG_HOME/john-slop/<config_name>.toml` when a
|
||||||
|
validated config name is selected;
|
||||||
|
5. an explicit `--config <path>` file;
|
||||||
|
6. explicit CLI flags for the current invocation.
|
||||||
|
|
||||||
|
Diagnostic catalog files may contain only `schema_version` and `[[rules]]`.
|
||||||
|
They can add or patch data-driven rules but cannot define executable matcher
|
||||||
|
algorithms, categories, replacements, thresholds, or input policy. Normal
|
||||||
|
configuration layers intentionally load afterward so they can disable or tune
|
||||||
|
catalog rules by stable ID.
|
||||||
|
|
||||||
|
Missing optional files are allowed; malformed files are fatal and identify their
|
||||||
|
origin. A config name accepts only ASCII letters, digits, `_`, and `-`; it cannot
|
||||||
|
contain separators, `.` segments, or an extension. Decide array behavior per
|
||||||
|
field: replacement/rule tables merge by stable ID, while ignore globs and file
|
||||||
|
extensions replace as complete arrays unless the schema explicitly says
|
||||||
|
otherwise. Do not rely on a library's default array-merge semantics implicitly.
|
||||||
|
|
||||||
|
`crates/john-slop-core/assets/unicode.toml` is the canonical generated Unicode
|
||||||
|
replacement and diagnostic catalog; `config.toml` owns other defaults. Each
|
||||||
|
replacement entry has an ID, Unicode source string/code point, ASCII target,
|
||||||
|
rationale, and enabled flag. Reject duplicate sources, empty sources, non-ASCII
|
||||||
|
targets, and conflicting edits. Longer sources win matching precedence. Provide a command
|
||||||
|
that prints the resolved configuration with secrets/redacted fields if secrets
|
||||||
|
are ever introduced.
|
||||||
|
|
||||||
|
Configuration schema changes require a `schema_version`, migration notes, and
|
||||||
|
tests for precedence. Embed assets with `include_str!` so installed binaries do
|
||||||
|
not depend on the source tree or current working directory.
|
||||||
|
|
||||||
|
## CLI, formatter, and LSP behavior
|
||||||
|
|
||||||
|
Use explicit Cargo binaries named `john-slop`, `john-slop-fmt`, and
|
||||||
|
`john-slop-lsp`.
|
||||||
|
|
||||||
|
- The detector reads files/directories/stdin, reports findings, and exits `0`
|
||||||
|
below the selected threshold, `1` when findings cross it, and `2` for usage or
|
||||||
|
operational/configuration errors.
|
||||||
|
- The formatter supports stdin/stdout, `--check`, and an explicit `--write`.
|
||||||
|
Default behavior must not rewrite a file. Writes must be atomic where the
|
||||||
|
platform permits, preserve original bytes when no edit applies, preserve line
|
||||||
|
endings, and be idempotent.
|
||||||
|
- The LSP communicates only over stdin/stdout; logs go to stderr. It publishes
|
||||||
|
applicable configured heuristic diagnostics and exposes quick-fix/code-action
|
||||||
|
edits only for safe replacements. It must honor document versions, clear
|
||||||
|
diagnostics on close, handle invalid URIs without panicking, and never write
|
||||||
|
editor-owned files.
|
||||||
|
|
||||||
|
Replacement edits are generated from the original snapshot, sorted, checked for
|
||||||
|
overlap, then applied from the end of the buffer toward the beginning. The
|
||||||
|
formatter and LSP must produce byte-for-byte equivalent results for the same
|
||||||
|
source and resolved config.
|
||||||
|
|
||||||
|
## Errors, safety, and observability
|
||||||
|
|
||||||
|
- Libraries return typed errors with `thiserror`; application boundaries attach
|
||||||
|
path/rule context and render with `miette`.
|
||||||
|
- Do not use `unwrap`, `expect`, `panic!`, `todo!`, or `unreachable!` on
|
||||||
|
user-controlled input. An `expect` is acceptable only for an invariant proved
|
||||||
|
during construction and explained in a comment.
|
||||||
|
- Treat invalid UTF-8 and binary content as explicit skipped-input diagnostics
|
||||||
|
according to policy; do not silently truncate or lossily rewrite it.
|
||||||
|
- Do not follow symlinks by default. Respect ignore configuration and establish
|
||||||
|
file-size, archive-size, archive-entry, and recursion limits.
|
||||||
|
- Do not make network calls during analysis. Results should be reproducible and
|
||||||
|
private by default.
|
||||||
|
- Structured logs go to stderr and are disabled or quiet by default. Never mix
|
||||||
|
logs with JSON, formatter stdout, or the LSP protocol stream.
|
||||||
|
|
||||||
|
## Testing and quality gates
|
||||||
|
|
||||||
|
Each behavior change requires tests at the lowest useful layer plus an
|
||||||
|
integration test when it crosses a binary boundary. Include positive,
|
||||||
|
near-miss/negative, Unicode, empty, large-input, malformed-config, and ordering
|
||||||
|
cases. Regression fixtures must state why they should or should not match.
|
||||||
|
|
||||||
|
Before considering a task complete, run:
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo clippy --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --all-targets --all-features
|
||||||
|
cargo test --all-targets --no-default-features
|
||||||
|
cargo doc --no-deps
|
||||||
|
```
|
||||||
|
|
||||||
|
Run `cargo tarpaulin --config tarpaulin.toml` when tests or executable Rust code
|
||||||
|
change. It enforces the 90% line-coverage floor. Jenkins renders the version,
|
||||||
|
test, and coverage badges from package metadata and CI results; generated badge
|
||||||
|
files and status snapshots do not belong in the repository.
|
||||||
|
|
||||||
|
Also run focused CLI golden tests and LSP protocol tests once those binaries
|
||||||
|
exist. Favor property tests for replacement idempotence/non-overlap and fuzz
|
||||||
|
targets for config parsing, span conversion, archives, and replacement planning.
|
||||||
|
Benchmarks should cover a representative large document and directory; optimize
|
||||||
|
only from measured results.
|
||||||
|
|
||||||
|
## Change discipline
|
||||||
|
|
||||||
|
Keep commits and patches small enough to review. Do not combine a structural
|
||||||
|
move with new detection behavior unless the new behavior requires it. Preserve
|
||||||
|
public behavior during refactors or document the deliberate break. Update sample
|
||||||
|
configuration, schema/version notes, CLI help, and tests together.
|
||||||
|
|
||||||
|
When implementing `tasks.md`, work in dependency order, mark only verified items
|
||||||
|
complete, and add newly discovered work under the relevant milestone. Record a
|
||||||
|
short decision note in the task file when changing configuration precedence,
|
||||||
|
the score formula, a public data contract, or safety limits.
|
||||||
|
|
||||||
|
After completing and verifying each individual task, create a dedicated commit
|
||||||
|
before starting the next task. Stage only the files that belong to that task;
|
||||||
|
never include unrelated or pre-existing user changes. Use a concise conventional
|
||||||
|
commit message that describes the completed behavior. If a task cannot be
|
||||||
|
verified or committed, leave it unchecked and report the blocker instead of
|
||||||
|
combining it with later work.
|
||||||
|
|
||||||
|
After every milestone's acceptance criteria and quality gates pass, create an
|
||||||
|
annotated Git tag named `v0.1.N`, where `N` is the milestone number, on its
|
||||||
|
completion commit. Before that completion commit, set the workspace package
|
||||||
|
version and internal registry dependency requirements to `0.1.N`, refresh
|
||||||
|
`Cargo.lock` and cut the matching changelog entry. Jenkins derives the version
|
||||||
|
badge from that metadata. Verify the manifests, lockfile, changelog, and binary
|
||||||
|
`--version` output agree before tagging. Never move an existing milestone tag;
|
||||||
|
add a corrective commit and a documented successor tag if a published milestone
|
||||||
|
needs amendment.
|
||||||
|
|
@ -0,0 +1,230 @@
|
||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to John Slop are recorded here. The format follows
|
||||||
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and releases use
|
||||||
|
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.1.27-rc.8] - 2026-09-30
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Selected current Cross target images so all Linux and Windows release
|
||||||
|
artifacts compile with the declared Rust 1.98.1-or-newer toolchain.
|
||||||
|
|
||||||
|
## [0.1.27-rc.7] - 2026-09-30
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Used the publisher's repository fallback mapping so releases reach the HTTPS
|
||||||
|
proxy while the developer SSH origin remains unchanged.
|
||||||
|
|
||||||
|
## [0.1.27-rc.6] - 2026-09-30
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Published cargo-binstall `.tgz` archives alongside raw binaries so the
|
||||||
|
package metadata added in RC5 resolves to release assets.
|
||||||
|
|
||||||
|
## [0.1.27-rc.5] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added `cargo-binstall` package metadata for downloading versioned target
|
||||||
|
archives from Gitea releases.
|
||||||
|
|
||||||
|
## [0.1.27-rc.4] - 2026-09-30
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Made the release wrapper temporarily select the canonical HTTPS origin for
|
||||||
|
publication and restore the developer SSH origin whether publication succeeds
|
||||||
|
or fails.
|
||||||
|
|
||||||
|
## [0.1.27-rc.3] - 2026-09-30
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Unified the root Cargo package, repository slug, release bundles, runtime
|
||||||
|
configuration directory, documentation, and Git remote around the canonical
|
||||||
|
`john-slop` spelling. Rust source continues to import the package as
|
||||||
|
`john_slop`, because Rust identifiers cannot contain hyphens.
|
||||||
|
|
||||||
|
## [0.1.27-rc.2] - 2026-09-30
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Switched canonical repository and release metadata to the TLS-valid
|
||||||
|
`proxy.dandaedre.com/gitea` endpoint so authenticated Gitea API publication
|
||||||
|
does not fail during connection setup.
|
||||||
|
|
||||||
|
## [0.1.27-rc.1] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added a token-file wrapper around `rust_realease` for publishing tagged raw
|
||||||
|
binary release candidates without placing Gitea credentials in arguments.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Removed unused SBOM generation and unpublished Tarpaulin execution from the
|
||||||
|
Jenkins pipeline while retaining local coverage validation.
|
||||||
|
- Made reused Jenkins workspaces tolerate legacy generated release artifacts.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Replaced the yanked transitive `yoke-derive 0.8.3` lockfile entry with
|
||||||
|
`0.8.4`.
|
||||||
|
|
||||||
|
## [0.1.26] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added conservative response-residue, editorial-placeholder, source-residue,
|
||||||
|
stock-closing, and clustered excess-vocabulary rules with positive and
|
||||||
|
realistic near-miss characterization fixtures.
|
||||||
|
- Added a research-backed review-signal methodology that separates directly
|
||||||
|
reviewable residue, low-weight population clues, verification targets, and
|
||||||
|
deliberately excluded language-proficiency proxies.
|
||||||
|
- Added LSP `--all-diagnostics` and `--minimum-diagnostic-weight WEIGHT`
|
||||||
|
switches for explicit full-catalog and weighted editor review.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- The LSP now publishes only configured Unicode diagnostics by default. The
|
||||||
|
visibility policy does not alter shared detector results, evidence scores,
|
||||||
|
replacement planning, or formatting.
|
||||||
|
|
||||||
|
## [0.1.25] - 2026-09-30
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- Fullwidth forms U+FF01 through U+FF5E now width-fold one-to-one to ASCII
|
||||||
|
`!` through `~`, instead of using visually motivated UTS #39 skeletons or
|
||||||
|
remaining unchanged. Ideographic space now maps to ASCII space.
|
||||||
|
- LSP formatting, quick fixes, the standalone formatter, and detector fixes
|
||||||
|
share the corrected mappings while retaining previously shipped stable IDs.
|
||||||
|
|
||||||
|
## [0.1.24] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added restricted runtime `diagnostics.toml` catalogs for defining
|
||||||
|
exact-phrase, regex, and repeated-regex rules without recompilation.
|
||||||
|
- Added repeatable `--diagnostics <FILE>` support across the detector,
|
||||||
|
formatter, and LSP, including LSP configuration reloads.
|
||||||
|
- Added `john-slop config init-diagnostics`, a commented starter catalog, and a
|
||||||
|
complete project example.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- User and explicit diagnostic catalogs load before normal configuration so
|
||||||
|
later base, named, and explicit layers can tune or disable their stable IDs.
|
||||||
|
- Runtime catalogs reject code-backed matcher engines and unrelated settings;
|
||||||
|
executable plugin behavior remains outside the declarative configuration
|
||||||
|
boundary.
|
||||||
|
|
||||||
|
## [0.1.23] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added typed category-level diagnostic switches for `typography`, `style`,
|
||||||
|
`prompt-leakage`, and `source-code`, composed with the existing individual
|
||||||
|
rule switches through every configuration layer.
|
||||||
|
- Added `john-slop config init` and its `config generate` alias to write the
|
||||||
|
commented embedded base configuration into the current directory.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Configuration generation protects an existing `config.toml` unless
|
||||||
|
`--force` is explicitly supplied.
|
||||||
|
|
||||||
|
## [0.1.22] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added a generated, versioned Unicode policy containing 1,829 mappings and
|
||||||
|
their shared diagnostic metadata, plus an exhaustive drift test and
|
||||||
|
deterministic regeneration example.
|
||||||
|
- Added `replace`, `review`, and `off` bulk policies for printable-ASCII UTS #39
|
||||||
|
mappings.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Printable-ASCII Unicode 16.0 UTS #39 lookalikes now receive consistent fixes
|
||||||
|
from CLI diagnostics, the standalone formatter, LSP quick fixes, and LSP
|
||||||
|
document formatting under the default `replace` policy.
|
||||||
|
- Invisible and bidirectional controls remain diagnostic-only.
|
||||||
|
|
||||||
|
## [0.1.21] - 2026-09-30
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Added real LSP document formatting backed by the same resolved replacement
|
||||||
|
plan as `john-slop-fmt`.
|
||||||
|
- Added non-mutating exact-span diagnostics for Unicode 16.0 UTS #39
|
||||||
|
ASCII-lookalikes and pinned Gitea invisible/bidirectional controls.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Expanded the shared fixtures and expected-failure diagnostic showcase with
|
||||||
|
deliberately non-fixable Unicode review cases.
|
||||||
|
|
||||||
|
## [0.1.20] - 2026-09-29
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- Synchronized workspace package versions, internal dependency requirements,
|
||||||
|
lockfile metadata, binary version output, milestone tags, and the generated
|
||||||
|
version badge, with a regression contract to prevent future drift.
|
||||||
|
- Made graphical Miette output easier to scan with marked rule-code titles,
|
||||||
|
attached diagnostic bodies, and spacing between complete findings.
|
||||||
|
- Expanded the expected-failure diagnostic showcase to include prose and source
|
||||||
|
rules plus the complete embedded replacement-character set.
|
||||||
|
- Documented Cargo's required `--path .` syntax when installing all binaries
|
||||||
|
from a local source checkout.
|
||||||
|
- Restructured the README and documentation index as a progressive path from
|
||||||
|
quickstart to usage guides, with engineering detail in a dedicated advanced
|
||||||
|
guide.
|
||||||
|
- Added an opt-in Rust example that maps fullwidth Unicode forms to all 95
|
||||||
|
printable ASCII targets while leaving production defaults unchanged.
|
||||||
|
- Extended the LSP from replacement-only warnings to every applicable heuristic
|
||||||
|
finding while retaining quick fixes only for safe ASCII replacements.
|
||||||
|
- Adopted the measured three-package Cargo workspace. Shared package metadata,
|
||||||
|
dependency requirements, and lint policy now use workspace inheritance while
|
||||||
|
the reusable core remains isolated from CLI, async, and LSP dependencies.
|
||||||
|
- Renamed the ambiguous confidence metric to low-care evidence score throughout
|
||||||
|
the Rust API, CLI, terminal output, and JSON schema version 2. Deprecated Rust
|
||||||
|
type aliases and `--fail-confidence` remain available for migration; the
|
||||||
|
scoring formula and values are unchanged.
|
||||||
|
|
||||||
|
## [0.1.0] - 2026-09-29
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Local, explainable heuristic analysis with a bounded low-care evidence score.
|
||||||
|
- Layered embedded, user, named, explicit, and CLI configuration.
|
||||||
|
- File, directory, ZIP, office-container, and stdin input adapters with limits.
|
||||||
|
- Human, graphical Miette, concise, and versioned JSON detector output.
|
||||||
|
- Conservative Unicode-to-ASCII formatter with check, diff, and atomic write.
|
||||||
|
- Stdio LSP replacement diagnostics and versioned quick fixes.
|
||||||
|
- Runnable examples, reviewed fixtures, fuzz targets, benchmarks, and CI gates.
|
||||||
|
|
||||||
|
[Unreleased]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.8...HEAD
|
||||||
|
[0.1.27-rc.8]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.7...v0.1.27-rc.8
|
||||||
|
[0.1.27-rc.7]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.6...v0.1.27-rc.7
|
||||||
|
[0.1.27-rc.6]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.5...v0.1.27-rc.6
|
||||||
|
[0.1.27-rc.5]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.4...v0.1.27-rc.5
|
||||||
|
[0.1.27-rc.4]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.3...v0.1.27-rc.4
|
||||||
|
[0.1.27-rc.3]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.2...v0.1.27-rc.3
|
||||||
|
[0.1.27-rc.2]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.1...v0.1.27-rc.2
|
||||||
|
[0.1.27-rc.1]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.26...v0.1.27-rc.1
|
||||||
|
[0.1.26]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.25...v0.1.26
|
||||||
|
[0.1.25]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.24...v0.1.25
|
||||||
|
[0.1.24]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.23...v0.1.24
|
||||||
|
[0.1.23]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.22...v0.1.23
|
||||||
|
[0.1.22]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.21...v0.1.22
|
||||||
|
[0.1.21]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.20...v0.1.21
|
||||||
|
[0.1.20]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.0...v0.1.20
|
||||||
|
[0.1.0]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/releases/tag/v0.1.0
|
||||||
|
|
@ -0,0 +1,15 @@
|
||||||
|
# Code of Conduct
|
||||||
|
|
||||||
|
John Slop uses the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
|
||||||
|
|
||||||
|
Be respectful, specific, and evidence-led. Harassment, discriminatory language,
|
||||||
|
personal attacks, doxxing, sexualized conduct, and sustained disruption are not
|
||||||
|
acceptable in project spaces. Maintainers may edit, remove, or reject comments,
|
||||||
|
commits, issues, and other contributions that violate these expectations, and
|
||||||
|
may temporarily or permanently restrict participation.
|
||||||
|
|
||||||
|
Report conduct concerns privately to the repository owner through the hosting
|
||||||
|
service profile. Reports will be reviewed promptly and handled with discretion;
|
||||||
|
the maintainer will disclose information only when necessary to investigate or
|
||||||
|
protect participants. This project does not tolerate retaliation against a
|
||||||
|
reporter acting in good faith.
|
||||||
|
|
@ -0,0 +1,22 @@
|
||||||
|
# Contributing
|
||||||
|
|
||||||
|
Contributions should preserve John Slop's explainable, local-first product
|
||||||
|
boundary. Findings must describe observable signals without asserting AI
|
||||||
|
authorship, model identity, or personal characteristics.
|
||||||
|
|
||||||
|
1. Open an issue for behavior or public-contract changes before investing in a
|
||||||
|
large patch.
|
||||||
|
2. Create a focused branch and add tests at the lowest useful layer.
|
||||||
|
3. Update user documentation, samples, and stable output snapshots together.
|
||||||
|
4. Run the quality, supply-chain, fuzz-smoke, or benchmark commands relevant to
|
||||||
|
the change as documented in [Development](docs/development.md).
|
||||||
|
5. Submit small conventional commits and explain rule false-positive tradeoffs.
|
||||||
|
|
||||||
|
New heuristic rules require a stable lowercase kebab-case ID, conservative
|
||||||
|
positive and realistic near-miss fixtures, scopes, a rationale and help text,
|
||||||
|
and evaluation-corpus review. Default weights or thresholds require the
|
||||||
|
documented sign-off process in [Evaluation](docs/evaluation.md).
|
||||||
|
|
||||||
|
By participating, contributors agree to follow the
|
||||||
|
[Code of Conduct](CODE_OF_CONDUCT.md). Report vulnerabilities through
|
||||||
|
[SECURITY.md](SECURITY.md), not a public issue.
|
||||||
File diff suppressed because it is too large
Load Diff
111
Cargo.toml
111
Cargo.toml
|
|
@ -1,8 +1,109 @@
|
||||||
[package]
|
[workspace]
|
||||||
name = "john_slop"
|
members = ["crates/john-slop-core", "crates/john-slop-lsp"]
|
||||||
version = "0.1.0"
|
default-members = [".", "crates/john-slop-core", "crates/john-slop-lsp"]
|
||||||
|
resolver = "3"
|
||||||
|
|
||||||
|
[workspace.package]
|
||||||
|
version = "0.1.27-rc.8"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
rust-version = "1.98.1"
|
||||||
|
license = "MIT"
|
||||||
|
repository = "https://proxy.dandaedre.com/gitea/idle-org/john-slop"
|
||||||
|
|
||||||
|
[workspace.dependencies]
|
||||||
|
clap = { version = "4.6.7", default-features = false, features = ["derive", "error-context", "help", "std", "suggestions", "usage"] }
|
||||||
|
criterion = { version = "0.8.2", default-features = false }
|
||||||
|
directories = "6.0.0"
|
||||||
|
figment = { version = "0.10.19", default-features = false, features = ["toml"] }
|
||||||
|
futures = { version = "0.3.34", default-features = false, features = ["std"] }
|
||||||
|
globset = { version = "0.4.20", default-features = false }
|
||||||
|
insta = { version = "1.48.0", default-features = false }
|
||||||
|
john-slop-core = { version = "0.1.27-rc.8", path = "crates/john-slop-core" }
|
||||||
|
john-slop-lsp-engine = { package = "john-slop-lsp", version = "0.1.27-rc.8", path = "crates/john-slop-lsp" }
|
||||||
|
miette = { version = "7.6.0", default-features = false, features = ["derive"] }
|
||||||
|
regex = "1.13.1"
|
||||||
|
serde = { version = "1.0.229", features = ["derive"] }
|
||||||
|
serde_json = "1.0.151"
|
||||||
|
sha2 = "0.11.0"
|
||||||
|
similar = "2.7.0"
|
||||||
|
tempfile = "3.27.0"
|
||||||
|
thiserror = "2.0.21"
|
||||||
|
tokio = { version = "1.53.1", default-features = false }
|
||||||
|
toml = "1.1.6"
|
||||||
|
tower = { version = "0.5.3", default-features = false, features = ["util"] }
|
||||||
|
tower-lsp-server = { version = "0.23.0", default-features = false, features = ["runtime-tokio"] }
|
||||||
|
unicode-security = "0.1.2"
|
||||||
|
walkdir = "2.5.0"
|
||||||
|
zip = { version = "8.6.0", default-features = false, features = ["deflate-flate2-zlib-rs"] }
|
||||||
|
|
||||||
|
[workspace.lints.rust]
|
||||||
|
unsafe_code = "forbid"
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "john-slop"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
rust-version.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
description = "Inspect files, directories, or ZIP-based documents for low-care signals"
|
||||||
|
autobins = false
|
||||||
|
|
||||||
|
[package.metadata.binstall]
|
||||||
|
pkg-url = "{ repo }/releases/download/v{ version }/{ name }-{ target }-v{ version }.tgz"
|
||||||
|
pkg-fmt = "tgz"
|
||||||
|
bin-dir = "{ name }-{ target }-v{ version }/{ bin }{ binary-ext }"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "john-slop"
|
||||||
|
path = "src/bin/john-slop.rs"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "john-slop-fmt"
|
||||||
|
path = "src/bin/john-slop-fmt.rs"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "john-slop-lsp"
|
||||||
|
path = "src/bin/john-slop-lsp.rs"
|
||||||
|
required-features = ["lsp"]
|
||||||
|
|
||||||
|
[features]
|
||||||
|
default = ["fancy-diagnostics", "lsp"]
|
||||||
|
fancy-diagnostics = ["miette/fancy"]
|
||||||
|
lsp = ["dep:john-slop-lsp-engine", "dep:tokio", "dep:tower-lsp-server"]
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
miette = { version = "7.6.0", features = ["fancy"] }
|
clap.workspace = true
|
||||||
thiserror = "2.0.17"
|
globset.workspace = true
|
||||||
|
john-slop-core.workspace = true
|
||||||
|
john-slop-lsp-engine = { workspace = true, optional = true }
|
||||||
|
miette.workspace = true
|
||||||
|
serde.workspace = true
|
||||||
|
serde_json.workspace = true
|
||||||
|
similar.workspace = true
|
||||||
|
tempfile.workspace = true
|
||||||
|
thiserror.workspace = true
|
||||||
|
tokio = { workspace = true, features = ["io-std", "macros", "rt-multi-thread", "sync"], optional = true }
|
||||||
|
toml.workspace = true
|
||||||
|
tower-lsp-server = { workspace = true, optional = true }
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
criterion.workspace = true
|
||||||
|
futures.workspace = true
|
||||||
|
insta.workspace = true
|
||||||
|
sha2.workspace = true
|
||||||
|
tower.workspace = true
|
||||||
|
zip.workspace = true
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
|
|
||||||
|
[[bench]]
|
||||||
|
name = "core"
|
||||||
|
harness = false
|
||||||
|
|
||||||
|
[profile.release]
|
||||||
|
codegen-units = 1
|
||||||
|
lto = "thin"
|
||||||
|
panic = "abort"
|
||||||
|
strip = "symbols"
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,13 @@
|
||||||
|
[build]
|
||||||
|
|
||||||
|
[build.env]
|
||||||
|
passthrough = ["RUSTUP_TOOLCHAIN"]
|
||||||
|
|
||||||
|
[target.x86_64-unknown-linux-musl]
|
||||||
|
image = "ghcr.io/cross-rs/x86_64-unknown-linux-musl:main"
|
||||||
|
|
||||||
|
[target.aarch64-unknown-linux-musl]
|
||||||
|
image = "ghcr.io/cross-rs/aarch64-unknown-linux-musl:main"
|
||||||
|
|
||||||
|
[target.x86_64-pc-windows-gnu]
|
||||||
|
image = "ghcr.io/cross-rs/x86_64-pc-windows-gnu:main"
|
||||||
|
|
@ -0,0 +1,363 @@
|
||||||
|
// Add this webhook in Gitea:
|
||||||
|
// http://192.168.1.37:8080/gitea-webhook/post
|
||||||
|
|
||||||
|
pipeline {
|
||||||
|
agent { label 'rust_no_docker' }
|
||||||
|
|
||||||
|
environment {
|
||||||
|
CARGO_TERM_COLOR = 'always'
|
||||||
|
RUST_BACKTRACE = '1'
|
||||||
|
MSRV = '1.98.1'
|
||||||
|
CARGO_TARPAULIN_VERSION = '0.37.5'
|
||||||
|
CARGO_AUDIT_VERSION = '0.22.2'
|
||||||
|
CARGO_DENY_VERSION = '0.20.2'
|
||||||
|
BADGE_PUBLISH_ROOT = '/mnt/assets/coverage'
|
||||||
|
}
|
||||||
|
|
||||||
|
stages {
|
||||||
|
stage('Checkout') {
|
||||||
|
steps {
|
||||||
|
checkout scm
|
||||||
|
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
git submodule sync --recursive
|
||||||
|
git submodule update --init --recursive
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Setup Rust') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
if ! command -v rustup >/dev/null 2>&1; then
|
||||||
|
echo "ERROR: rustup is not installed on this worker." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Ensuring the stable Rust toolchain is available..."
|
||||||
|
|
||||||
|
rustup toolchain install stable \
|
||||||
|
--profile minimal \
|
||||||
|
--component rustfmt \
|
||||||
|
--component clippy
|
||||||
|
|
||||||
|
rustup toolchain install "${MSRV}" \
|
||||||
|
--profile minimal
|
||||||
|
|
||||||
|
rustup default stable
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "Using:"
|
||||||
|
rustc --version
|
||||||
|
cargo --version
|
||||||
|
rustfmt --version
|
||||||
|
cargo clippy --version
|
||||||
|
|
||||||
|
INSTALLED_AUDIT="$(cargo audit --version 2>/dev/null || true)"
|
||||||
|
if [ "${INSTALLED_AUDIT}" != "cargo-audit ${CARGO_AUDIT_VERSION}" ]; then
|
||||||
|
cargo install cargo-audit --locked --version "${CARGO_AUDIT_VERSION}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
INSTALLED_DENY="$(cargo deny --version 2>/dev/null || true)"
|
||||||
|
if [ "${INSTALLED_DENY}" != "cargo-deny ${CARGO_DENY_VERSION}" ]; then
|
||||||
|
cargo install cargo-deny --locked --version "${CARGO_DENY_VERSION}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
INSTALLED_TARPAULIN="$(cargo tarpaulin --version 2>/dev/null || true)"
|
||||||
|
if [ "${INSTALLED_TARPAULIN}" != "cargo-tarpaulin-tarpaulin ${CARGO_TARPAULIN_VERSION}" ]; then
|
||||||
|
cargo install cargo-tarpaulin --locked --version "${CARGO_TARPAULIN_VERSION}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('MSRV') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Checking all targets with Rust ${MSRV}..."
|
||||||
|
|
||||||
|
cargo "+${MSRV}" check \
|
||||||
|
--locked \
|
||||||
|
--all-targets \
|
||||||
|
--all-features
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Format') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Checking Rust formatting..."
|
||||||
|
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Check') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Checking all targets..."
|
||||||
|
|
||||||
|
cargo check \
|
||||||
|
--locked \
|
||||||
|
--all-targets \
|
||||||
|
--all-features
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Test') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Running tests..."
|
||||||
|
|
||||||
|
cargo test \
|
||||||
|
--locked \
|
||||||
|
--all-targets \
|
||||||
|
--all-features
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Test Minimal Features') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Running tests without default features..."
|
||||||
|
|
||||||
|
cargo test \
|
||||||
|
--locked \
|
||||||
|
--all-targets \
|
||||||
|
--no-default-features
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Fixtures and Goldens') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
cargo test --locked --test example_fixtures --all-features
|
||||||
|
cargo test --locked --test cli --all-features
|
||||||
|
|
||||||
|
mkdir -p coverage
|
||||||
|
: > "coverage/.tests-passed-${BUILD_NUMBER}"
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Coverage') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
cargo tarpaulin \
|
||||||
|
--config tarpaulin.toml \
|
||||||
|
--out Json \
|
||||||
|
--output-dir coverage
|
||||||
|
|
||||||
|
mv coverage/tarpaulin-report.json coverage/summary.json
|
||||||
|
coverage_percent=$(sed -n \
|
||||||
|
's/^.*],"coverage":\\([0-9][0-9.]*\\),"covered":[0-9][0-9]*,"coverable":[0-9][0-9]*}$/\\1/p' \
|
||||||
|
coverage/summary.json)
|
||||||
|
scripts/render-coverage-badge.sh \
|
||||||
|
"${coverage_percent}" \
|
||||||
|
coverage/coverage.svg
|
||||||
|
rm coverage/summary.json
|
||||||
|
: > "coverage/.coverage-ready-${BUILD_NUMBER}"
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Clippy') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Running Clippy..."
|
||||||
|
|
||||||
|
cargo clippy \
|
||||||
|
--locked \
|
||||||
|
--all-targets \
|
||||||
|
--all-features \
|
||||||
|
-- \
|
||||||
|
-D warnings
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Documentation') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
cargo doc --locked --no-deps --all-features
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Supply Chain') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
cargo audit --deny warnings
|
||||||
|
cargo deny --locked check
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Workspace Packages') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Assembling all publishable workspace packages..."
|
||||||
|
|
||||||
|
scripts/package-workspace.sh --no-verify
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stage('Build Release') {
|
||||||
|
steps {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
echo "Building John Slop release binaries..."
|
||||||
|
|
||||||
|
# Jenkins reuses workspaces. Remove only ignored release
|
||||||
|
# output so rerunning the same commit stays deterministic.
|
||||||
|
rm -rf -- dist
|
||||||
|
|
||||||
|
# The packaging script reads and validates Cargo.toml itself.
|
||||||
|
scripts/package-release.sh
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "Release build completed."
|
||||||
|
|
||||||
|
ls -lh \
|
||||||
|
target/release/john-slop \
|
||||||
|
target/release/john-slop-fmt \
|
||||||
|
target/release/john-slop-lsp \
|
||||||
|
dist/*.tar.gz \
|
||||||
|
dist/*.sha256
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
post {
|
||||||
|
success {
|
||||||
|
publishChecks(
|
||||||
|
conclusion: 'SUCCESS',
|
||||||
|
name: 'Rust CI',
|
||||||
|
summary: 'JohnSlop passed all CI checks.',
|
||||||
|
text: 'The Linux MSRV/stable matrix, formatting, feature tests, fixtures, coverage and badge generation, docs, Clippy, supply-chain policy, and release build completed successfully.',
|
||||||
|
title: 'Build Successful'
|
||||||
|
)
|
||||||
|
|
||||||
|
echo '✅ JohnSlop passed all CI checks.'
|
||||||
|
}
|
||||||
|
|
||||||
|
failure {
|
||||||
|
publishChecks(
|
||||||
|
conclusion: 'FAILURE',
|
||||||
|
name: 'Rust CI',
|
||||||
|
summary: 'JohnSlop failed CI.',
|
||||||
|
text: 'A toolchain, formatting, test, fixture, coverage, badge generation, documentation, Clippy, supply-chain, or release check failed. Check the Jenkins log.',
|
||||||
|
title: 'Build Failed'
|
||||||
|
)
|
||||||
|
|
||||||
|
echo '❌ JohnSlop CI failed.'
|
||||||
|
}
|
||||||
|
|
||||||
|
unstable {
|
||||||
|
publishChecks(
|
||||||
|
conclusion: 'ACTION_REQUIRED',
|
||||||
|
name: 'Rust CI',
|
||||||
|
summary: 'JohnSlop CI is unstable.',
|
||||||
|
text: 'The Jenkins build completed with an unstable status.',
|
||||||
|
title: 'Build Unstable'
|
||||||
|
)
|
||||||
|
|
||||||
|
echo '⚠️ JohnSlop CI is unstable.'
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup {
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
mkdir -p coverage
|
||||||
|
test_status=failing
|
||||||
|
if [ -f "coverage/.tests-passed-${BUILD_NUMBER}" ]; then
|
||||||
|
test_status=passing
|
||||||
|
fi
|
||||||
|
scripts/render-test-badge.sh \
|
||||||
|
"${test_status}" \
|
||||||
|
coverage/test.svg
|
||||||
|
|
||||||
|
if [ ! -f "coverage/.coverage-ready-${BUILD_NUMBER}" ]; then
|
||||||
|
scripts/render-coverage-badge.sh - coverage/coverage.svg
|
||||||
|
fi
|
||||||
|
|
||||||
|
package_version=$(awk '
|
||||||
|
$0 == "[workspace.package]" { in_package = 1; next }
|
||||||
|
substr($0, 1, 1) == "[" { in_package = 0 }
|
||||||
|
in_package && $1 == "version" && $2 == "=" {
|
||||||
|
gsub(/^"|"$/, "", $3)
|
||||||
|
print $3
|
||||||
|
exit
|
||||||
|
}
|
||||||
|
' Cargo.toml)
|
||||||
|
if [ -z "${package_version}" ]; then
|
||||||
|
echo "could not determine workspace package version" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
scripts/render-version-badge.sh \
|
||||||
|
"${package_version}" \
|
||||||
|
coverage/version.svg
|
||||||
|
|
||||||
|
rm -f \
|
||||||
|
coverage/summary.json \
|
||||||
|
coverage/tarpaulin-report.json \
|
||||||
|
"coverage/.tests-passed-${BUILD_NUMBER}" \
|
||||||
|
"coverage/.coverage-ready-${BUILD_NUMBER}"
|
||||||
|
'''
|
||||||
|
|
||||||
|
archiveArtifacts(
|
||||||
|
allowEmptyArchive: false,
|
||||||
|
artifacts: 'coverage/coverage.svg,coverage/test.svg,coverage/version.svg',
|
||||||
|
fingerprint: true
|
||||||
|
)
|
||||||
|
|
||||||
|
sh '''
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
if [ "${BRANCH_NAME:-}" = "main" ]; then
|
||||||
|
scripts/publish-badges.sh \
|
||||||
|
"${BADGE_PUBLISH_ROOT}" \
|
||||||
|
coverage \
|
||||||
|
test \
|
||||||
|
version
|
||||||
|
fi
|
||||||
|
'''
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
128
README.md
128
README.md
|
|
@ -1,9 +1,125 @@
|
||||||
# JohnSlop
|
# John Slop
|
||||||
|
|
||||||
Detect AI slop markers
|

|
||||||
|

|
||||||
|

|
||||||
|
|
||||||
|
John Slop is a local-first linter for concrete signs of low-effort or
|
||||||
|
insufficiently reviewed text and source code. It reports explainable evidence
|
||||||
|
for human review; it does not prove AI authorship or identify a model.
|
||||||
|
|
||||||
# Links
|
## Quickstart
|
||||||
* [https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing]
|
|
||||||
* [https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup]
|
With `cargo-binstall` available, install all three prebuilt binaries from the
|
||||||
* [https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion]
|
latest release candidate:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo binstall --git https://proxy.dandaedre.com/gitea/idle-org/john-slop \
|
||||||
|
john-slop
|
||||||
|
```
|
||||||
|
|
||||||
|
To build from source, Rust 1.98.1 or newer is required. Install from the
|
||||||
|
upstream Git repository:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo install --git https://proxy.dandaedre.com/gitea/idle-org/john-slop \
|
||||||
|
--locked --all-features
|
||||||
|
```
|
||||||
|
|
||||||
|
Or install a checkout from its repository root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo install --path . --locked --all-features
|
||||||
|
```
|
||||||
|
|
||||||
|
Scan a document or source directory:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop path/to/document path/to/source-directory
|
||||||
|
```
|
||||||
|
|
||||||
|
Generate a commented `config.toml` in the current directory, then edit category
|
||||||
|
or individual-rule switches as needed:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config init
|
||||||
|
john-slop config init-diagnostics
|
||||||
|
```
|
||||||
|
|
||||||
|
The second command creates a runtime rule catalog: exact-phrase and regex
|
||||||
|
diagnostics can then be added without changing or recompiling the program.
|
||||||
|
|
||||||
|
Diagnostics are colored Miette reports by default. Make the command fail when a
|
||||||
|
document reaches a chosen low-care evidence score with `--fail-score 0..100`:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop --fail-score 40 path/to/document
|
||||||
|
```
|
||||||
|
|
||||||
|
Preview configured Unicode-to-ASCII replacements, or check them in automation:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop-fmt path/to/document
|
||||||
|
john-slop-fmt --check path/to/document
|
||||||
|
```
|
||||||
|
|
||||||
|
Formatting is read-only unless `--write` is explicit. Start the editor server
|
||||||
|
with `john-slop-lsp`; the [editor guide](docs/editors.md) has ready-to-use Helix
|
||||||
|
and Vim configuration.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
John Slop provides three binaries backed by the same configuration and source
|
||||||
|
spans:
|
||||||
|
|
||||||
|
- `john-slop` detects copied assistant scaffolding, generic style residue,
|
||||||
|
source placeholders, tutorial narration, configured replacements, ASCII-like
|
||||||
|
Unicode, and invisible or bidirectional controls.
|
||||||
|
- `john-slop-fmt` previews, checks, or explicitly applies configured ASCII
|
||||||
|
replacements, including the default printable-ASCII UTS #39 profile.
|
||||||
|
- `john-slop-lsp` publishes Unicode diagnostics over stdio by default, offers
|
||||||
|
explicit weighted or all-diagnostic opt-ins, provides quick fixes for safe
|
||||||
|
replacements, and formats without writing editor-owned files.
|
||||||
|
|
||||||
|
The detector accepts UTF-8 files, directories, ZIP archives, and common
|
||||||
|
ZIP-based office documents. Each finding includes its source span, stable rule
|
||||||
|
ID, severity, message, and remediation help. A bounded evidence score aggregates
|
||||||
|
configured signals; it is not the probability that AI wrote the content.
|
||||||
|
|
||||||
|
Analysis is local and deterministic, with no network requests or telemetry.
|
||||||
|
False positives remain possible—especially in transcripts, tutorials,
|
||||||
|
multilingual text, and intentional typography—so findings are prompts for
|
||||||
|
review, not verdicts. Projects with multilingual text should consider the
|
||||||
|
Unicode profile's non-mutating `review` mode before writing formatted output.
|
||||||
|
|
||||||
|
## Learn progressively
|
||||||
|
|
||||||
|
Start with the guide closest to what you want to do:
|
||||||
|
|
||||||
|
1. [Use the CLI and understand exit statuses](docs/usage.md).
|
||||||
|
2. [Read or integrate the available output formats](docs/output.md).
|
||||||
|
3. [Configure rules, replacements, and layered overrides](docs/configuration.md),
|
||||||
|
with [commented configuration examples](examples/config/).
|
||||||
|
4. [Run the Rust and expected-failure examples](examples/README.md).
|
||||||
|
5. [Connect the formatter and LSP to Helix or Vim](docs/editors.md).
|
||||||
|
6. [Understand the review-signal methodology](docs/review-signals.md), the
|
||||||
|
[heuristics and evidence score](docs/heuristics.md), and then the
|
||||||
|
[interpretation limits](docs/limitations.md).
|
||||||
|
|
||||||
|
The [documentation index](docs/README.md) offers the same guided path with a
|
||||||
|
little more context.
|
||||||
|
|
||||||
|
## Advanced topics
|
||||||
|
|
||||||
|
Architecture, the public library API, evaluation results, performance,
|
||||||
|
dependency policy, design decisions, Unicode research, development, and release
|
||||||
|
engineering are collected in the [advanced guide](docs/advanced.md).
|
||||||
|
|
||||||
|
## Project
|
||||||
|
|
||||||
|
See [CONTRIBUTING.md](CONTRIBUTING.md) to work on the project and
|
||||||
|
[SECURITY.md](SECURITY.md) to report a vulnerability. The ordered implementation
|
||||||
|
plan lives in [tasks.md](tasks.md), and agent contribution rules live in
|
||||||
|
[AGENTS.md](AGENTS.md).
|
||||||
|
|
||||||
|
John Slop is distributed under the [MIT license](LICENSE).
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,22 @@
|
||||||
|
# Security policy
|
||||||
|
|
||||||
|
The current supported line is `0.1.x`. Older snapshots and untagged forks do not
|
||||||
|
receive security fixes.
|
||||||
|
|
||||||
|
Do not open a public issue for a suspected vulnerability. Use the repository
|
||||||
|
host's private security-advisory/reporting interface when available. Otherwise,
|
||||||
|
contact the repository owner privately through the hosting profile and request
|
||||||
|
a secure channel. Include the affected version/commit, operating system, impact,
|
||||||
|
minimal reproduction, and whether details have been shared elsewhere. Do not
|
||||||
|
include private documents processed by John Slop.
|
||||||
|
|
||||||
|
The maintainer will acknowledge a report within seven days, assess scope and
|
||||||
|
severity, coordinate a fix and disclosure date when confirmed, and credit the
|
||||||
|
reporter if requested. This is a best-effort volunteer response target, not a
|
||||||
|
service-level guarantee.
|
||||||
|
|
||||||
|
John Slop processes untrusted text and archives but is not a sandbox. Archive
|
||||||
|
limits and path validation reduce exposure; reports that bypass those limits,
|
||||||
|
cause unintended writes, leak document contents, corrupt protocol output, or
|
||||||
|
execute code are in scope. False-positive disagreements and requests for new
|
||||||
|
heuristics belong in the normal issue tracker.
|
||||||
|
|
@ -0,0 +1,119 @@
|
||||||
|
use std::{
|
||||||
|
fs::{self, File},
|
||||||
|
io::Write,
|
||||||
|
};
|
||||||
|
|
||||||
|
use criterion::{Criterion, Throughput, criterion_group, criterion_main};
|
||||||
|
use john_slop::{AnalysisContext, Config, Detector, read_inputs};
|
||||||
|
use tempfile::TempDir;
|
||||||
|
use zip::write::FileOptions;
|
||||||
|
|
||||||
|
fn benchmark_large_document(criterion: &mut Criterion) {
|
||||||
|
let config = Config::embedded();
|
||||||
|
let detector = Detector::new(&config).expect("embedded detector");
|
||||||
|
let source = "Reviewed prose without configured signals.\n".repeat(25_000);
|
||||||
|
let context = AnalysisContext::for_path("large.txt");
|
||||||
|
let mut group = criterion.benchmark_group("analysis");
|
||||||
|
group.throughput(Throughput::Bytes(source.len() as u64));
|
||||||
|
group.bench_function("one-megabyte-document", |bencher| {
|
||||||
|
bencher.iter(|| detector.analyze_with(&source, &context));
|
||||||
|
});
|
||||||
|
group.finish();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn benchmark_many_small_files(criterion: &mut Criterion) {
|
||||||
|
let directory = small_file_tree(250);
|
||||||
|
let config = Config::embedded();
|
||||||
|
let mut group = criterion.benchmark_group("input");
|
||||||
|
group.throughput(Throughput::Elements(250));
|
||||||
|
group.bench_function("250-small-files", |bencher| {
|
||||||
|
bencher.iter(|| {
|
||||||
|
read_inputs(&config, directory.path())
|
||||||
|
.expect("open fixture directory")
|
||||||
|
.collect::<Result<Vec<_>, _>>()
|
||||||
|
.expect("read fixture directory")
|
||||||
|
});
|
||||||
|
});
|
||||||
|
group.finish();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn benchmark_rule_scaling(criterion: &mut Criterion) {
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
let original = config.rules.clone();
|
||||||
|
for generation in 1..10 {
|
||||||
|
config
|
||||||
|
.rules
|
||||||
|
.extend(original.iter().cloned().map(|mut rule| {
|
||||||
|
rule.id = format!("{}-{generation}", rule.id);
|
||||||
|
rule
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
config.validate().expect("scaled config remains valid");
|
||||||
|
let source = "Assistant: Certainly! Here's a result—review it.\n".repeat(100);
|
||||||
|
let context = AnalysisContext::for_path("scaled.txt");
|
||||||
|
|
||||||
|
criterion.bench_function("rules/build-and-analyze-110-rules", |bencher| {
|
||||||
|
bencher.iter(|| {
|
||||||
|
Detector::new(&config)
|
||||||
|
.expect("build scaled detector")
|
||||||
|
.analyze_with(&source, &context)
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn benchmark_archive_limits(criterion: &mut Criterion) {
|
||||||
|
let directory = archive_with_members(100);
|
||||||
|
let path = directory.path().join("many.zip");
|
||||||
|
let config = Config::embedded();
|
||||||
|
let mut group = criterion.benchmark_group("input");
|
||||||
|
group.throughput(Throughput::Elements(100));
|
||||||
|
group.bench_function("100-member-archive", |bencher| {
|
||||||
|
bencher.iter(|| {
|
||||||
|
read_inputs(&config, &path)
|
||||||
|
.expect("open fixture archive")
|
||||||
|
.collect::<Result<Vec<_>, _>>()
|
||||||
|
.expect("read fixture archive")
|
||||||
|
});
|
||||||
|
});
|
||||||
|
group.finish();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn small_file_tree(count: usize) -> TempDir {
|
||||||
|
let directory = tempfile::tempdir().expect("create benchmark directory");
|
||||||
|
for index in 0..count {
|
||||||
|
fs::write(
|
||||||
|
directory.path().join(format!("document-{index:04}.txt")),
|
||||||
|
"Reviewed prose without configured signals.\n",
|
||||||
|
)
|
||||||
|
.expect("write benchmark document");
|
||||||
|
}
|
||||||
|
directory
|
||||||
|
}
|
||||||
|
|
||||||
|
fn archive_with_members(count: usize) -> TempDir {
|
||||||
|
let directory = tempfile::tempdir().expect("create benchmark directory");
|
||||||
|
let file = File::create(directory.path().join("many.zip")).expect("create benchmark archive");
|
||||||
|
let mut archive = zip::ZipWriter::new(file);
|
||||||
|
let options: FileOptions<()> = FileOptions::default()
|
||||||
|
.compression_method(zip::CompressionMethod::Stored)
|
||||||
|
.unix_permissions(0o644);
|
||||||
|
for index in 0..count {
|
||||||
|
archive
|
||||||
|
.start_file(format!("document-{index:04}.txt"), options)
|
||||||
|
.expect("start benchmark member");
|
||||||
|
archive
|
||||||
|
.write_all(b"Reviewed prose without configured signals.\n")
|
||||||
|
.expect("write benchmark member");
|
||||||
|
}
|
||||||
|
archive.finish().expect("finish benchmark archive");
|
||||||
|
directory
|
||||||
|
}
|
||||||
|
|
||||||
|
criterion_group!(
|
||||||
|
benches,
|
||||||
|
benchmark_large_document,
|
||||||
|
benchmark_many_small_files,
|
||||||
|
benchmark_rule_scaling,
|
||||||
|
benchmark_archive_limits
|
||||||
|
);
|
||||||
|
criterion_main!(benches);
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
[package]
|
||||||
|
name = "john-slop-core"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
rust-version.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
description = "Reusable heuristic analysis, configuration, input, and replacement engine for John Slop"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
directories.workspace = true
|
||||||
|
figment.workspace = true
|
||||||
|
globset.workspace = true
|
||||||
|
miette.workspace = true
|
||||||
|
regex.workspace = true
|
||||||
|
serde.workspace = true
|
||||||
|
thiserror.workspace = true
|
||||||
|
unicode-security.workspace = true
|
||||||
|
walkdir.workspace = true
|
||||||
|
zip.workspace = true
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
tempfile.workspace = true
|
||||||
|
toml.workspace = true
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
# Embedded input defaults. Unicode replacements live in the generated
|
||||||
|
# `unicode.toml` policy asset so detection, diagnostics, and fixes stay aligned.
|
||||||
|
schema_version = 1
|
||||||
|
replacements = []
|
||||||
|
|
||||||
|
# Diagnostic category gates are applied in addition to each rule's `enabled`
|
||||||
|
# field. Set a category to false to silence that family without copying every
|
||||||
|
# embedded rule into this file.
|
||||||
|
[diagnostics.categories]
|
||||||
|
typography = true
|
||||||
|
style = true
|
||||||
|
prompt-leakage = true
|
||||||
|
source-code = true
|
||||||
|
|
||||||
|
# Disable or re-enable one diagnostic by its stable ID. Uncomment and change
|
||||||
|
# the ID as needed; `john-slop config show` lists the complete resolved rules.
|
||||||
|
# [[rules]]
|
||||||
|
# id = "templated-conclusion"
|
||||||
|
# enabled = false
|
||||||
|
|
||||||
|
[input]
|
||||||
|
max_document_bytes = 8388608
|
||||||
|
max_archive_bytes = 67108864
|
||||||
|
max_expanded_bytes = 268435456
|
||||||
|
max_archive_entries = 10000
|
||||||
|
max_compression_ratio = 100
|
||||||
|
max_walk_depth = 64
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
# Runtime diagnostic catalog. Copy or generate this as `diagnostics.toml` and
|
||||||
|
# add data-driven rules without rebuilding John Slop.
|
||||||
|
schema_version = 1
|
||||||
|
|
||||||
|
# This starter rule is disabled until its phrase and metadata fit your policy.
|
||||||
|
# Duplicate this complete table for additional diagnostics; every ID must be
|
||||||
|
# unique lowercase kebab-case.
|
||||||
|
[[rules]]
|
||||||
|
id = "custom-review-marker"
|
||||||
|
category = "style"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "REVIEW BEFORE PUBLISHING"
|
||||||
|
weight = 8.0
|
||||||
|
message = "a project review marker remains in the source"
|
||||||
|
help = "Resolve or remove the marker before publishing."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = false
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose", "source-code"]
|
||||||
|
extensions = []
|
||||||
|
include = []
|
||||||
|
exclude = []
|
||||||
|
skip_generated = true
|
||||||
|
skip_vendor = true
|
||||||
|
skip_minified = true
|
||||||
|
|
@ -0,0 +1,280 @@
|
||||||
|
# Embedded detection-rule metadata. Milestone 3 compiles and executes these
|
||||||
|
# matchers; Milestone 2 makes their configuration typed, layered, and validated.
|
||||||
|
schema_version = 1
|
||||||
|
|
||||||
|
[thresholds]
|
||||||
|
low = 20
|
||||||
|
medium = 50
|
||||||
|
high = 75
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-self-identification"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "As an AI language model"
|
||||||
|
weight = 30.0
|
||||||
|
message = "assistant self-identification appears in the source"
|
||||||
|
help = "Review whether an assistant response was copied into the document."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "knowledge-cutoff-disclaimer"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "knowledge cutoff"
|
||||||
|
weight = 20.0
|
||||||
|
message = "a knowledge-cutoff disclaimer appears in the source"
|
||||||
|
help = "Verify the statement and remove assistant boilerplate if it is not relevant."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-capability-disclaimer"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "I cannot browse the internet"
|
||||||
|
weight = 20.0
|
||||||
|
message = "an assistant capability disclaimer appears in the source"
|
||||||
|
help = "Replace generic capability boilerplate with a source-specific limitation if needed."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-answer-preface"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "Certainly! Here's"
|
||||||
|
weight = 15.0
|
||||||
|
message = "a stock assistant answer preface appears in the source"
|
||||||
|
help = "Review whether conversational response scaffolding was copied into the document."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "prompt-response-scaffolding"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?m)^(User|Assistant):[ \\t]"
|
||||||
|
weight = 25.0
|
||||||
|
message = "prompt/response scaffolding appears in the source"
|
||||||
|
help = "Confirm that chat transcript markers are intentional."
|
||||||
|
occurrence_cap = 2
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose", "source-code"]
|
||||||
|
extensions = []
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-follow-up-offer"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?i)\\b(?:let me know|tell me) if you(?:'d| would) like me to\\b"
|
||||||
|
weight = 12.0
|
||||||
|
message = "a conversational assistant follow-up offer appears in the source"
|
||||||
|
help = "Review whether a response-closing offer was copied into the document."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-revision-preface"
|
||||||
|
category = "prompt-leakage"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?i)\\bhere(?:'s| is) (?:a|the) (?:revised|polished|rewritten) (?:version|draft)\\b"
|
||||||
|
weight = 12.0
|
||||||
|
message = "an assistant-style revision preface appears in the source"
|
||||||
|
help = "Remove conversational revision scaffolding if it is not part of the document."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "templated-conclusion"
|
||||||
|
category = "style"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?i)\\bin conclusion,? (it is|we can)"
|
||||||
|
weight = 5.0
|
||||||
|
message = "a templated conclusion phrase appears in the source"
|
||||||
|
help = "Review whether the conclusion adds specific information."
|
||||||
|
occurrence_cap = 2
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "templated-filler"
|
||||||
|
category = "style"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?i)\\bit is (?:important|worth) (?:to note|noting) that\\b"
|
||||||
|
weight = 4.0
|
||||||
|
message = "a generic filler transition appears in the source"
|
||||||
|
help = "Prefer a direct, source-specific statement when the transition adds no meaning."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "repeated-transition"
|
||||||
|
category = "style"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "repeated-regex"
|
||||||
|
pattern = "(?im)^(?:Moreover|Furthermore|Additionally),[ \\t]"
|
||||||
|
weight = 4.0
|
||||||
|
message = "the same family of stock transitions is repeated"
|
||||||
|
help = "Review repeated transitions and keep only those that clarify the argument."
|
||||||
|
occurrence_cap = 3
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "editorial-insertion-placeholder"
|
||||||
|
category = "style"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?i)\\[(?:insert|add) (?:citation|reference|source|example|details?)(?: here)?\\]"
|
||||||
|
weight = 10.0
|
||||||
|
message = "an unresolved editorial insertion placeholder appears in the source"
|
||||||
|
help = "Supply the requested material or remove the placeholder before publishing."
|
||||||
|
occurrence_cap = 2
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "stock-helpful-closing"
|
||||||
|
category = "style"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?im)^[ \\t]*I hope this helps[.!][ \\t]*$"
|
||||||
|
weight = 2.0
|
||||||
|
message = "a stock conversational closing appears in the source"
|
||||||
|
help = "Keep the closing only when the document is intentionally conversational."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "excess-vocabulary-cluster"
|
||||||
|
category = "style"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "repeated-regex"
|
||||||
|
pattern = "(?i)\\b(?:delv(?:e|es|ed|ing)|intricate|multifaceted|pivotal|underscore[sd]?|realm|landscape)\\b"
|
||||||
|
weight = 1.0
|
||||||
|
message = "several terms from a population-level excess-vocabulary cluster appear"
|
||||||
|
help = "This weak corpus-level clue is not authorship evidence; review only for fit and specificity."
|
||||||
|
occurrence_cap = 3
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt", "rst", "adoc"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "source-placeholder-todo"
|
||||||
|
category = "source-code"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?i)\\bTODO:[ \\t]+(implement|add implementation)\\b"
|
||||||
|
weight = 10.0
|
||||||
|
message = "an implementation placeholder remains in source code"
|
||||||
|
help = "Implement the missing behavior or link the placeholder to tracked work."
|
||||||
|
occurrence_cap = 3
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["source-code"]
|
||||||
|
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "source-tutorial-narration"
|
||||||
|
category = "source-code"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?im)^[ \\t]*//[ \\t]*(?:first|next|finally),?[ \\t]+we\\b"
|
||||||
|
weight = 8.0
|
||||||
|
message = "a tutorial-style narration comment appears in source code"
|
||||||
|
help = "Describe why the code exists rather than narrating the next obvious operation."
|
||||||
|
occurrence_cap = 2
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["source-code"]
|
||||||
|
extensions = ["c", "cpp", "go", "java", "js", "rs", "ts"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "source-omitted-code-placeholder"
|
||||||
|
category = "source-code"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?im)^[ \\t]*(?://|#)[ \\t]*(?:\\.\\.\\.[ \\t]*)?(?:existing|remaining|rest of (?:the )?)[ \\t]+code(?:[ \\t]+(?:here|remains?|unchanged))?[ \\t]*$"
|
||||||
|
weight = 10.0
|
||||||
|
message = "an omitted-code placeholder remains in source code"
|
||||||
|
help = "Restore the omitted implementation or remove the placeholder."
|
||||||
|
occurrence_cap = 2
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["source-code"]
|
||||||
|
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "source-markdown-fence"
|
||||||
|
category = "source-code"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "(?m)^[ \\t]*```(?:[A-Za-z0-9_+-]+)?[ \\t]*$"
|
||||||
|
weight = 6.0
|
||||||
|
message = "a Markdown code fence appears in a source-code file"
|
||||||
|
help = "Remove response-formatting fences unless they are intentionally embedded data."
|
||||||
|
occurrence_cap = 2
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["source-code"]
|
||||||
|
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,192 @@
|
||||||
|
//! Regenerate `assets/unicode.toml` from reviewed mappings and Unicode Security.
|
||||||
|
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
|
||||||
|
use serde::Serialize;
|
||||||
|
|
||||||
|
const CURATED: &[(&str, &str, &str, &str)] = &[
|
||||||
|
("hyphen", "‐", "-", "Unicode hyphen"),
|
||||||
|
("non-breaking-hyphen", "‑", "-", "non-breaking hyphen"),
|
||||||
|
("figure-dash", "‒", "-", "figure dash"),
|
||||||
|
("en-dash", "–", "-", "en dash"),
|
||||||
|
("em-dash", "—", "-", "em dash"),
|
||||||
|
("horizontal-bar", "―", "-", "horizontal bar"),
|
||||||
|
("minus-sign", "−", "-", "mathematical minus sign"),
|
||||||
|
("small-em-dash", "﹘", "-", "small em dash"),
|
||||||
|
("small-hyphen-minus", "﹣", "-", "small hyphen-minus"),
|
||||||
|
];
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct Catalog {
|
||||||
|
schema_version: u32,
|
||||||
|
unicode: UnicodePolicy,
|
||||||
|
replacements: Vec<Replacement>,
|
||||||
|
rules: Vec<Rule>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct UnicodePolicy {
|
||||||
|
confusables: &'static str,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct Replacement {
|
||||||
|
id: String,
|
||||||
|
source: String,
|
||||||
|
replacement: String,
|
||||||
|
rationale: String,
|
||||||
|
enabled: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct Rule {
|
||||||
|
id: &'static str,
|
||||||
|
category: &'static str,
|
||||||
|
severity: &'static str,
|
||||||
|
matcher: &'static str,
|
||||||
|
pattern: &'static str,
|
||||||
|
weight: f64,
|
||||||
|
message: &'static str,
|
||||||
|
help: &'static str,
|
||||||
|
occurrence_cap: usize,
|
||||||
|
enabled: bool,
|
||||||
|
scope: Scope,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct Scope {
|
||||||
|
content: [&'static str; 2],
|
||||||
|
extensions: [String; 0],
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
let mut replacements = CURATED
|
||||||
|
.iter()
|
||||||
|
.map(|(id, source, replacement, name)| Replacement {
|
||||||
|
id: (*id).to_owned(),
|
||||||
|
source: (*source).to_owned(),
|
||||||
|
replacement: (*replacement).to_owned(),
|
||||||
|
rationale: format!("Replace the {name} with reviewed ASCII `{replacement}`."),
|
||||||
|
enabled: true,
|
||||||
|
})
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
|
||||||
|
replacements.push(Replacement {
|
||||||
|
id: "ideographic-space".to_owned(),
|
||||||
|
source: '\u{3000}'.to_string(),
|
||||||
|
replacement: " ".to_owned(),
|
||||||
|
rationale: "Width-fold the ideographic space to ASCII space.".to_owned(),
|
||||||
|
enabled: true,
|
||||||
|
});
|
||||||
|
for ascii in b'!'..=b'~' {
|
||||||
|
// U+FF01..=U+FF5E is a contiguous block corresponding to ASCII !..=~.
|
||||||
|
let value = u32::from(ascii) + 0xfee0;
|
||||||
|
let Some(fullwidth) = char::from_u32(value) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
let id = if ascii == b'-' {
|
||||||
|
// Preserve the public ID shipped before the complete width-folding set.
|
||||||
|
"fullwidth-hyphen-minus".to_owned()
|
||||||
|
} else if printable_ascii_skeleton(fullwidth).is_some() {
|
||||||
|
// Preserve IDs already shipped as generated UTS #39 mappings.
|
||||||
|
format!("uts39-u{value:04x}")
|
||||||
|
} else {
|
||||||
|
format!("fullwidth-u{value:04x}")
|
||||||
|
};
|
||||||
|
let replacement = char::from(ascii).to_string();
|
||||||
|
replacements.push(Replacement {
|
||||||
|
id,
|
||||||
|
source: fullwidth.to_string(),
|
||||||
|
rationale: format!(
|
||||||
|
"Width-fold Unicode fullwidth form U+{value:04X} to ASCII {replacement:?}."
|
||||||
|
),
|
||||||
|
replacement,
|
||||||
|
enabled: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let reviewed_sources = replacements
|
||||||
|
.iter()
|
||||||
|
.filter_map(|rule| rule.source.chars().next())
|
||||||
|
.collect::<BTreeSet<_>>();
|
||||||
|
|
||||||
|
for value in 0..=char::MAX as u32 {
|
||||||
|
let Some(character) = char::from_u32(value) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
if character.is_ascii() || reviewed_sources.contains(&character) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let source = character.to_string();
|
||||||
|
let Some(skeleton) = printable_ascii_skeleton(character) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
replacements.push(Replacement {
|
||||||
|
id: format!("uts39-u{value:04x}"),
|
||||||
|
source,
|
||||||
|
replacement: skeleton.clone(),
|
||||||
|
rationale: format!(
|
||||||
|
"Replace Unicode 16.0 UTS #39 lookalike U+{value:04X} with its printable ASCII skeleton {skeleton:?}."
|
||||||
|
),
|
||||||
|
enabled: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let catalog = Catalog {
|
||||||
|
schema_version: 1,
|
||||||
|
unicode: UnicodePolicy {
|
||||||
|
confusables: "replace",
|
||||||
|
},
|
||||||
|
replacements,
|
||||||
|
rules: vec![
|
||||||
|
Rule {
|
||||||
|
id: "unicode-ascii-replacement",
|
||||||
|
category: "typography",
|
||||||
|
severity: "advice",
|
||||||
|
matcher: "replacement",
|
||||||
|
pattern: "configured-replacements",
|
||||||
|
weight: 10.0,
|
||||||
|
message: "a Unicode character or sequence has a configured ASCII replacement",
|
||||||
|
help: "Review the character or apply the configured ASCII replacement.",
|
||||||
|
occurrence_cap: 5,
|
||||||
|
enabled: true,
|
||||||
|
scope: Scope {
|
||||||
|
content: ["prose", "source-code"],
|
||||||
|
extensions: [],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Rule {
|
||||||
|
id: "unicode-invisible-control",
|
||||||
|
category: "typography",
|
||||||
|
severity: "warning",
|
||||||
|
matcher: "invisible",
|
||||||
|
pattern: "gitea-invisible-9c715c0",
|
||||||
|
weight: 12.0,
|
||||||
|
message: "an invisible or bidirectional Unicode control appears in the source",
|
||||||
|
help: "Review the hidden or direction-changing character; it is not removed automatically.",
|
||||||
|
occurrence_cap: 5,
|
||||||
|
enabled: true,
|
||||||
|
scope: Scope {
|
||||||
|
content: ["prose", "source-code"],
|
||||||
|
extensions: [],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
|
||||||
|
println!(
|
||||||
|
"# Generated by `cargo run -p john-slop-core --example generate_unicode_catalog`.\n\
|
||||||
|
# Unicode Security 0.1.2 / Unicode 16.0.0 UTS #39, plus reviewed punctuation and width folding.\n\
|
||||||
|
# Do not edit generated mappings by hand; edit this generator and regenerate.\n{}",
|
||||||
|
toml::to_string_pretty(&catalog).expect("the catalog model must serialize")
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn printable_ascii_skeleton(character: char) -> Option<String> {
|
||||||
|
let source = character.to_string();
|
||||||
|
let skeleton = unicode_security::skeleton(&source).collect::<String>();
|
||||||
|
(!skeleton.is_empty()
|
||||||
|
&& skeleton.bytes().all(|byte| (b' '..=b'~').contains(&byte))
|
||||||
|
&& skeleton != source)
|
||||||
|
.then_some(skeleton)
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,943 @@
|
||||||
|
//! Compiled, reusable registry of configured detection rules.
|
||||||
|
|
||||||
|
use std::{
|
||||||
|
collections::BTreeMap,
|
||||||
|
path::{Path, PathBuf},
|
||||||
|
};
|
||||||
|
|
||||||
|
use globset::{Glob, GlobSet, GlobSetBuilder};
|
||||||
|
use regex::Regex;
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
Analysis, Category, Config, ConfigValidationError, ContentKind, EvidenceBand, EvidenceScore,
|
||||||
|
Finding, HeuristicRule, MatcherType, RuleId, SourceId, replacement, unicode_review,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// A validated heuristic definition paired with its reusable matcher.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct Rule {
|
||||||
|
definition: HeuristicRule,
|
||||||
|
matcher: CompiledMatcher,
|
||||||
|
scope: CompiledScope,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Rule {
|
||||||
|
/// Return this rule's stable identifier.
|
||||||
|
pub fn id(&self) -> &str {
|
||||||
|
&self.definition.id
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return this rule's evidence family.
|
||||||
|
pub const fn category(&self) -> Category {
|
||||||
|
self.definition.category
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return whether this rule is enabled in the resolved configuration.
|
||||||
|
pub const fn is_enabled(&self) -> bool {
|
||||||
|
self.definition.enabled
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the literal, regex source, or replacement-catalog sentinel.
|
||||||
|
pub fn pattern(&self) -> &str {
|
||||||
|
match &self.matcher {
|
||||||
|
CompiledMatcher::ExactPhrase(pattern) => pattern,
|
||||||
|
CompiledMatcher::Regex(regex) => regex.as_str(),
|
||||||
|
CompiledMatcher::RepeatedRegex(regex) => regex.as_str(),
|
||||||
|
CompiledMatcher::Replacement => "configured-replacements",
|
||||||
|
CompiledMatcher::Invisible => "gitea-invisible-9c715c0",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Input metadata used to select applicable rules without reading the source twice.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct AnalysisContext {
|
||||||
|
path: Option<PathBuf>,
|
||||||
|
content_kind: ContentKind,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AnalysisContext {
|
||||||
|
/// Describe prose without a known filesystem path, such as standard input.
|
||||||
|
pub const fn prose() -> Self {
|
||||||
|
Self {
|
||||||
|
path: None,
|
||||||
|
content_kind: ContentKind::Prose,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Describe source code without a known filesystem path.
|
||||||
|
pub const fn source_code() -> Self {
|
||||||
|
Self {
|
||||||
|
path: None,
|
||||||
|
content_kind: ContentKind::SourceCode,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Infer the broad content kind from a path's extension.
|
||||||
|
pub fn for_path(path: impl Into<PathBuf>) -> Self {
|
||||||
|
let path = path.into();
|
||||||
|
let content_kind = infer_content_kind(&path);
|
||||||
|
Self {
|
||||||
|
path: Some(path),
|
||||||
|
content_kind,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Describe a loaded document using its exact source identity.
|
||||||
|
pub fn for_source(source: &SourceId) -> Self {
|
||||||
|
match source {
|
||||||
|
SourceId::Stdin => Self::prose(),
|
||||||
|
SourceId::File(path) => Self::for_path(path),
|
||||||
|
SourceId::ArchiveMember { member, .. } => Self::for_path(member),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the path used by extension and glob filters, when known.
|
||||||
|
pub fn path(&self) -> Option<&Path> {
|
||||||
|
self.path.as_deref()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the broad content kind selected for this input.
|
||||||
|
pub const fn content_kind(&self) -> ContentKind {
|
||||||
|
self.content_kind
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for AnalysisContext {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::prose()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Matcher compiled once when the detector registry is built.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
enum CompiledMatcher {
|
||||||
|
ExactPhrase(String),
|
||||||
|
Regex(Regex),
|
||||||
|
RepeatedRegex(Regex),
|
||||||
|
Replacement,
|
||||||
|
Invisible,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
struct CompiledScope {
|
||||||
|
include: Option<GlobSet>,
|
||||||
|
exclude: GlobSet,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reusable detection engine built from one resolved configuration snapshot.
|
||||||
|
#[derive(Debug, Clone)]
|
||||||
|
pub struct Detector {
|
||||||
|
config: Config,
|
||||||
|
rules: Vec<Rule>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Detector {
|
||||||
|
/// Validate a resolved configuration and compile every reusable matcher.
|
||||||
|
pub fn new(config: &Config) -> Result<Self, DetectorBuildError> {
|
||||||
|
config
|
||||||
|
.validate()
|
||||||
|
.map_err(|error| DetectorBuildError::InvalidConfig(Box::new(error)))?;
|
||||||
|
let rules = config
|
||||||
|
.rules
|
||||||
|
.iter()
|
||||||
|
.cloned()
|
||||||
|
.map(Rule::compile)
|
||||||
|
.collect::<Result<Vec<_>, _>>()?;
|
||||||
|
Ok(Self {
|
||||||
|
config: config.clone(),
|
||||||
|
rules,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the compiled registry in deterministic configuration order.
|
||||||
|
pub fn rules(&self) -> &[Rule] {
|
||||||
|
&self.rules
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Analyze text with the compiled registry.
|
||||||
|
pub fn analyze(&self, source: &str) -> Analysis {
|
||||||
|
self.analyze_with(source, &AnalysisContext::default())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Analyze text with explicit content and path applicability metadata.
|
||||||
|
pub fn analyze_with(&self, source: &str, context: &AnalysisContext) -> Analysis {
|
||||||
|
let mut findings = self
|
||||||
|
.rules
|
||||||
|
.iter()
|
||||||
|
.filter(|rule| {
|
||||||
|
rule.is_enabled()
|
||||||
|
&& self.config.diagnostics.category_enabled(rule.category())
|
||||||
|
&& rule.applies(context)
|
||||||
|
})
|
||||||
|
.flat_map(|rule| rule.findings(&self.config, source))
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
normalize_findings(&mut findings);
|
||||||
|
let evidence_score = aggregate_evidence_score(&self.config, &findings);
|
||||||
|
let evidence_band = (!findings.is_empty())
|
||||||
|
.then(|| band_for_score(self.config.thresholds, evidence_score.score()));
|
||||||
|
|
||||||
|
Analysis {
|
||||||
|
findings,
|
||||||
|
evidence_score,
|
||||||
|
evidence_band,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn normalize_findings(findings: &mut Vec<Finding>) {
|
||||||
|
findings.sort_by(|left, right| {
|
||||||
|
(left.byte_span.start, left.byte_span.end, &left.rule_id).cmp(&(
|
||||||
|
right.byte_span.start,
|
||||||
|
right.byte_span.end,
|
||||||
|
&right.rule_id,
|
||||||
|
))
|
||||||
|
});
|
||||||
|
findings
|
||||||
|
.dedup_by(|right, left| right.rule_id == left.rule_id && right.byte_span == left.byte_span);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn aggregate_evidence_score(config: &Config, findings: &[Finding]) -> EvidenceScore {
|
||||||
|
let mut family_rule_counts = BTreeMap::<Category, usize>::new();
|
||||||
|
let mut evidence = 0.0;
|
||||||
|
|
||||||
|
for rule in &config.rules {
|
||||||
|
let occurrences = findings
|
||||||
|
.iter()
|
||||||
|
.filter(|finding| finding.rule_id.as_str() == rule.id)
|
||||||
|
.count()
|
||||||
|
.min(rule.occurrence_cap);
|
||||||
|
if occurrences == 0 {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let occurrence_factor = (0..occurrences)
|
||||||
|
.map(|index| 0.5_f64.powi(i32::try_from(index).unwrap_or(i32::MAX)))
|
||||||
|
.sum::<f64>();
|
||||||
|
let family_index = family_rule_counts.entry(rule.category).or_default();
|
||||||
|
let family_factor =
|
||||||
|
family_decay(rule.category).powi(i32::try_from(*family_index).unwrap_or(i32::MAX));
|
||||||
|
evidence += rule.weight * occurrence_factor * family_factor;
|
||||||
|
*family_index += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let normalized = if evidence == 0.0 {
|
||||||
|
0.0
|
||||||
|
} else {
|
||||||
|
100.0 * evidence / (evidence + 50.0)
|
||||||
|
};
|
||||||
|
EvidenceScore::from_score(normalized.round() as u8)
|
||||||
|
}
|
||||||
|
|
||||||
|
const fn family_decay(category: Category) -> f64 {
|
||||||
|
match category {
|
||||||
|
Category::PromptLeakage => 0.90,
|
||||||
|
Category::Typography | Category::Style | Category::SourceCode => 0.75,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn band_for_score(thresholds: crate::EvidenceThresholds, score: u8) -> EvidenceBand {
|
||||||
|
if score >= thresholds.high {
|
||||||
|
EvidenceBand::High
|
||||||
|
} else if score >= thresholds.medium {
|
||||||
|
EvidenceBand::Medium
|
||||||
|
} else if score >= thresholds.low {
|
||||||
|
EvidenceBand::Low
|
||||||
|
} else {
|
||||||
|
EvidenceBand::BelowThreshold
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Rule {
|
||||||
|
fn compile(definition: HeuristicRule) -> Result<Self, DetectorBuildError> {
|
||||||
|
let matcher = match definition.matcher {
|
||||||
|
MatcherType::ExactPhrase => CompiledMatcher::ExactPhrase(definition.pattern.clone()),
|
||||||
|
MatcherType::Regex => {
|
||||||
|
let regex = Regex::new(&definition.pattern).map_err(|source| {
|
||||||
|
DetectorBuildError::InvalidRegex {
|
||||||
|
id: definition.id.clone(),
|
||||||
|
source,
|
||||||
|
}
|
||||||
|
})?;
|
||||||
|
CompiledMatcher::Regex(regex)
|
||||||
|
}
|
||||||
|
MatcherType::RepeatedRegex => {
|
||||||
|
let regex = Regex::new(&definition.pattern).map_err(|source| {
|
||||||
|
DetectorBuildError::InvalidRegex {
|
||||||
|
id: definition.id.clone(),
|
||||||
|
source,
|
||||||
|
}
|
||||||
|
})?;
|
||||||
|
CompiledMatcher::RepeatedRegex(regex)
|
||||||
|
}
|
||||||
|
MatcherType::Replacement => CompiledMatcher::Replacement,
|
||||||
|
MatcherType::Invisible => CompiledMatcher::Invisible,
|
||||||
|
};
|
||||||
|
let scope = CompiledScope::compile(&definition)?;
|
||||||
|
Ok(Self {
|
||||||
|
definition,
|
||||||
|
matcher,
|
||||||
|
scope,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn applies(&self, context: &AnalysisContext) -> bool {
|
||||||
|
if !self
|
||||||
|
.definition
|
||||||
|
.scope
|
||||||
|
.content
|
||||||
|
.contains(&context.content_kind)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
let Some(path) = context.path() else {
|
||||||
|
return self.scope.include.is_none();
|
||||||
|
};
|
||||||
|
if !self.definition.scope.extensions.is_empty()
|
||||||
|
&& !path
|
||||||
|
.extension()
|
||||||
|
.and_then(|extension| extension.to_str())
|
||||||
|
.is_some_and(|extension| {
|
||||||
|
self.definition
|
||||||
|
.scope
|
||||||
|
.extensions
|
||||||
|
.iter()
|
||||||
|
.any(|allowed| extension.eq_ignore_ascii_case(allowed))
|
||||||
|
})
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if self
|
||||||
|
.scope
|
||||||
|
.include
|
||||||
|
.as_ref()
|
||||||
|
.is_some_and(|include| !include.is_match(path))
|
||||||
|
|| self.scope.exclude.is_match(path)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
!(self.definition.scope.skip_generated && is_generated(path)
|
||||||
|
|| self.definition.scope.skip_vendor && is_vendor(path)
|
||||||
|
|| self.definition.scope.skip_minified && is_minified(path))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn findings(&self, config: &Config, source: &str) -> Vec<Finding> {
|
||||||
|
match &self.matcher {
|
||||||
|
CompiledMatcher::ExactPhrase(pattern) => source
|
||||||
|
.match_indices(pattern)
|
||||||
|
.map(|(start, matched)| self.finding(source, start..start + matched.len(), None))
|
||||||
|
.collect(),
|
||||||
|
CompiledMatcher::Regex(regex) => regex
|
||||||
|
.find_iter(source)
|
||||||
|
.map(|matched| self.finding(source, matched.range(), None))
|
||||||
|
.collect(),
|
||||||
|
CompiledMatcher::RepeatedRegex(regex) => {
|
||||||
|
let matches = regex.find_iter(source).collect::<Vec<_>>();
|
||||||
|
if matches.len() < 2 {
|
||||||
|
Vec::new()
|
||||||
|
} else {
|
||||||
|
matches
|
||||||
|
.into_iter()
|
||||||
|
.map(|matched| self.finding(source, matched.range(), None))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
CompiledMatcher::Replacement => {
|
||||||
|
let mut findings = replacement::plan(config, source)
|
||||||
|
.into_iter()
|
||||||
|
.map(|edit| self.finding(source, edit.byte_span.clone(), Some(edit)))
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
findings.extend(
|
||||||
|
replacement::plan_confusables_for_review(config, source)
|
||||||
|
.into_iter()
|
||||||
|
.map(|edit| self.finding(source, edit.byte_span, None)),
|
||||||
|
);
|
||||||
|
findings
|
||||||
|
}
|
||||||
|
CompiledMatcher::Invisible => source
|
||||||
|
.char_indices()
|
||||||
|
.filter(|(start, character)| {
|
||||||
|
unicode_review::is_gitea_invisible(*character)
|
||||||
|
&& !(*character == '\r'
|
||||||
|
&& source.as_bytes().get(start + 1) == Some(&b'\n'))
|
||||||
|
})
|
||||||
|
.map(|(start, character)| {
|
||||||
|
let mut finding =
|
||||||
|
self.finding(source, start..start + character.len_utf8(), None);
|
||||||
|
finding.help = Some(format!(
|
||||||
|
"Review the hidden or direction-changing character U+{:04X}. It is never removed automatically.",
|
||||||
|
character as u32
|
||||||
|
));
|
||||||
|
finding
|
||||||
|
})
|
||||||
|
.collect(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finding(
|
||||||
|
&self,
|
||||||
|
source: &str,
|
||||||
|
byte_span: std::ops::Range<usize>,
|
||||||
|
fix: Option<crate::TextEdit>,
|
||||||
|
) -> Finding {
|
||||||
|
Finding {
|
||||||
|
rule_id: RuleId::new(self.id()),
|
||||||
|
category: self.category(),
|
||||||
|
severity: self.definition.severity,
|
||||||
|
message: self.definition.message.clone(),
|
||||||
|
help: Some(self.definition.help.clone()),
|
||||||
|
matched_text: source[byte_span.clone()].to_owned(),
|
||||||
|
byte_span,
|
||||||
|
weight: self.definition.weight,
|
||||||
|
fix,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CompiledScope {
|
||||||
|
fn compile(rule: &HeuristicRule) -> Result<Self, DetectorBuildError> {
|
||||||
|
Ok(Self {
|
||||||
|
include: if rule.scope.include.is_empty() {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
Some(compile_globs(rule, "include", &rule.scope.include)?)
|
||||||
|
},
|
||||||
|
exclude: compile_globs(rule, "exclude", &rule.scope.exclude)?,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn compile_globs(
|
||||||
|
rule: &HeuristicRule,
|
||||||
|
field: &'static str,
|
||||||
|
patterns: &[String],
|
||||||
|
) -> Result<GlobSet, DetectorBuildError> {
|
||||||
|
let mut builder = GlobSetBuilder::new();
|
||||||
|
for pattern in patterns {
|
||||||
|
builder.add(
|
||||||
|
Glob::new(pattern).map_err(|source| DetectorBuildError::InvalidGlob {
|
||||||
|
id: rule.id.clone(),
|
||||||
|
field,
|
||||||
|
pattern: pattern.clone(),
|
||||||
|
source,
|
||||||
|
})?,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
builder
|
||||||
|
.build()
|
||||||
|
.map_err(|source| DetectorBuildError::InvalidGlob {
|
||||||
|
id: rule.id.clone(),
|
||||||
|
field,
|
||||||
|
pattern: patterns.join(", "),
|
||||||
|
source,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn infer_content_kind(path: &Path) -> ContentKind {
|
||||||
|
const SOURCE_EXTENSIONS: &[&str] = &[
|
||||||
|
"c", "cc", "cpp", "cs", "go", "h", "hpp", "java", "js", "jsx", "kt", "php", "py", "rb",
|
||||||
|
"rs", "swift", "ts", "tsx",
|
||||||
|
];
|
||||||
|
if path
|
||||||
|
.extension()
|
||||||
|
.and_then(|extension| extension.to_str())
|
||||||
|
.is_some_and(|extension| {
|
||||||
|
SOURCE_EXTENSIONS
|
||||||
|
.iter()
|
||||||
|
.any(|candidate| extension.eq_ignore_ascii_case(candidate))
|
||||||
|
})
|
||||||
|
{
|
||||||
|
ContentKind::SourceCode
|
||||||
|
} else {
|
||||||
|
ContentKind::Prose
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn has_component(path: &Path, candidates: &[&str]) -> bool {
|
||||||
|
path.components().any(|component| {
|
||||||
|
component.as_os_str().to_str().is_some_and(|value| {
|
||||||
|
candidates
|
||||||
|
.iter()
|
||||||
|
.any(|candidate| value.eq_ignore_ascii_case(candidate))
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_generated(path: &Path) -> bool {
|
||||||
|
has_component(path, &["generated", "gen", "dist", "build", "target"])
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_vendor(path: &Path) -> bool {
|
||||||
|
has_component(
|
||||||
|
path,
|
||||||
|
&[
|
||||||
|
"vendor",
|
||||||
|
"vendors",
|
||||||
|
"node_modules",
|
||||||
|
"third_party",
|
||||||
|
"third-party",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_minified(path: &Path) -> bool {
|
||||||
|
path.file_name()
|
||||||
|
.and_then(|name| name.to_str())
|
||||||
|
.is_some_and(|name| {
|
||||||
|
let name = name.to_ascii_lowercase();
|
||||||
|
name.contains(".min.") || name.ends_with("-min.js") || name.ends_with("-min.css")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Failure while constructing a reusable detector registry.
|
||||||
|
#[derive(Debug, thiserror::Error)]
|
||||||
|
pub enum DetectorBuildError {
|
||||||
|
/// The resolved configuration was not semantically valid.
|
||||||
|
#[error("cannot build detector from invalid configuration")]
|
||||||
|
InvalidConfig(#[source] Box<ConfigValidationError>),
|
||||||
|
/// A configured regular expression could not be compiled.
|
||||||
|
#[error("failed to compile matcher for rule `{id}`")]
|
||||||
|
InvalidRegex {
|
||||||
|
/// Stable rule identifier.
|
||||||
|
id: String,
|
||||||
|
/// Regex compiler failure.
|
||||||
|
#[source]
|
||||||
|
source: regex::Error,
|
||||||
|
},
|
||||||
|
/// A configured path applicability glob could not be compiled.
|
||||||
|
#[error("failed to compile {field} glob `{pattern}` for rule `{id}`")]
|
||||||
|
InvalidGlob {
|
||||||
|
/// Stable rule identifier.
|
||||||
|
id: String,
|
||||||
|
/// Scope field containing the glob.
|
||||||
|
field: &'static str,
|
||||||
|
/// Invalid glob source.
|
||||||
|
pattern: String,
|
||||||
|
/// Glob compiler failure.
|
||||||
|
#[source]
|
||||||
|
source: globset::Error,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn registry_is_built_from_the_validated_configuration() {
|
||||||
|
let config = Config::embedded();
|
||||||
|
let detector = Detector::new(&config).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(detector.rules().len(), config.rules.len());
|
||||||
|
assert_eq!(detector.rules()[0].id(), config.rules[0].id);
|
||||||
|
assert_eq!(detector.rules()[0].pattern(), config.rules[0].pattern);
|
||||||
|
assert_eq!(
|
||||||
|
detector
|
||||||
|
.rules()
|
||||||
|
.iter()
|
||||||
|
.filter(|rule| rule.is_enabled())
|
||||||
|
.count(),
|
||||||
|
config.rules.iter().filter(|rule| rule.enabled).count()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn replacement_rule_preserves_explainable_findings() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let analysis = detector.analyze("ordinary-word and suspicious—dash");
|
||||||
|
|
||||||
|
assert_eq!(analysis.findings.len(), 1);
|
||||||
|
assert_eq!(
|
||||||
|
analysis.findings[0].rule_id.as_str(),
|
||||||
|
"unicode-ascii-replacement"
|
||||||
|
);
|
||||||
|
assert_eq!(analysis.findings[0].matched_text, "—");
|
||||||
|
assert_eq!(analysis.findings[0].fix.as_ref().unwrap().replacement, "-");
|
||||||
|
assert_eq!(analysis.evidence_score.score(), 17);
|
||||||
|
assert_eq!(analysis.evidence_band, Some(EvidenceBand::BelowThreshold));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn category_and_individual_switches_are_both_required() {
|
||||||
|
let source = "In conclusion, it is complete.";
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
config.diagnostics.categories.style = false;
|
||||||
|
assert!(
|
||||||
|
Detector::new(&config)
|
||||||
|
.unwrap()
|
||||||
|
.analyze(source)
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
|
||||||
|
config.diagnostics.categories.style = true;
|
||||||
|
config
|
||||||
|
.rules
|
||||||
|
.iter_mut()
|
||||||
|
.find(|rule| rule.id == "templated-conclusion")
|
||||||
|
.unwrap()
|
||||||
|
.enabled = false;
|
||||||
|
assert!(
|
||||||
|
Detector::new(&config)
|
||||||
|
.unwrap()
|
||||||
|
.analyze(source)
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
|
||||||
|
config
|
||||||
|
.rules
|
||||||
|
.iter_mut()
|
||||||
|
.find(|rule| rule.id == "templated-conclusion")
|
||||||
|
.unwrap()
|
||||||
|
.enabled = true;
|
||||||
|
assert_eq!(
|
||||||
|
Detector::new(&config)
|
||||||
|
.unwrap()
|
||||||
|
.analyze(source)
|
||||||
|
.findings
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn unicode_review_findings_have_exact_spans_and_safe_fixes() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let source = "é pаss\u{200b}word\u{202e}\r\nnext";
|
||||||
|
let findings = detector.analyze(source).findings;
|
||||||
|
|
||||||
|
assert_eq!(findings.len(), 3);
|
||||||
|
assert_eq!(findings[0].rule_id.as_str(), "unicode-ascii-replacement");
|
||||||
|
assert_eq!(findings[0].matched_text, "а");
|
||||||
|
assert_eq!(&source[findings[0].byte_span.clone()], "а");
|
||||||
|
assert_eq!(findings[0].fix.as_ref().unwrap().replacement, "a");
|
||||||
|
assert_eq!(
|
||||||
|
findings
|
||||||
|
.iter()
|
||||||
|
.filter(|finding| finding.rule_id.as_str() == "unicode-invisible-control")
|
||||||
|
.count(),
|
||||||
|
2
|
||||||
|
);
|
||||||
|
assert!(findings[1..].iter().all(|finding| finding.fix.is_none()));
|
||||||
|
assert!(findings.iter().all(|finding| finding.matched_text != "\r"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn confusable_policy_switches_between_fix_review_and_off() {
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
|
||||||
|
config.unicode.confusables = crate::ConfusablePolicy::Review;
|
||||||
|
let review = Detector::new(&config).unwrap().analyze("pаss");
|
||||||
|
assert_eq!(review.findings.len(), 1);
|
||||||
|
assert!(review.findings[0].fix.is_none());
|
||||||
|
|
||||||
|
config.unicode.confusables = crate::ConfusablePolicy::Off;
|
||||||
|
assert!(
|
||||||
|
Detector::new(&config)
|
||||||
|
.unwrap()
|
||||||
|
.analyze("pаss")
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn safe_replacement_characters_are_not_duplicated_as_confusable_findings() {
|
||||||
|
let findings = Detector::new(&Config::embedded())
|
||||||
|
.unwrap()
|
||||||
|
.analyze("one—two")
|
||||||
|
.findings;
|
||||||
|
|
||||||
|
assert_eq!(findings.len(), 1);
|
||||||
|
assert_eq!(findings[0].rule_id.as_str(), "unicode-ascii-replacement");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn compiled_regex_uses_utf8_byte_offsets() {
|
||||||
|
let regex = Regex::new(r"\u{1D306}").unwrap();
|
||||||
|
let matched = regex.find("a𝌆b").unwrap();
|
||||||
|
|
||||||
|
assert_eq!(matched.range(), 1..5);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn exact_phrase_findings_use_configured_metadata_and_byte_spans() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let source = "préface: As an AI language model, I cannot verify this.";
|
||||||
|
|
||||||
|
let finding = detector
|
||||||
|
.analyze(source)
|
||||||
|
.findings
|
||||||
|
.into_iter()
|
||||||
|
.find(|finding| finding.rule_id.as_str() == "assistant-self-identification")
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let start = source.find("As an AI language model").unwrap();
|
||||||
|
assert_eq!(finding.byte_span, start..start + finding.matched_text.len());
|
||||||
|
assert_eq!(finding.matched_text, "As an AI language model");
|
||||||
|
assert_eq!(finding.category, Category::PromptLeakage);
|
||||||
|
assert_eq!(finding.weight, 30.0);
|
||||||
|
assert!(finding.message.contains("self-identification"));
|
||||||
|
assert!(finding.help.unwrap().contains("copied"));
|
||||||
|
assert!(finding.fix.is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn regex_findings_cover_the_exact_utf8_source_slice() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let source = "préface\nAssistant: generated response";
|
||||||
|
|
||||||
|
let finding = detector
|
||||||
|
.analyze(source)
|
||||||
|
.findings
|
||||||
|
.into_iter()
|
||||||
|
.find(|finding| finding.rule_id.as_str() == "prompt-response-scaffolding")
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
assert_eq!(&source[finding.byte_span.clone()], finding.matched_text);
|
||||||
|
assert_eq!(finding.matched_text, "Assistant: ");
|
||||||
|
assert_eq!(finding.category, Category::PromptLeakage);
|
||||||
|
assert_eq!(finding.weight, 25.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn embedded_catalog_covers_each_conservative_signal_family() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let ids = detector
|
||||||
|
.rules()
|
||||||
|
.iter()
|
||||||
|
.map(Rule::id)
|
||||||
|
.collect::<std::collections::BTreeSet<_>>();
|
||||||
|
|
||||||
|
for expected in [
|
||||||
|
"assistant-self-identification",
|
||||||
|
"assistant-capability-disclaimer",
|
||||||
|
"assistant-answer-preface",
|
||||||
|
"knowledge-cutoff-disclaimer",
|
||||||
|
"prompt-response-scaffolding",
|
||||||
|
"templated-filler",
|
||||||
|
"templated-conclusion",
|
||||||
|
"repeated-transition",
|
||||||
|
"source-placeholder-todo",
|
||||||
|
"source-tutorial-narration",
|
||||||
|
"unicode-ascii-replacement",
|
||||||
|
] {
|
||||||
|
assert!(ids.contains(expected), "missing {expected}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn repeated_transition_requires_multiple_occurrences() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let one = detector.analyze("Moreover, one transition is ordinary.");
|
||||||
|
let repeated =
|
||||||
|
detector.analyze("Moreover, the first claim.\nAdditionally, the second claim.");
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
!one.findings
|
||||||
|
.iter()
|
||||||
|
.any(|finding| finding.rule_id.as_str() == "repeated-transition")
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
repeated
|
||||||
|
.findings
|
||||||
|
.iter()
|
||||||
|
.filter(|finding| finding.rule_id.as_str() == "repeated-transition")
|
||||||
|
.count(),
|
||||||
|
2
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn content_and_extension_scopes_select_the_expected_rules() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let text = "TODO: implement the parser";
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::prose())
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::for_path("src/parser.rs"))
|
||||||
|
.findings
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::for_path("src/parser.md"))
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn include_and_exclude_globs_are_compiled_and_applied() {
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
let rule = config
|
||||||
|
.rules
|
||||||
|
.iter_mut()
|
||||||
|
.find(|rule| rule.id == "assistant-self-identification")
|
||||||
|
.unwrap();
|
||||||
|
rule.scope.include = vec!["docs/**".to_owned()];
|
||||||
|
rule.scope.exclude = vec!["docs/archive/**".to_owned()];
|
||||||
|
let detector = Detector::new(&config).unwrap();
|
||||||
|
let text = "As an AI language model, I can help.";
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::for_path("docs/guide.md"))
|
||||||
|
.findings
|
||||||
|
.len(),
|
||||||
|
1
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::for_path("notes/guide.md"))
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::for_path("docs/archive/guide.md"))
|
||||||
|
.findings
|
||||||
|
.is_empty()
|
||||||
|
);
|
||||||
|
assert!(detector.analyze(text).findings.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn generated_vendor_and_minified_paths_are_skipped_by_default() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let text = "TODO: implement the parser";
|
||||||
|
|
||||||
|
for path in [
|
||||||
|
"generated/parser.rs",
|
||||||
|
"vendor/parser.rs",
|
||||||
|
"node_modules/parser.js",
|
||||||
|
"src/parser.min.js",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
detector
|
||||||
|
.analyze_with(text, &AnalysisContext::for_path(path))
|
||||||
|
.findings
|
||||||
|
.is_empty(),
|
||||||
|
"unexpected finding for {path}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn archive_members_use_the_inner_path_for_applicability() {
|
||||||
|
let context = AnalysisContext::for_source(&SourceId::ArchiveMember {
|
||||||
|
archive: PathBuf::from("bundle.zip"),
|
||||||
|
member: PathBuf::from("src/lib.rs"),
|
||||||
|
});
|
||||||
|
|
||||||
|
assert_eq!(context.path(), Some(Path::new("src/lib.rs")));
|
||||||
|
assert_eq!(context.content_kind(), ContentKind::SourceCode);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn evidence_score_caps_occurrences_and_applies_diminishing_returns() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let five = detector.analyze(&"—".repeat(5));
|
||||||
|
let twenty = detector.analyze(&"—".repeat(20));
|
||||||
|
|
||||||
|
assert_eq!(five.evidence_score, twenty.evidence_score);
|
||||||
|
assert_eq!(five.evidence_score.score(), 28);
|
||||||
|
assert_eq!(twenty.findings.len(), 20);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn evidence_score_damps_distinct_rules_in_the_same_family() {
|
||||||
|
let source = "As an AI language model, I cannot browse the internet.";
|
||||||
|
let correlated = Detector::new(&Config::embedded()).unwrap().analyze(source);
|
||||||
|
let mut independent_config = Config::embedded();
|
||||||
|
independent_config
|
||||||
|
.rules
|
||||||
|
.iter_mut()
|
||||||
|
.find(|rule| rule.id == "assistant-capability-disclaimer")
|
||||||
|
.unwrap()
|
||||||
|
.category = Category::Style;
|
||||||
|
let independent = Detector::new(&independent_config).unwrap().analyze(source);
|
||||||
|
|
||||||
|
assert!(correlated.evidence_score < independent.evidence_score);
|
||||||
|
assert_eq!(correlated.findings.len(), independent.findings.len());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn prompt_leakage_uses_lighter_family_damping_than_other_categories() {
|
||||||
|
assert_eq!(family_decay(Category::PromptLeakage), 0.90);
|
||||||
|
assert_eq!(family_decay(Category::Style), 0.75);
|
||||||
|
assert_eq!(family_decay(Category::Typography), 0.75);
|
||||||
|
assert_eq!(family_decay(Category::SourceCode), 0.75);
|
||||||
|
|
||||||
|
let source = concat!(
|
||||||
|
"User: request\n",
|
||||||
|
"Assistant: Certainly! Here's a response.\n",
|
||||||
|
"As an AI language model, I cannot browse the internet. ",
|
||||||
|
"My knowledge cutoff applies."
|
||||||
|
);
|
||||||
|
let analysis = Detector::new(&Config::embedded()).unwrap().analyze(source);
|
||||||
|
|
||||||
|
assert_eq!(analysis.findings.len(), 6);
|
||||||
|
assert_eq!(analysis.evidence_score.score(), 67);
|
||||||
|
assert_eq!(analysis.evidence_band, Some(EvidenceBand::Medium));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn configured_thresholds_define_named_band_boundaries() {
|
||||||
|
let thresholds = crate::EvidenceThresholds {
|
||||||
|
low: 20,
|
||||||
|
medium: 50,
|
||||||
|
high: 75,
|
||||||
|
};
|
||||||
|
|
||||||
|
assert_eq!(band_for_score(thresholds, 19), EvidenceBand::BelowThreshold);
|
||||||
|
assert_eq!(band_for_score(thresholds, 20), EvidenceBand::Low);
|
||||||
|
assert_eq!(band_for_score(thresholds, 50), EvidenceBand::Medium);
|
||||||
|
assert_eq!(band_for_score(thresholds, 75), EvidenceBand::High);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn one_weak_repeated_signal_cannot_reach_high_evidence() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let source = "Moreover, first.\nAdditionally, second.\nFurthermore, third.";
|
||||||
|
let analysis = detector.analyze(source);
|
||||||
|
|
||||||
|
assert_eq!(analysis.findings.len(), 3);
|
||||||
|
assert_ne!(analysis.evidence_band, Some(EvidenceBand::High));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn empty_and_clean_text_have_zero_evidence_and_no_verdict() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
|
||||||
|
for source in ["", "A specific, carefully sourced statement."] {
|
||||||
|
let analysis = detector.analyze(source);
|
||||||
|
assert!(analysis.findings.is_empty());
|
||||||
|
assert_eq!(analysis.evidence_score.score(), 0);
|
||||||
|
assert_eq!(analysis.evidence_band, None);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn findings_are_sorted_and_identical_rule_spans_are_deduplicated() {
|
||||||
|
let detector = Detector::new(&Config::embedded()).unwrap();
|
||||||
|
let mut findings = detector.analyze("As an AI language model—").findings;
|
||||||
|
findings.reverse();
|
||||||
|
findings.push(findings[0].clone());
|
||||||
|
|
||||||
|
normalize_findings(&mut findings);
|
||||||
|
|
||||||
|
assert_eq!(findings.len(), 2);
|
||||||
|
assert!(findings[0].byte_span.start < findings[1].byte_span.start);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,156 @@
|
||||||
|
//! Rich source diagnostics derived from presentation-neutral findings.
|
||||||
|
|
||||||
|
use std::{error::Error, fmt};
|
||||||
|
|
||||||
|
use miette::{Diagnostic, LabeledSpan, NamedSource, Severity, SourceCode, SourceSpan};
|
||||||
|
|
||||||
|
use crate::{Finding, FindingSeverity, RuleId, SourceId};
|
||||||
|
|
||||||
|
/// Owned `miette` diagnostic for one finding against an exact source snapshot.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct FindingDiagnostic {
|
||||||
|
rule_id: RuleId,
|
||||||
|
severity: FindingSeverity,
|
||||||
|
message: String,
|
||||||
|
help: Option<String>,
|
||||||
|
matched_text: String,
|
||||||
|
source: NamedSource<String>,
|
||||||
|
span: SourceSpan,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FindingDiagnostic {
|
||||||
|
/// Create a diagnostic by binding a finding to its named source snapshot.
|
||||||
|
pub fn new(source_id: &SourceId, source: &str, finding: &Finding) -> Self {
|
||||||
|
Self {
|
||||||
|
rule_id: finding.rule_id.clone(),
|
||||||
|
severity: finding.severity,
|
||||||
|
message: finding.message.clone(),
|
||||||
|
help: finding.help.clone(),
|
||||||
|
matched_text: finding.matched_text.clone(),
|
||||||
|
source: NamedSource::new(source_id.to_string(), source.to_owned()),
|
||||||
|
span: finding.byte_span.clone().into(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the stable rule identifier used as the diagnostic code.
|
||||||
|
pub fn rule_id(&self) -> &RuleId {
|
||||||
|
&self.rule_id
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the configured finding severity.
|
||||||
|
pub const fn finding_severity(&self) -> FindingSeverity {
|
||||||
|
self.severity
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the exact source span labeled by this diagnostic.
|
||||||
|
pub const fn span(&self) -> SourceSpan {
|
||||||
|
self.span
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the named source snapshot used for rendering.
|
||||||
|
pub fn named_source(&self) -> &NamedSource<String> {
|
||||||
|
&self.source
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for FindingDiagnostic {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
self.message.fmt(formatter)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Error for FindingDiagnostic {}
|
||||||
|
|
||||||
|
impl Diagnostic for FindingDiagnostic {
|
||||||
|
fn code(&self) -> Option<Box<dyn fmt::Display + '_>> {
|
||||||
|
Some(Box::new(&self.rule_id))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn severity(&self) -> Option<Severity> {
|
||||||
|
Some(match self.severity {
|
||||||
|
FindingSeverity::Advice => Severity::Advice,
|
||||||
|
FindingSeverity::Warning => Severity::Warning,
|
||||||
|
FindingSeverity::Error => Severity::Error,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn help(&self) -> Option<Box<dyn fmt::Display + '_>> {
|
||||||
|
self.help
|
||||||
|
.as_deref()
|
||||||
|
.map(|help| Box::new(help) as Box<dyn fmt::Display>)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn source_code(&self) -> Option<&dyn SourceCode> {
|
||||||
|
Some(&self.source)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn labels(&self) -> Option<Box<dyn Iterator<Item = LabeledSpan> + '_>> {
|
||||||
|
Some(Box::new(std::iter::once(
|
||||||
|
LabeledSpan::new_primary_with_span(Some(self.matched_text.clone()), self.span),
|
||||||
|
)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::{AnalysisContext, Config, Detector};
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn finding_diagnostic_preserves_code_severity_source_span_and_help() {
|
||||||
|
let source = "préface: As an AI language model, I can help.";
|
||||||
|
let finding = Detector::new(&Config::embedded())
|
||||||
|
.unwrap()
|
||||||
|
.analyze_with(source, &AnalysisContext::for_path("article.md"))
|
||||||
|
.findings
|
||||||
|
.into_iter()
|
||||||
|
.find(|finding| finding.rule_id.as_str() == "assistant-self-identification")
|
||||||
|
.unwrap();
|
||||||
|
let diagnostic =
|
||||||
|
FindingDiagnostic::new(&SourceId::File("article.md".into()), source, &finding);
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
diagnostic.code().unwrap().to_string(),
|
||||||
|
finding.rule_id.as_str()
|
||||||
|
);
|
||||||
|
assert_eq!(diagnostic.severity(), Some(Severity::Warning));
|
||||||
|
assert!(diagnostic.help().unwrap().to_string().contains("copied"));
|
||||||
|
assert_eq!(diagnostic.named_source().name(), "article.md");
|
||||||
|
let label = diagnostic.labels().unwrap().next().unwrap();
|
||||||
|
assert_eq!(label.offset(), finding.byte_span.start);
|
||||||
|
assert_eq!(label.len(), finding.byte_span.len());
|
||||||
|
assert_eq!(
|
||||||
|
&source[label.offset()..label.offset() + label.len()],
|
||||||
|
finding.matched_text
|
||||||
|
);
|
||||||
|
assert_eq!(diagnostic.rule_id(), &finding.rule_id);
|
||||||
|
assert_eq!(diagnostic.finding_severity(), FindingSeverity::Warning);
|
||||||
|
assert_eq!(diagnostic.span(), finding.byte_span.clone().into());
|
||||||
|
assert_eq!(diagnostic.to_string(), finding.message);
|
||||||
|
assert!(diagnostic.source_code().is_some());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_finding_severity_maps_to_miette_and_help_remains_optional() {
|
||||||
|
for (finding_severity, miette_severity) in [
|
||||||
|
(FindingSeverity::Advice, Severity::Advice),
|
||||||
|
(FindingSeverity::Warning, Severity::Warning),
|
||||||
|
(FindingSeverity::Error, Severity::Error),
|
||||||
|
] {
|
||||||
|
let finding = Finding {
|
||||||
|
rule_id: RuleId::from_static("severity-test"),
|
||||||
|
category: crate::Category::Style,
|
||||||
|
severity: finding_severity,
|
||||||
|
message: "review this".into(),
|
||||||
|
help: None,
|
||||||
|
byte_span: 0..1,
|
||||||
|
matched_text: "x".into(),
|
||||||
|
weight: 1.0,
|
||||||
|
fix: None,
|
||||||
|
};
|
||||||
|
let diagnostic = FindingDiagnostic::new(&SourceId::Stdin, "x", &finding);
|
||||||
|
assert_eq!(diagnostic.severity(), Some(miette_severity));
|
||||||
|
assert!(diagnostic.help().is_none());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,171 @@
|
||||||
|
//! Typed failures returned by the input layer.
|
||||||
|
|
||||||
|
use std::{io, path::PathBuf, string::FromUtf8Error};
|
||||||
|
|
||||||
|
use thiserror::Error;
|
||||||
|
|
||||||
|
use crate::SourceId;
|
||||||
|
|
||||||
|
/// Failure while discovering or reading one input document.
|
||||||
|
#[derive(Debug, Error)]
|
||||||
|
pub enum InputError {
|
||||||
|
/// Reading standard input failed.
|
||||||
|
#[error("failed to read standard input")]
|
||||||
|
StdinIo {
|
||||||
|
/// Underlying operating-system error.
|
||||||
|
#[source]
|
||||||
|
source: io::Error,
|
||||||
|
},
|
||||||
|
/// The requested path does not identify a file or directory.
|
||||||
|
#[error("path `{path}` is neither a file nor a directory")]
|
||||||
|
MissingPath {
|
||||||
|
/// Requested path.
|
||||||
|
path: PathBuf,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// A filesystem operation failed.
|
||||||
|
#[error("failed to {action} `{path}`")]
|
||||||
|
Io {
|
||||||
|
/// Operation being attempted.
|
||||||
|
action: &'static str,
|
||||||
|
/// Affected filesystem path.
|
||||||
|
path: PathBuf,
|
||||||
|
/// Underlying operating-system error.
|
||||||
|
#[source]
|
||||||
|
source: io::Error,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Recursive directory traversal failed for one entry.
|
||||||
|
#[error("failed while walking an input directory")]
|
||||||
|
Walk {
|
||||||
|
/// Underlying traversal error.
|
||||||
|
#[source]
|
||||||
|
source: walkdir::Error,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// A ZIP container or member could not be decoded.
|
||||||
|
#[error("failed to read ZIP data from `{archive}`")]
|
||||||
|
Zip {
|
||||||
|
/// Outer archive path.
|
||||||
|
archive: PathBuf,
|
||||||
|
/// Underlying ZIP format error.
|
||||||
|
#[source]
|
||||||
|
source: zip::result::ZipError,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// A source exceeded a configured byte or count limit.
|
||||||
|
#[error("{source_id} uses {actual} {resource}, exceeding the limit of {limit}")]
|
||||||
|
LimitExceeded {
|
||||||
|
/// Source that violated the limit.
|
||||||
|
source_id: SourceId,
|
||||||
|
/// Human-readable resource unit.
|
||||||
|
resource: &'static str,
|
||||||
|
/// Observed value, or `u64::MAX` after arithmetic overflow.
|
||||||
|
actual: u64,
|
||||||
|
/// Configured maximum.
|
||||||
|
limit: u64,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Archive expansion is disproportionate to compressed size.
|
||||||
|
#[error("archive member {source_id} exceeds the configured {limit}:1 compression ratio")]
|
||||||
|
CompressionRatio {
|
||||||
|
/// Archive member identity.
|
||||||
|
source_id: SourceId,
|
||||||
|
/// Maximum allowed ratio.
|
||||||
|
limit: u64,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// An archive member name is unsafe to treat as a relative identity.
|
||||||
|
#[error("unsafe archive member path `{member}` in `{archive}`")]
|
||||||
|
UnsafeArchiveMember {
|
||||||
|
/// Outer archive path.
|
||||||
|
archive: PathBuf,
|
||||||
|
/// Untrusted member name from the archive.
|
||||||
|
member: String,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Encrypted members are unsupported.
|
||||||
|
#[error("archive member {source_id} is encrypted")]
|
||||||
|
EncryptedMember {
|
||||||
|
/// Encrypted archive member identity.
|
||||||
|
source_id: SourceId,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Input contained a NUL byte and is treated as binary.
|
||||||
|
#[error("input {source_id} contains a NUL byte")]
|
||||||
|
BinaryContent {
|
||||||
|
/// Source treated as binary.
|
||||||
|
source_id: SourceId,
|
||||||
|
},
|
||||||
|
|
||||||
|
/// Input was not strict UTF-8.
|
||||||
|
#[error("input {source_id} is not valid UTF-8")]
|
||||||
|
InvalidUtf8 {
|
||||||
|
/// Source that failed decoding.
|
||||||
|
source_id: SourceId,
|
||||||
|
/// Underlying UTF-8 conversion error.
|
||||||
|
#[source]
|
||||||
|
source: FromUtf8Error,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
impl InputError {
|
||||||
|
/// Return the exact document identity when the failure occurred after discovery.
|
||||||
|
pub const fn source_id(&self) -> Option<&SourceId> {
|
||||||
|
match self {
|
||||||
|
Self::LimitExceeded { source_id, .. }
|
||||||
|
| Self::CompressionRatio { source_id, .. }
|
||||||
|
| Self::EncryptedMember { source_id }
|
||||||
|
| Self::BinaryContent { source_id }
|
||||||
|
| Self::InvalidUtf8 { source_id, .. } => Some(source_id),
|
||||||
|
Self::StdinIo { .. }
|
||||||
|
| Self::MissingPath { .. }
|
||||||
|
| Self::Io { .. }
|
||||||
|
| Self::Walk { .. }
|
||||||
|
| Self::Zip { .. }
|
||||||
|
| Self::UnsafeArchiveMember { .. } => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Result type used by the input layer.
|
||||||
|
pub type InputResult<T> = Result<T, InputError>;
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn source_identity_is_available_only_for_document_specific_failures() {
|
||||||
|
let source_id = SourceId::Stdin;
|
||||||
|
for error in [
|
||||||
|
InputError::LimitExceeded {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
resource: "bytes",
|
||||||
|
actual: 2,
|
||||||
|
limit: 1,
|
||||||
|
},
|
||||||
|
InputError::CompressionRatio {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
limit: 2,
|
||||||
|
},
|
||||||
|
InputError::EncryptedMember {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
},
|
||||||
|
InputError::BinaryContent {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
},
|
||||||
|
InputError::InvalidUtf8 {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
source: String::from_utf8(vec![0xff]).unwrap_err(),
|
||||||
|
},
|
||||||
|
] {
|
||||||
|
assert_eq!(error.source_id(), Some(&source_id));
|
||||||
|
}
|
||||||
|
|
||||||
|
let error = InputError::MissingPath {
|
||||||
|
path: PathBuf::from("missing"),
|
||||||
|
};
|
||||||
|
assert_eq!(error.source_id(), None);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,426 @@
|
||||||
|
//! Streaming ZIP-based archive input.
|
||||||
|
|
||||||
|
use std::{
|
||||||
|
fs::{self, File},
|
||||||
|
io::Read,
|
||||||
|
path::{Component, Path, PathBuf},
|
||||||
|
};
|
||||||
|
|
||||||
|
use zip::ZipArchive;
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
SourceId,
|
||||||
|
config::InputLimits,
|
||||||
|
error::{InputError, InputResult},
|
||||||
|
input::Document,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Iterator over text members in one ZIP archive.
|
||||||
|
pub(super) struct ArchiveDocuments {
|
||||||
|
/// Outer archive identity.
|
||||||
|
outer: PathBuf,
|
||||||
|
/// Open ZIP reader retained between iterator calls.
|
||||||
|
archive: ZipArchive<File>,
|
||||||
|
/// Index of the next member to inspect.
|
||||||
|
index: usize,
|
||||||
|
/// Cumulative declared expanded bytes accepted so far.
|
||||||
|
expanded_bytes: u64,
|
||||||
|
/// Resource policy copied from resolved configuration.
|
||||||
|
limits: InputLimits,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for ArchiveDocuments {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
formatter
|
||||||
|
.debug_struct("ArchiveDocuments")
|
||||||
|
.field("outer", &self.outer)
|
||||||
|
.field("members", &self.archive.len())
|
||||||
|
.field("index", &self.index)
|
||||||
|
.finish()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ArchiveDocuments {
|
||||||
|
/// Open an archive for streaming member reads.
|
||||||
|
pub(super) fn open(outer: PathBuf, limits: &InputLimits) -> InputResult<Self> {
|
||||||
|
let archive_id = SourceId::File(outer.clone());
|
||||||
|
let archive_size = fs::metadata(&outer)
|
||||||
|
.map_err(|source| InputError::Io {
|
||||||
|
action: "inspect",
|
||||||
|
path: outer.clone(),
|
||||||
|
source,
|
||||||
|
})?
|
||||||
|
.len();
|
||||||
|
if archive_size > limits.max_archive_bytes {
|
||||||
|
return Err(InputError::LimitExceeded {
|
||||||
|
source_id: archive_id.clone(),
|
||||||
|
resource: "bytes",
|
||||||
|
actual: archive_size,
|
||||||
|
limit: limits.max_archive_bytes,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let file = File::open(&outer).map_err(|source| InputError::Io {
|
||||||
|
action: "open",
|
||||||
|
path: outer.clone(),
|
||||||
|
source,
|
||||||
|
})?;
|
||||||
|
let archive = ZipArchive::new(file).map_err(|source| InputError::Zip {
|
||||||
|
archive: outer.clone(),
|
||||||
|
source,
|
||||||
|
})?;
|
||||||
|
if archive.len() > limits.max_archive_entries {
|
||||||
|
return Err(InputError::LimitExceeded {
|
||||||
|
source_id: archive_id,
|
||||||
|
resource: "entries",
|
||||||
|
actual: u64::try_from(archive.len()).unwrap_or(u64::MAX),
|
||||||
|
limit: u64::try_from(limits.max_archive_entries).unwrap_or(u64::MAX),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
outer,
|
||||||
|
archive,
|
||||||
|
index: 0,
|
||||||
|
expanded_bytes: 0,
|
||||||
|
limits: limits.clone(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Iterator for ArchiveDocuments {
|
||||||
|
type Item = InputResult<Document>;
|
||||||
|
|
||||||
|
fn next(&mut self) -> Option<Self::Item> {
|
||||||
|
while self.index < self.archive.len() {
|
||||||
|
let index = self.index;
|
||||||
|
self.index += 1;
|
||||||
|
let mut member = match self
|
||||||
|
.archive
|
||||||
|
.by_index(index)
|
||||||
|
.map_err(|source| InputError::Zip {
|
||||||
|
archive: self.outer.clone(),
|
||||||
|
source,
|
||||||
|
}) {
|
||||||
|
Ok(member) => member,
|
||||||
|
Err(error) => return Some(Err(error)),
|
||||||
|
};
|
||||||
|
|
||||||
|
if member.is_dir() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let name = match safe_member_path(&self.outer, member.name()) {
|
||||||
|
Ok(name) => name,
|
||||||
|
Err(error) => return Some(Err(error)),
|
||||||
|
};
|
||||||
|
let source_id = SourceId::ArchiveMember {
|
||||||
|
archive: self.outer.clone(),
|
||||||
|
member: name.clone(),
|
||||||
|
};
|
||||||
|
if member.encrypted() {
|
||||||
|
return Some(Err(InputError::EncryptedMember { source_id }));
|
||||||
|
}
|
||||||
|
|
||||||
|
let expanded_size = member.size();
|
||||||
|
if expanded_size > self.limits.max_document_bytes {
|
||||||
|
return Some(Err(InputError::LimitExceeded {
|
||||||
|
source_id,
|
||||||
|
resource: "bytes",
|
||||||
|
actual: expanded_size,
|
||||||
|
limit: self.limits.max_document_bytes,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
let compressed_size = member.compressed_size();
|
||||||
|
let allowed_expanded =
|
||||||
|
compressed_size.saturating_mul(self.limits.max_compression_ratio);
|
||||||
|
if expanded_size > 0 && expanded_size > allowed_expanded {
|
||||||
|
return Some(Err(InputError::CompressionRatio {
|
||||||
|
source_id,
|
||||||
|
limit: self.limits.max_compression_ratio,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
self.expanded_bytes = match self.expanded_bytes.checked_add(expanded_size) {
|
||||||
|
Some(total) if total <= self.limits.max_expanded_bytes => total,
|
||||||
|
_ => {
|
||||||
|
return Some(Err(InputError::LimitExceeded {
|
||||||
|
source_id: SourceId::File(self.outer.clone()),
|
||||||
|
resource: "expanded bytes",
|
||||||
|
actual: u64::MAX,
|
||||||
|
limit: self.limits.max_expanded_bytes,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let read_limit = self.limits.max_document_bytes.saturating_add(1);
|
||||||
|
let mut bytes = Vec::new();
|
||||||
|
if let Err(source) = member.by_ref().take(read_limit).read_to_end(&mut bytes) {
|
||||||
|
return Some(Err(InputError::Io {
|
||||||
|
action: "read archive member from",
|
||||||
|
path: self.outer.clone(),
|
||||||
|
source,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > self.limits.max_document_bytes {
|
||||||
|
return Some(Err(InputError::LimitExceeded {
|
||||||
|
source_id,
|
||||||
|
resource: "bytes",
|
||||||
|
actual: u64::try_from(bytes.len()).unwrap_or(u64::MAX),
|
||||||
|
limit: self.limits.max_document_bytes,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
if bytes.contains(&0) {
|
||||||
|
return Some(Err(InputError::BinaryContent { source_id }));
|
||||||
|
}
|
||||||
|
let content = match String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
source,
|
||||||
|
}) {
|
||||||
|
Ok(content) => content,
|
||||||
|
Err(error) => return Some(Err(error)),
|
||||||
|
};
|
||||||
|
|
||||||
|
return Some(Ok(Document {
|
||||||
|
source: SourceId::ArchiveMember {
|
||||||
|
archive: self.outer.clone(),
|
||||||
|
member: name,
|
||||||
|
},
|
||||||
|
content,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse an archive name into a safe relative member identity.
|
||||||
|
fn safe_member_path(archive: &Path, name: &str) -> InputResult<PathBuf> {
|
||||||
|
let normalized = name.replace('\\', "/");
|
||||||
|
let path = Path::new(&normalized);
|
||||||
|
if normalized.is_empty()
|
||||||
|
|| path
|
||||||
|
.components()
|
||||||
|
.any(|component| !matches!(component, Component::Normal(_)))
|
||||||
|
{
|
||||||
|
return Err(InputError::UnsafeArchiveMember {
|
||||||
|
archive: archive.to_path_buf(),
|
||||||
|
member: name.to_owned(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(path.to_path_buf())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use std::io::Write;
|
||||||
|
|
||||||
|
use tempfile::tempdir;
|
||||||
|
use zip::write::FileOptions;
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn create_archive(path: &Path, entries: &[(&str, &[u8])], compression: zip::CompressionMethod) {
|
||||||
|
let file = File::create(path).unwrap();
|
||||||
|
let mut writer = zip::ZipWriter::new(file);
|
||||||
|
let options: FileOptions<()> = FileOptions::default().compression_method(compression);
|
||||||
|
for (name, bytes) in entries {
|
||||||
|
writer.start_file(*name, options).unwrap();
|
||||||
|
writer.write_all(bytes).unwrap();
|
||||||
|
}
|
||||||
|
writer.finish().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_absolute_and_traversing_member_paths() {
|
||||||
|
for path in [
|
||||||
|
"../escape.txt",
|
||||||
|
"/absolute.txt",
|
||||||
|
"a/../../escape",
|
||||||
|
r"a\..\escape",
|
||||||
|
] {
|
||||||
|
assert!(
|
||||||
|
safe_member_path(Path::new("archive.zip"), path).is_err(),
|
||||||
|
"accepted {path}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
safe_member_path(Path::new("archive.zip"), "safe/member.txt").unwrap(),
|
||||||
|
Path::new("safe/member.txt")
|
||||||
|
);
|
||||||
|
assert!(safe_member_path(Path::new("archive.zip"), "").is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn opening_archives_reports_filesystem_format_size_and_entry_limits() {
|
||||||
|
let directory = tempdir().unwrap();
|
||||||
|
let missing = directory.path().join("missing.zip");
|
||||||
|
assert!(matches!(
|
||||||
|
ArchiveDocuments::open(missing.clone(), &InputLimits::default()),
|
||||||
|
Err(InputError::Io { action: "inspect", path, .. }) if path == missing
|
||||||
|
));
|
||||||
|
|
||||||
|
let malformed = directory.path().join("malformed.zip");
|
||||||
|
fs::write(&malformed, b"not a zip").unwrap();
|
||||||
|
assert!(matches!(
|
||||||
|
ArchiveDocuments::open(malformed.clone(), &InputLimits::default()),
|
||||||
|
Err(InputError::Zip { archive, .. }) if archive == malformed
|
||||||
|
));
|
||||||
|
|
||||||
|
let path = directory.path().join("limits.zip");
|
||||||
|
create_archive(
|
||||||
|
&path,
|
||||||
|
&[("one.txt", b"one"), ("two.txt", b"two")],
|
||||||
|
zip::CompressionMethod::Stored,
|
||||||
|
);
|
||||||
|
let limits = InputLimits {
|
||||||
|
max_archive_bytes: 1,
|
||||||
|
..InputLimits::default()
|
||||||
|
};
|
||||||
|
assert!(matches!(
|
||||||
|
ArchiveDocuments::open(path.clone(), &limits),
|
||||||
|
Err(InputError::LimitExceeded {
|
||||||
|
resource: "bytes",
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
|
||||||
|
let limits = InputLimits {
|
||||||
|
max_archive_bytes: u64::MAX,
|
||||||
|
max_archive_entries: 1,
|
||||||
|
..InputLimits::default()
|
||||||
|
};
|
||||||
|
assert!(matches!(
|
||||||
|
ArchiveDocuments::open(path, &limits),
|
||||||
|
Err(InputError::LimitExceeded {
|
||||||
|
resource: "entries",
|
||||||
|
actual: 2,
|
||||||
|
limit: 1,
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn iterator_skips_directories_and_preserves_safe_member_identity() {
|
||||||
|
let directory = tempdir().unwrap();
|
||||||
|
let path = directory.path().join("documents.zip");
|
||||||
|
let file = File::create(&path).unwrap();
|
||||||
|
let mut writer = zip::ZipWriter::new(file);
|
||||||
|
let options: FileOptions<()> = FileOptions::default();
|
||||||
|
writer.add_directory("docs/", options).unwrap();
|
||||||
|
writer.start_file("docs/article.txt", options).unwrap();
|
||||||
|
writer.write_all("héllo".as_bytes()).unwrap();
|
||||||
|
writer.finish().unwrap();
|
||||||
|
|
||||||
|
let mut archive = ArchiveDocuments::open(path.clone(), &InputLimits::default()).unwrap();
|
||||||
|
let debug = format!("{archive:?}");
|
||||||
|
assert!(debug.contains("members: 2"));
|
||||||
|
let document = archive.next().unwrap().unwrap();
|
||||||
|
assert_eq!(document.content, "héllo");
|
||||||
|
assert_eq!(
|
||||||
|
document.source,
|
||||||
|
SourceId::ArchiveMember {
|
||||||
|
archive: path,
|
||||||
|
member: PathBuf::from("docs/article.txt"),
|
||||||
|
}
|
||||||
|
);
|
||||||
|
assert!(archive.next().is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn iterator_rejects_unsafe_binary_and_non_utf8_members_independently() {
|
||||||
|
for (name, bytes, expected) in [
|
||||||
|
("../escape.txt", b"text".as_slice(), "unsafe"),
|
||||||
|
("binary.txt", b"a\0b".as_slice(), "binary"),
|
||||||
|
("invalid.txt", [0xff].as_slice(), "utf8"),
|
||||||
|
] {
|
||||||
|
let directory = tempdir().unwrap();
|
||||||
|
let path = directory.path().join("invalid-member.zip");
|
||||||
|
create_archive(&path, &[(name, bytes)], zip::CompressionMethod::Stored);
|
||||||
|
let error = ArchiveDocuments::open(path, &InputLimits::default())
|
||||||
|
.unwrap()
|
||||||
|
.next()
|
||||||
|
.unwrap()
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(
|
||||||
|
matches!(
|
||||||
|
(expected, error),
|
||||||
|
("unsafe", InputError::UnsafeArchiveMember { .. })
|
||||||
|
| ("binary", InputError::BinaryContent { .. })
|
||||||
|
| ("utf8", InputError::InvalidUtf8 { .. })
|
||||||
|
),
|
||||||
|
"wrong error for {name}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn iterator_enforces_member_expansion_and_compression_limits() {
|
||||||
|
let directory = tempdir().unwrap();
|
||||||
|
let oversized = directory.path().join("oversized.zip");
|
||||||
|
create_archive(
|
||||||
|
&oversized,
|
||||||
|
&[("large.txt", b"12345")],
|
||||||
|
zip::CompressionMethod::Stored,
|
||||||
|
);
|
||||||
|
let limits = InputLimits {
|
||||||
|
max_document_bytes: 4,
|
||||||
|
..InputLimits::default()
|
||||||
|
};
|
||||||
|
let error = ArchiveDocuments::open(oversized, &limits)
|
||||||
|
.unwrap()
|
||||||
|
.next()
|
||||||
|
.unwrap()
|
||||||
|
.unwrap_err();
|
||||||
|
assert!(matches!(
|
||||||
|
error,
|
||||||
|
InputError::LimitExceeded {
|
||||||
|
resource: "bytes",
|
||||||
|
actual: 5,
|
||||||
|
limit: 4,
|
||||||
|
..
|
||||||
|
}
|
||||||
|
));
|
||||||
|
|
||||||
|
let compressed = directory.path().join("compressed.zip");
|
||||||
|
create_archive(
|
||||||
|
&compressed,
|
||||||
|
&[("repeated.txt", &[b'a'; 4096])],
|
||||||
|
zip::CompressionMethod::Deflated,
|
||||||
|
);
|
||||||
|
let limits = InputLimits {
|
||||||
|
max_compression_ratio: 1,
|
||||||
|
..InputLimits::default()
|
||||||
|
};
|
||||||
|
assert!(matches!(
|
||||||
|
ArchiveDocuments::open(compressed, &limits)
|
||||||
|
.unwrap()
|
||||||
|
.next()
|
||||||
|
.unwrap(),
|
||||||
|
Err(InputError::CompressionRatio { limit: 1, .. })
|
||||||
|
));
|
||||||
|
|
||||||
|
let cumulative = directory.path().join("cumulative.zip");
|
||||||
|
create_archive(
|
||||||
|
&cumulative,
|
||||||
|
&[("one.txt", b"1234"), ("two.txt", b"5678")],
|
||||||
|
zip::CompressionMethod::Stored,
|
||||||
|
);
|
||||||
|
let limits = InputLimits {
|
||||||
|
max_expanded_bytes: 6,
|
||||||
|
..InputLimits::default()
|
||||||
|
};
|
||||||
|
let mut archive = ArchiveDocuments::open(cumulative, &limits).unwrap();
|
||||||
|
assert!(archive.next().unwrap().is_ok());
|
||||||
|
assert!(matches!(
|
||||||
|
archive.next().unwrap(),
|
||||||
|
Err(InputError::LimitExceeded {
|
||||||
|
resource: "expanded bytes",
|
||||||
|
actual: u64::MAX,
|
||||||
|
limit: 6,
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
//! Regular-file input support.
|
||||||
|
|
||||||
|
use std::{fs, path::Path};
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
SourceId,
|
||||||
|
config::InputLimits,
|
||||||
|
error::{InputError, InputResult},
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Read a regular file as strict UTF-8 text.
|
||||||
|
pub(super) fn read_text(path: &Path, limits: &InputLimits) -> InputResult<String> {
|
||||||
|
let source_id = SourceId::File(path.to_path_buf());
|
||||||
|
let metadata = fs::metadata(path).map_err(|source| InputError::Io {
|
||||||
|
action: "inspect",
|
||||||
|
path: path.to_path_buf(),
|
||||||
|
source,
|
||||||
|
})?;
|
||||||
|
if metadata.len() > limits.max_document_bytes {
|
||||||
|
return Err(InputError::LimitExceeded {
|
||||||
|
source_id,
|
||||||
|
resource: "bytes",
|
||||||
|
actual: metadata.len(),
|
||||||
|
limit: limits.max_document_bytes,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let bytes = fs::read(path).map_err(|source| InputError::Io {
|
||||||
|
action: "read",
|
||||||
|
path: path.to_path_buf(),
|
||||||
|
source,
|
||||||
|
})?;
|
||||||
|
if bytes.contains(&0) {
|
||||||
|
return Err(InputError::BinaryContent { source_id });
|
||||||
|
}
|
||||||
|
String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 { source_id, source })
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,127 @@
|
||||||
|
//! Streaming input acquisition for files, directories, and ZIP containers.
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests;
|
||||||
|
|
||||||
|
mod archive;
|
||||||
|
mod file;
|
||||||
|
mod walk;
|
||||||
|
|
||||||
|
use std::{io::Read, path::Path, vec};
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
SourceId,
|
||||||
|
config::InputLimits,
|
||||||
|
error::{InputError, InputResult},
|
||||||
|
};
|
||||||
|
|
||||||
|
use self::{archive::ArchiveDocuments, walk::WalkDocuments};
|
||||||
|
|
||||||
|
/// One strict UTF-8 source document.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct Document {
|
||||||
|
/// Stable identity of the file or archive member.
|
||||||
|
pub source: SourceId,
|
||||||
|
/// Complete UTF-8 content of this document.
|
||||||
|
pub content: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Streaming documents produced from one requested path.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct Documents {
|
||||||
|
/// Input-specific iterator implementation.
|
||||||
|
inner: DocumentsInner,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private iterator variants hidden behind the stable public wrapper.
|
||||||
|
#[derive(Debug)]
|
||||||
|
enum DocumentsInner {
|
||||||
|
/// One regular-file result.
|
||||||
|
File(vec::IntoIter<InputResult<Document>>),
|
||||||
|
/// Recursive directory results.
|
||||||
|
Directory(WalkDocuments),
|
||||||
|
/// ZIP member results.
|
||||||
|
Archive(ArchiveDocuments),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Iterator for Documents {
|
||||||
|
type Item = InputResult<Document>;
|
||||||
|
|
||||||
|
fn next(&mut self) -> Option<Self::Item> {
|
||||||
|
match &mut self.inner {
|
||||||
|
DocumentsInner::File(document) => document.next(),
|
||||||
|
DocumentsInner::Directory(documents) => documents.next(),
|
||||||
|
DocumentsInner::Archive(documents) => documents.next(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Open a file, directory, or recognized ZIP-based container for streaming.
|
||||||
|
pub fn read(path: &Path, limits: &InputLimits) -> InputResult<Documents> {
|
||||||
|
if path.is_file() {
|
||||||
|
if has_zip_extension(path) {
|
||||||
|
return ArchiveDocuments::open(path.to_path_buf(), limits).map(|archive| Documents {
|
||||||
|
inner: DocumentsInner::Archive(archive),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let result = file::read_text(path, limits).map(|content| Document {
|
||||||
|
source: SourceId::File(path.to_path_buf()),
|
||||||
|
content,
|
||||||
|
});
|
||||||
|
return Ok(Documents {
|
||||||
|
inner: DocumentsInner::File(vec![result].into_iter()),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if path.is_dir() {
|
||||||
|
return Ok(Documents {
|
||||||
|
inner: DocumentsInner::Directory(WalkDocuments::new(path, limits)),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Err(InputError::MissingPath {
|
||||||
|
path: path.to_path_buf(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read standard input as one bounded strict UTF-8 document.
|
||||||
|
pub fn read_stdin(reader: impl Read, limits: &InputLimits) -> InputResult<Document> {
|
||||||
|
let source_id = SourceId::Stdin;
|
||||||
|
let mut bytes = Vec::new();
|
||||||
|
reader
|
||||||
|
.take(limits.max_document_bytes.saturating_add(1))
|
||||||
|
.read_to_end(&mut bytes)
|
||||||
|
.map_err(|source| InputError::StdinIo { source })?;
|
||||||
|
if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > limits.max_document_bytes {
|
||||||
|
return Err(InputError::LimitExceeded {
|
||||||
|
source_id,
|
||||||
|
resource: "bytes",
|
||||||
|
actual: u64::try_from(bytes.len()).unwrap_or(u64::MAX),
|
||||||
|
limit: limits.max_document_bytes,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if bytes.contains(&0) {
|
||||||
|
return Err(InputError::BinaryContent { source_id });
|
||||||
|
}
|
||||||
|
let content = String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 {
|
||||||
|
source_id: source_id.clone(),
|
||||||
|
source,
|
||||||
|
})?;
|
||||||
|
Ok(Document {
|
||||||
|
source: source_id,
|
||||||
|
content,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether a path has an extension treated as a ZIP container.
|
||||||
|
fn has_zip_extension(path: &Path) -> bool {
|
||||||
|
path.extension()
|
||||||
|
.and_then(|extension| extension.to_str())
|
||||||
|
.is_some_and(|extension| {
|
||||||
|
matches!(
|
||||||
|
extension.to_ascii_lowercase().as_str(),
|
||||||
|
"zip" | "docx" | "xlsx" | "pptx"
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,197 @@
|
||||||
|
use std::{
|
||||||
|
fs::{self, File},
|
||||||
|
io::Write,
|
||||||
|
path::Path,
|
||||||
|
};
|
||||||
|
|
||||||
|
use tempfile::tempdir;
|
||||||
|
use zip::write::FileOptions;
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
struct FailingReader;
|
||||||
|
|
||||||
|
impl std::io::Read for FailingReader {
|
||||||
|
fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {
|
||||||
|
Err(std::io::Error::other("fixture read failure"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn limits() -> InputLimits {
|
||||||
|
InputLimits::default()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn write_file(path: &Path, content: &str) {
|
||||||
|
let mut file = File::create(path).expect("create fixture");
|
||||||
|
file.write_all(content.as_bytes()).expect("write fixture");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn create_zip(path: &Path) -> std::io::Result<()> {
|
||||||
|
let file = File::create(path)?;
|
||||||
|
let mut zip = zip::ZipWriter::new(file);
|
||||||
|
let options: FileOptions<()> = FileOptions::default()
|
||||||
|
.compression_method(zip::CompressionMethod::Deflated)
|
||||||
|
.unix_permissions(0o644);
|
||||||
|
|
||||||
|
zip.start_file("foo.txt", options)?;
|
||||||
|
zip.write_all(b"hello foo")?;
|
||||||
|
zip.start_file("bar.txt", options)?;
|
||||||
|
zip.write_all(b"hello bar")?;
|
||||||
|
zip.finish()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn reads_one_file() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let path = directory.path().join("test.txt");
|
||||||
|
write_file(&path, "line1\nline2");
|
||||||
|
|
||||||
|
let documents = read(&path, &limits())
|
||||||
|
.expect("open input")
|
||||||
|
.collect::<InputResult<Vec<_>>>()
|
||||||
|
.expect("read input");
|
||||||
|
|
||||||
|
assert_eq!(documents.len(), 1);
|
||||||
|
assert_eq!(documents[0].source, SourceId::File(path));
|
||||||
|
assert_eq!(documents[0].content, "line1\nline2");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn yields_a_directory_error_without_discarding_other_files() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
write_file(&directory.path().join("a.txt"), "a");
|
||||||
|
fs::write(directory.path().join("b.txt"), [0xff]).expect("write invalid fixture");
|
||||||
|
write_file(&directory.path().join("c.txt"), "c");
|
||||||
|
|
||||||
|
let results = read(directory.path(), &limits())
|
||||||
|
.expect("open directory")
|
||||||
|
.collect::<Vec<_>>();
|
||||||
|
|
||||||
|
assert_eq!(results.len(), 3);
|
||||||
|
assert_eq!(results.iter().filter(|result| result.is_ok()).count(), 2);
|
||||||
|
assert_eq!(results.iter().filter(|result| result.is_err()).count(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn streams_archive_members_with_outer_identity() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let path = directory.path().join("archive.zip");
|
||||||
|
create_zip(&path).expect("create zip");
|
||||||
|
|
||||||
|
let documents = read(&path, &limits())
|
||||||
|
.expect("open archive")
|
||||||
|
.collect::<InputResult<Vec<_>>>()
|
||||||
|
.expect("read archive");
|
||||||
|
|
||||||
|
assert_eq!(documents.len(), 2);
|
||||||
|
assert!(documents.iter().all(|document| matches!(
|
||||||
|
&document.source,
|
||||||
|
SourceId::ArchiveMember { archive, .. } if archive == &path
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn invalid_utf8_is_a_per_document_error() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let path = directory.path().join("invalid.txt");
|
||||||
|
fs::write(&path, [0xff, 0xfe]).expect("write fixture");
|
||||||
|
|
||||||
|
let mut documents = read(&path, &limits()).expect("open input");
|
||||||
|
assert!(documents.next().expect("one result").is_err());
|
||||||
|
assert!(documents.next().is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn binary_content_is_an_input_error_not_a_clean_document() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let path = directory.path().join("binary.txt");
|
||||||
|
fs::write(&path, b"clean prefix\0binary suffix").expect("write fixture");
|
||||||
|
|
||||||
|
let mut documents = read(&path, &limits()).expect("open input");
|
||||||
|
let error = documents.next().expect("one result").unwrap_err();
|
||||||
|
|
||||||
|
assert!(matches!(
|
||||||
|
error,
|
||||||
|
InputError::BinaryContent {
|
||||||
|
source_id: SourceId::File(error_path)
|
||||||
|
} if error_path == path
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn missing_path_is_rejected() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let path = directory.path().join("missing.txt");
|
||||||
|
|
||||||
|
let error = read(&path, &limits()).expect_err("missing input should fail");
|
||||||
|
assert!(matches!(error, InputError::MissingPath { path: error_path } if error_path == path));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn enforces_document_size_limit_before_reading() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let path = directory.path().join("large.txt");
|
||||||
|
write_file(&path, "too large");
|
||||||
|
let mut limits = limits();
|
||||||
|
limits.max_document_bytes = 3;
|
||||||
|
|
||||||
|
let mut documents = read(&path, &limits).expect("open input");
|
||||||
|
let error = documents
|
||||||
|
.next()
|
||||||
|
.expect("one result")
|
||||||
|
.expect_err("oversized input");
|
||||||
|
|
||||||
|
assert!(matches!(
|
||||||
|
error,
|
||||||
|
InputError::LimitExceeded {
|
||||||
|
actual: 9,
|
||||||
|
limit: 3,
|
||||||
|
..
|
||||||
|
}
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn stdin_is_bounded_and_decoded_as_strict_utf8() {
|
||||||
|
let document = read_stdin("hello".as_bytes(), &limits()).unwrap();
|
||||||
|
assert_eq!(document.source, SourceId::Stdin);
|
||||||
|
assert_eq!(document.content, "hello");
|
||||||
|
|
||||||
|
let mut small = limits();
|
||||||
|
small.max_document_bytes = 3;
|
||||||
|
assert!(matches!(
|
||||||
|
read_stdin("hello".as_bytes(), &small),
|
||||||
|
Err(InputError::LimitExceeded {
|
||||||
|
source_id: SourceId::Stdin,
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
assert!(matches!(
|
||||||
|
read_stdin([0xff].as_slice(), &limits()),
|
||||||
|
Err(InputError::InvalidUtf8 {
|
||||||
|
source_id: SourceId::Stdin,
|
||||||
|
..
|
||||||
|
})
|
||||||
|
));
|
||||||
|
assert!(matches!(
|
||||||
|
read_stdin(b"nul\0byte".as_slice(), &limits()),
|
||||||
|
Err(InputError::BinaryContent {
|
||||||
|
source_id: SourceId::Stdin
|
||||||
|
})
|
||||||
|
));
|
||||||
|
assert!(matches!(
|
||||||
|
read_stdin(FailingReader, &limits()),
|
||||||
|
Err(InputError::StdinIo { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn regular_file_metadata_failures_retain_operation_context() {
|
||||||
|
let directory = tempdir().expect("temp dir");
|
||||||
|
let missing = directory.path().join("missing.txt");
|
||||||
|
assert!(matches!(
|
||||||
|
file::read_text(&missing, &limits()),
|
||||||
|
Err(InputError::Io { action: "inspect", path, .. }) if path == missing
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,62 @@
|
||||||
|
//! Streaming directory traversal.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use walkdir::{IntoIter, WalkDir};
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
SourceId,
|
||||||
|
config::InputLimits,
|
||||||
|
error::{InputError, InputResult},
|
||||||
|
input::Document,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Iterator over readable regular files below a directory.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub(super) struct WalkDocuments {
|
||||||
|
/// Underlying streaming directory iterator.
|
||||||
|
entries: IntoIter,
|
||||||
|
/// Limits applied to every yielded file.
|
||||||
|
limits: InputLimits,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl WalkDocuments {
|
||||||
|
/// Begin a recursive walk without following symbolic links.
|
||||||
|
pub(super) fn new(root: &Path, limits: &InputLimits) -> Self {
|
||||||
|
Self {
|
||||||
|
entries: WalkDir::new(root)
|
||||||
|
.follow_links(false)
|
||||||
|
.max_depth(limits.max_walk_depth)
|
||||||
|
.sort_by_file_name()
|
||||||
|
.into_iter(),
|
||||||
|
limits: limits.clone(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Iterator for WalkDocuments {
|
||||||
|
type Item = InputResult<Document>;
|
||||||
|
|
||||||
|
fn next(&mut self) -> Option<Self::Item> {
|
||||||
|
loop {
|
||||||
|
let entry = match self.entries.next()? {
|
||||||
|
Ok(entry) => entry,
|
||||||
|
Err(source) => return Some(Err(InputError::Walk { source })),
|
||||||
|
};
|
||||||
|
if !entry.file_type().is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let path = entry.into_path();
|
||||||
|
let content = match super::file::read_text(&path, &self.limits) {
|
||||||
|
Ok(content) => content,
|
||||||
|
Err(error) => return Some(Err(error)),
|
||||||
|
};
|
||||||
|
|
||||||
|
return Some(Ok(Document {
|
||||||
|
source: SourceId::File(path),
|
||||||
|
content,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,106 @@
|
||||||
|
//! Shared analysis library for John Slop's command-line, formatter, and LSP
|
||||||
|
//! adapters.
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
mod config;
|
||||||
|
mod detector;
|
||||||
|
mod diagnostic;
|
||||||
|
mod error;
|
||||||
|
mod input;
|
||||||
|
mod model;
|
||||||
|
mod replacement;
|
||||||
|
mod unicode_review;
|
||||||
|
|
||||||
|
pub use config::{
|
||||||
|
Config, ConfigError, ConfigLayerError, ConfigLoader, ConfigName, ConfigNameError,
|
||||||
|
ConfigOverrides, ConfigPaths, ConfigValidationError, ConfusablePolicy, ContentKind,
|
||||||
|
DiagnosticCategories, DiagnosticPolicy, EvidenceThresholds, HeuristicRule, InputLimits,
|
||||||
|
InputOverrides, MatcherType, ReplacementRule, RuleScope, ThresholdOverrides, UnicodePolicy,
|
||||||
|
default_config_template, default_diagnostics_template,
|
||||||
|
};
|
||||||
|
pub use detector::{AnalysisContext, Detector, DetectorBuildError, Rule};
|
||||||
|
pub use diagnostic::FindingDiagnostic;
|
||||||
|
pub use error::{InputError, InputResult};
|
||||||
|
pub use input::{Document, Documents, read_stdin};
|
||||||
|
pub use model::{
|
||||||
|
Analysis, AnalysisSummary, Category, EvidenceBand, EvidenceScore, Finding, FindingSeverity,
|
||||||
|
RuleId, SeverityCounts, SourceId, TextEdit,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Deprecated name for [`EvidenceScore`].
|
||||||
|
#[deprecated(since = "0.1.0", note = "use EvidenceScore")]
|
||||||
|
pub type Confidence = EvidenceScore;
|
||||||
|
|
||||||
|
/// Deprecated name for [`EvidenceBand`].
|
||||||
|
#[deprecated(since = "0.1.0", note = "use EvidenceBand")]
|
||||||
|
pub type ConfidenceBand = EvidenceBand;
|
||||||
|
|
||||||
|
/// Deprecated name for [`EvidenceThresholds`].
|
||||||
|
#[deprecated(since = "0.1.0", note = "use EvidenceThresholds")]
|
||||||
|
pub type ConfidenceThresholds = EvidenceThresholds;
|
||||||
|
pub use replacement::ReplacementError;
|
||||||
|
|
||||||
|
/// Load embedded defaults plus optional platform user configuration.
|
||||||
|
pub fn load_config() -> Result<Config, ConfigError> {
|
||||||
|
ConfigLoader::discover(None, None)?.load()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Analyze a UTF-8 source snapshot with the resolved configuration.
|
||||||
|
pub fn analyze_source(config: &Config, source: &str) -> Analysis {
|
||||||
|
Detector::new(config)
|
||||||
|
.expect("validated configuration must build a detector")
|
||||||
|
.analyze(source)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Plan safe ASCII replacements without modifying the source.
|
||||||
|
pub fn plan_replacements(config: &Config, source: &str) -> Vec<TextEdit> {
|
||||||
|
replacement::plan(config, source)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Validate and apply text edits against their original UTF-8 source snapshot.
|
||||||
|
pub fn apply_replacements<'source>(
|
||||||
|
source: &'source str,
|
||||||
|
edits: &[TextEdit],
|
||||||
|
) -> Result<std::borrow::Cow<'source, str>, ReplacementError> {
|
||||||
|
replacement::apply(source, edits)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read documents from a file, directory, or supported ZIP-based container.
|
||||||
|
pub fn read_inputs(config: &Config, path: &Path) -> InputResult<Documents> {
|
||||||
|
input::read(path, &config.input)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn facade_uses_one_configuration_for_analysis_and_edits() {
|
||||||
|
let config = load_config().unwrap();
|
||||||
|
let source = "review—this";
|
||||||
|
let analysis = analyze_source(&config, source);
|
||||||
|
let edits = plan_replacements(&config, source);
|
||||||
|
|
||||||
|
assert_eq!(analysis.findings.len(), 1);
|
||||||
|
assert_eq!(analysis.findings[0].fix.as_ref(), edits.first());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
#[allow(deprecated)]
|
||||||
|
fn legacy_score_type_aliases_remain_source_compatible() {
|
||||||
|
let score: Confidence = EvidenceScore::from_score(42);
|
||||||
|
let band: ConfidenceBand = EvidenceBand::Medium;
|
||||||
|
let thresholds: ConfidenceThresholds = EvidenceThresholds {
|
||||||
|
low: 20,
|
||||||
|
medium: 50,
|
||||||
|
high: 75,
|
||||||
|
};
|
||||||
|
|
||||||
|
assert_eq!(score.score(), 42);
|
||||||
|
assert_eq!(band, EvidenceBand::Medium);
|
||||||
|
assert_eq!(thresholds.low, 20);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,377 @@
|
||||||
|
//! Presentation-neutral domain values shared by analysis adapters.
|
||||||
|
|
||||||
|
use std::{fmt, ops::Range, path::PathBuf};
|
||||||
|
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
|
||||||
|
/// Stable identifier for a detection rule.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||||
|
pub struct RuleId(String);
|
||||||
|
|
||||||
|
impl RuleId {
|
||||||
|
/// Construct a rule identifier from validated configuration text.
|
||||||
|
pub fn new(value: impl Into<String>) -> Self {
|
||||||
|
Self(value.into())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Construct a rule identifier from a static, trusted value.
|
||||||
|
pub fn from_static(value: &'static str) -> Self {
|
||||||
|
Self(value.to_owned())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the identifier as text.
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
&self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for RuleId {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
self.0.fmt(formatter)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Broad family of evidence reported by a rule.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "kebab-case")]
|
||||||
|
pub enum Category {
|
||||||
|
/// Suspicious typography or character usage.
|
||||||
|
Typography,
|
||||||
|
/// Repeated or templated writing style.
|
||||||
|
Style,
|
||||||
|
/// Prompt, response, or assistant scaffolding left in the source.
|
||||||
|
PromptLeakage,
|
||||||
|
/// Low-care patterns specific to source code.
|
||||||
|
SourceCode,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Category {
|
||||||
|
/// Return the stable kebab-case presentation name.
|
||||||
|
pub const fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::Typography => "typography",
|
||||||
|
Self::Style => "style",
|
||||||
|
Self::PromptLeakage => "prompt-leakage",
|
||||||
|
Self::SourceCode => "source-code",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for Category {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
formatter.write_str(self.as_str())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Review severity attached to an individual configured finding.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "lowercase")]
|
||||||
|
pub enum FindingSeverity {
|
||||||
|
/// Informational signal worth considering during review.
|
||||||
|
Advice,
|
||||||
|
/// Stronger signal that should be reviewed before acceptance.
|
||||||
|
Warning,
|
||||||
|
/// Definite policy violation under the selected configuration.
|
||||||
|
Error,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FindingSeverity {
|
||||||
|
/// Return the stable lowercase presentation name.
|
||||||
|
pub const fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::Advice => "advice",
|
||||||
|
Self::Warning => "warning",
|
||||||
|
Self::Error => "error",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for FindingSeverity {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
formatter.write_str(self.as_str())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Bounded strength of the configured low-care evidence found in a document.
|
||||||
|
///
|
||||||
|
/// This is deliberately not an AI-authorship probability.
|
||||||
|
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord)]
|
||||||
|
pub struct EvidenceScore(u8);
|
||||||
|
|
||||||
|
impl EvidenceScore {
|
||||||
|
/// Construct an evidence score, clamping values above 100.
|
||||||
|
pub const fn from_score(score: u8) -> Self {
|
||||||
|
Self(if score > 100 { 100 } else { score })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return the normalized score in the inclusive range `0..=100`.
|
||||||
|
pub const fn score(self) -> u8 {
|
||||||
|
self.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Named interpretation of a non-empty analysis score.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "kebab-case")]
|
||||||
|
pub enum EvidenceBand {
|
||||||
|
/// Evidence exists but remains below the configured low threshold.
|
||||||
|
BelowThreshold,
|
||||||
|
/// Score meets the configured low threshold.
|
||||||
|
Low,
|
||||||
|
/// Score meets the configured medium threshold.
|
||||||
|
Medium,
|
||||||
|
/// Score meets the configured high threshold.
|
||||||
|
High,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EvidenceBand {
|
||||||
|
/// Return the stable kebab-case presentation name.
|
||||||
|
pub const fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Self::BelowThreshold => "below-threshold",
|
||||||
|
Self::Low => "low",
|
||||||
|
Self::Medium => "medium",
|
||||||
|
Self::High => "high",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for EvidenceBand {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
formatter.write_str(self.as_str())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Identity of the exact source snapshot being analyzed.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub enum SourceId {
|
||||||
|
/// Content supplied through standard input.
|
||||||
|
Stdin,
|
||||||
|
/// A regular file on disk.
|
||||||
|
File(PathBuf),
|
||||||
|
/// A member contained in an archive file.
|
||||||
|
ArchiveMember {
|
||||||
|
/// Path of the outer archive.
|
||||||
|
archive: PathBuf,
|
||||||
|
/// Path-like member name stored in the archive.
|
||||||
|
member: PathBuf,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
impl fmt::Display for SourceId {
|
||||||
|
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::Stdin => formatter.write_str("<stdin>"),
|
||||||
|
Self::File(path) => path.display().fmt(formatter),
|
||||||
|
Self::ArchiveMember { archive, member } => {
|
||||||
|
write!(formatter, "{}::{}", archive.display(), member.display())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A non-overlapping replacement against an analyzed source snapshot.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub struct TextEdit {
|
||||||
|
/// Zero-based, half-open UTF-8 byte range to replace.
|
||||||
|
pub byte_span: Range<usize>,
|
||||||
|
/// Text to insert in place of the range.
|
||||||
|
pub replacement: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One explainable piece of evidence produced by a detection rule.
|
||||||
|
#[derive(Debug, Clone, PartialEq)]
|
||||||
|
pub struct Finding {
|
||||||
|
/// Stable rule identifier.
|
||||||
|
pub rule_id: RuleId,
|
||||||
|
/// Evidence family.
|
||||||
|
pub category: Category,
|
||||||
|
/// Configured review severity.
|
||||||
|
pub severity: FindingSeverity,
|
||||||
|
/// Concise description of the observed signal.
|
||||||
|
pub message: String,
|
||||||
|
/// Optional remediation or review guidance.
|
||||||
|
pub help: Option<String>,
|
||||||
|
/// Zero-based, half-open UTF-8 byte range in the analyzed source.
|
||||||
|
pub byte_span: Range<usize>,
|
||||||
|
/// Exact source text covered by `byte_span`.
|
||||||
|
pub matched_text: String,
|
||||||
|
/// Configured evidence weight before aggregate normalization.
|
||||||
|
pub weight: f64,
|
||||||
|
/// Optional safe replacement for this finding.
|
||||||
|
pub fix: Option<TextEdit>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete deterministic analysis of one source snapshot.
|
||||||
|
#[derive(Debug, Clone, Default, PartialEq)]
|
||||||
|
pub struct Analysis {
|
||||||
|
/// Findings sorted by source offset and rule identifier.
|
||||||
|
pub findings: Vec<Finding>,
|
||||||
|
/// Aggregate strength of the configured evidence found.
|
||||||
|
pub evidence_score: EvidenceScore,
|
||||||
|
/// Named score band, or no verdict when no evidence was found.
|
||||||
|
pub evidence_band: Option<EvidenceBand>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Analysis {
|
||||||
|
/// Summarize score, named band, and severity counts without consuming findings.
|
||||||
|
pub fn summary(&self) -> AnalysisSummary {
|
||||||
|
AnalysisSummary {
|
||||||
|
evidence_score: self.evidence_score.score(),
|
||||||
|
evidence_band: self.evidence_band,
|
||||||
|
findings: SeverityCounts::from_findings(&self.findings),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compact deterministic summary of one completed analysis.
|
||||||
|
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||||
|
pub struct AnalysisSummary {
|
||||||
|
/// Normalized evidence score in `0..=100`.
|
||||||
|
pub evidence_score: u8,
|
||||||
|
/// Named band, or no verdict when the source has no evidence.
|
||||||
|
pub evidence_band: Option<EvidenceBand>,
|
||||||
|
/// Finding totals grouped by configured severity.
|
||||||
|
pub findings: SeverityCounts,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Finding counts grouped by configured presentation severity.
|
||||||
|
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
||||||
|
pub struct SeverityCounts {
|
||||||
|
/// Informational findings.
|
||||||
|
pub advice: usize,
|
||||||
|
/// Warning findings.
|
||||||
|
pub warnings: usize,
|
||||||
|
/// Error findings.
|
||||||
|
pub errors: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SeverityCounts {
|
||||||
|
/// Return the total number of findings across severities.
|
||||||
|
pub const fn total(self) -> usize {
|
||||||
|
self.advice + self.warnings + self.errors
|
||||||
|
}
|
||||||
|
|
||||||
|
fn from_findings(findings: &[Finding]) -> Self {
|
||||||
|
let mut counts = Self::default();
|
||||||
|
for finding in findings {
|
||||||
|
match finding.severity {
|
||||||
|
FindingSeverity::Advice => counts.advice += 1,
|
||||||
|
FindingSeverity::Warning => counts.warnings += 1,
|
||||||
|
FindingSeverity::Error => counts.errors += 1,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
counts
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn evidence_score_is_bounded() {
|
||||||
|
assert_eq!(EvidenceScore::from_score(0).score(), 0);
|
||||||
|
assert_eq!(EvidenceScore::from_score(73).score(), 73);
|
||||||
|
assert_eq!(EvidenceScore::from_score(255).score(), 100);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn source_id_display_distinguishes_all_source_kinds() {
|
||||||
|
assert_eq!(SourceId::Stdin.to_string(), "<stdin>");
|
||||||
|
assert_eq!(
|
||||||
|
SourceId::File(PathBuf::from("article.md")).to_string(),
|
||||||
|
"article.md"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
SourceId::ArchiveMember {
|
||||||
|
archive: PathBuf::from("bundle.zip"),
|
||||||
|
member: PathBuf::from("docs/article.md"),
|
||||||
|
}
|
||||||
|
.to_string(),
|
||||||
|
"bundle.zip::docs/article.md"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rule_id_has_stable_text_and_ordering() {
|
||||||
|
let first = RuleId::new("alpha-rule");
|
||||||
|
let second = RuleId::from_static("beta-rule");
|
||||||
|
|
||||||
|
assert_eq!(first.as_str(), "alpha-rule");
|
||||||
|
assert_eq!(first.to_string(), "alpha-rule");
|
||||||
|
assert!(first < second);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn analysis_summary_counts_severity_without_removing_findings() {
|
||||||
|
let analysis = Analysis {
|
||||||
|
findings: vec![
|
||||||
|
Finding {
|
||||||
|
rule_id: RuleId::from_static("advice-rule"),
|
||||||
|
category: Category::Style,
|
||||||
|
severity: FindingSeverity::Advice,
|
||||||
|
message: "advice".to_owned(),
|
||||||
|
help: None,
|
||||||
|
byte_span: 0..1,
|
||||||
|
matched_text: "a".to_owned(),
|
||||||
|
weight: 1.0,
|
||||||
|
fix: None,
|
||||||
|
},
|
||||||
|
Finding {
|
||||||
|
rule_id: RuleId::from_static("warning-rule"),
|
||||||
|
category: Category::PromptLeakage,
|
||||||
|
severity: FindingSeverity::Warning,
|
||||||
|
message: "warning".to_owned(),
|
||||||
|
help: None,
|
||||||
|
byte_span: 2..3,
|
||||||
|
matched_text: "b".to_owned(),
|
||||||
|
weight: 10.0,
|
||||||
|
fix: None,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
evidence_score: EvidenceScore::from_score(32),
|
||||||
|
evidence_band: Some(EvidenceBand::Low),
|
||||||
|
};
|
||||||
|
|
||||||
|
let summary = analysis.summary();
|
||||||
|
|
||||||
|
assert_eq!(analysis.findings.len(), 2);
|
||||||
|
assert_eq!(summary.evidence_score, 32);
|
||||||
|
assert_eq!(summary.evidence_band, Some(EvidenceBand::Low));
|
||||||
|
assert_eq!(summary.findings.advice, 1);
|
||||||
|
assert_eq!(summary.findings.warnings, 1);
|
||||||
|
assert_eq!(summary.findings.errors, 0);
|
||||||
|
assert_eq!(summary.findings.total(), analysis.findings.len());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn enum_names_are_stable_for_display_and_serialization() {
|
||||||
|
for (category, expected) in [
|
||||||
|
(Category::Typography, "typography"),
|
||||||
|
(Category::Style, "style"),
|
||||||
|
(Category::PromptLeakage, "prompt-leakage"),
|
||||||
|
(Category::SourceCode, "source-code"),
|
||||||
|
] {
|
||||||
|
assert_eq!(category.as_str(), expected);
|
||||||
|
assert_eq!(category.to_string(), expected);
|
||||||
|
}
|
||||||
|
for (severity, expected) in [
|
||||||
|
(FindingSeverity::Advice, "advice"),
|
||||||
|
(FindingSeverity::Warning, "warning"),
|
||||||
|
(FindingSeverity::Error, "error"),
|
||||||
|
] {
|
||||||
|
assert_eq!(severity.as_str(), expected);
|
||||||
|
assert_eq!(severity.to_string(), expected);
|
||||||
|
}
|
||||||
|
for (band, expected) in [
|
||||||
|
(EvidenceBand::BelowThreshold, "below-threshold"),
|
||||||
|
(EvidenceBand::Low, "low"),
|
||||||
|
(EvidenceBand::Medium, "medium"),
|
||||||
|
(EvidenceBand::High, "high"),
|
||||||
|
] {
|
||||||
|
assert_eq!(band.as_str(), expected);
|
||||||
|
assert_eq!(band.to_string(), expected);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,367 @@
|
||||||
|
//! Pure planning and application of configured text replacements.
|
||||||
|
|
||||||
|
use std::{borrow::Cow, collections::BTreeMap, ops::Range};
|
||||||
|
|
||||||
|
use miette::Diagnostic;
|
||||||
|
|
||||||
|
use crate::{Config, ConfusablePolicy, ReplacementRule, TextEdit};
|
||||||
|
|
||||||
|
/// An edit set cannot be applied safely to its source snapshot.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error, Diagnostic)]
|
||||||
|
#[diagnostic(code(john_slop::replacement::invalid_edit))]
|
||||||
|
pub enum ReplacementError {
|
||||||
|
/// A range is empty, reversed, or extends beyond the source.
|
||||||
|
#[error("replacement byte range {start}..{end} is invalid for a {source_len}-byte source")]
|
||||||
|
InvalidRange {
|
||||||
|
/// Inclusive start byte supplied by the edit.
|
||||||
|
start: usize,
|
||||||
|
/// Exclusive end byte supplied by the edit.
|
||||||
|
end: usize,
|
||||||
|
/// Source snapshot length in bytes.
|
||||||
|
source_len: usize,
|
||||||
|
},
|
||||||
|
/// A range endpoint does not lie on a UTF-8 character boundary.
|
||||||
|
#[error("replacement byte offset {offset} is not a UTF-8 character boundary")]
|
||||||
|
InvalidCharacterBoundary {
|
||||||
|
/// Invalid endpoint.
|
||||||
|
offset: usize,
|
||||||
|
},
|
||||||
|
/// Two edits cover at least one common source byte.
|
||||||
|
#[error("replacement ranges overlap at byte {offset}")]
|
||||||
|
OverlappingEdits {
|
||||||
|
/// Start of the later edit inside an earlier edit.
|
||||||
|
offset: usize,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Plan deterministic non-overlapping edits without modifying the source.
|
||||||
|
pub fn plan(config: &Config, source: &str) -> Vec<TextEdit> {
|
||||||
|
plan_matching(config, source, |rule| {
|
||||||
|
rule.enabled
|
||||||
|
&& (!is_bulk_confusable(rule)
|
||||||
|
|| config.unicode.confusables == ConfusablePolicy::Replace)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn plan_confusables_for_review(config: &Config, source: &str) -> Vec<TextEdit> {
|
||||||
|
if config.unicode.confusables != ConfusablePolicy::Review {
|
||||||
|
return Vec::new();
|
||||||
|
}
|
||||||
|
plan_matching(config, source, |rule| {
|
||||||
|
rule.enabled && is_bulk_confusable(rule)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn is_bulk_confusable(rule: &ReplacementRule) -> bool {
|
||||||
|
let mut characters = rule.source.chars();
|
||||||
|
let is_fullwidth_scalar =
|
||||||
|
matches!(characters.next(), Some('\u{ff01}'..='\u{ff5e}')) && characters.next().is_none();
|
||||||
|
rule.id.starts_with("uts39-") && !is_fullwidth_scalar
|
||||||
|
}
|
||||||
|
|
||||||
|
fn plan_matching(
|
||||||
|
config: &Config,
|
||||||
|
source: &str,
|
||||||
|
include: impl Fn(&ReplacementRule) -> bool,
|
||||||
|
) -> Vec<TextEdit> {
|
||||||
|
let mut single_character_rules = BTreeMap::new();
|
||||||
|
let mut sequence_rules = Vec::new();
|
||||||
|
for rule in config.replacements.iter().filter(|rule| include(rule)) {
|
||||||
|
let mut characters = rule.source.chars();
|
||||||
|
let first = characters.next();
|
||||||
|
if let (Some(character), None) = (first, characters.next()) {
|
||||||
|
single_character_rules.insert(character, rule);
|
||||||
|
} else {
|
||||||
|
sequence_rules.push(rule);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sequence_rules.sort_by(|left, right| {
|
||||||
|
right
|
||||||
|
.source
|
||||||
|
.len()
|
||||||
|
.cmp(&left.source.len())
|
||||||
|
.then_with(|| left.id.cmp(&right.id))
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut edits = Vec::new();
|
||||||
|
let mut offset = 0;
|
||||||
|
while offset < source.len() {
|
||||||
|
if let Some(rule) = sequence_rules
|
||||||
|
.iter()
|
||||||
|
.find(|rule| source[offset..].starts_with(&rule.source))
|
||||||
|
{
|
||||||
|
let end = offset + rule.source.len();
|
||||||
|
edits.push(TextEdit {
|
||||||
|
byte_span: offset..end,
|
||||||
|
replacement: rule.replacement.clone(),
|
||||||
|
});
|
||||||
|
offset = end;
|
||||||
|
} else {
|
||||||
|
let Some(character) = source[offset..].chars().next() else {
|
||||||
|
break;
|
||||||
|
};
|
||||||
|
let end = offset + character.len_utf8();
|
||||||
|
if let Some(rule) = single_character_rules.get(&character) {
|
||||||
|
edits.push(TextEdit {
|
||||||
|
byte_span: offset..end,
|
||||||
|
replacement: rule.replacement.clone(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
offset = end;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
edits
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Validate and apply edits from the end of the source toward the beginning.
|
||||||
|
///
|
||||||
|
/// With no edits, the returned [`Cow`] borrows the original source without an
|
||||||
|
/// allocation. All ranges refer to the original source snapshot.
|
||||||
|
pub fn apply<'source>(
|
||||||
|
source: &'source str,
|
||||||
|
edits: &[TextEdit],
|
||||||
|
) -> Result<Cow<'source, str>, ReplacementError> {
|
||||||
|
if edits.is_empty() {
|
||||||
|
return Ok(Cow::Borrowed(source));
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut ordered = edits.iter().collect::<Vec<_>>();
|
||||||
|
ordered.sort_by(|left, right| range_order(&left.byte_span, &right.byte_span));
|
||||||
|
let mut previous_end = 0;
|
||||||
|
for edit in &ordered {
|
||||||
|
let range = &edit.byte_span;
|
||||||
|
if range.start >= range.end || range.end > source.len() {
|
||||||
|
return Err(ReplacementError::InvalidRange {
|
||||||
|
start: range.start,
|
||||||
|
end: range.end,
|
||||||
|
source_len: source.len(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
for offset in [range.start, range.end] {
|
||||||
|
if !source.is_char_boundary(offset) {
|
||||||
|
return Err(ReplacementError::InvalidCharacterBoundary { offset });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if range.start < previous_end {
|
||||||
|
return Err(ReplacementError::OverlappingEdits {
|
||||||
|
offset: range.start,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
previous_end = range.end;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut output = source.to_owned();
|
||||||
|
for edit in ordered.into_iter().rev() {
|
||||||
|
output.replace_range(edit.byte_span.clone(), &edit.replacement);
|
||||||
|
}
|
||||||
|
Ok(Cow::Owned(output))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn range_order(left: &Range<usize>, right: &Range<usize>) -> std::cmp::Ordering {
|
||||||
|
(left.start, left.end).cmp(&(right.start, right.end))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn plans_only_configured_non_ascii_characters() {
|
||||||
|
let edits = plan(&Config::embedded(), "plain-word—fixed");
|
||||||
|
|
||||||
|
assert_eq!(edits.len(), 1);
|
||||||
|
assert_eq!(edits[0].byte_span, 10..13);
|
||||||
|
assert_eq!(edits[0].replacement, "-");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn plans_multiple_edits_in_source_order_with_utf8_ranges() {
|
||||||
|
let edits = plan(&Config::embedded(), "a—b−c");
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
edits,
|
||||||
|
vec![
|
||||||
|
TextEdit {
|
||||||
|
byte_span: 1..4,
|
||||||
|
replacement: "-".to_owned(),
|
||||||
|
},
|
||||||
|
TextEdit {
|
||||||
|
byte_span: 5..8,
|
||||||
|
replacement: "-".to_owned(),
|
||||||
|
},
|
||||||
|
]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn generated_catalog_replaces_letters_digits_punctuation_and_multi_byte_skeletons() {
|
||||||
|
let source = "а𝐀!𝟘ⅿ🯹";
|
||||||
|
let edits = plan(&Config::embedded(), source);
|
||||||
|
|
||||||
|
assert_eq!(edits.len(), 6);
|
||||||
|
assert_eq!(apply(source, &edits).unwrap(), "aA!Orn9");
|
||||||
|
assert_eq!(edits[4].replacement, "rn");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn generated_catalog_width_folds_every_printable_fullwidth_ascii_character() {
|
||||||
|
let source = std::iter::once('\u{3000}')
|
||||||
|
.chain((b'!'..=b'~').filter_map(|ascii| char::from_u32(u32::from(ascii) + 0xfee0)))
|
||||||
|
.collect::<String>();
|
||||||
|
let expected = (b' '..=b'~').map(char::from).collect::<String>();
|
||||||
|
let edits = plan(&Config::embedded(), &source);
|
||||||
|
|
||||||
|
assert_eq!(edits.len(), 95);
|
||||||
|
assert_eq!(apply(&source, &edits).unwrap(), expected);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bulk_confusable_policy_preserves_curated_replacements() {
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
config.unicode.confusables = ConfusablePolicy::Review;
|
||||||
|
assert_eq!(apply("а—I ", &plan(&config, "а—I ")).unwrap(), "а-I ");
|
||||||
|
assert_eq!(plan_confusables_for_review(&config, "а—I ").len(), 1);
|
||||||
|
|
||||||
|
config.unicode.confusables = ConfusablePolicy::Off;
|
||||||
|
assert_eq!(apply("а—I ", &plan(&config, "а—I ")).unwrap(), "а-I ");
|
||||||
|
assert!(plan_confusables_for_review(&config, "а—I ").is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn empty_or_ascii_sources_need_no_edits() {
|
||||||
|
let config = Config::embedded();
|
||||||
|
|
||||||
|
assert!(plan(&config, "").is_empty());
|
||||||
|
assert!(plan(&config, "ordinary-ascii").is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn longest_source_wins_independently_of_configuration_order() {
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
config.replacements.insert(
|
||||||
|
0,
|
||||||
|
crate::ReplacementRule {
|
||||||
|
id: "double-em-dash".to_owned(),
|
||||||
|
source: "——".to_owned(),
|
||||||
|
replacement: "--".to_owned(),
|
||||||
|
rationale: "Preserve a doubled dash.".to_owned(),
|
||||||
|
enabled: true,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
config.validate().unwrap();
|
||||||
|
|
||||||
|
let edits = plan(&config, "a——b—c");
|
||||||
|
|
||||||
|
assert_eq!(edits.len(), 2);
|
||||||
|
assert_eq!(edits[0].byte_span, 1..7);
|
||||||
|
assert_eq!(edits[0].replacement, "--");
|
||||||
|
assert_eq!(edits[1].byte_span, 8..11);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn applies_edits_in_reverse_and_preserves_line_endings() {
|
||||||
|
let source = "a—b\r\nc−d\n";
|
||||||
|
let edits = plan(&Config::embedded(), source);
|
||||||
|
|
||||||
|
assert_eq!(apply(source, &edits).unwrap(), "a-b\r\nc-d\n");
|
||||||
|
assert_eq!(
|
||||||
|
apply(&apply(source, &edits).unwrap(), &[]).unwrap(),
|
||||||
|
"a-b\r\nc-d\n"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_edits_borrow_the_original_source() {
|
||||||
|
let source = "ordinary ASCII";
|
||||||
|
|
||||||
|
assert!(matches!(apply(source, &[]).unwrap(), Cow::Borrowed(value) if value == source));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn rejects_invalid_boundaries_ranges_and_overlaps() {
|
||||||
|
let invalid_boundary = [TextEdit {
|
||||||
|
byte_span: 1..2,
|
||||||
|
replacement: "-".to_owned(),
|
||||||
|
}];
|
||||||
|
assert!(matches!(
|
||||||
|
apply("é", &invalid_boundary),
|
||||||
|
Err(ReplacementError::InvalidCharacterBoundary { offset: 1 })
|
||||||
|
));
|
||||||
|
|
||||||
|
let invalid_range = [TextEdit {
|
||||||
|
byte_span: 0..3,
|
||||||
|
replacement: "-".to_owned(),
|
||||||
|
}];
|
||||||
|
assert!(matches!(
|
||||||
|
apply("ab", &invalid_range),
|
||||||
|
Err(ReplacementError::InvalidRange { .. })
|
||||||
|
));
|
||||||
|
|
||||||
|
let overlapping = [
|
||||||
|
TextEdit {
|
||||||
|
byte_span: 0..2,
|
||||||
|
replacement: "x".to_owned(),
|
||||||
|
},
|
||||||
|
TextEdit {
|
||||||
|
byte_span: 1..3,
|
||||||
|
replacement: "y".to_owned(),
|
||||||
|
},
|
||||||
|
];
|
||||||
|
assert!(matches!(
|
||||||
|
apply("abc", &overlapping),
|
||||||
|
Err(ReplacementError::OverlappingEdits { offset: 1 })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn generated_sequences_are_stable_non_overlapping_and_idempotent() {
|
||||||
|
let config = Config::embedded();
|
||||||
|
let alphabet = ["a", "é", "—", "−", "\r", "\n"];
|
||||||
|
|
||||||
|
for first in alphabet {
|
||||||
|
for second in alphabet {
|
||||||
|
for third in alphabet {
|
||||||
|
for fourth in alphabet {
|
||||||
|
let source = [first, second, third, fourth].concat();
|
||||||
|
let edits = plan(&config, &source);
|
||||||
|
assert_eq!(edits, plan(&config, &source));
|
||||||
|
for edit in &edits {
|
||||||
|
assert!(source.is_char_boundary(edit.byte_span.start));
|
||||||
|
assert!(source.is_char_boundary(edit.byte_span.end));
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
edits
|
||||||
|
.windows(2)
|
||||||
|
.all(|pair| { pair[0].byte_span.end <= pair[1].byte_span.start })
|
||||||
|
);
|
||||||
|
|
||||||
|
let formatted = apply(&source, &edits).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
source
|
||||||
|
.chars()
|
||||||
|
.filter(|character| *character == '\r')
|
||||||
|
.count(),
|
||||||
|
formatted
|
||||||
|
.chars()
|
||||||
|
.filter(|character| *character == '\r')
|
||||||
|
.count()
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
source
|
||||||
|
.chars()
|
||||||
|
.filter(|character| *character == '\n')
|
||||||
|
.count(),
|
||||||
|
formatted
|
||||||
|
.chars()
|
||||||
|
.filter(|character| *character == '\n')
|
||||||
|
.count()
|
||||||
|
);
|
||||||
|
let second_edits = plan(&config, &formatted);
|
||||||
|
assert!(second_edits.is_empty());
|
||||||
|
assert_eq!(apply(&formatted, &second_edits).unwrap(), formatted);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
//! Pinned invisible and bidirectional-control review profile.
|
||||||
|
|
||||||
|
/// Match the invisible/control profile generated by Gitea at commit 9c715c0.
|
||||||
|
///
|
||||||
|
/// Gitea removes ordinary space, tab, and line feed before generating its
|
||||||
|
/// table. Callers remain responsible for treating CRLF as one ordinary line
|
||||||
|
/// ending rather than a suspicious standalone carriage return.
|
||||||
|
pub(crate) const fn is_gitea_invisible(character: char) -> bool {
|
||||||
|
matches!(
|
||||||
|
character as u32,
|
||||||
|
11..=13
|
||||||
|
| 127
|
||||||
|
| 160
|
||||||
|
| 173
|
||||||
|
| 847
|
||||||
|
| 1564
|
||||||
|
| 4447
|
||||||
|
| 4448
|
||||||
|
| 6068
|
||||||
|
| 6069
|
||||||
|
| 6155..=6158
|
||||||
|
| 7355..=7356
|
||||||
|
| 8192..=8207
|
||||||
|
| 8234..=8239
|
||||||
|
| 8287..=8303
|
||||||
|
| 10240
|
||||||
|
| 12288
|
||||||
|
| 12644
|
||||||
|
| 65024..=65039
|
||||||
|
| 65279
|
||||||
|
| 65440
|
||||||
|
| 65520..=65528
|
||||||
|
| 65532
|
||||||
|
| 78844
|
||||||
|
| 119155..=119162
|
||||||
|
| 917504..=917631
|
||||||
|
| 917760..=917999
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn gitea_profile_covers_invisible_and_bidi_ranges_without_normal_whitespace() {
|
||||||
|
assert!(is_gitea_invisible('\u{200b}'));
|
||||||
|
assert!(is_gitea_invisible('\u{202e}'));
|
||||||
|
assert!(is_gitea_invisible('\u{2066}'));
|
||||||
|
assert!(is_gitea_invisible('\u{e0001}'));
|
||||||
|
assert!(!is_gitea_invisible(' '));
|
||||||
|
assert!(!is_gitea_invisible('\t'));
|
||||||
|
assert!(!is_gitea_invisible('\n'));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
[package]
|
||||||
|
name = "john-slop-lsp"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
rust-version.workspace = true
|
||||||
|
license.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
description = "Language Server Protocol engine for John Slop"
|
||||||
|
|
||||||
|
[lib]
|
||||||
|
name = "john_slop_lsp_engine"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
john-slop-core.workspace = true
|
||||||
|
serde_json.workspace = true
|
||||||
|
thiserror.workspace = true
|
||||||
|
tokio = { workspace = true, features = ["sync"] }
|
||||||
|
tower-lsp-server.workspace = true
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,32 @@
|
||||||
|
[graph]
|
||||||
|
all-features = true
|
||||||
|
|
||||||
|
[advisories]
|
||||||
|
version = 2
|
||||||
|
yanked = "deny"
|
||||||
|
|
||||||
|
[licenses]
|
||||||
|
version = 2
|
||||||
|
confidence-threshold = 0.8
|
||||||
|
allow = [
|
||||||
|
"0BSD",
|
||||||
|
"Apache-2.0",
|
||||||
|
"Apache-2.0 WITH LLVM-exception",
|
||||||
|
"ISC",
|
||||||
|
"MIT",
|
||||||
|
"MIT-0",
|
||||||
|
"MPL-2.0",
|
||||||
|
"Unicode-3.0",
|
||||||
|
"Unlicense",
|
||||||
|
"Zlib",
|
||||||
|
]
|
||||||
|
|
||||||
|
[bans]
|
||||||
|
multiple-versions = "warn"
|
||||||
|
wildcards = "deny"
|
||||||
|
highlight = "all"
|
||||||
|
|
||||||
|
[sources]
|
||||||
|
unknown-registry = "deny"
|
||||||
|
unknown-git = "deny"
|
||||||
|
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
|
||||||
|
|
@ -0,0 +1,39 @@
|
||||||
|
# John Slop documentation
|
||||||
|
|
||||||
|
The root [README](../README.md) gets you from installation to a first scan. Move
|
||||||
|
through these guides as your needs grow; you do not need to understand the
|
||||||
|
internals to use the tools.
|
||||||
|
|
||||||
|
## Start using it
|
||||||
|
|
||||||
|
1. [Usage](usage.md) covers accepted inputs, detector options, stable exit
|
||||||
|
statuses, formatter modes, and the LSP command.
|
||||||
|
2. [Output](output.md) explains graphical Miette reports, plain human output,
|
||||||
|
concise records, versioned JSON Lines, and partial input errors.
|
||||||
|
3. [Configuration](configuration.md) documents source precedence, named
|
||||||
|
profiles, stable-ID merges, validation, and inspection commands. Start from
|
||||||
|
the [commented examples](../examples/config/).
|
||||||
|
|
||||||
|
## Put it into practice
|
||||||
|
|
||||||
|
4. [Runnable examples](../examples/README.md) demonstrate in-memory analysis,
|
||||||
|
streaming path input, shared replacement planning, and an expected-failure
|
||||||
|
diagnostic showcase.
|
||||||
|
5. [Editor setup](editors.md) provides working Helix (`hx`) and Vim formatter
|
||||||
|
and LSP configurations with validation steps.
|
||||||
|
6. [Review signals and evidence methodology](review-signals.md) explains the
|
||||||
|
research boundary, evidence tiers, and deliberately excluded proxies.
|
||||||
|
7. [Heuristics and evidence scoring](heuristics.md) explains what each rule
|
||||||
|
contributes and how the bounded score is calculated.
|
||||||
|
8. [Limitations](limitations.md) covers false positives, evasion, domain shift,
|
||||||
|
score semantics, and safe interpretation.
|
||||||
|
|
||||||
|
## Go deeper
|
||||||
|
|
||||||
|
The [advanced guide](advanced.md) is the single entry point for architecture,
|
||||||
|
the public library API, evaluation, benchmarks, dependencies, design records,
|
||||||
|
research references, development, security, and releases.
|
||||||
|
|
||||||
|
Project work is tracked in [tasks.md](../tasks.md), while contribution policy is
|
||||||
|
defined in [CONTRIBUTING.md](../CONTRIBUTING.md) and
|
||||||
|
[AGENTS.md](../AGENTS.md).
|
||||||
|
|
@ -0,0 +1,58 @@
|
||||||
|
# Advanced concepts and project internals
|
||||||
|
|
||||||
|
This guide collects material that is useful after the normal CLI,
|
||||||
|
configuration, output, and editor workflows are familiar. The root
|
||||||
|
[README](../README.md) and [documentation path](README.md) remain the best entry
|
||||||
|
points for day-to-day use.
|
||||||
|
|
||||||
|
## Detection model and evaluation
|
||||||
|
|
||||||
|
- [Heuristics and evidence scoring](heuristics.md) describes applicability,
|
||||||
|
occurrence damping, normalization, and why the score is not authorship
|
||||||
|
confidence.
|
||||||
|
- [Review signals and evidence methodology](review-signals.md) records the
|
||||||
|
research basis, evidence tiers, and signals deliberately kept out.
|
||||||
|
- [Evaluation](evaluation.md) records corpus provenance, per-rule measurements,
|
||||||
|
aggregate false positives, and the limits of the current metrics.
|
||||||
|
- [Limitations](limitations.md) covers evasion, domain shift, interpretation,
|
||||||
|
and operational boundaries.
|
||||||
|
|
||||||
|
## Architecture and embedding
|
||||||
|
|
||||||
|
- [Architecture and public API](architecture.md) describes the Cargo workspace,
|
||||||
|
module boundaries, shared domain values, streaming input, and reusable facade.
|
||||||
|
- [Output contracts](output.md) specifies human presentation and the versioned
|
||||||
|
JSON Lines schema for downstream integrations.
|
||||||
|
- [Decision records](decisions/) preserve the reasoning behind core contracts,
|
||||||
|
input limits, the LSP protocol, and the supported workspace.
|
||||||
|
- [Crate-boundary experiment](experiments/crate-boundary.md) contains the
|
||||||
|
measurements that preceded the workspace decision.
|
||||||
|
|
||||||
|
## Performance, dependencies, and safety
|
||||||
|
|
||||||
|
- [Performance benchmarks](benchmarks.md) defines representative workloads and
|
||||||
|
the evidence required before adding parallel execution.
|
||||||
|
- [Dependency and binary review](dependencies.md) records feature choices,
|
||||||
|
advisory and license policy, unsafe boundaries, and release-size measurements.
|
||||||
|
- [Security policy](../SECURITY.md) explains supported versions and private
|
||||||
|
vulnerability reporting.
|
||||||
|
|
||||||
|
## Research references
|
||||||
|
|
||||||
|
- [Background](background.md) collects external context on observable signs of
|
||||||
|
low-care AI-assisted writing.
|
||||||
|
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
|
||||||
|
Gitea's locale-aware catalog, its upstream source, and safe adoption
|
||||||
|
boundaries.
|
||||||
|
|
||||||
|
## Development and release engineering
|
||||||
|
|
||||||
|
- [Development](development.md) covers the Rust toolchain, feature matrix,
|
||||||
|
tests, snapshots, coverage, badges, and CI gates.
|
||||||
|
- [Contributing](../CONTRIBUTING.md) explains the patch workflow and review
|
||||||
|
expectations; [AGENTS.md](../AGENTS.md) defines repository-specific rules for
|
||||||
|
coding agents.
|
||||||
|
- [Release procedure](releasing.md) defines tagged reproducible packaging,
|
||||||
|
checksums, and provenance metadata.
|
||||||
|
- [Implementation plan](../tasks.md) records completed milestones and deferred
|
||||||
|
product work.
|
||||||
|
|
@ -0,0 +1,109 @@
|
||||||
|
# Architecture
|
||||||
|
|
||||||
|
John Slop is a workspace with a compatibility facade and two reusable libraries.
|
||||||
|
Executables remain thin adapters responsible for argument parsing, presentation,
|
||||||
|
and process exit statuses.
|
||||||
|
|
||||||
|
The current flow is:
|
||||||
|
|
||||||
|
```text
|
||||||
|
path -> input stream -> Document -> analysis -> Analysis + Finding
|
||||||
|
-> diagnostic/output adapters
|
||||||
|
-> replacement planner -> TextEdit -> applier
|
||||||
|
-> formatter I/O
|
||||||
|
-> LSP UTF-16 edit
|
||||||
|
```
|
||||||
|
|
||||||
|
The crate boundary separates dependency families:
|
||||||
|
|
||||||
|
- `john-slop-core` owns configuration, input, detection, diagnostics,
|
||||||
|
replacement planning, and presentation-neutral domain values;
|
||||||
|
- `john-slop-lsp` owns asynchronous editor state and protocol conversion while
|
||||||
|
depending only on `john-slop-core`;
|
||||||
|
- the root `john-slop` package re-exports both libraries to preserve its public
|
||||||
|
API and distributes the detector, formatter, and LSP binaries.
|
||||||
|
|
||||||
|
The root manifest is intentionally both a package and the workspace root. A
|
||||||
|
virtual manifest would relocate the compatibility facade, binaries, examples,
|
||||||
|
and fixtures without improving dependency isolation. `[workspace.package]`,
|
||||||
|
`[workspace.dependencies]`, and `[workspace.lints]` provide one source of truth
|
||||||
|
for shared release metadata, dependency requirements, and first-party unsafe
|
||||||
|
code policy. Member manifests retain only package-specific descriptions,
|
||||||
|
dependency selections, feature activation, and optionality. In particular, the
|
||||||
|
workspace Tokio requirement enables no features: the LSP library selects
|
||||||
|
`sync`, and the root LSP adapter selects its runtime and stdio features.
|
||||||
|
|
||||||
|
The modules within those crates retain deliberately narrow responsibilities:
|
||||||
|
|
||||||
|
- `config` owns validated embedded defaults, restricted runtime diagnostic
|
||||||
|
catalogs, the generated Unicode policy, and input limits;
|
||||||
|
- `input` streams files, directory entries, and archive members as documents;
|
||||||
|
- `detector` compiles configured matchers and path scopes once, then produces
|
||||||
|
explainable findings and a bounded low-care evidence score;
|
||||||
|
- `diagnostic` binds findings to exact named snapshots as typed `miette`
|
||||||
|
diagnostics;
|
||||||
|
- `replacement` produces non-overlapping edits and validates/applies arbitrary
|
||||||
|
edit sets without performing I/O;
|
||||||
|
- `unicode_review` pins the UTS #39 ASCII-skeleton version and Gitea-compatible
|
||||||
|
invisible/control ranges; generated printable-ASCII mappings feed the shared
|
||||||
|
replacement planner while controls remain non-mutating;
|
||||||
|
- `lsp` indexes open snapshots, runs the compiled detector, converts canonical
|
||||||
|
finding and replacement spans to UTF-16 coordinates, and implements
|
||||||
|
versioned quick fixes plus whole-document formatting;
|
||||||
|
- `model` contains presentation-neutral domain values;
|
||||||
|
- `error` exposes typed input failures for library consumers;
|
||||||
|
- `src/bin` contains the detector presentation/exit policy and formatter/LSP
|
||||||
|
adapters.
|
||||||
|
|
||||||
|
All source spans are zero-based UTF-8 byte ranges into the exact analyzed source
|
||||||
|
snapshot. Presentation layers will convert those canonical spans to terminal or
|
||||||
|
UTF-16 LSP coordinates at their boundaries. Archive sources retain both their
|
||||||
|
outer path and inner member identity. Rule applicability uses the member path
|
||||||
|
so language and path filters behave the same inside and outside archives.
|
||||||
|
|
||||||
|
## Public facade
|
||||||
|
|
||||||
|
The root compatibility facade and `john-slop-core` expose the same core entry
|
||||||
|
points and presentation-neutral types:
|
||||||
|
|
||||||
|
- `load_config` resolves the configuration embedded in the current build;
|
||||||
|
- `read_inputs` returns a streaming iterator of per-document results;
|
||||||
|
- `read_stdin` returns one bounded strict-UTF-8 standard-input document;
|
||||||
|
- `analyze_source` returns findings and an evidence score for UTF-8 text;
|
||||||
|
- `plan_replacements` returns safe edits against the same source snapshot;
|
||||||
|
- `apply_replacements` validates boundaries and overlap, then applies edits in
|
||||||
|
descending byte order while borrowing no-op input unchanged.
|
||||||
|
|
||||||
|
`FindingDiagnostic` is the library bridge to `miette`; CLI-specific concise and
|
||||||
|
JSON rendering stays in the detector adapter so it cannot alter analysis.
|
||||||
|
|
||||||
|
`ConfigLoader` applies embedded catalogs, runtime diagnostic catalogs, user
|
||||||
|
base, named user, explicit file, and CLI layers in order. Stable-ID merging and
|
||||||
|
semantic validation happen before a configuration reaches analysis. Keeping
|
||||||
|
these operations behind the facade avoids duplicating policy across the three
|
||||||
|
binaries. Runtime catalogs are declarative: they can add rules using existing
|
||||||
|
matcher implementations but cannot load executable code.
|
||||||
|
|
||||||
|
The formatter adapter owns filesystem mutation. It resolves configuration and
|
||||||
|
preflights every requested document before writing, then uses same-directory
|
||||||
|
temporary files and preserves permissions. Archive mutation remains outside the
|
||||||
|
library and adapter contract.
|
||||||
|
|
||||||
|
The LSP adapter owns no filesystem mutation. `tower-lsp-server` supplies stdio
|
||||||
|
JSON-RPC framing while `LspBackend` tracks authoritative open-document versions
|
||||||
|
behind asynchronous locks. Every publication is checked against the current
|
||||||
|
version after analysis so a replaced snapshot cannot overwrite newer results.
|
||||||
|
Configuration reload swaps only a fully validated model and republishes open
|
||||||
|
documents in deterministic URI order. The detector is compiled once per
|
||||||
|
resolved configuration rather than once per edit. File URI paths drive normal
|
||||||
|
scope applicability; language IDs supply source/prose context for untitled or
|
||||||
|
extensionless buffers. The default publication filter retains only Unicode
|
||||||
|
findings; explicit all-diagnostic and minimum-weight modes expose broader
|
||||||
|
findings, while only findings with core `TextEdit`s become code actions. Document formatting calls the same
|
||||||
|
replacement planner directly and never derives mutations from diagnostics. See
|
||||||
|
[ADR 0003](decisions/0003-lsp-framework-and-protocol.md) for framework and
|
||||||
|
protocol choices.
|
||||||
|
|
||||||
|
See [ADR 0001](decisions/0001-core-analysis-contracts.md) for the foundational
|
||||||
|
contracts and [ADR 0002](decisions/0002-input-policy-and-limits.md) for traversal
|
||||||
|
and archive safety decisions.
|
||||||
|
|
@ -0,0 +1,14 @@
|
||||||
|
# Background reading
|
||||||
|
|
||||||
|
John Slop focuses on observable low-care signals and avoids making unsupported
|
||||||
|
authorship claims. These resources provide context for that product boundary and
|
||||||
|
the kinds of review problems the project aims to help surface:
|
||||||
|
|
||||||
|
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
|
||||||
|
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
|
||||||
|
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
|
||||||
|
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
|
||||||
|
|
||||||
|
Project-specific decisions take precedence over external guidance. See
|
||||||
|
[ADR 0001](decisions/0001-core-analysis-contracts.md) for evidence scoring and
|
||||||
|
authorship semantics.
|
||||||
|
|
@ -0,0 +1,41 @@
|
||||||
|
# Performance benchmarks
|
||||||
|
|
||||||
|
Criterion benchmarks cover the scaling boundaries most likely to change
|
||||||
|
architectural decisions:
|
||||||
|
|
||||||
|
- analysis of an approximately 1 MiB clean prose document;
|
||||||
|
- streaming and decoding 250 small files in deterministic path order;
|
||||||
|
- compiling and analyzing with 110 rules (10× the embedded catalog); and
|
||||||
|
- opening, validating, and reading a 100-member stored ZIP archive.
|
||||||
|
|
||||||
|
Run them on an otherwise idle machine:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo bench --locked --bench core
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `-- --quick` for a smoke measurement. Criterion output under `target/` is
|
||||||
|
not a portable performance artifact: record the CPU, OS, Rust version, commit,
|
||||||
|
and exact command when comparing results.
|
||||||
|
|
||||||
|
The Milestone 7 smoke baseline was measured on 2026-09-29 with
|
||||||
|
`cargo bench --locked --bench core -- --quick`, Rust 1.100.0-nightly, Linux
|
||||||
|
7.2.7, and an AMD Ryzen 9 9950X3D:
|
||||||
|
|
||||||
|
| Workload | Criterion estimate |
|
||||||
|
| --- | ---: |
|
||||||
|
| approximately 1 MiB document | 15.686 ms / 65.36 MiB/s |
|
||||||
|
| 250 small files | 286.54 µs / 872.48 Kfiles/s |
|
||||||
|
| build and analyze 110 rules | 9.474 ms |
|
||||||
|
| 100-member stored archive | 116.37 µs / 859.34 Kmembers/s |
|
||||||
|
|
||||||
|
These quick-mode figures are orientation data, not a cross-machine performance
|
||||||
|
guarantee. A release decision should use Criterion's normal sampling mode.
|
||||||
|
|
||||||
|
Milestone 7 retains the ordered single-threaded implementation. The baseline
|
||||||
|
workloads complete in milliseconds on the development machine, and directory
|
||||||
|
and archive results already stream rather than accumulating the entire input
|
||||||
|
tree before analysis. Parallel traversal would add scheduling and output-order
|
||||||
|
coordination without an established user-visible bottleneck. Reconsider it only
|
||||||
|
with a representative workload, a reproducible before/after benchmark, and
|
||||||
|
tests proving byte-for-byte output order remains unchanged.
|
||||||
|
|
@ -0,0 +1,213 @@
|
||||||
|
# Configuration
|
||||||
|
|
||||||
|
John Slop embeds schema-versioned base, heuristic, and Unicode catalogs in every
|
||||||
|
binary. Installed binaries therefore work without a source checkout or external
|
||||||
|
default files.
|
||||||
|
|
||||||
|
## Precedence
|
||||||
|
|
||||||
|
Configuration layers are applied from lowest to highest priority:
|
||||||
|
|
||||||
|
1. embedded `assets/config.toml` and `assets/heuristics.toml`;
|
||||||
|
2. the optional platform user diagnostic catalog `diagnostics.toml`, followed
|
||||||
|
by explicitly selected diagnostic catalogs in declaration order;
|
||||||
|
3. the platform user file `config.toml`;
|
||||||
|
4. a selected user `<config_name>.toml`;
|
||||||
|
5. an explicit `--config <FILE>`;
|
||||||
|
6. configuration flags for the current invocation.
|
||||||
|
|
||||||
|
Diagnostic catalogs load before normal configuration so a base, named, or
|
||||||
|
explicit configuration can tune or disable a catalog rule by stable ID. A
|
||||||
|
catalog accepts only `schema_version` and `[[rules]]`; unrelated settings are a
|
||||||
|
configuration error rather than being silently applied from the wrong file.
|
||||||
|
|
||||||
|
On Linux, the user directory conventionally resolves to
|
||||||
|
`~/.config/john-slop`. John Slop uses operating-system APIs rather than reading
|
||||||
|
`HOME` directly, so Windows and macOS receive their native application config
|
||||||
|
directories. Run this to inspect the exact paths on the current machine:
|
||||||
|
|
||||||
|
The RC3 naming migration changes the former `john_slop` application directory
|
||||||
|
to `john-slop`. Move existing `config.toml`, named configurations, and
|
||||||
|
`diagnostics.toml` into the new directory before upgrading; the old directory
|
||||||
|
is no longer loaded automatically.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config paths
|
||||||
|
```
|
||||||
|
|
||||||
|
Named configurations accept only ASCII letters, digits, `_`, and `-`. Do not
|
||||||
|
include a directory, dot component, or `.toml` extension. Missing user and named
|
||||||
|
files are optional; a selected explicit file must exist.
|
||||||
|
|
||||||
|
## Generate a project configuration
|
||||||
|
|
||||||
|
Create the embedded, commented base configuration as `./config.toml`:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config init
|
||||||
|
```
|
||||||
|
|
||||||
|
The command refuses to overwrite an existing file. Use
|
||||||
|
`john-slop config init --force` only when replacing that file is intentional.
|
||||||
|
`config generate` is an alias for `config init`. Load a project-local file with
|
||||||
|
`john-slop --config config.toml path/to/input` or inspect it with the command
|
||||||
|
shown below.
|
||||||
|
|
||||||
|
Generate a dedicated runtime diagnostic catalog separately:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config init-diagnostics
|
||||||
|
```
|
||||||
|
|
||||||
|
Place the result at the `user_diagnostics` path printed by `config paths` for
|
||||||
|
automatic loading, or select one or more project/team catalogs explicitly:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop --diagnostics team.toml --diagnostics project.toml article.md
|
||||||
|
john-slop-lsp --diagnostics project.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicit catalogs are applied in command-line order. All catalogs load before
|
||||||
|
`config.toml`, named configuration, and `--config`, allowing those normal
|
||||||
|
layers to override their rules.
|
||||||
|
|
||||||
|
## Schema
|
||||||
|
|
||||||
|
The current `schema_version` is `1`. The complete defaults live in
|
||||||
|
[`config.toml`](../crates/john-slop-core/assets/config.toml),
|
||||||
|
[`heuristics.toml`](../crates/john-slop-core/assets/heuristics.toml), and the
|
||||||
|
generated [`unicode.toml`](../crates/john-slop-core/assets/unicode.toml).
|
||||||
|
Ready-to-copy, commented partial layers live in
|
||||||
|
[`examples/config`](../examples/config/): `config.toml` demonstrates a user-wide
|
||||||
|
base and `editorial.toml` demonstrates a named overlay.
|
||||||
|
|
||||||
|
Replacement and heuristic arrays merge by stable `id`. A partial record changes
|
||||||
|
only the supplied fields:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[diagnostics.categories]
|
||||||
|
# Disable every style diagnostic as one family.
|
||||||
|
style = false
|
||||||
|
|
||||||
|
[[replacements]]
|
||||||
|
id = "em-dash"
|
||||||
|
enabled = false
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "templated-conclusion"
|
||||||
|
# An individual rule switch is independent of the category switch.
|
||||||
|
enabled = false
|
||||||
|
```
|
||||||
|
|
||||||
|
The category keys are `typography`, `style`, `prompt-leakage`, and
|
||||||
|
`source-code`. A diagnostic runs only when its category switch and its
|
||||||
|
individual `[[rules]]` `enabled` field are both `true`. Consequently, enabling
|
||||||
|
a category does not silently re-enable a rule that another layer disabled
|
||||||
|
individually; set that rule's `enabled = true` explicitly when that is intended.
|
||||||
|
Later files override earlier category values using the normal precedence above.
|
||||||
|
|
||||||
|
A new ID must provide every field. Scope arrays replace the prior array as a
|
||||||
|
whole. A rule scope can select `prose` or `source-code`, restrict lowercase
|
||||||
|
extensions, and use `include`/`exclude` path globs. The `skip_generated`,
|
||||||
|
`skip_vendor`, and `skip_minified` switches default to `true`; set one to
|
||||||
|
`false` only when those paths are intentionally in scope. Archive rules match
|
||||||
|
the inner member path. Unknown keys, malformed TOML, duplicate IDs within one
|
||||||
|
layer, invalid regular expressions or globs, unsafe matchers, non-ASCII
|
||||||
|
replacement targets, and invalid thresholds are fatal configuration
|
||||||
|
diagnostics.
|
||||||
|
|
||||||
|
The built-in matcher names are `exact-phrase`, `regex`, `repeated-regex`,
|
||||||
|
`replacement`, and `invisible`. The last two use validated sentinel patterns
|
||||||
|
because their implementations are data-backed rather than free-form strings:
|
||||||
|
`configured-replacements` and `gitea-invisible-9c715c0`. They remain ordinary
|
||||||
|
rule records, so later layers
|
||||||
|
can disable them or adjust severity, weight, occurrence cap, and scope without
|
||||||
|
changing code. Invisible findings never authorize edits; generated confusable
|
||||||
|
mappings follow the bulk policy below.
|
||||||
|
|
||||||
|
## Runtime diagnostic catalog
|
||||||
|
|
||||||
|
`diagnostics.toml` supports new data-driven diagnostics without recompilation.
|
||||||
|
Every new rule must provide the complete rule shape:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
schema_version = 1
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "custom-review-marker"
|
||||||
|
category = "style"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "REVIEW BEFORE PUBLISHING"
|
||||||
|
weight = 8.0
|
||||||
|
message = "a project review marker remains"
|
||||||
|
help = "Resolve or remove the marker before publishing."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
scope = { content = ["prose", "source-code"], extensions = [] }
|
||||||
|
```
|
||||||
|
|
||||||
|
Runtime rules may select `exact-phrase`, `regex`, or `repeated-regex`.
|
||||||
|
`replacement` and `invisible` are code-backed catalog sentinels and cannot
|
||||||
|
define new matching behavior. Categories remain the four stable values listed
|
||||||
|
above. Unknown fields, invalid or empty-match regular expressions, duplicate
|
||||||
|
IDs within one catalog, incomplete new rules, and missing explicit catalogs are
|
||||||
|
fatal. These constraints keep startup deterministic and prevent a configuration
|
||||||
|
file from executing code.
|
||||||
|
|
||||||
|
The generated UTS #39 mappings have one bulk policy:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[unicode]
|
||||||
|
confusables = "review" # replace (default), review, or off
|
||||||
|
```
|
||||||
|
|
||||||
|
`replace` enables diagnostics and edits, `review` retains diagnostics without
|
||||||
|
formatter/LSP edits, and `off` suppresses generated lookalikes. The reviewed
|
||||||
|
punctuation and width-folding replacements and invisible/control diagnostics
|
||||||
|
are unaffected.
|
||||||
|
|
||||||
|
Set `enabled = false` on a known rule or replacement ID to disable it. Because
|
||||||
|
records merge by ID, do not copy the full embedded table merely to change one
|
||||||
|
field; a later partial record preserves every unspecified value.
|
||||||
|
|
||||||
|
Replacement `source` values are non-empty Unicode strings containing at least
|
||||||
|
one non-ASCII byte; `replacement` values are non-empty ASCII strings. Sources
|
||||||
|
must be unique. When enabled sources share a prefix, the longest matching source
|
||||||
|
wins at each byte offset, with stable rule ID ordering as the deterministic
|
||||||
|
tie-break. Edits always refer to the original UTF-8 snapshot.
|
||||||
|
|
||||||
|
Generated confusable IDs use `uts39-u<lowercase-codepoint>`. New fullwidth
|
||||||
|
mappings use `fullwidth-u<lowercase-codepoint>`; mappings shipped previously
|
||||||
|
retain their `uts39-*` or `fullwidth-hyphen-minus` IDs for compatibility, while
|
||||||
|
ideographic space uses `ideographic-space`. Every entry can be disabled or
|
||||||
|
overridden through normal merge-by-ID behavior.
|
||||||
|
Maintain the embedded list by editing
|
||||||
|
`crates/john-slop-core/examples/generate_unicode_catalog.rs` and running:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run -p john-slop-core --example generate_unicode_catalog \
|
||||||
|
> crates/john-slop-core/assets/unicode.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
The exhaustive catalog test walks every Unicode scalar and rejects drift from
|
||||||
|
the pinned Unicode 16.0 tables.
|
||||||
|
|
||||||
|
Each heuristic also configures `severity` as `advice`, `warning`, or `error`.
|
||||||
|
Severity controls CLI presentation, LSP severity, and summary counts; the
|
||||||
|
evidence score still comes from the independently configured weight and
|
||||||
|
aggregation formula. In the LSP, advice is information while warnings and errors
|
||||||
|
retain their corresponding levels.
|
||||||
|
|
||||||
|
## Inspection
|
||||||
|
|
||||||
|
Print the fully layered and validated model as deterministic TOML:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config show
|
||||||
|
john-slop config show --config project.toml
|
||||||
|
john-slop config show --config-name strict
|
||||||
|
```
|
||||||
|
|
||||||
|
The output contains no secrets in schema version 1. If secret-bearing fields are
|
||||||
|
introduced later, inspection output must redact them by default.
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
# ADR 0001: Core analysis contracts
|
||||||
|
|
||||||
|
- Status: accepted
|
||||||
|
- Date: 2026-09-29
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
John Slop needs stable semantics before its proof-of-concept modules are split
|
||||||
|
into a library, CLI, formatter, and language server. In particular, consumers
|
||||||
|
need to know what the aggregate evidence score means, how source locations are represented,
|
||||||
|
whether formatting is destructive by default, and whether archive members can
|
||||||
|
be rewritten.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### The low-care evidence score represents evidence, not authorship
|
||||||
|
|
||||||
|
The detector reports the bounded strength of configured, observable low-care
|
||||||
|
signals. It does not report the probability that content is AI-generated and
|
||||||
|
must not present a score as an authorship determination. Every score must be
|
||||||
|
decomposable into source-labelled findings from stable rule IDs.
|
||||||
|
|
||||||
|
### UTF-8 byte spans are canonical
|
||||||
|
|
||||||
|
Internal source spans are zero-based, half-open UTF-8 byte ranges into the exact
|
||||||
|
source snapshot analyzed. This matches Rust string and regex offsets and avoids
|
||||||
|
lossy conversions inside the detection engine. Presentation adapters validate
|
||||||
|
boundaries and convert to their required representation: `miette::SourceSpan`
|
||||||
|
for terminal diagnostics and UTF-16 line/character positions for LSP.
|
||||||
|
|
||||||
|
### Formatter writes are opt-in
|
||||||
|
|
||||||
|
The formatter's default file behavior is read-only. It may print transformed
|
||||||
|
text or a diff, while an explicit `--write` is required to mutate files.
|
||||||
|
`--check` only reports whether an edit would be made. A failed analysis,
|
||||||
|
configuration load, or edit validation must leave the original file untouched.
|
||||||
|
|
||||||
|
### Archive mutation is deferred
|
||||||
|
|
||||||
|
The first formatter release may inspect text members in supported ZIP-based
|
||||||
|
containers, but it will not rewrite archives, DOCX, XLSX, or PPTX files. In-place
|
||||||
|
archive editing has additional integrity, metadata-preservation, atomicity, and
|
||||||
|
decompression-safety concerns. It requires a separate decision record and test
|
||||||
|
plan before implementation.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- User-facing terminology must describe evidence and request review rather than
|
||||||
|
accuse an author.
|
||||||
|
- The domain model stores byte ranges, while the LSP owns a tested UTF-16 mapping
|
||||||
|
layer tied to document versions.
|
||||||
|
- Replacement planning remains a pure operation; filesystem mutation lives in
|
||||||
|
an explicit formatter adapter.
|
||||||
|
- Archive inputs can produce findings but an attempted archive write produces a
|
||||||
|
clear unsupported-operation diagnostic.
|
||||||
|
|
@ -0,0 +1,65 @@
|
||||||
|
# ADR 0002: Input policy and resource limits
|
||||||
|
|
||||||
|
- Status: accepted
|
||||||
|
- Date: 2026-09-29
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The detector accepts individual files, directory trees, and ZIP-based
|
||||||
|
containers. Unbounded traversal or decompression can exhaust memory, CPU, file
|
||||||
|
descriptors, or disk. Extension-only detection can also misclassify binary files
|
||||||
|
as text. The input layer needs deterministic defaults before it is redesigned as
|
||||||
|
a streaming API.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Limits apply per invocation and can be lowered by configuration or CLI flags.
|
||||||
|
Raising them requires an explicit option; there is no unlimited sentinel.
|
||||||
|
|
||||||
|
| Resource | Default limit |
|
||||||
|
| --- | ---: |
|
||||||
|
| Plain file or archive member | 8 MiB |
|
||||||
|
| Input archive file | 64 MiB |
|
||||||
|
| Total expanded archive bytes | 256 MiB |
|
||||||
|
| Archive entries inspected | 10,000 |
|
||||||
|
| Expanded/compressed ratio per member | 100:1 |
|
||||||
|
| Directory traversal depth | 64 levels |
|
||||||
|
| Nested archive traversal | disabled (depth 0) |
|
||||||
|
|
||||||
|
An exact file passed by the user is considered regardless of extension. During
|
||||||
|
directory traversal, configurable extension allowlists select likely prose,
|
||||||
|
markup, configuration, and source-code files. Content still has to decode as
|
||||||
|
strict UTF-8 and must not contain a NUL byte. Unsupported/binary content produces
|
||||||
|
a skipped-input result rather than a clean analysis or a lossy conversion.
|
||||||
|
|
||||||
|
ZIP-based containers are recognized from both a supported extension (`zip`,
|
||||||
|
`docx`, `xlsx`, or `pptx`) and ZIP magic bytes. An extension/magic mismatch is a
|
||||||
|
typed input error. Members are streamed and counted before allocation. Directory
|
||||||
|
members and encrypted members are skipped with a reason. Absolute paths,
|
||||||
|
platform prefixes, and any path containing a parent (`..`) component are
|
||||||
|
rejected. Archive member names are identities only; members are never extracted
|
||||||
|
to the filesystem by the analysis path.
|
||||||
|
|
||||||
|
Directory walking does not follow symbolic links. Hidden paths are skipped by
|
||||||
|
default. The default ignore set includes `.git`, `.hg`, `.svn`, `target`,
|
||||||
|
`node_modules`, `vendor`, common generated directories, source maps, and minified
|
||||||
|
assets. Repository ignore files are respected when the eventual walker supports
|
||||||
|
them. An explicitly named file can bypass path/extension ignores, but cannot
|
||||||
|
bypass content or resource-safety limits.
|
||||||
|
|
||||||
|
One unreadable, invalid, or oversized path yields a per-document input error and
|
||||||
|
does not discard successful documents from the same directory or archive.
|
||||||
|
Results remain ordered by normalized source identity even if reading is later
|
||||||
|
parallelized.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The input API must stream per-document results and track cumulative archive
|
||||||
|
counters.
|
||||||
|
- `Document` needs distinct outer-path and optional archive-member identities.
|
||||||
|
- Size checks must use checked arithmetic and occur before allocating from
|
||||||
|
metadata supplied by an archive.
|
||||||
|
- Tests need boundary cases immediately below, at, and above each configurable
|
||||||
|
limit, plus malicious archive names and misleading extensions.
|
||||||
|
- The README and CLI help must distinguish skipped inputs, operational errors,
|
||||||
|
and clean analyzed documents.
|
||||||
|
|
@ -0,0 +1,101 @@
|
||||||
|
# ADR 0003: LSP framework and protocol contract
|
||||||
|
|
||||||
|
- Status: accepted, amended by Milestone 19
|
||||||
|
- Date: 2026-09-29
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
John Slop needs a small stdio language server that reuses the validated
|
||||||
|
replacement configuration and exact `TextEdit`s from the library. Implementing
|
||||||
|
JSON-RPC framing and the Language Server Protocol state machine locally would
|
||||||
|
add protocol risk without improving detection. The original `tower-lsp` crate's
|
||||||
|
latest release is 0.20.0, while its maintained community continuation publishes
|
||||||
|
the same Tower-oriented API as `tower-lsp-server`.
|
||||||
|
|
||||||
|
The dependency review compared the crates' published metadata, documentation,
|
||||||
|
runtime features, and minimum Rust versions. `tower-lsp-server` 0.23.0 is the
|
||||||
|
latest stable release; 0.24.0-rc.1 is a prerelease. Version 0.23.0 requires Rust
|
||||||
|
1.85, below this project's Rust 1.98.1 baseline, and provides the Tokio stdio
|
||||||
|
transport, generated LSP types, client notifications, and in-process service
|
||||||
|
interface needed by the milestone. Its MIT OR Apache-2.0 licensing is compatible
|
||||||
|
with this project.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Use `tower-lsp-server` 0.23 and its Tokio runtime feature. Use Tokio directly
|
||||||
|
only for the stdio runtime and synchronized in-memory document state. Do not
|
||||||
|
enable proposed LSP extensions or an alternate URI implementation.
|
||||||
|
|
||||||
|
The initial server targets stable LSP 3.17 behavior with these assumptions:
|
||||||
|
|
||||||
|
- transport is stdio only, with protocol bytes reserved for stdout;
|
||||||
|
- document positions are zero-based UTF-16 code-unit offsets;
|
||||||
|
- clients send full-document synchronization, simplifying authoritative version
|
||||||
|
handling while replacement behavior is established;
|
||||||
|
- the server owns snapshots of open documents only and never writes their URIs;
|
||||||
|
- diagnostics and quick fixes cover configured ASCII replacements, not heuristic
|
||||||
|
authorship or aggregate heuristic findings;
|
||||||
|
- quick fixes use versioned workspace edits and are rejected when the request
|
||||||
|
refers to a stale or unknown document version;
|
||||||
|
- `workspace/didChangeConfiguration` reloads the same layered configuration used
|
||||||
|
by the CLI and formatter, then republishes diagnostics for open snapshots;
|
||||||
|
- malformed messages and invalid URIs are protocol/input errors, never panics.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The replacement engine remains the single source of edit ranges and text.
|
||||||
|
- A tested line index converts canonical UTF-8 byte offsets to LSP UTF-16
|
||||||
|
positions at the adapter boundary.
|
||||||
|
- Protocol tests can drive `LspService` in process without launching a child
|
||||||
|
process, while a small stdio smoke test protects stream purity.
|
||||||
|
- Incremental synchronization may be added later, but must preserve version and
|
||||||
|
stale-result guarantees before it is advertised.
|
||||||
|
- Upgrading to the 0.24 line waits for a stable release and a separate API and
|
||||||
|
minimum-Rust review.
|
||||||
|
|
||||||
|
## Amendment: full heuristic diagnostics
|
||||||
|
|
||||||
|
On 2026-09-29, Milestone 19 superseded the initial replacement-only diagnostic
|
||||||
|
limit after the core detector and configuration contracts had stabilized. The
|
||||||
|
server now compiles the same detector as the CLI once per resolved configuration
|
||||||
|
and publishes every applicable finding for each open snapshot.
|
||||||
|
|
||||||
|
This amendment does not broaden mutation behavior:
|
||||||
|
|
||||||
|
- safe configured replacements remain the only findings with quick fixes;
|
||||||
|
- style, prompt-leakage, and source-code review findings are diagnostic-only;
|
||||||
|
- stable heuristic IDs replace replacement-entry IDs as published diagnostic
|
||||||
|
codes, avoiding duplicate typography diagnostics;
|
||||||
|
- configured advice, warning, and error severities map to LSP information,
|
||||||
|
warning, and error respectively;
|
||||||
|
- file URI paths retain extension and path-scope behavior, while editor language
|
||||||
|
IDs determine prose/source applicability for untitled or extensionless
|
||||||
|
buffers; and
|
||||||
|
- configuration reload atomically swaps the validated configuration and compiled
|
||||||
|
detector before open documents are republished.
|
||||||
|
|
||||||
|
The original full-snapshot synchronization, UTF-16 conversion, version checks,
|
||||||
|
stdio purity, no-write boundary, and stale-action guarantees remain unchanged.
|
||||||
|
|
||||||
|
## Amendment: quiet diagnostic publication
|
||||||
|
|
||||||
|
On 2026-09-30, Milestone 26 changed the presentation default without changing
|
||||||
|
the shared detector. The LSP now publishes only Unicode findings unless it is
|
||||||
|
started with `--all-diagnostics` or `--minimum-diagnostic-weight WEIGHT`.
|
||||||
|
Minimum-weight mode always retains Unicode findings and filters only the other
|
||||||
|
categories. This makes exact, fixable problems the quiet editor default while
|
||||||
|
keeping the full catalog easy to exercise in tests. Filtering occurs after
|
||||||
|
analysis, so CLI results, evidence scores, formatting, configuration reload,
|
||||||
|
and the core finding contract remain unchanged.
|
||||||
|
|
||||||
|
## Amendment: document formatting parity
|
||||||
|
|
||||||
|
Milestone 21 adds the standard `textDocument/formatting` capability. The server
|
||||||
|
plans edits from its current open-document snapshot with the same resolved
|
||||||
|
configuration and core replacement planner used by `john-slop-fmt`, then
|
||||||
|
converts only the edit coordinates from UTF-8 bytes to LSP UTF-16 positions.
|
||||||
|
Formatting remains limited to configured safe ASCII replacements; the server
|
||||||
|
does not write files, infer formatting from diagnostics, or apply broader
|
||||||
|
Unicode review findings. Protocol tests compare the result of an actual
|
||||||
|
formatting request with the standalone formatter contract, including Unicode,
|
||||||
|
CRLF, no-op, and configuration-reload cases.
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
# ADR 0004: Supported Cargo workspace
|
||||||
|
|
||||||
|
- Status: accepted
|
||||||
|
- Date: 2026-09-29
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The crate-boundary experiment extracted reusable analysis/configuration/input
|
||||||
|
code into `john-slop-core` and LSP protocol state into `john-slop-lsp`, leaving
|
||||||
|
the root `john-slop` package as a compatibility facade and binary distribution.
|
||||||
|
Cold core and no-LSP builds used roughly 35–55% less CPU and disk, while LSP and
|
||||||
|
whole-project builds were effectively unchanged. The experiment initially
|
||||||
|
recommended waiting because three publishable packages require synchronized
|
||||||
|
versions, ordered publication, workspace-aware CI, and clear asset ownership.
|
||||||
|
|
||||||
|
The project has now chosen to offer the reusable package boundary and accepts
|
||||||
|
that release overhead. Repeating package metadata and dependency versions in
|
||||||
|
three manifests would make that overhead unnecessarily error-prone.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Adopt the three-package layout as the supported architecture:
|
||||||
|
|
||||||
|
- `john-slop-core` owns embedded catalogs and reusable synchronous domain logic;
|
||||||
|
- `john-slop-lsp` depends on core and owns protocol state/conversion;
|
||||||
|
- root `john-slop` remains the compatibility facade and owns all three binaries.
|
||||||
|
|
||||||
|
Keep the root manifest as both a package and the workspace root. Moving its
|
||||||
|
sources under another directory solely to create a virtual manifest adds path,
|
||||||
|
fixture, and packaging churn without changing the dependency graph.
|
||||||
|
|
||||||
|
Centralize shared version, edition, MSRV, license, repository, dependency
|
||||||
|
requirements, and lint policy in the root workspace tables. Every member
|
||||||
|
inherits them. Dependency feature activation and optionality remain in member
|
||||||
|
manifests: workspace inheritance must not cause the core package to compile
|
||||||
|
Tokio/LSP dependencies or make the facade's LSP stack unconditional.
|
||||||
|
|
||||||
|
Keep all three packages as default members so existing root quality commands
|
||||||
|
cover the whole workspace. Enforce membership, synchronized resolved versions,
|
||||||
|
inheritance, and feature isolation with an integration test. Assemble packages
|
||||||
|
with Cargo's workspace packaging command and publish in dependency order: core,
|
||||||
|
LSP, then facade.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Core/no-LSP consumers retain the measured lean dependency graph.
|
||||||
|
- Existing `john_slop` API paths and binary names remain compatible by default.
|
||||||
|
- Shared requirements have one update point, while member feature choices stay
|
||||||
|
reviewable at their point of use.
|
||||||
|
- Full builds gain no material speedup from the boundary.
|
||||||
|
- A release coordinates three matching package versions. Before matching
|
||||||
|
dependency versions exist in the registry, package assembly uses
|
||||||
|
`--no-verify`; registry-form verification follows publication.
|
||||||
|
- Embedded catalog changes belong to `john-slop-core`, and workspace coverage,
|
||||||
|
CI, dependency review, and packaging remain permanent maintenance costs.
|
||||||
|
|
@ -0,0 +1,62 @@
|
||||||
|
# Dependency and binary review
|
||||||
|
|
||||||
|
Milestone 7 reviewed the locked graph on 2026-09-29 with Rust 1.100.0-nightly
|
||||||
|
on x86-64 Linux.
|
||||||
|
|
||||||
|
## Features and scope
|
||||||
|
|
||||||
|
The normal dependency graph contains 135 distinct package names. Direct crates
|
||||||
|
serve narrow boundaries: Clap parses adapters, Figment/Serde/TOML load typed
|
||||||
|
configuration, Regex/Globset compile rules and scopes, Miette renders
|
||||||
|
diagnostics, Walkdir/Zip acquire inputs, Tokio/tower-lsp-server implement the
|
||||||
|
stdio protocol, Unicode Security supplies the pinned UTS #39 skeleton data, and
|
||||||
|
Tempfile/Similar support atomic writes and diffs.
|
||||||
|
|
||||||
|
Default features are disabled where they would add unrelated behavior. In
|
||||||
|
particular, Zip previously enabled AES, Bzip2, Deflate64, LZMA, PPMd, Zstandard,
|
||||||
|
time, and multiple encoders. John Slop reads unencrypted office/ZIP inputs and
|
||||||
|
now enables only the zlib-rs Deflate path required by common containers. A
|
||||||
|
Deflate archive regression test pins that support. This removed 394 lockfile
|
||||||
|
lines. Regex retains Unicode and performance defaults because configured rules
|
||||||
|
use Unicode text and case matching; Miette's graphical stack remains isolated
|
||||||
|
behind `fancy-diagnostics`.
|
||||||
|
|
||||||
|
All publishable packages inherit dependency requirements from the root
|
||||||
|
`[workspace.dependencies]` table. This centralizes versions and baseline feature
|
||||||
|
policy without globally enabling member-specific features: `john-slop-core`
|
||||||
|
does not select Clap, Tokio, or the LSP stack; `john-slop-lsp` selects only
|
||||||
|
Tokio synchronization; and the root adapter enables runtime/stdio support only
|
||||||
|
behind its optional `lsp` feature. `tests/workspace_contract.rs` pins this
|
||||||
|
boundary and the synchronized package metadata.
|
||||||
|
|
||||||
|
## Security, licenses, and unsafe code
|
||||||
|
|
||||||
|
`cargo audit --deny warnings` scanned 193 locked packages against 1,277 RustSec
|
||||||
|
advisories without a finding. `cargo deny --locked check` passed advisories,
|
||||||
|
licenses, and sources; it reports seven transitive duplicate-version families
|
||||||
|
as warnings for ongoing review. [`deny.toml`](../deny.toml) contains the reviewed
|
||||||
|
license allow-list and rejects wildcard versions, yanked packages, and unknown
|
||||||
|
registries or Git sources.
|
||||||
|
|
||||||
|
A workspace lint forbids first-party Rust `unsafe` code, and the source scan
|
||||||
|
found no existing blocks. Dependencies may use
|
||||||
|
unsafe code internally, notably for compression, operating-system integration,
|
||||||
|
and async/runtime primitives; this review does not claim otherwise. Lockfile,
|
||||||
|
advisory, source, and license checks remain CI gates because this result changes
|
||||||
|
when dependencies change.
|
||||||
|
|
||||||
|
## Release size baseline
|
||||||
|
|
||||||
|
Before release-profile stripping, the all-feature Linux binaries measured 6.21,
|
||||||
|
5.84, and 9.38 MB. The finalized thin-LTO, single-codegen-unit,
|
||||||
|
abort-on-panic, symbol-stripped profile measures:
|
||||||
|
|
||||||
|
| Binary | Bytes |
|
||||||
|
| --- | ---: |
|
||||||
|
| `john-slop` | 3,542,544 |
|
||||||
|
| `john-slop-fmt` | 3,322,944 |
|
||||||
|
| `john-slop-lsp` | 4,756,688 |
|
||||||
|
|
||||||
|
These are build-specific engineering measurements, not promised download sizes.
|
||||||
|
The release procedure records final sizes and emits checksums beside any
|
||||||
|
distributed binaries.
|
||||||
|
|
@ -0,0 +1,119 @@
|
||||||
|
# Development
|
||||||
|
|
||||||
|
The project uses Rust edition 2024 and requires Rust 1.98.1 or newer. The
|
||||||
|
committed lockfile makes application builds reproducible. Linux is the supported
|
||||||
|
and exercised platform; other platforms are not declared supported until CI
|
||||||
|
covers them. Jenkins checks Linux on both Rust 1.98.1 (the MSRV) and current
|
||||||
|
stable, with all features and without default features.
|
||||||
|
|
||||||
|
## Quality gates
|
||||||
|
|
||||||
|
Run the same core checks required by the contributor guide:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo fmt --all -- --check
|
||||||
|
cargo clippy --locked --all-targets --all-features -- -D warnings
|
||||||
|
cargo test --locked --all-targets --all-features
|
||||||
|
cargo test --locked --all-targets --no-default-features
|
||||||
|
cargo doc --locked --no-deps --all-features
|
||||||
|
cargo audit --deny warnings
|
||||||
|
cargo deny --locked check
|
||||||
|
```
|
||||||
|
|
||||||
|
The root package is also the workspace root, and all three packages are default
|
||||||
|
members. These unqualified commands therefore cover the compatibility/binary
|
||||||
|
package, `john-slop-core`, and `john-slop-lsp`. Use `-p john-slop-core` for the
|
||||||
|
isolated reusable dependency graph or `-p john-slop-lsp` for protocol-only work.
|
||||||
|
The workspace contract test validates membership, version synchronization,
|
||||||
|
manifest inheritance, and feature isolation. Before a release, assemble every
|
||||||
|
publishable crate with `scripts/package-workspace.sh --no-verify`; see the
|
||||||
|
[release procedure](releasing.md) for the registry verification boundary.
|
||||||
|
|
||||||
|
To smoke-test installation from the current source checkout, identify the local
|
||||||
|
package explicitly (recent Cargo versions no longer infer it from an omitted
|
||||||
|
package argument):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo install --path . --locked --all-features
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `--root <temporary-directory>` when verifying this command without changing
|
||||||
|
the normal Cargo installation root.
|
||||||
|
|
||||||
|
The default `fancy-diagnostics` feature enables the detector's colored Miette
|
||||||
|
renderer and default output mode. The default `lsp` feature exposes the
|
||||||
|
compatibility facade's protocol module and builds the LSP binary. The
|
||||||
|
no-default-features test run exercises the core, detector, and formatter without
|
||||||
|
the async protocol dependency graph and with the plain diagnostic fallback. Focused
|
||||||
|
unit and integration tests cover domain invariants, input failures, archives,
|
||||||
|
CLI exit behavior, and feature configurations.
|
||||||
|
|
||||||
|
CI runs the executable fixture manifest and Insta CLI snapshots explicitly in
|
||||||
|
addition to the full suite. `cargo-audit` rejects RustSec advisories and yanked
|
||||||
|
packages. [`deny.toml`](../deny.toml) rejects unknown dependency sources,
|
||||||
|
wildcard versions, and licenses outside the reviewed allow-list; duplicate
|
||||||
|
versions are reported for review while upstream compatibility is assessed.
|
||||||
|
|
||||||
|
Insta snapshots are reserved for stable user-facing text such as CLI help and
|
||||||
|
usage errors. Structured domain values use direct assertions so failures remain
|
||||||
|
precise and easy to review. Accept snapshot changes only after inspecting the
|
||||||
|
generated diff.
|
||||||
|
|
||||||
|
## Coverage and badges
|
||||||
|
|
||||||
|
Run the same line-coverage command used by Jenkins:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo tarpaulin --config tarpaulin.toml --out Json --output-dir coverage
|
||||||
|
```
|
||||||
|
|
||||||
|
The configuration enforces a 90% floor. The current suite measured 92.72%
|
||||||
|
(1,604/1,730 lines) on 2026-09-30. The remaining lines are dominated by
|
||||||
|
platform/I/O failure injection, malformed archive internals, coordinate
|
||||||
|
overflow, LSP logging/reload failures, and subprocess attribution. Do not add
|
||||||
|
brittle tests or production seams solely to make Tarpaulin count those paths.
|
||||||
|
|
||||||
|
`Cargo.toml` is the authoritative version source. Jenkins temporarily renames
|
||||||
|
the Tarpaulin JSON output to `coverage/summary.json`, extracts and validates its
|
||||||
|
overall percentage, renders the badges locally, and then removes the summary.
|
||||||
|
It archives only `coverage/version.svg`, `coverage/test.svg`, and
|
||||||
|
`coverage/coverage.svg`, with fingerprinting. Successful `main` builds publish
|
||||||
|
those files atomically under
|
||||||
|
`/mnt/assets/coverage/idle-org/john-slop/main/`, which serves these mutable URLs:
|
||||||
|
|
||||||
|
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/version.svg`
|
||||||
|
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/test.svg`
|
||||||
|
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/coverage.svg`
|
||||||
|
|
||||||
|
The asset server is deployment-owned: it must mount assets read-only, disable
|
||||||
|
directory listings, serve SVG with a restrictive content security policy, and
|
||||||
|
revalidate these URLs or apply a short cache lifetime.
|
||||||
|
|
||||||
|
## Test scope
|
||||||
|
|
||||||
|
Add behavior tests at the lowest useful layer and an integration test when a
|
||||||
|
change crosses a binary boundary. Important edge cases include malformed input,
|
||||||
|
invalid UTF-8, empty text, Unicode byte spans, deterministic ordering, archive
|
||||||
|
limits, and near misses for detection rules. The detailed expectations and
|
||||||
|
task-level commit discipline are in [AGENTS.md](../AGENTS.md).
|
||||||
|
|
||||||
|
Each completed milestone receives an immutable annotated `v0.1.N` Git tag,
|
||||||
|
where `N` is the milestone number, after its acceptance criteria and quality
|
||||||
|
gates pass. Its completion commit must also use package version `0.1.N` in the
|
||||||
|
workspace manifest, internal registry requirements, lockfile, changelog, binary
|
||||||
|
version output. Jenkins derives the current version badge from that metadata.
|
||||||
|
|
||||||
|
## Fuzzing
|
||||||
|
|
||||||
|
The standalone [`fuzz`](../fuzz/) package contains five `cargo-fuzz` targets and
|
||||||
|
reviewed seeds for configuration merging, rule validation/compilation, UTF-8
|
||||||
|
span conversion, replacement invariants, and hostile archive bytes. See its
|
||||||
|
README for invocation. Fuzz artifacts remain ignored until a failure has been
|
||||||
|
minimized and promoted to a normal regression test.
|
||||||
|
|
||||||
|
## Benchmarks
|
||||||
|
|
||||||
|
Run `cargo bench --locked --bench core` for the large-document, many-file,
|
||||||
|
rule-scaling, and archive-limit workloads described in
|
||||||
|
[Performance benchmarks](benchmarks.md). Benchmark reports are local artifacts;
|
||||||
|
only methodology and contextualized measurements belong in version control.
|
||||||
|
|
@ -0,0 +1,141 @@
|
||||||
|
# Editor setup
|
||||||
|
|
||||||
|
Both `john-slop-fmt` and `john-slop-lsp` are operational. The LSP publishes only
|
||||||
|
configured Unicode replacement and control findings by default, keeping noisy
|
||||||
|
or error-prone heuristics out of the normal editor loop. Safe ASCII-replacement
|
||||||
|
findings additionally provide versioned quick fixes, and the server implements
|
||||||
|
whole-document formatting with the exact same replacement planner as
|
||||||
|
`john-slop-fmt`.
|
||||||
|
|
||||||
|
Install all three binaries somewhere on `PATH`, then confirm the editor-facing
|
||||||
|
help is available:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop-fmt --help
|
||||||
|
john-slop-lsp --help
|
||||||
|
```
|
||||||
|
|
||||||
|
## Helix (`hx` / `helix-term`)
|
||||||
|
|
||||||
|
Helix configures formatters and language servers in `languages.toml`. Put this
|
||||||
|
in `~/.config/helix/languages.toml` on Linux, or in `.helix/languages.toml` for
|
||||||
|
project-local configuration:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[language-server.john-slop]
|
||||||
|
command = "john-slop-lsp"
|
||||||
|
|
||||||
|
[[language]]
|
||||||
|
name = "john-slop-text"
|
||||||
|
language-id = "plaintext"
|
||||||
|
scope = "text.john-slop"
|
||||||
|
file-types = ["txt"]
|
||||||
|
formatter = { command = "john-slop-fmt" }
|
||||||
|
language-servers = ["john-slop"]
|
||||||
|
auto-format = false
|
||||||
|
```
|
||||||
|
|
||||||
|
This complete custom language entry makes the repository's `example.txt`
|
||||||
|
fixtures useful without depending on a built-in plain-text grammar. It selects
|
||||||
|
the external `john-slop-fmt` adapter. To exercise formatting through the LSP
|
||||||
|
instead, omit the `formatter = ...` line; `:format` then requests
|
||||||
|
`textDocument/formatting` from `john-slop-lsp`. Both routes resolve the same
|
||||||
|
configuration and produce the same text. Set `auto-format = true` only if
|
||||||
|
save-time replacement is wanted.
|
||||||
|
|
||||||
|
For an existing language, merge John Slop into that language's entry rather
|
||||||
|
than replacing its useful defaults. For example, Rust should retain
|
||||||
|
`rust-analyzer`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[[language]]
|
||||||
|
name = "rust"
|
||||||
|
formatter = { command = "john-slop-fmt" }
|
||||||
|
language-servers = ["rust-analyzer", "john-slop"]
|
||||||
|
```
|
||||||
|
|
||||||
|
When several language servers advertise formatting, retain an explicit Helix
|
||||||
|
`formatter` or select the intended server interactively; John Slop must not
|
||||||
|
silently replace language-specific formatting from `rust-analyzer` or another
|
||||||
|
primary server.
|
||||||
|
|
||||||
|
Run `hx --health rust` (or the selected language name) to confirm both binaries
|
||||||
|
are visible. Helix documents that a formatter reads the buffer from stdin and
|
||||||
|
writes it to stdout, while an LSP command must be on `PATH`; those are the
|
||||||
|
contracts the two John Slop milestones target. See the official
|
||||||
|
[Helix language configuration reference](https://docs.helix-editor.com/languages.html).
|
||||||
|
|
||||||
|
## Vim
|
||||||
|
|
||||||
|
Vim's native `formatprg` option is enough for formatting. Add this to
|
||||||
|
`~/.vimrc`:
|
||||||
|
|
||||||
|
```vim
|
||||||
|
augroup john-slop-format
|
||||||
|
autocmd!
|
||||||
|
autocmd BufRead,BufNewFile *.txt setfiletype text
|
||||||
|
autocmd FileType text,markdown,rust setlocal formatprg=john-slop-fmt
|
||||||
|
augroup END
|
||||||
|
```
|
||||||
|
|
||||||
|
Use `gggqG` to filter the whole buffer through the formatter, or select a range
|
||||||
|
and press `gq`. Vim requires `formatprg` programs to read stdin and write stdout;
|
||||||
|
see the official [`formatprg` and `gq` help](https://vimhelp.org/options.txt.html#%27formatprg%27).
|
||||||
|
|
||||||
|
Classic Vim needs an LSP client plugin. With
|
||||||
|
[`prabirshrestha/vim-lsp`](https://github.com/prabirshrestha/vim-lsp), register
|
||||||
|
John Slop in `~/.vimrc`:
|
||||||
|
|
||||||
|
```vim
|
||||||
|
if executable('john-slop-lsp')
|
||||||
|
au User lsp_setup call lsp#register_server({
|
||||||
|
\ 'name': 'john-slop',
|
||||||
|
\ 'cmd': {server_info->['john-slop-lsp']},
|
||||||
|
\ 'allowlist': ['text', 'markdown', 'rust'],
|
||||||
|
\ })
|
||||||
|
endif
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the plugin's `:LspStatus`, `:LspNextDiagnostic`, and `:LspCodeAction`
|
||||||
|
commands to inspect server state, move through heuristic diagnostics, and apply
|
||||||
|
available ASCII quick fixes. The registration shape and `allowlist` follow the
|
||||||
|
plugin's current upstream documentation.
|
||||||
|
|
||||||
|
Clients that expose LSP document formatting can request it directly from John
|
||||||
|
Slop instead of configuring `formatprg`; both paths use the same replacement
|
||||||
|
catalog. Keep `formatprg=john-slop-fmt` when the Vim client cannot select one
|
||||||
|
formatter among multiple attached language servers.
|
||||||
|
|
||||||
|
The server uses the same configuration layers as the other binaries. Add
|
||||||
|
`args = ["--config-name", "editorial"]` to the Helix language-server table, or
|
||||||
|
append those arguments to the Vim `cmd` list, when an editor-specific named
|
||||||
|
configuration is wanted. Use `args = ["--diagnostics", "diagnostics.toml"]` (and
|
||||||
|
the equivalent Vim command argument) for a project runtime diagnostic catalog.
|
||||||
|
Add `--all-diagnostics` to exercise every configured rule during testing, or
|
||||||
|
`--minimum-diagnostic-weight 10` to retain Unicode findings and opt into only
|
||||||
|
non-Unicode findings with weight 10 or greater. This visibility filter does not
|
||||||
|
change detector scoring or formatting.
|
||||||
|
`--log-json` is available for troubleshooting and writes lifecycle metadata to
|
||||||
|
stderr without document contents.
|
||||||
|
|
||||||
|
## Try the shared fixtures
|
||||||
|
|
||||||
|
Open `examples/fixtures/typography/example.txt` in either editor. Its expected
|
||||||
|
formatter result is the adjacent `formatted.txt`; the shared manifest records
|
||||||
|
three replacements and one typography rule. Compare detector and formatter
|
||||||
|
output with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run --bin john-slop -- --output concise \
|
||||||
|
examples/fixtures/typography/example.txt
|
||||||
|
cargo run --bin john-slop-fmt -- \
|
||||||
|
examples/fixtures/typography/example.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
After starting the server with `--all-diagnostics`, open
|
||||||
|
`examples/fixtures/style/example.txt` to see `repeated-transition` and other
|
||||||
|
style findings, or `examples/fixtures/source/example.rs` to see source-only
|
||||||
|
placeholder and narration findings alongside typography. Advice maps to LSP
|
||||||
|
information, warnings to warnings, and configured errors to errors. Every
|
||||||
|
diagnostic carries its stable heuristic rule ID as the code and includes the
|
||||||
|
configured remediation text.
|
||||||
|
|
@ -0,0 +1,83 @@
|
||||||
|
# Evaluation corpus
|
||||||
|
|
||||||
|
Milestone 3 ships a small, reviewed characterization corpus under
|
||||||
|
`tests/fixtures/evaluation`. Its text was authored for this repository and is
|
||||||
|
covered by the MIT license. Labels name observable configured signals, never a
|
||||||
|
presumed author or generation method.
|
||||||
|
|
||||||
|
With the embedded schema-version 1 catalog, the corpus currently reports:
|
||||||
|
|
||||||
|
| Rule | True positive | False positive | False negative | Precision | Recall |
|
||||||
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||||
|
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| assistant-follow-up-offer | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| assistant-revision-preface | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| templated-filler | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| editorial-insertion-placeholder | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| stock-helpful-closing | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| excess-vocabulary-cluster | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| source-omitted-code-placeholder | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| source-markdown-fence | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
| unicode-ascii-replacement | 2 | 0 | 0 | 100% | 100% |
|
||||||
|
| unicode-invisible-control | 1 | 0 | 0 | 100% | 100% |
|
||||||
|
|
||||||
|
Aggregate labeled-rule counts are 20 true positives, 0 false positives, and 0
|
||||||
|
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
|
||||||
|
suite recomputes these values from the manifest and fails on drift.
|
||||||
|
|
||||||
|
The same manifest pins evidence scores for regression characterization:
|
||||||
|
|
||||||
|
| Corpus case | Score |
|
||||||
|
| --- | ---: |
|
||||||
|
| six assistant disclaimer/preface rules | 64 |
|
||||||
|
| one prompt/response marker | 33 |
|
||||||
|
| style signals | 25 |
|
||||||
|
| source-code signals | 33 |
|
||||||
|
| typography signal | 17 |
|
||||||
|
| one ASCII lookalike and one invisible character | 28 |
|
||||||
|
| each of five negative documents | 0 |
|
||||||
|
|
||||||
|
These perfect values describe a tiny purpose-built regression set, not expected
|
||||||
|
performance on real documents. They must not be used as an accuracy or
|
||||||
|
authorship claim. Broader, independently sourced data and domain-specific review
|
||||||
|
are required before tuning default weights or thresholds.
|
||||||
|
|
||||||
|
## Default-policy sign-off
|
||||||
|
|
||||||
|
Each shipped rule has at least one explicitly labeled realistic near miss. The
|
||||||
|
initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds
|
||||||
|
unchanged. The prompt-leakage review accepted lighter `0.90ⁿ` correlation
|
||||||
|
damping for distinct prompt-leakage rules while retaining `0.75ⁿ` for other
|
||||||
|
categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings
|
||||||
|
replaceable by default, retains a non-mutating `review` policy, and keeps the
|
||||||
|
Gitea-compatible invisible/control profile diagnostic-only, with explicit
|
||||||
|
multilingual limitations. The Milestone 25 review gives one-to-one fullwidth
|
||||||
|
ASCII compatibility folds precedence over visual skeletons under every policy.
|
||||||
|
The Milestone 26 review adds narrow response and unfinished-editing residue,
|
||||||
|
keeps population-level vocabulary and stock closings deliberately low-weight,
|
||||||
|
and excludes language-proficiency proxies.
|
||||||
|
The checked-in `signoff.toml` records those decisions
|
||||||
|
and contains the SHA-256 digest of `config.toml`, `heuristics.toml`, and
|
||||||
|
`unicode.toml`. This binds the reviewed scoring behavior to both rule catalogs
|
||||||
|
and the default category policy.
|
||||||
|
|
||||||
|
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
|
||||||
|
change invalidates the sign-off test. Renewal requires reviewing positive and
|
||||||
|
near-miss cases, reporting changed per-rule/aggregate metrics, explaining the
|
||||||
|
false-positive tradeoff, updating the digest and decision text, and obtaining
|
||||||
|
explicit maintainer approval in the change review. A green test alone is not
|
||||||
|
approval to tune defaults.
|
||||||
|
|
||||||
|
Runtime diagnostic catalogs are deployment policy and are not part of the
|
||||||
|
embedded-catalog digest. Their rules use the same validation and scoring
|
||||||
|
formula, but teams must evaluate their own added patterns and near misses for
|
||||||
|
their corpus rather than treating the embedded sign-off as validation of custom
|
||||||
|
rules.
|
||||||
|
|
@ -0,0 +1,99 @@
|
||||||
|
# Crate-boundary experiment
|
||||||
|
|
||||||
|
Status: **adopted for Milestone 13**. The experiment originally recommended
|
||||||
|
waiting because full builds do not improve and three publishable packages add
|
||||||
|
release coordination. The project subsequently chose to support the reusable
|
||||||
|
package boundary and accepted those measured costs; the durable policy is in
|
||||||
|
[ADR 0004](../decisions/0004-supported-cargo-workspace.md).
|
||||||
|
|
||||||
|
## Candidate design
|
||||||
|
|
||||||
|
The experiment keeps the root `john-slop` package as a compatibility facade and
|
||||||
|
binary distribution. It moves reusable analysis into `john-slop-core`, moves
|
||||||
|
protocol state into `john-slop-lsp`, and makes the facade's LSP dependency
|
||||||
|
optional behind the default-enabled `lsp` feature. Existing default API paths
|
||||||
|
and executable names remain intact.
|
||||||
|
|
||||||
|
The embedded catalogs live with `john-slop-core`, which makes that package
|
||||||
|
self-contained. Workspace default members include all three packages so the
|
||||||
|
existing unqualified quality commands continue to cover them.
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
Baseline commit `c12b6b8` (`v0.1.11`) and the experimental branch were
|
||||||
|
checked in separate worktrees. Each cold command used an empty, dedicated
|
||||||
|
`CARGO_TARGET_DIR`; dependency sources and the operating-system file cache were
|
||||||
|
warm. Timings are one directional sample rather than a statistical benchmark.
|
||||||
|
Artifact size is the resulting target directory size reported by `du -sh`.
|
||||||
|
|
||||||
|
Environment: Rust/Cargo 1.100.0-nightly from 2026-09-15, x86-64 Linux, 32 logical
|
||||||
|
CPUs. Commands used `--locked`; the full checks used all targets and features.
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Baseline wall / CPU | Candidate wall / CPU | Baseline / candidate artifacts |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Reusable core | 7.51s / 29.96s¹ | 3.82s / 13.56s | 256 / 116 MiB¹ |
|
||||||
|
| Root library, no defaults | 7.51s / 29.96s | 4.09s / 19.04s | 256 / 158 MiB |
|
||||||
|
| Formatter, no defaults | 6.98s / 29.50s | 4.28s / 19.19s | 244 / 160 MiB |
|
||||||
|
| LSP | 7.92s / 30.13s | 6.97s / 29.66s | 243 / 244 MiB |
|
||||||
|
| Whole project | 7.90s / 49.68s | 7.73s / 50.69s | 415 / 416 MiB |
|
||||||
|
| No-change library check | 0.043s wall | 0.036s wall | not material |
|
||||||
|
|
||||||
|
¹ The single-crate baseline has no isolated core target, so its library result
|
||||||
|
is the nearest comparison.
|
||||||
|
|
||||||
|
The useful changes are therefore concentrated in non-LSP builds:
|
||||||
|
|
||||||
|
- the directly consumable core uses about 55% less CPU and disk and 49% less
|
||||||
|
wall time than the former monolithic library check;
|
||||||
|
- the compatibility library without default features uses about 36% less CPU,
|
||||||
|
38% less disk, and 46% less wall time;
|
||||||
|
- the formatter without default features uses about 35% less CPU and disk and
|
||||||
|
39% less wall time;
|
||||||
|
- LSP, full-project, and no-change incremental costs are effectively unchanged.
|
||||||
|
|
||||||
|
Tarpaulin still reports 92.19% (1,334/1,447 lines) with workspace coverage, so
|
||||||
|
the boundary neither gains nor loses measured test coverage.
|
||||||
|
|
||||||
|
## Gains
|
||||||
|
|
||||||
|
- Async protocol dependencies no longer enter core or no-LSP builds.
|
||||||
|
- The module ownership matches the conceptual detector/LSP boundary.
|
||||||
|
- A future library consumer can depend on `john-slop-core` without CLI, JSON,
|
||||||
|
diff, Tokio, or LSP dependencies.
|
||||||
|
- The root facade avoids a forced source migration for default-feature users.
|
||||||
|
|
||||||
|
## Costs and risks
|
||||||
|
|
||||||
|
- One package and manifest become three, with synchronized versions and the
|
||||||
|
publish order core → LSP → facade.
|
||||||
|
- The embedded catalogs must move into the core package, changing repository
|
||||||
|
paths and release ownership.
|
||||||
|
- Coverage and CI must remain workspace-aware.
|
||||||
|
- `john_slop::lsp` is unavailable when default features are disabled; this is a
|
||||||
|
feature-contract change even though default builds are compatible.
|
||||||
|
- Ordinary default and full-project builds receive no material speedup, and
|
||||||
|
runtime behavior and performance do not improve.
|
||||||
|
|
||||||
|
## Original recommendation and adoption
|
||||||
|
|
||||||
|
Do not merge this branch while John Slop is primarily distributed as one set of
|
||||||
|
three binaries and has no demonstrated external core-library consumers. The
|
||||||
|
measured savings are real but apply only when deliberately excluding LSP or
|
||||||
|
depending directly on the new core package; the normal full build is unchanged.
|
||||||
|
|
||||||
|
Reconsider the branch if `john-slop-core` becomes a supported published API, if
|
||||||
|
the adapters gain independent release schedules, or if the full build grows
|
||||||
|
enough that the package boundaries enable additional parallelism. Until then,
|
||||||
|
optionalizing the LSP module in the existing single crate would capture most of
|
||||||
|
the no-default-feature compile benefit with substantially less release overhead.
|
||||||
|
|
||||||
|
Milestone 13 deliberately chose the first reconsideration condition: treat
|
||||||
|
`john-slop-core` and `john-slop-lsp` as supported reusable packages. The measured
|
||||||
|
full-build result has not changed. Adoption instead values the enforceable
|
||||||
|
dependency boundary, lean core consumers, and clearer ownership enough to
|
||||||
|
accept synchronized releases and workspace-aware CI/package checks. Shared
|
||||||
|
metadata and dependency requirements now use Cargo workspace inheritance, while
|
||||||
|
member-specific feature activation remains local so centralization does not
|
||||||
|
erase the isolation this experiment measured.
|
||||||
|
|
@ -0,0 +1,64 @@
|
||||||
|
# Heuristics and evidence scoring
|
||||||
|
|
||||||
|
John Slop reports observable low-care signals. It does not classify authorship.
|
||||||
|
Every contribution to the score is represented by a finding with an exact
|
||||||
|
source span, stable rule ID, category, message, help, and configured weight.
|
||||||
|
|
||||||
|
The detector compiles the resolved rule catalog once. Exact phrases, regular
|
||||||
|
expressions, repeated-pattern rules, configured Unicode replacements, and
|
||||||
|
invisible/control characters share the same finding
|
||||||
|
model. Content kind, extension, path globs, and conventional generated/vendor/
|
||||||
|
minified paths decide whether each rule applies. Findings are deduplicated by
|
||||||
|
rule and byte span, then sorted by start offset, end offset, and rule ID so
|
||||||
|
every adapter observes the same order.
|
||||||
|
|
||||||
|
Additional exact-phrase, regex, and repeated-regex rules can be supplied at
|
||||||
|
runtime through `diagnostics.toml` or repeatable `--diagnostics` arguments.
|
||||||
|
They receive the same validation, scoping, scoring, ordering, CLI presentation,
|
||||||
|
and LSP publication as embedded rules. New matcher algorithms and categories
|
||||||
|
remain code changes rather than executable configuration.
|
||||||
|
|
||||||
|
The generated Unicode 16.0 profile maps every non-ASCII scalar with a printable
|
||||||
|
ASCII UTS #39 skeleton after explicitly reviewed compatibility-width and
|
||||||
|
punctuation mappings.
|
||||||
|
The default `replace` policy gives each of these findings a formatter/LSP edit;
|
||||||
|
`review` keeps the same exact diagnostics without edits, and `off` suppresses
|
||||||
|
the generated lookalikes. Invisible and direction-changing controls remain
|
||||||
|
diagnostic-only because deleting them silently is not a generally safe edit.
|
||||||
|
|
||||||
|
## Evidence-score formula
|
||||||
|
|
||||||
|
For each rule in deterministic catalog order, only the configured
|
||||||
|
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
|
||||||
|
full weight, then later occurrences receive one half, one quarter, and so on.
|
||||||
|
Findings beyond the cap remain visible but do not increase the score.
|
||||||
|
|
||||||
|
Rules are grouped by category as correlated evidence families. The first
|
||||||
|
matching rule in a category receives its full occurrence-adjusted contribution;
|
||||||
|
later distinct prompt-leakage rules receive successive factors of `0.90`, while
|
||||||
|
style, typography, and source-code rules receive successive factors of `0.75`.
|
||||||
|
Prompt/response markers and copied assistant disclaimers are stronger concrete
|
||||||
|
artifacts than generic stylistic similarities, but the remaining damping still
|
||||||
|
prevents a cluster of related phrases from being treated as fully independent
|
||||||
|
proof.
|
||||||
|
|
||||||
|
If the resulting evidence sum is `E`, the displayed score is
|
||||||
|
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
|
||||||
|
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
|
||||||
|
is explicitly reported as below threshold. Empty input and text with no
|
||||||
|
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
|
||||||
|
oversized, and unsupported inputs remain typed input errors; they are never
|
||||||
|
reported as clean analyses.
|
||||||
|
|
||||||
|
Every `Analysis` retains its complete finding list and can produce a compact
|
||||||
|
summary containing the numeric score, optional named band, and separate advice,
|
||||||
|
warning, and error counts. Summary generation never filters the evidence used
|
||||||
|
to explain the score.
|
||||||
|
|
||||||
|
Weights and thresholds are review policy, not calibrated probabilities. A
|
||||||
|
score of 75 does not mean a 75% chance that AI wrote the input.
|
||||||
|
|
||||||
|
The diagnostic showcase exercises five distinct prompt-leakage rules, including
|
||||||
|
two prompt/response markers. Together with its style and typography findings it
|
||||||
|
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
|
||||||
|
an estimated probability.
|
||||||
|
|
@ -0,0 +1,94 @@
|
||||||
|
# Limitations
|
||||||
|
|
||||||
|
John Slop is a review aid for observable low-care signals. It is not an
|
||||||
|
AI-authorship detector, plagiarism detector, fact checker, security scanner, or
|
||||||
|
quality oracle. A finding says only that configured text matched at a particular
|
||||||
|
source span. Human context remains decisive.
|
||||||
|
|
||||||
|
## False positives
|
||||||
|
|
||||||
|
Legitimate material can contain every shipped signal. Documentation may quote a
|
||||||
|
chat transcript, an article may discuss knowledge cutoffs, a tutorial may use
|
||||||
|
step-by-step comments deliberately, and a tracked source-code TODO may be valid.
|
||||||
|
Unicode punctuation can be correct typography even when a downstream format
|
||||||
|
requires ASCII. The UTS #39 ASCII-skeleton profile intentionally flags many
|
||||||
|
legitimate Greek, Cyrillic, mathematical, and other non-Latin characters for
|
||||||
|
review because it has no locale or identifier context. Generic transitions are
|
||||||
|
weak style evidence, not defects by themselves.
|
||||||
|
|
||||||
|
Use path scopes, rule disabling, and project-specific weights to reflect local
|
||||||
|
policy. Do not automatically reject, shame, or attribute a document based on a
|
||||||
|
score. Review the visible findings and original context.
|
||||||
|
|
||||||
|
## False negatives and evasion
|
||||||
|
|
||||||
|
Exact and regular-expression rules are inherently easy to evade through
|
||||||
|
paraphrasing, case changes, spelling variation, inserted characters, or removal
|
||||||
|
of obvious scaffolding. Careless work need not contain any known signal, while
|
||||||
|
well-reviewed AI-assisted work may contain none. John Slop deliberately avoids
|
||||||
|
opaque statistical authorship guesses, so a clean result is not evidence of
|
||||||
|
human authorship or high quality.
|
||||||
|
|
||||||
|
Adversarial inputs can also target applicability assumptions: misleading file
|
||||||
|
extensions, unusual generated-directory names, or stdin without a filename may
|
||||||
|
select a different scope. Archive analysis uses the inner member path but does
|
||||||
|
not infer semantic language from content.
|
||||||
|
|
||||||
|
## Domain and language shift
|
||||||
|
|
||||||
|
The starter prose rules primarily encode English phrases. Other languages,
|
||||||
|
specialized editorial styles, historical corpora, transcripts, generated API
|
||||||
|
documentation, notebooks, and uncommon programming languages are
|
||||||
|
underrepresented. File-extension inference covers a conservative common set and
|
||||||
|
defaults unknown paths to prose. Teams should validate rules against their own
|
||||||
|
reviewed material before changing enforcement thresholds. Disable or narrow
|
||||||
|
set `[unicode] confusables = "review"` to keep findings without edits, or use
|
||||||
|
`"off"` when ordinary multilingual text makes character-level review too noisy.
|
||||||
|
Locale-aware suppression is not implemented.
|
||||||
|
|
||||||
|
## Evidence score is not probability
|
||||||
|
|
||||||
|
The score is a deterministic compression of configured weights, occurrence
|
||||||
|
caps, and category damping. Those values are policy choices and have not been
|
||||||
|
calibrated against population-level authorship data. A score of 75 does not mean
|
||||||
|
there is a 75% probability of AI authorship, low care, or an error. Thresholds
|
||||||
|
name review bands only, and changing the catalog changes their meaning.
|
||||||
|
|
||||||
|
Every score is traceable to findings, but that explainability does not make the
|
||||||
|
underlying heuristic universally valid. Related signals can remain correlated
|
||||||
|
even after category damping.
|
||||||
|
|
||||||
|
## Evaluation limits
|
||||||
|
|
||||||
|
The checked-in evaluation corpus is small, purpose-built, and intended to catch
|
||||||
|
behavioral regressions. Its current perfect precision and recall reflect fixture
|
||||||
|
construction, not real-world accuracy. It contains only one clean document and
|
||||||
|
must not justify deployment claims or default-weight tuning. Representative,
|
||||||
|
licensed, domain-specific samples and independent human review are required for
|
||||||
|
such decisions.
|
||||||
|
|
||||||
|
## Input and operational boundaries
|
||||||
|
|
||||||
|
Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8,
|
||||||
|
encrypted archives, unsafe archive members, and configured resource-limit
|
||||||
|
violations are errors rather than clean results. Generated, vendor, dependency,
|
||||||
|
and minified paths are skipped by default. The formatter handles strict UTF-8
|
||||||
|
files and configured ASCII replacements only; it does not rewrite ZIP-based
|
||||||
|
containers, prose, or source structure. Recursive writes require explicit
|
||||||
|
`--write --recursive` and preflight the complete input set.
|
||||||
|
|
||||||
|
The LSP intentionally handles open editor snapshots only and uses full-document
|
||||||
|
synchronization. It publishes the configured heuristic catalog but does not
|
||||||
|
publish a document-level evidence score; editor diagnostics remain individual
|
||||||
|
review signals. It does not read or write a document URI, mutate archives,
|
||||||
|
provide prose rewrites, or implement incremental range changes. Only configured
|
||||||
|
ASCII replacements receive quick fixes or formatting edits. Under the default
|
||||||
|
policy this includes printable-ASCII UTS #39 skeletons, which can alter valid
|
||||||
|
Greek, Cyrillic, mathematical, or other text; use `review` for non-mutating
|
||||||
|
diagnostics. Invisible and bidirectional-control findings are diagnostic-only, and
|
||||||
|
configuration reload errors retain the last valid configuration and compiled
|
||||||
|
detector.
|
||||||
|
|
||||||
|
Analysis is local and performs no network lookup. Consequently, it cannot check
|
||||||
|
citations, current facts, provenance, licensing, or similarity to external
|
||||||
|
sources.
|
||||||
|
|
@ -0,0 +1,93 @@
|
||||||
|
# Detector output contracts
|
||||||
|
|
||||||
|
The detector exposes four formats through `--output`. All formats preserve the
|
||||||
|
same deterministic document and finding order. Requested paths retain command
|
||||||
|
line order, directory entries are sorted by filename, archive order follows the
|
||||||
|
container, and findings are sorted by byte span and rule ID.
|
||||||
|
|
||||||
|
## Human
|
||||||
|
|
||||||
|
`--output human` renders every finding as a plain `miette` diagnostic with the
|
||||||
|
source name, exact excerpt and primary span, severity, stable rule code,
|
||||||
|
message, and remediation help. A final line per document reports the low-care
|
||||||
|
evidence score/band and advice, warning, and error counts. The narratable renderer
|
||||||
|
deliberately avoids ANSI escapes.
|
||||||
|
|
||||||
|
## Miette
|
||||||
|
|
||||||
|
`--output miette` is the default. It renders the same typed diagnostics with
|
||||||
|
Miette's graphical Unicode source annotations and ANSI severity colors,
|
||||||
|
followed by the same summary. Each diagnostic starts with a `◆`-marked rule-code
|
||||||
|
title immediately followed by its severity/message, and ends with an empty line
|
||||||
|
that separates it from the next diagnostic or summary. This mode requires the
|
||||||
|
default `fancy-diagnostics` Cargo feature; builds made with
|
||||||
|
`--no-default-features`
|
||||||
|
accept the option but fall back to the plain human renderer. Select `human`,
|
||||||
|
`concise`, or `json` when ANSI escapes are not appropriate.
|
||||||
|
|
||||||
|
## Concise
|
||||||
|
|
||||||
|
`--output concise` emits one line per finding followed by one summary line. A
|
||||||
|
finding line contains the source identity, half-open UTF-8 byte range, severity,
|
||||||
|
rule ID, and message. Recoverable input errors go to stderr in human and concise
|
||||||
|
modes.
|
||||||
|
|
||||||
|
## JSON Lines schema version 2
|
||||||
|
|
||||||
|
`--output json` emits one compact JSON object per line. An analyzed document has
|
||||||
|
`kind: "analysis"` and this shape:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schema_version": 2,
|
||||||
|
"kind": "analysis",
|
||||||
|
"source": "article.md",
|
||||||
|
"summary": {
|
||||||
|
"evidence_score": 17,
|
||||||
|
"evidence_band": "below-threshold",
|
||||||
|
"advice": 1,
|
||||||
|
"warnings": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"total": 1
|
||||||
|
},
|
||||||
|
"findings": [
|
||||||
|
{
|
||||||
|
"rule_id": "unicode-ascii-replacement",
|
||||||
|
"category": "typography",
|
||||||
|
"severity": "advice",
|
||||||
|
"message": "a Unicode character or sequence has a configured ASCII replacement",
|
||||||
|
"help": "Review the character or apply the configured ASCII replacement.",
|
||||||
|
"byte_span": { "start": 8, "end": 11 },
|
||||||
|
"matched_text": "—",
|
||||||
|
"weight": 10.0,
|
||||||
|
"fix": {
|
||||||
|
"byte_span": { "start": 8, "end": 11 },
|
||||||
|
"replacement": "-"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`evidence_band` is `null` when there is no evidence. Spans are zero-based,
|
||||||
|
half-open UTF-8 byte offsets into the exact source snapshot, not character or
|
||||||
|
UTF-16 positions. `fix` is `null` for findings without a safe configured edit.
|
||||||
|
|
||||||
|
A recoverable input failure has `kind: "input-error"`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schema_version": 2,
|
||||||
|
"kind": "input-error",
|
||||||
|
"source": "broken.txt",
|
||||||
|
"message": "input broken.txt is not valid UTF-8"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
JSON records are the only stdout content in JSON mode and never contain ANSI
|
||||||
|
escapes or logs. Fatal argument/configuration/setup failures can occur before
|
||||||
|
the JSON stream starts; they return status 2, leave stdout empty, and report on
|
||||||
|
stderr. Additive fields may appear within schema version 2; incompatible changes
|
||||||
|
require a new schema version. Schema version 2 renamed the version 1 summary
|
||||||
|
fields `confidence_score` and `confidence_band` to `evidence_score` and
|
||||||
|
`evidence_band`; version 1 records are no longer emitted.
|
||||||
|
|
@ -0,0 +1,63 @@
|
||||||
|
# Release procedure
|
||||||
|
|
||||||
|
Releases are built from a clean, reviewed Linux checkout with the exact Rust
|
||||||
|
toolchain recorded in `BUILD-INFO`.
|
||||||
|
|
||||||
|
If publishing Rust packages, publish and verify `john-slop-core` first,
|
||||||
|
`john-slop-lsp` second, and the root `john-slop` facade last. All three versions
|
||||||
|
must remain synchronized; path dependencies carry matching registry versions.
|
||||||
|
Before publication, assemble all three crates in Cargo's dependency order with
|
||||||
|
`scripts/package-workspace.sh --no-verify`. The flag is required until the
|
||||||
|
matching core and LSP versions exist in the registry; after publication, rerun
|
||||||
|
the script without it to verify the registry-form dependency graph.
|
||||||
|
|
||||||
|
1. Ensure `CHANGELOG.md`, `Cargo.toml`, `Cargo.lock`, documentation, and the
|
||||||
|
version badge agree; run every command in `docs/development.md`. Milestone
|
||||||
|
`v0.1.N` completion commits use package version `0.1.N`; regenerate the badge
|
||||||
|
only after updating the manifest and lockfile.
|
||||||
|
2. Run normal Criterion sampling and the fuzz targets appropriate to changes.
|
||||||
|
3. Commit the release preparation and create signed or annotated tag `vX.Y.Z`.
|
||||||
|
4. Check out that tag and build the release archive:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
scripts/package-release.sh X.Y.Z
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Run the packaging command a second time in a fresh checkout with the same
|
||||||
|
Rust version and compare the archive SHA-256 digest.
|
||||||
|
6. Inspect `BUILD-INFO`, `SHA256SUMS`, binary help, archive contents, and recorded
|
||||||
|
sizes. Sign the archive and checksum if a signing identity is available.
|
||||||
|
7. Publish the archive, archive checksum, changelog excerpt, and signature
|
||||||
|
together. Never publish an untagged or dirty build as a release.
|
||||||
|
|
||||||
|
`SOURCE_DATE_EPOCH` comes from the tagged commit. The release profile uses thin
|
||||||
|
LTO, one codegen unit, abort-on-panic, and symbol stripping; GNU tar normalizes
|
||||||
|
entry order, timestamps, ownership, and gzip metadata. Reproducibility still
|
||||||
|
depends on the same target, Rust toolchain, linker, and system libraries, all of
|
||||||
|
which must be recorded or controlled by the release environment.
|
||||||
|
|
||||||
|
The packaging smoke test produced byte-identical archives in two consecutive
|
||||||
|
runs on the Milestone 7 development host and validated both archive and internal
|
||||||
|
checksums.
|
||||||
|
|
||||||
|
## Publish with rust_realease
|
||||||
|
|
||||||
|
Install [`rust_realease`](https://proxy.dandaedre.com/gitea/idle-org/rust_realease),
|
||||||
|
then pass the path of a readable Gitea token file to the repository wrapper:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./release.sh /path/to/token.txt --dry-run
|
||||||
|
./release.sh /path/to/token.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
The wrapper keeps the token out of command-line arguments, exports it as
|
||||||
|
`GITEA_TOKEN` only to `rust_realease`, selects the `john-slop` workspace package,
|
||||||
|
and publishes cargo-binstall archives alongside raw binary artifacts. The
|
||||||
|
project-local `rust-release.toml` maps the SSH development origin to the
|
||||||
|
canonical HTTPS proxy for release API and download operations without modifying
|
||||||
|
the Git remote. A hyphenated Cargo version such as
|
||||||
|
`0.1.27-rc.8` automatically marks the Gitea release as a prerelease. Additional
|
||||||
|
arguments are forwarded, so use `--dry-run` before publication and
|
||||||
|
`--replace-assets` only after reviewing an existing release collision. The
|
||||||
|
checkout must be clean and its local and remote `vX.Y.Z` tag must match the
|
||||||
|
package version as required by `rust_realease`.
|
||||||
|
|
@ -0,0 +1,106 @@
|
||||||
|
# Review signals and evidence methodology
|
||||||
|
|
||||||
|
John Slop finds concrete text that may deserve review. It does **not** determine
|
||||||
|
whether AI wrote a document, identify a model, or assign an authorship
|
||||||
|
probability. A finding names the exact matched text and the configured reason
|
||||||
|
for reviewing it; the evidence score is only a bounded aggregation of those
|
||||||
|
findings.
|
||||||
|
|
||||||
|
This boundary matters because general-purpose AI-text detection is not reliable
|
||||||
|
enough to support an authorship claim. Sadasivan et al. show both practical
|
||||||
|
paraphrasing attacks and a theoretical trade-off between detection and false
|
||||||
|
positives as human and model distributions become closer [1]. A comparative
|
||||||
|
evaluation of 14 detection tools likewise found them neither accurate nor
|
||||||
|
reliable, with performance worsening after obfuscation [2]. Liang et al. found
|
||||||
|
that perplexity-based detectors disproportionately classified non-native English
|
||||||
|
writing as AI-generated [3]. John Slop therefore does not use perplexity,
|
||||||
|
burstiness, grammar, spelling, vocabulary richness, dialect, or English fluency
|
||||||
|
as evidence.
|
||||||
|
|
||||||
|
## Signal tiers
|
||||||
|
|
||||||
|
### Direct response residue
|
||||||
|
|
||||||
|
These are relatively strong, locally explainable signs that conversational
|
||||||
|
material may have been pasted into a document:
|
||||||
|
|
||||||
|
- assistant self-identification, capability, or knowledge-cutoff disclaimers;
|
||||||
|
- `User:` / `Assistant:` transcript scaffolding;
|
||||||
|
- answer or revision prefaces; and
|
||||||
|
- follow-up offers such as asking whether the reader wants an expansion.
|
||||||
|
|
||||||
|
They still do not prove authorship. A quotation, article about assistants, or
|
||||||
|
intentional transcript can contain the same text, so applicability scopes and
|
||||||
|
near-miss fixtures are part of every rule's review.
|
||||||
|
|
||||||
|
### Unfinished editing and source residue
|
||||||
|
|
||||||
|
Unresolved insertion notes, implementation placeholders, omitted-code comments,
|
||||||
|
tutorial narration, and stray Markdown fences are useful low-care signals
|
||||||
|
regardless of how they were produced. These rules ask whether work is complete;
|
||||||
|
they do not infer who or what produced it.
|
||||||
|
|
||||||
|
### Subtle population-level clues
|
||||||
|
|
||||||
|
Some patterns are meaningful across a large corpus but weak in one document.
|
||||||
|
Kobak et al. measured abrupt, population-level increases in selected style words
|
||||||
|
across more than 15 million biomedical abstracts [4]. John Slop represents this
|
||||||
|
only as `excess-vocabulary-cluster`: it requires multiple matches, has weight
|
||||||
|
`1.0`, is occurrence-capped, and says explicitly that it is not authorship
|
||||||
|
evidence. A stock closing has similarly low weight. These clues should prompt a
|
||||||
|
specificity edit at most, never an accusation.
|
||||||
|
|
||||||
|
Repeated transitions and templated phrases sit between the tiers. They activate
|
||||||
|
only on an exact conservative pattern; repetition from one rule is damped and
|
||||||
|
cannot alone create a high evidence band.
|
||||||
|
|
||||||
|
### Verification targets, not offline shape rules
|
||||||
|
|
||||||
|
Generated references and package names can be wrong. Walters and Wilder found
|
||||||
|
fabricated and substantially erroneous citations in a controlled set of
|
||||||
|
generated bibliographies [5]. Spracklen et al. measured package hallucinations
|
||||||
|
in a large code-generation study [6]. But a citation's typography or a package
|
||||||
|
name's shape cannot establish that it is fictitious. Reliable checks require an
|
||||||
|
authoritative catalog and contextual validation, while John Slop analysis is
|
||||||
|
local and makes no network calls. These remain human review targets rather than
|
||||||
|
shipped heuristic rules.
|
||||||
|
|
||||||
|
## How a rule is admitted
|
||||||
|
|
||||||
|
A shipped rule must have a stable ID, a narrow pattern, a rationale, review
|
||||||
|
help, an applicability scope, a finite non-negative weight, and an occurrence
|
||||||
|
cap. It also needs:
|
||||||
|
|
||||||
|
1. a positive fixture showing the intended match;
|
||||||
|
2. a realistic near miss showing what must not match;
|
||||||
|
3. deterministic byte spans and ordering; and
|
||||||
|
4. review of likely harm, especially language-proficiency and demographic
|
||||||
|
proxies.
|
||||||
|
|
||||||
|
Exact residue and fixable Unicode receive higher confidence than broad stylistic
|
||||||
|
clues. New research is used to propose testable review signals, not to import a
|
||||||
|
paper's population-level conclusion into an individual-document verdict.
|
||||||
|
|
||||||
|
## Scoring and editor visibility
|
||||||
|
|
||||||
|
Rule weight contributes bounded evidence to the shared score. Per-rule caps and
|
||||||
|
correlation damping prevent repeated copies of one pattern from dominating it.
|
||||||
|
The number is neither accuracy nor probability; see
|
||||||
|
[Heuristics and evidence scoring](heuristics.md) for the formula.
|
||||||
|
|
||||||
|
LSP visibility is a separate presentation choice. By default the editor shows
|
||||||
|
only configured Unicode findings, which are exact and may have safe replacement
|
||||||
|
edits. `--all-diagnostics` enables the complete detector catalog for testing or
|
||||||
|
deliberate review. `--minimum-diagnostic-weight <WEIGHT>` is the quieter opt-in:
|
||||||
|
it keeps Unicode findings and adds only non-Unicode findings whose configured
|
||||||
|
weight meets the threshold. Filtering affects editor diagnostics, not CLI
|
||||||
|
analysis or scoring.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
1. Sadasivan et al., [Can AI-Generated Text be Reliably Detected?](https://arxiv.org/abs/2303.11156)
|
||||||
|
2. Weber-Wulff et al., [Testing of detection tools for AI-generated text](https://link.springer.com/article/10.1007/s40979-023-00146-z)
|
||||||
|
3. Liang et al., [GPT detectors are biased against non-native English writers](https://pmc.ncbi.nlm.nih.gov/articles/PMC10382961/)
|
||||||
|
4. Kobak et al., [Delving into LLM-assisted writing in biomedical publications through excess vocabulary](https://arxiv.org/abs/2406.07016)
|
||||||
|
5. Walters and Wilder, [Fabrication and errors in the bibliographic citations generated by ChatGPT](https://pmc.ncbi.nlm.nih.gov/articles/PMC10484980/)
|
||||||
|
6. Spracklen et al., [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen)
|
||||||
|
|
@ -0,0 +1,72 @@
|
||||||
|
# Ambiguous and invisible Unicode references
|
||||||
|
|
||||||
|
Gitea maintains the concrete list that most closely matches the requested
|
||||||
|
"ambiguous Unicode characters" catalog:
|
||||||
|
|
||||||
|
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
||||||
|
is the generator input;
|
||||||
|
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
||||||
|
are easier to browse as code;
|
||||||
|
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
||||||
|
combines a locale-specific table with `_common` at runtime; and
|
||||||
|
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
||||||
|
records the upstream source and a separate invisible-rune catalog.
|
||||||
|
|
||||||
|
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
||||||
|
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
||||||
|
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
||||||
|
The array values alternate between a confusable code point and the code point
|
||||||
|
it resembles. Counts across tables are not necessarily unique global mappings.
|
||||||
|
|
||||||
|
Gitea's generator identifies
|
||||||
|
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
||||||
|
as its upstream. For standards-based identifier spoofing, the canonical broader
|
||||||
|
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
||||||
|
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
||||||
|
data file. GitHub separately documents a warning for hidden bidirectional text;
|
||||||
|
that protection is about bidi controls rather than a published Gitea-style
|
||||||
|
locale-aware lookalike table.
|
||||||
|
|
||||||
|
## Implemented John Slop profile
|
||||||
|
|
||||||
|
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
|
||||||
|
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
|
||||||
|
scalar whose single-character skeleton is non-empty printable ASCII, plus nine
|
||||||
|
reviewed punctuation mappings and 95 compatibility width folds. Width folding
|
||||||
|
takes precedence when a UTS #39 visual skeleton would produce a different
|
||||||
|
target, such as fullwidth `I` mapping to `I` rather than `l`. That is 1,884
|
||||||
|
mappings in the current profile.
|
||||||
|
Each entry has a stable ID, source, target, rationale, and enabled switch, and
|
||||||
|
the same asset owns the diagnostic metadata.
|
||||||
|
|
||||||
|
The default `[unicode] confusables = "replace"` policy emits
|
||||||
|
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
|
||||||
|
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
|
||||||
|
`review` for identical findings without edits, or `off` to suppress generated
|
||||||
|
lookalikes. The reviewed punctuation and width-folding mappings remain active
|
||||||
|
under all three settings and can still be disabled individually.
|
||||||
|
|
||||||
|
This broad mapping is intentionally more aggressive than Gitea's locale-aware
|
||||||
|
display warning. Greek, Cyrillic, mathematical, and other characters may be
|
||||||
|
legitimate text, so projects handling multilingual content should prefer
|
||||||
|
`review` unless ASCII normalization is their explicit policy.
|
||||||
|
|
||||||
|
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
|
||||||
|
pinned commit above, including bidi embedding/override and isolate controls.
|
||||||
|
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
|
||||||
|
reported. Each other match receives an exact UTF-8 source span that adapters
|
||||||
|
convert normally. Controls never feed the formatter or LSP quick fixes because
|
||||||
|
silent deletion has no generally safe meaning.
|
||||||
|
|
||||||
|
Regenerate the checked-in policy deterministically with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run -p john-slop-core --example generate_unicode_catalog \
|
||||||
|
> crates/john-slop-core/assets/unicode.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
|
||||||
|
|
||||||
|
Future profile updates must pin their upstream Unicode/data version, retain
|
||||||
|
provenance and licensing, test ordinary multilingual near misses, and renew the
|
||||||
|
hash-bound default-policy sign-off.
|
||||||
|
|
@ -0,0 +1,184 @@
|
||||||
|
# Usage
|
||||||
|
|
||||||
|
## Detector
|
||||||
|
|
||||||
|
Run the detector over one or more paths:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run --bin john-slop -- [--fail-score 0..100] <PATH>...
|
||||||
|
cargo run --bin john-slop -- --stdin [--stdin-filename article.md]
|
||||||
|
```
|
||||||
|
|
||||||
|
Each path may be a UTF-8 text file, a directory, a ZIP archive, or a supported
|
||||||
|
ZIP-based office document (`.docx`, `.xlsx`, or `.pptx`). Directory traversal and
|
||||||
|
archive reading stream one result at a time, so callers do not need to retain an
|
||||||
|
entire input tree in memory.
|
||||||
|
|
||||||
|
`--stdin` reads one bounded strict UTF-8 document from standard input. Supply
|
||||||
|
`--stdin-filename PATH` when extension, language, and path scopes should use a
|
||||||
|
virtual filename. Standard input can be analyzed alongside filesystem paths.
|
||||||
|
Repeat `--ignore GLOB` to skip matching file or archive-member paths. Resource
|
||||||
|
limits can be overridden with `--max-document-bytes`, `--max-archive-bytes`,
|
||||||
|
`--max-expanded-bytes`, `--max-archive-entries`,
|
||||||
|
`--max-compression-ratio`, and `--max-walk-depth`.
|
||||||
|
|
||||||
|
Select output with `--output human`, `--output miette`, `--output concise`, or
|
||||||
|
`--output json`. Miette is the default and uses colored Unicode graphical source
|
||||||
|
annotations when the default `fancy-diagnostics` feature is enabled. Its marked
|
||||||
|
rule-code titles stay attached to their messages, while an empty line separates
|
||||||
|
complete diagnostics; it falls
|
||||||
|
back to human mode without that feature. Human mode renders uncolored plain
|
||||||
|
source excerpts, exact labels, remediation help, and a summary. Concise mode
|
||||||
|
emits stable single-line finding and summary records. JSON mode emits one object
|
||||||
|
per analyzed document with `schema_version: 2`, summary fields, exact byte spans,
|
||||||
|
and every finding. Machine modes never emit ANSI escapes or logs on stdout.
|
||||||
|
|
||||||
|
Findings cover the embedded or overridden heuristic catalog and retain exact
|
||||||
|
UTF-8 byte spans through every output mode. See [Output contracts](output.md)
|
||||||
|
for the human/concise formats and complete JSON Lines schema.
|
||||||
|
|
||||||
|
Exit statuses are stable:
|
||||||
|
|
||||||
|
- `0`: every analyzed document is below the requested evidence-score threshold;
|
||||||
|
- `1`: at least one document reaches the threshold;
|
||||||
|
- `2`: arguments are invalid or an operational/input error prevents completion.
|
||||||
|
|
||||||
|
`--fail-score SCORE` selects an exact inclusive score threshold and
|
||||||
|
defaults to 100. `--fail-level low|medium|high` instead uses the corresponding
|
||||||
|
threshold from the fully resolved configuration. The two options are mutually
|
||||||
|
exclusive. The old `--fail-confidence` spelling remains a visible alias for
|
||||||
|
migration.
|
||||||
|
|
||||||
|
Input failures are isolated per document whenever iteration can continue. Valid
|
||||||
|
documents still emit all findings, while any partial input failure makes the
|
||||||
|
final status `2` (taking precedence over threshold status `1`). Human and
|
||||||
|
concise input errors go to stderr. JSON mode emits versioned `input-error`
|
||||||
|
records on stdout alongside `analysis` records and leaves stderr empty unless a
|
||||||
|
fatal setup failure prevents the output contract from starting.
|
||||||
|
|
||||||
|
The low-care evidence score summarizes observable configured evidence. It is
|
||||||
|
not an AI-authorship probability. See [Heuristics and evidence scoring](heuristics.md) for
|
||||||
|
the formula and [Limitations](limitations.md) before using thresholds in review
|
||||||
|
workflows.
|
||||||
|
|
||||||
|
Configuration layers can be selected with `--config-name NAME` and `--config
|
||||||
|
FILE`; repeat `--diagnostics FILE` to load runtime rule catalogs without
|
||||||
|
recompilation. Command-line resource limits are the highest-precedence input overrides.
|
||||||
|
Use `john-slop config show` to print the resolved TOML and `john-slop config
|
||||||
|
paths` to inspect source precedence. See [Configuration](configuration.md) for
|
||||||
|
the schema and merge rules.
|
||||||
|
|
||||||
|
The LSP publishes Unicode findings by default. Start `john-slop-lsp` with
|
||||||
|
`--all-diagnostics` to publish the complete configured detector catalog, or use
|
||||||
|
`--minimum-diagnostic-weight WEIGHT` to retain Unicode findings and add only
|
||||||
|
non-Unicode findings at or above that weight. This editor-only filter does not
|
||||||
|
change CLI analysis, evidence scores, replacement planning, or formatting.
|
||||||
|
|
||||||
|
## Rust examples
|
||||||
|
|
||||||
|
The [`examples` directory](../examples/README.md) contains runnable Cargo
|
||||||
|
examples for direct text analysis, streaming path input, and replacement edit
|
||||||
|
planning. Run all example targets as part of the normal test matrix with
|
||||||
|
`cargo test --examples --all-features`.
|
||||||
|
|
||||||
|
Its [shared fixture manifest](../examples/fixtures/manifest.toml) pairs several
|
||||||
|
`example.txt` inputs with expected detector rules, replacement counts, and
|
||||||
|
formatted output. The executable fixture and protocol tests validate the
|
||||||
|
library, detector, formatter, and LSP against the shared replacement contract.
|
||||||
|
The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase)
|
||||||
|
prints every current prose finding in graphical Miette form and intentionally
|
||||||
|
exits `1` when its low evidence threshold is reached.
|
||||||
|
|
||||||
|
## Supported input policy
|
||||||
|
|
||||||
|
Input reads use strict UTF-8 and reject NUL-containing text. They do not follow
|
||||||
|
symlinks by default. Limits on files, walk depth, archive entries, expanded
|
||||||
|
bytes, compression ratio, and member paths protect analysis from unbounded or
|
||||||
|
unsafe input. The exact policy is recorded in
|
||||||
|
[ADR 0002](decisions/0002-input-policy-and-limits.md).
|
||||||
|
|
||||||
|
## Formatter
|
||||||
|
|
||||||
|
With no path, `john-slop-fmt` reads one bounded strict-UTF-8 document from stdin
|
||||||
|
and writes only formatted bytes to stdout. With a file path, the default remains
|
||||||
|
read-only and prints the transformed file without changing it:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop-fmt < article.txt > article.ascii.txt
|
||||||
|
john-slop-fmt article.txt
|
||||||
|
john-slop-fmt --check article.txt
|
||||||
|
john-slop-fmt --diff article.txt
|
||||||
|
john-slop-fmt --write article.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
The modes are mutually exclusive:
|
||||||
|
|
||||||
|
- default mode emits formatted text and exits `0`;
|
||||||
|
- `--check` emits nothing and exits `1` when any replacement is needed;
|
||||||
|
- `--diff` emits a unified diff and exits `1` when any replacement is needed;
|
||||||
|
- `--write` emits nothing and atomically replaces changed files, exiting `0`;
|
||||||
|
- argument, configuration, UTF-8, input, or write failures exit `2`.
|
||||||
|
|
||||||
|
All modes use the same embedded/user/named/explicit configuration precedence as
|
||||||
|
the detector. `--config-name NAME`, `--config FILE`, and repeatable
|
||||||
|
`--diagnostics FILE` select those layers. Diagnostic rules do not change the
|
||||||
|
formatter's replacement plan, but catalog validation stays consistent across
|
||||||
|
all adapters.
|
||||||
|
Replacement sources use deterministic longest-match semantics, and formatting
|
||||||
|
twice is byte-for-byte identical to formatting once.
|
||||||
|
|
||||||
|
Directories require `--recursive`, and recursive operation is deliberately
|
||||||
|
limited to `--check --recursive` or `--write --recursive`. Before a recursive
|
||||||
|
write, every input is read, decoded, planned, and edit-validated; a failure
|
||||||
|
prevents all writes. Changed files use a temporary file in the same directory,
|
||||||
|
copy original permissions, sync content, and replace by rename where the
|
||||||
|
platform supports it. Files needing no edit are never rewritten.
|
||||||
|
|
||||||
|
ZIP, DOCX, XLSX, and PPTX mutation is not supported. Extract and format an
|
||||||
|
individual UTF-8 file instead.
|
||||||
|
|
||||||
|
## LSP
|
||||||
|
|
||||||
|
`john-slop-lsp` is a stdio Language Server Protocol 3.17 adapter. Editors launch
|
||||||
|
it without positional arguments:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop-lsp
|
||||||
|
john-slop-lsp --config-name editorial
|
||||||
|
john-slop-lsp --config ./john-slop.toml
|
||||||
|
john-slop-lsp --diagnostics ./diagnostics.toml
|
||||||
|
john-slop-lsp --all-diagnostics
|
||||||
|
john-slop-lsp --minimum-diagnostic-weight 10
|
||||||
|
```
|
||||||
|
|
||||||
|
The server advertises full-document synchronization and quick-fix code actions.
|
||||||
|
It stores only currently open UTF-8 snapshots, their URI, language ID, and
|
||||||
|
monotonically increasing version. On open or an accepted newer full-document
|
||||||
|
change it runs the same configured detector as the CLI and publishes every
|
||||||
|
applicable Unicode finding. `--all-diagnostics` publishes the complete result;
|
||||||
|
`--minimum-diagnostic-weight WEIGHT` adds only non-Unicode findings meeting the
|
||||||
|
threshold. File URI extensions select prose/source and path scopes; editor
|
||||||
|
language IDs provide the prose/source fallback for untitled or extensionless
|
||||||
|
buffers.
|
||||||
|
|
||||||
|
Each diagnostic has source `john-slop`, the stable heuristic rule ID as its
|
||||||
|
code, the configured severity and remediation text, document-version metadata,
|
||||||
|
and an exact UTF-16 range. Advice maps to LSP information, while warning and
|
||||||
|
error retain their corresponding LSP severities.
|
||||||
|
|
||||||
|
Runtime catalog rules are reloaded by
|
||||||
|
`workspace/didChangeConfiguration` together with normal configuration. Quick
|
||||||
|
fixes remain intentionally narrower than diagnostics: they are recomputed
|
||||||
|
from the current snapshot with the shared replacement planner and returned only
|
||||||
|
for findings carrying safe configured ASCII edits. Style, prompt-leakage, and
|
||||||
|
source-review findings are diagnostic-only. A request carrying stale John Slop
|
||||||
|
diagnostic metadata is rejected. Closing a document removes its snapshot and
|
||||||
|
publishes an empty diagnostic set. `workspace/didChangeConfiguration` reloads
|
||||||
|
the normal embedded, user, named, and explicit layers, recompiles the detector,
|
||||||
|
and republishes every open snapshot under the unchanged visibility filter; a failed reload keeps the last valid
|
||||||
|
configuration and detector.
|
||||||
|
|
||||||
|
Stdout is reserved for JSON-RPC framing. The server is quiet by default; pass
|
||||||
|
`--log-json` to emit lifecycle-only JSON Lines to stderr. Those records contain
|
||||||
|
event metadata, URI, and version but never document contents. Complete Helix and
|
||||||
|
Vim setup and validation steps live in [Editor setup](editors.md).
|
||||||
|
|
@ -0,0 +1,100 @@
|
||||||
|
# Runnable Rust examples
|
||||||
|
|
||||||
|
These Cargo examples exercise the same library boundaries used by the three
|
||||||
|
John Slop binaries:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run --example analyze_text
|
||||||
|
cargo run --example analyze_path -- examples/fixtures
|
||||||
|
cargo run --example plan_replacements
|
||||||
|
cargo run --example ascii_replacements
|
||||||
|
```
|
||||||
|
|
||||||
|
- `analyze_text` analyzes an in-memory prose snapshot and prints its explainable
|
||||||
|
findings.
|
||||||
|
- `analyze_path` streams a file, directory, or supported ZIP container. Its
|
||||||
|
optional path defaults to `examples/fixtures`.
|
||||||
|
- `plan_replacements` prints the exact UTF-8 byte edits shared by the formatter
|
||||||
|
and LSP quick fixes.
|
||||||
|
- `ascii_replacements` demonstrates the embedded width-folding catalog across
|
||||||
|
all 95 printable ASCII targets.
|
||||||
|
|
||||||
|
These examples use embedded defaults so their behavior does not depend on a
|
||||||
|
developer's user configuration. Applications that want normal configuration
|
||||||
|
layering should call `john_slop::load_config()` instead.
|
||||||
|
|
||||||
|
## Shared fixture corpus
|
||||||
|
|
||||||
|
[`fixtures/manifest.toml`](fixtures/manifest.toml) describes small inputs shared
|
||||||
|
by detector CLI, formatter, and LSP testing. Several directories deliberately
|
||||||
|
contain an `example.txt`, covering clean prose, copied assistant residue, stock
|
||||||
|
style, safe Unicode punctuation replacements, and diagnostic-only Unicode
|
||||||
|
lookalikes/invisibles. A Rust source example covers source-only rules. Each
|
||||||
|
input has a checked-in expected formatted result; the Unicode review fixture's
|
||||||
|
expected result is intentionally byte-identical because those findings do not
|
||||||
|
authorize rewriting.
|
||||||
|
|
||||||
|
Exercise the currently implemented boundaries with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run --bin john-slop -- --output concise examples/fixtures/assistant/example.txt
|
||||||
|
cargo run --bin john-slop-fmt -- examples/fixtures/typography/example.txt
|
||||||
|
cargo run --bin john-slop-fmt -- --check examples/fixtures/clean/example.txt
|
||||||
|
cargo run --example analyze_path -- examples/fixtures
|
||||||
|
cargo test --test example_fixtures
|
||||||
|
cargo test --test lsp
|
||||||
|
```
|
||||||
|
|
||||||
|
The formatter integration test executes every case and compares stdout with its
|
||||||
|
expected ASCII file. The manifest also names `lsp` as a consumer, and the
|
||||||
|
in-process protocol suite proves that applying LSP edits produces the same bytes
|
||||||
|
as the formatter.
|
||||||
|
|
||||||
|
## Printable ASCII replacement showcase
|
||||||
|
|
||||||
|
[`ascii_replacements.rs`](ascii_replacements.rs) applies the embedded fullwidth
|
||||||
|
mappings for ASCII `!` through `~` and maps ideographic space to ASCII space.
|
||||||
|
Its [Unicode input](ascii-replacement-showcase/fullwidth.txt)
|
||||||
|
and [expected output](ascii-replacement-showcase/ascii.txt) display all 95
|
||||||
|
printable ASCII targets in one place:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run --example ascii_replacements
|
||||||
|
```
|
||||||
|
|
||||||
|
The example reports 95 planned edits on stderr and writes only the transformed
|
||||||
|
fixture to stdout. These compatibility width folds are production defaults;
|
||||||
|
broader script-confusable replacements remain controlled separately by the
|
||||||
|
`[unicode] confusables` policy.
|
||||||
|
|
||||||
|
## Expected-failure diagnostic showcase
|
||||||
|
|
||||||
|
[`diagnostic-showcase.txt`](diagnostic-showcase.txt) contains every currently
|
||||||
|
shipped prose signal, while
|
||||||
|
[`diagnostic-showcase/source.rs`](diagnostic-showcase/source.rs) adds the
|
||||||
|
source-placeholder and tutorial-narration rules plus the reviewed
|
||||||
|
punctuation replacements, a fixable Cyrillic ASCII lookalike, a zero-width space, and a
|
||||||
|
right-to-left override. The adjacent example configuration promotes assistant
|
||||||
|
self-identification from warning to error and lifts the typography occurrence
|
||||||
|
cap for this display, making one run show Miette advice, warning, error,
|
||||||
|
fixable, and deliberately non-fixable control diagnostics together:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo run --bin john-slop -- \
|
||||||
|
--config examples/diagnostic-showcase.toml \
|
||||||
|
--output miette --fail-level low \
|
||||||
|
examples/diagnostic-showcase.txt \
|
||||||
|
examples/diagnostic-showcase/source.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
That command is expected to exit with status `1`: reaching the configured
|
||||||
|
threshold is the behavior under demonstration. The repository test asserts the
|
||||||
|
nonzero result and representative prose/source categories rather than requiring
|
||||||
|
exact equality with an evolving catalog. Keeping new diagnostic kinds visible
|
||||||
|
here is nevertheless part of the intended maintenance workflow. Run the test
|
||||||
|
with capture disabled to print the complete report:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo test --test example_fixtures \
|
||||||
|
diagnostic_showcase_spans_prose_source_and_severities -- --nocapture
|
||||||
|
```
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
//! Stream and analyze every supported document under a path.
|
||||||
|
|
||||||
|
use std::{env, error::Error, path::PathBuf};
|
||||||
|
|
||||||
|
use john_slop::{AnalysisContext, Config, Detector, read_inputs};
|
||||||
|
|
||||||
|
fn main() -> Result<(), Box<dyn Error>> {
|
||||||
|
let path = env::args_os()
|
||||||
|
.nth(1)
|
||||||
|
.map_or_else(|| PathBuf::from("examples/fixtures"), PathBuf::from);
|
||||||
|
let config = Config::embedded();
|
||||||
|
let detector = Detector::new(&config)?;
|
||||||
|
|
||||||
|
for document in read_inputs(&config, &path)? {
|
||||||
|
match document {
|
||||||
|
Ok(document) => {
|
||||||
|
let context = AnalysisContext::for_source(&document.source);
|
||||||
|
let analysis = detector.analyze_with(&document.content, &context);
|
||||||
|
println!(
|
||||||
|
"{}: evidence_score={}, findings={}",
|
||||||
|
document.source,
|
||||||
|
analysis.evidence_score.score(),
|
||||||
|
analysis.findings.len()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Err(error) => eprintln!("input error: {error}"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,30 @@
|
||||||
|
//! Analyze an in-memory document with the embedded heuristic catalog.
|
||||||
|
|
||||||
|
use std::error::Error;
|
||||||
|
|
||||||
|
use john_slop::{AnalysisContext, Config, Detector};
|
||||||
|
|
||||||
|
fn main() -> Result<(), Box<dyn Error>> {
|
||||||
|
let source = "As an AI language model, I can help—but this draft needs review.";
|
||||||
|
let detector = Detector::new(&Config::embedded())?;
|
||||||
|
let analysis = detector.analyze_with(source, &AnalysisContext::for_path("article.md"));
|
||||||
|
let summary = analysis.summary();
|
||||||
|
|
||||||
|
println!(
|
||||||
|
"evidence_score={} band={} findings={}",
|
||||||
|
summary.evidence_score,
|
||||||
|
summary
|
||||||
|
.evidence_band
|
||||||
|
.map_or("none", john_slop::EvidenceBand::as_str),
|
||||||
|
summary.findings.total()
|
||||||
|
);
|
||||||
|
|
||||||
|
for finding in analysis.findings {
|
||||||
|
println!(
|
||||||
|
"{} {}..{} {:?}",
|
||||||
|
finding.rule_id, finding.byte_span.start, finding.byte_span.end, finding.matched_text
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
Fullwidth ASCII showcase:
|
||||||
|
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
Fullwidth ASCII showcase:
|
||||||
|
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
//! Showcase embedded width folding for every printable ASCII target.
|
||||||
|
|
||||||
|
use std::io::Write;
|
||||||
|
|
||||||
|
use john_slop::{Config, apply_replacements, plan_replacements};
|
||||||
|
|
||||||
|
const SOURCE: &str = include_str!("ascii-replacement-showcase/fullwidth.txt");
|
||||||
|
|
||||||
|
fn showcase_config() -> Config {
|
||||||
|
Config::embedded()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||||
|
let config = showcase_config();
|
||||||
|
let edits = plan_replacements(&config, SOURCE);
|
||||||
|
let formatted = apply_replacements(SOURCE, &edits)?;
|
||||||
|
|
||||||
|
eprintln!(
|
||||||
|
"planned {} embedded replacements covering ASCII space through tilde",
|
||||||
|
edits.len()
|
||||||
|
);
|
||||||
|
std::io::stdout().write_all(formatted.as_bytes())?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
const EXPECTED: &str = include_str!("ascii-replacement-showcase/ascii.txt");
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn showcase_maps_each_printable_ascii_target_once() {
|
||||||
|
let config = showcase_config();
|
||||||
|
let edits = plan_replacements(&config, SOURCE);
|
||||||
|
let targets = edits
|
||||||
|
.iter()
|
||||||
|
.map(|edit| edit.replacement.as_bytes()[0])
|
||||||
|
.collect::<BTreeSet<_>>();
|
||||||
|
|
||||||
|
assert_eq!(edits.len(), 95);
|
||||||
|
assert_eq!(targets, (b' '..=b'~').collect());
|
||||||
|
assert_eq!(
|
||||||
|
apply_replacements(SOURCE, &edits).expect("apply non-overlapping showcase edits"),
|
||||||
|
EXPECTED
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,35 @@
|
||||||
|
# Commented configuration examples
|
||||||
|
|
||||||
|
Copy [`config.toml`](config.toml) to the platform path printed by
|
||||||
|
`john-slop config paths` for user-wide policy. Copy [`editorial.toml`](editorial.toml)
|
||||||
|
beside it and select it with `--config-name editorial` for a named overlay.
|
||||||
|
|
||||||
|
Both are deliberately partial. Embedded defaults remain active for fields and
|
||||||
|
stable IDs they do not mention. Inspect the exact merged result before enforcing
|
||||||
|
it:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config show --config-name editorial
|
||||||
|
```
|
||||||
|
|
||||||
|
Project-local policy can use either file with `--config path/to/file.toml`.
|
||||||
|
The CLI can also create the embedded commented base in the current directory:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop config init
|
||||||
|
```
|
||||||
|
|
||||||
|
It preserves an existing `config.toml` unless `--force` is explicit.
|
||||||
|
|
||||||
|
[`diagnostics.toml`](diagnostics.toml) is a complete runtime rule definition.
|
||||||
|
Place it at the `user_diagnostics` path reported by `john-slop config paths` for
|
||||||
|
automatic loading, or keep it with a project and select it explicitly:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
john-slop --diagnostics examples/config/diagnostics.toml article.md
|
||||||
|
john-slop-lsp --diagnostics examples/config/diagnostics.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
Generate the commented starter with `john-slop config init-diagnostics`. The
|
||||||
|
starter rule is disabled until its phrase and metadata are intentionally
|
||||||
|
customized.
|
||||||
|
|
@ -0,0 +1,36 @@
|
||||||
|
# User-wide base layer. `schema_version` is optional in overlays, but spelling
|
||||||
|
# it out makes compatibility expectations explicit.
|
||||||
|
schema_version = 1
|
||||||
|
|
||||||
|
[diagnostics.categories]
|
||||||
|
# Category gates can silence a complete diagnostic family. All four categories
|
||||||
|
# (`typography`, `style`, `prompt-leakage`, and `source-code`) default to true.
|
||||||
|
style = true
|
||||||
|
prompt-leakage = true
|
||||||
|
|
||||||
|
[input]
|
||||||
|
# Bound each decoded document to 4 MiB for this installation.
|
||||||
|
max_document_bytes = 4194304
|
||||||
|
|
||||||
|
[thresholds]
|
||||||
|
# Evidence bands affect labels and `--fail-level`; they are not probabilities.
|
||||||
|
low = 20
|
||||||
|
medium = 50
|
||||||
|
high = 75
|
||||||
|
|
||||||
|
# Arrays merge by stable ID. A known ID needs only the fields being changed.
|
||||||
|
[[replacements]]
|
||||||
|
id = "em-dash"
|
||||||
|
# Keep typographic em dashes instead of offering an ASCII replacement.
|
||||||
|
enabled = false
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "templated-conclusion"
|
||||||
|
# Disable one rule even while its `style` category remains enabled.
|
||||||
|
enabled = false
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "source-placeholder-todo"
|
||||||
|
# Severity changes presentation; weight changes evidence-score contribution.
|
||||||
|
severity = "error"
|
||||||
|
weight = 15.0
|
||||||
|
|
@ -0,0 +1,24 @@
|
||||||
|
# Runtime diagnostic catalog. Add data-driven rules here without rebuilding
|
||||||
|
# John Slop, then pass this file with `--diagnostics`.
|
||||||
|
schema_version = 1
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "custom-review-marker"
|
||||||
|
category = "style"
|
||||||
|
severity = "warning"
|
||||||
|
matcher = "exact-phrase"
|
||||||
|
pattern = "REVIEW BEFORE PUBLISHING"
|
||||||
|
weight = 8.0
|
||||||
|
message = "a project review marker remains in the source"
|
||||||
|
help = "Resolve or remove the marker before publishing."
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = false
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose", "source-code"]
|
||||||
|
extensions = []
|
||||||
|
include = []
|
||||||
|
exclude = []
|
||||||
|
skip_generated = true
|
||||||
|
skip_vendor = true
|
||||||
|
skip_minified = true
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
# Named overlay selected with `--config-name editorial`. It is applied after the
|
||||||
|
# user-wide config.toml and before an explicit `--config` file.
|
||||||
|
|
||||||
|
[thresholds]
|
||||||
|
low = 15
|
||||||
|
medium = 40
|
||||||
|
high = 70
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "templated-filler"
|
||||||
|
weight = 2.0
|
||||||
|
|
||||||
|
# Scope is one value, so supplied arrays replace the embedded scope arrays.
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-self-identification"
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["md", "txt"]
|
||||||
|
include = ["articles/**", "docs/**"]
|
||||||
|
exclude = ["docs/quoted-transcripts/**"]
|
||||||
|
skip_generated = true
|
||||||
|
skip_vendor = true
|
||||||
|
skip_minified = true
|
||||||
|
|
||||||
|
[[replacements]]
|
||||||
|
id = "en-dash"
|
||||||
|
enabled = false
|
||||||
|
|
@ -0,0 +1,11 @@
|
||||||
|
# This example-only override demonstrates Miette's error presentation. The
|
||||||
|
# embedded catalog otherwise classifies this finding as a warning.
|
||||||
|
[[rules]]
|
||||||
|
id = "assistant-self-identification"
|
||||||
|
severity = "error"
|
||||||
|
|
||||||
|
# Let the companion source fixture display every embedded replacement character
|
||||||
|
# instead of stopping at the production catalog's normal per-document cap.
|
||||||
|
[[rules]]
|
||||||
|
id = "unicode-ascii-replacement"
|
||||||
|
occurrence_cap = 10
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
User: Write a polished summary without checking any sources.
|
||||||
|
Assistant: Certainly! Here's a comprehensive answer.
|
||||||
|
|
||||||
|
As an AI language model, I cannot browse the internet. My knowledge cutoff may
|
||||||
|
therefore make this inaccurate. It is important to note that the result is
|
||||||
|
seamless—production-ready, even.
|
||||||
|
|
||||||
|
Moreover, the design is robust.
|
||||||
|
Furthermore, the implementation is scalable.
|
||||||
|
Additionally, the solution is intuitive.
|
||||||
|
|
||||||
|
In conclusion, it is important to use this result as-is.
|
||||||
|
|
@ -0,0 +1,7 @@
|
||||||
|
fn main() {
|
||||||
|
// First, we declare an intentionally incomplete showcase value.
|
||||||
|
// Next, we leave the implementation placeholder visible for review.
|
||||||
|
// TODO: implement
|
||||||
|
let _replacement_characters = "‐ ‑ ‒ – — ― − ﹘ ﹣ -";
|
||||||
|
let _unicode_review_only = "pаssword";
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
Certainly! Here's a polished summary—ready to publish.
|
||||||
|
As an AI language model, I cannot browse the internet.
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
Certainly! Here's a polished summary-ready to publish.
|
||||||
|
As an AI language model, I cannot browse the internet.
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
The release checklist names an owner, a deadline, and a verification command.
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
The release checklist names an owner, a deadline, and a verification command.
|
||||||
|
|
@ -0,0 +1,51 @@
|
||||||
|
schema_version = 1
|
||||||
|
targets = ["cli", "fmt", "lsp"]
|
||||||
|
|
||||||
|
[[cases]]
|
||||||
|
path = "clean/example.txt"
|
||||||
|
formatted = "clean/formatted.txt"
|
||||||
|
expected_rules = []
|
||||||
|
expected_replacements = 0
|
||||||
|
|
||||||
|
[[cases]]
|
||||||
|
path = "assistant/example.txt"
|
||||||
|
formatted = "assistant/formatted.txt"
|
||||||
|
expected_rules = [
|
||||||
|
"assistant-answer-preface",
|
||||||
|
"assistant-capability-disclaimer",
|
||||||
|
"assistant-self-identification",
|
||||||
|
"unicode-ascii-replacement",
|
||||||
|
]
|
||||||
|
expected_replacements = 1
|
||||||
|
|
||||||
|
[[cases]]
|
||||||
|
path = "style/example.txt"
|
||||||
|
formatted = "style/formatted.txt"
|
||||||
|
expected_rules = [
|
||||||
|
"repeated-transition",
|
||||||
|
"templated-conclusion",
|
||||||
|
"templated-filler",
|
||||||
|
]
|
||||||
|
expected_replacements = 0
|
||||||
|
|
||||||
|
[[cases]]
|
||||||
|
path = "typography/example.txt"
|
||||||
|
formatted = "typography/formatted.txt"
|
||||||
|
expected_rules = ["unicode-ascii-replacement"]
|
||||||
|
expected_replacements = 3
|
||||||
|
|
||||||
|
[[cases]]
|
||||||
|
path = "unicode-review/example.txt"
|
||||||
|
formatted = "unicode-review/formatted.txt"
|
||||||
|
expected_rules = ["unicode-ascii-replacement", "unicode-invisible-control"]
|
||||||
|
expected_replacements = 7
|
||||||
|
|
||||||
|
[[cases]]
|
||||||
|
path = "source/example.rs"
|
||||||
|
formatted = "source/formatted.rs"
|
||||||
|
expected_rules = [
|
||||||
|
"source-placeholder-todo",
|
||||||
|
"source-tutorial-narration",
|
||||||
|
"unicode-ascii-replacement",
|
||||||
|
]
|
||||||
|
expected_replacements = 1
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
fn parse() {
|
||||||
|
// First, we create the parser.
|
||||||
|
// TODO: implement the parser
|
||||||
|
let status = "draft—only";
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
fn parse() {
|
||||||
|
// First, we create the parser.
|
||||||
|
// TODO: implement the parser
|
||||||
|
let status = "draft-only";
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
Moreover, the draft repeats a transition.
|
||||||
|
Furthermore, it repeats the same pattern.
|
||||||
|
It is important to note that the claim still needs a source.
|
||||||
|
In conclusion, we can publish it without adding specifics.
|
||||||
|
|
@ -0,0 +1,4 @@
|
||||||
|
Moreover, the draft repeats a transition.
|
||||||
|
Furthermore, it repeats the same pattern.
|
||||||
|
It is important to note that the claim still needs a source.
|
||||||
|
In conclusion, we can publish it without adding specifics.
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
Résumé — coût 5−2; portée – limitée.
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
Résumé - coût 5-2; portée - limitée.
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
The pаssword contains lookalike and invisible characters.
|
||||||
|
Catalog sample: а𝐀!𝟘ⅿ🯹.
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
The password contains lookalike and invisible characters.
|
||||||
|
Catalog sample: aA!Orn9.
|
||||||
|
|
@ -0,0 +1,18 @@
|
||||||
|
//! Inspect the edits shared by detector, formatter, and LSP adapters.
|
||||||
|
|
||||||
|
use john_slop::{Config, apply_replacements, plan_replacements};
|
||||||
|
|
||||||
|
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||||
|
let source = "Review—then subtract 5−2.";
|
||||||
|
let edits = plan_replacements(&Config::embedded(), source);
|
||||||
|
|
||||||
|
println!("source: {source}");
|
||||||
|
for edit in &edits {
|
||||||
|
println!(
|
||||||
|
"replace bytes {}..{} with {:?}",
|
||||||
|
edit.byte_span.start, edit.byte_span.end, edit.replacement
|
||||||
|
);
|
||||||
|
}
|
||||||
|
println!("result: {}", apply_replacements(source, &edits)?);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,52 @@
|
||||||
|
[package]
|
||||||
|
name = "john-slop-fuzz"
|
||||||
|
version = "0.0.0"
|
||||||
|
publish = false
|
||||||
|
edition = "2024"
|
||||||
|
|
||||||
|
[package.metadata]
|
||||||
|
cargo-fuzz = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
john-slop = { path = "..", default-features = false, features = ["lsp"] }
|
||||||
|
libfuzzer-sys = "0.4.13"
|
||||||
|
tempfile = "3.27.0"
|
||||||
|
toml = "1.1.6"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "config-merge"
|
||||||
|
path = "fuzz_targets/config_merge.rs"
|
||||||
|
test = false
|
||||||
|
doc = false
|
||||||
|
bench = false
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "rule-loading"
|
||||||
|
path = "fuzz_targets/rule_loading.rs"
|
||||||
|
test = false
|
||||||
|
doc = false
|
||||||
|
bench = false
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "utf8-span-mapping"
|
||||||
|
path = "fuzz_targets/utf8_span_mapping.rs"
|
||||||
|
test = false
|
||||||
|
doc = false
|
||||||
|
bench = false
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "replacement-planning"
|
||||||
|
path = "fuzz_targets/replacement_planning.rs"
|
||||||
|
test = false
|
||||||
|
doc = false
|
||||||
|
bench = false
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "archive-metadata"
|
||||||
|
path = "fuzz_targets/archive_metadata.rs"
|
||||||
|
test = false
|
||||||
|
doc = false
|
||||||
|
bench = false
|
||||||
|
|
||||||
|
[workspace]
|
||||||
|
members = ["."]
|
||||||
|
|
@ -0,0 +1,21 @@
|
||||||
|
# Fuzzing
|
||||||
|
|
||||||
|
The five harnesses exercise configuration layering, complete rule loading,
|
||||||
|
UTF-8-to-UTF-16 span conversion, replacement planning/application, and hostile
|
||||||
|
ZIP bytes under deliberately small resource limits.
|
||||||
|
|
||||||
|
Install `cargo-fuzz` and run one target from the repository root:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cargo install cargo-fuzz --locked
|
||||||
|
cargo fuzz run config-merge
|
||||||
|
cargo fuzz run rule-loading
|
||||||
|
cargo fuzz run utf8-span-mapping
|
||||||
|
cargo fuzz run replacement-planning
|
||||||
|
cargo fuzz run archive-metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
Checked-in seeds cover a valid partial override, a complete rule catalog,
|
||||||
|
multibyte/CRLF text, overlapping replacement opportunities, and malformed ZIP
|
||||||
|
metadata. Crashes and minimized artifacts are local build output and must not be
|
||||||
|
committed until converted into a reviewed regression test.
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
PKtruncated-central-directory
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
[[rules]]
|
||||||
|
id = "templated-filler"
|
||||||
|
enabled = false
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
one—two–three−four‐five
|
||||||
|
|
@ -0,0 +1,31 @@
|
||||||
|
schema_version = 1
|
||||||
|
replacements = []
|
||||||
|
|
||||||
|
[thresholds]
|
||||||
|
low = 20
|
||||||
|
medium = 50
|
||||||
|
high = 75
|
||||||
|
|
||||||
|
[input]
|
||||||
|
max_document_bytes = 1024
|
||||||
|
max_archive_bytes = 2048
|
||||||
|
max_expanded_bytes = 4096
|
||||||
|
max_archive_entries = 4
|
||||||
|
max_compression_ratio = 10
|
||||||
|
max_walk_depth = 4
|
||||||
|
|
||||||
|
[[rules]]
|
||||||
|
id = "seed-rule"
|
||||||
|
category = "style"
|
||||||
|
severity = "advice"
|
||||||
|
matcher = "regex"
|
||||||
|
pattern = "seed+"
|
||||||
|
weight = 1.0
|
||||||
|
message = "seed match"
|
||||||
|
help = "review the seed"
|
||||||
|
occurrence_cap = 1
|
||||||
|
enabled = true
|
||||||
|
|
||||||
|
[rules.scope]
|
||||||
|
content = ["prose"]
|
||||||
|
extensions = ["txt"]
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
alpha
|
||||||
|
é😀é
|
||||||
|
|
@ -0,0 +1,27 @@
|
||||||
|
#![no_main]
|
||||||
|
|
||||||
|
use std::fs;
|
||||||
|
|
||||||
|
use john_slop::{Config, read_inputs};
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
|
||||||
|
fuzz_target!(|data: &[u8]| {
|
||||||
|
let Ok(directory) = tempfile::tempdir() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let path = directory.path().join("input.zip");
|
||||||
|
if fs::write(&path, data).is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let mut config = Config::embedded();
|
||||||
|
config.input.max_archive_bytes = 64 * 1024;
|
||||||
|
config.input.max_expanded_bytes = 128 * 1024;
|
||||||
|
config.input.max_document_bytes = 16 * 1024;
|
||||||
|
config.input.max_archive_entries = 32;
|
||||||
|
config.input.max_compression_ratio = 20;
|
||||||
|
if let Ok(documents) = read_inputs(&config, &path) {
|
||||||
|
for document in documents.take(33) {
|
||||||
|
let _ = document;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,23 @@
|
||||||
|
#![no_main]
|
||||||
|
|
||||||
|
use std::fs;
|
||||||
|
|
||||||
|
use john_slop::{ConfigLoader, ConfigPaths};
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
|
||||||
|
fuzz_target!(|data: &[u8]| {
|
||||||
|
let Ok(directory) = tempfile::tempdir() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let explicit = directory.path().join("fuzz.toml");
|
||||||
|
if fs::write(&explicit, data).is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let loader = ConfigLoader::from_paths(ConfigPaths {
|
||||||
|
directory: directory.path().to_path_buf(),
|
||||||
|
base: directory.path().join("absent-base.toml"),
|
||||||
|
named: Some(directory.path().join("absent-named.toml")),
|
||||||
|
explicit: Some(explicit),
|
||||||
|
});
|
||||||
|
let _ = loader.load();
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,17 @@
|
||||||
|
#![no_main]
|
||||||
|
|
||||||
|
use john_slop::{Config, apply_replacements, plan_replacements};
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
|
||||||
|
fuzz_target!(|data: &[u8]| {
|
||||||
|
let Ok(source) = std::str::from_utf8(data) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let config = Config::embedded();
|
||||||
|
let edits = plan_replacements(&config, source);
|
||||||
|
let Ok(formatted) = apply_replacements(source, &edits) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let second = plan_replacements(&config, &formatted);
|
||||||
|
let _ = apply_replacements(&formatted, &second);
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
#![no_main]
|
||||||
|
|
||||||
|
use john_slop::{Config, Detector};
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
|
||||||
|
fuzz_target!(|data: &[u8]| {
|
||||||
|
let Ok(source) = std::str::from_utf8(data) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let Ok(config) = toml::from_str::<Config>(source) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
if config.validate().is_ok() {
|
||||||
|
let _ = Detector::new(&config);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,19 @@
|
||||||
|
#![no_main]
|
||||||
|
|
||||||
|
use john_slop::lsp::LineIndex;
|
||||||
|
use libfuzzer_sys::fuzz_target;
|
||||||
|
|
||||||
|
fuzz_target!(|data: &[u8]| {
|
||||||
|
let Ok(source) = std::str::from_utf8(data) else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let index = LineIndex::new(source);
|
||||||
|
for offset in 0..=source.len().saturating_add(1) {
|
||||||
|
let _ = index.position(offset);
|
||||||
|
}
|
||||||
|
if !source.is_empty() {
|
||||||
|
let midpoint = source.len() / 2;
|
||||||
|
let _ = index.range(midpoint..source.len());
|
||||||
|
let _ = index.range(source.len()..midpoint);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
@ -0,0 +1,43 @@
|
||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: $0 <gitea-token-file> [rust_realease options...]" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
[ "$#" -ge 1 ] || usage
|
||||||
|
token_file=$1
|
||||||
|
shift
|
||||||
|
|
||||||
|
if [ ! -f "$token_file" ] || [ ! -r "$token_file" ]; then
|
||||||
|
echo "token file is not a readable regular file: $token_file" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v rust_realease >/dev/null 2>&1; then
|
||||||
|
echo "rust_realease is not installed or not available in PATH" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
GITEA_TOKEN=
|
||||||
|
IFS= read -r GITEA_TOKEN <"$token_file" || :
|
||||||
|
if [ -z "$GITEA_TOKEN" ]; then
|
||||||
|
echo "token file is empty: $token_file" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
export GITEA_TOKEN
|
||||||
|
|
||||||
|
repository_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||||
|
manifest_version=$(sed -n 's/^version = "\([^"]*\)"/\1/p' "$repository_dir/Cargo.toml" | head -n 1)
|
||||||
|
|
||||||
|
set -- \
|
||||||
|
"$repository_dir" \
|
||||||
|
--package john-slop \
|
||||||
|
--artifact-format both \
|
||||||
|
"$@"
|
||||||
|
case "$manifest_version" in
|
||||||
|
*-*) set -- "$@" --prerelease ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
exec rust_realease "$@"
|
||||||
|
|
@ -0,0 +1,3 @@
|
||||||
|
[[repository.fallbacks]]
|
||||||
|
origin = "git@git.dandaedre.com:idle-org/john-slop.git"
|
||||||
|
fallback-url = "https://proxy.dandaedre.com/gitea/idle-org/john-slop"
|
||||||
|
|
@ -0,0 +1,270 @@
|
||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat >&2 <<'EOF'
|
||||||
|
usage: $0 <version|test|coverage> [options]
|
||||||
|
|
||||||
|
Render one deterministic SVG badge. Defaults:
|
||||||
|
version --text version --value unknown --color #007ec6
|
||||||
|
test --text tests --value passing --color #4c1
|
||||||
|
coverage --text coverage --value - --color #9f9f9f --suffix %
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--text TEXT left-hand label
|
||||||
|
--value VALUE right-hand value
|
||||||
|
--color COLOR value background (#RGB or #RRGGBB)
|
||||||
|
--size PX font size, from 6 through 24 (default: 11)
|
||||||
|
--height PX badge height, from 14 through 64 (default: 20)
|
||||||
|
--label-width PX fixed label width instead of automatic sizing
|
||||||
|
--value-width PX fixed value width instead of automatic sizing
|
||||||
|
--suffix TEXT text appended to the displayed value
|
||||||
|
--output PATH output file (default: coverage/<kind>.svg)
|
||||||
|
-h, --help show this help
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
./ci-coverage-badges.sh version --value 1.2.3
|
||||||
|
./ci-coverage-badges.sh test --value failing
|
||||||
|
./ci-coverage-badges.sh coverage --value 87.4
|
||||||
|
./ci-coverage-badges.sh version --text release --value rc.12 --size 12 --color '#8a2be2'
|
||||||
|
EOF
|
||||||
|
exit "${1:-2}"
|
||||||
|
}
|
||||||
|
|
||||||
|
[ "$#" -ge 1 ] || usage
|
||||||
|
kind=$1
|
||||||
|
shift
|
||||||
|
|
||||||
|
case "$kind" in
|
||||||
|
version)
|
||||||
|
text=version
|
||||||
|
value=unknown
|
||||||
|
color='#007ec6'
|
||||||
|
suffix=
|
||||||
|
output=coverage/version.svg
|
||||||
|
;;
|
||||||
|
test)
|
||||||
|
text=tests
|
||||||
|
value=passing
|
||||||
|
color=
|
||||||
|
suffix=
|
||||||
|
output=coverage/test.svg
|
||||||
|
;;
|
||||||
|
coverage)
|
||||||
|
text=coverage
|
||||||
|
value=-
|
||||||
|
color=
|
||||||
|
suffix=%
|
||||||
|
output=coverage/coverage.svg
|
||||||
|
;;
|
||||||
|
-h|--help) usage 0 ;;
|
||||||
|
*)
|
||||||
|
echo "unsupported badge kind: $kind" >&2
|
||||||
|
usage
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
font_size=11
|
||||||
|
height=20
|
||||||
|
label_width=
|
||||||
|
value_width=
|
||||||
|
color_set=false
|
||||||
|
|
||||||
|
while [ "$#" -gt 0 ]; do
|
||||||
|
option=$1
|
||||||
|
case "$option" in
|
||||||
|
--text|--value|--color|--size|--height|--label-width|--value-width|--suffix|--output)
|
||||||
|
[ "$#" -ge 2 ] || {
|
||||||
|
echo "missing value for $option" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
option_value=$2
|
||||||
|
shift 2
|
||||||
|
case "$option" in
|
||||||
|
--text) text=$option_value ;;
|
||||||
|
--value) value=$option_value ;;
|
||||||
|
--color) color=$option_value; color_set=true ;;
|
||||||
|
--size) font_size=$option_value ;;
|
||||||
|
--height) height=$option_value ;;
|
||||||
|
--label-width) label_width=$option_value ;;
|
||||||
|
--value-width) value_width=$option_value ;;
|
||||||
|
--suffix) suffix=$option_value ;;
|
||||||
|
--output) output=$option_value ;;
|
||||||
|
esac
|
||||||
|
;;
|
||||||
|
-h|--help) usage 0 ;;
|
||||||
|
*)
|
||||||
|
echo "unknown option: $option" >&2
|
||||||
|
usage
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
validate_text() {
|
||||||
|
field_name=$1
|
||||||
|
field_value=$2
|
||||||
|
if ! printf '%sX\n' "$field_value" | LC_ALL=C awk '
|
||||||
|
BEGIN { valid = 1 }
|
||||||
|
NR != 1 || $0 !~ /^[ -~]+X$/ || length($0) <= 1 { valid = 0 }
|
||||||
|
END { exit(valid == 0 ? 1 : 0) }
|
||||||
|
'; then
|
||||||
|
echo "$field_name must contain printable ASCII characters: $field_value" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
validate_integer() {
|
||||||
|
field_name=$1
|
||||||
|
field_value=$2
|
||||||
|
minimum=$3
|
||||||
|
maximum=$4
|
||||||
|
case "$field_value" in
|
||||||
|
''|*[!0-9]*)
|
||||||
|
echo "$field_name must be an integer from $minimum through $maximum: $field_value" >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
if [ "$field_value" -lt "$minimum" ] || [ "$field_value" -gt "$maximum" ]; then
|
||||||
|
echo "$field_name must be an integer from $minimum through $maximum: $field_value" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
validate_text text "$text"
|
||||||
|
validate_text value "$value"
|
||||||
|
if [ -n "$suffix" ]; then
|
||||||
|
validate_text suffix "$suffix"
|
||||||
|
fi
|
||||||
|
validate_integer size "$font_size" 6 24
|
||||||
|
validate_integer height "$height" 14 64
|
||||||
|
if [ -n "$label_width" ]; then
|
||||||
|
validate_integer label-width "$label_width" 20 1000
|
||||||
|
fi
|
||||||
|
if [ -n "$value_width" ]; then
|
||||||
|
validate_integer value-width "$value_width" 20 1000
|
||||||
|
fi
|
||||||
|
case "$output" in
|
||||||
|
''|*/) echo "output must name an SVG file: $output" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
case "$kind" in
|
||||||
|
test)
|
||||||
|
if [ "$color_set" = false ]; then
|
||||||
|
case "$value" in
|
||||||
|
passing) color='#4c1' ;;
|
||||||
|
failing) color='#e05d44' ;;
|
||||||
|
*) color='#9f9f9f' ;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
coverage)
|
||||||
|
if [ "$value" = '-' ]; then
|
||||||
|
[ "$color_set" = true ] || color='#9f9f9f'
|
||||||
|
else
|
||||||
|
if ! awk -v candidate="$value" 'BEGIN {
|
||||||
|
valid = candidate ~ /^[0-9]+([.][0-9]+)?$/ && candidate >= 0 && candidate <= 100
|
||||||
|
exit(valid ? 0 : 1)
|
||||||
|
}'; then
|
||||||
|
echo "coverage value must be - or a number from 0 through 100: $value" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
value=$(awk -v candidate="$value" 'BEGIN { printf "%.0f", candidate }')
|
||||||
|
if [ "$color_set" = false ]; then
|
||||||
|
if [ "$value" -ge 90 ]; then
|
||||||
|
color='#4c1'
|
||||||
|
elif [ "$value" -ge 70 ]; then
|
||||||
|
color='#dfb317'
|
||||||
|
else
|
||||||
|
color='#e05d44'
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
case "$color" in
|
||||||
|
\#[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]|\#[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]) ;;
|
||||||
|
*) echo "color must be #RGB or #RRGGBB: $color" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
display_value=$value$suffix
|
||||||
|
if [ -z "$label_width" ]; then
|
||||||
|
label_width=$(printf '%s\\n' "$text" | awk -v size="$font_size" '{
|
||||||
|
width = int(length($0) * size * 0.64 + 14)
|
||||||
|
print (width < 35 ? 35 : width)
|
||||||
|
}')
|
||||||
|
fi
|
||||||
|
if [ -z "$value_width" ]; then
|
||||||
|
value_width=$(printf '%s\\n' "$display_value" | awk -v size="$font_size" '{
|
||||||
|
width = int(length($0) * size * 0.64 + 14)
|
||||||
|
print (width < 35 ? 35 : width)
|
||||||
|
}')
|
||||||
|
fi
|
||||||
|
|
||||||
|
badge_width=$((label_width + value_width))
|
||||||
|
label_center=$(awk -v width="$label_width" 'BEGIN { printf "%.1f", width / 2 }')
|
||||||
|
value_center=$(awk -v left="$label_width" -v width="$value_width" 'BEGIN {
|
||||||
|
printf "%.1f", left + width / 2
|
||||||
|
}')
|
||||||
|
vertical_center=$(awk -v badge_height="$height" 'BEGIN { printf "%.1f", badge_height / 2 }')
|
||||||
|
|
||||||
|
xml_escape() {
|
||||||
|
sed \
|
||||||
|
-e 's/&/\&/g' \
|
||||||
|
-e 's/</\</g' \
|
||||||
|
-e 's/>/\>/g' \
|
||||||
|
-e 's/"/\"/g'
|
||||||
|
}
|
||||||
|
|
||||||
|
sed_replacement_escape() {
|
||||||
|
sed -e 's/[\\&|]/\\&/g'
|
||||||
|
}
|
||||||
|
|
||||||
|
escaped_text=$(printf '%s' "$text" | xml_escape | sed_replacement_escape)
|
||||||
|
escaped_value=$(printf '%s' "$display_value" | xml_escape | sed_replacement_escape)
|
||||||
|
escaped_title=$(printf '%s: %s' "$text" "$display_value" | xml_escape | sed_replacement_escape)
|
||||||
|
|
||||||
|
output_dir=${output%/*}
|
||||||
|
if [ "$output_dir" != "$output" ]; then
|
||||||
|
mkdir -p "$output_dir"
|
||||||
|
fi
|
||||||
|
temporary="${output}.tmp.$$"
|
||||||
|
trap 'rm -f "$temporary"' EXIT HUP INT TERM
|
||||||
|
|
||||||
|
sed \
|
||||||
|
-e "s/@BADGE_WIDTH@/$badge_width/g" \
|
||||||
|
-e "s/@HEIGHT@/$height/g" \
|
||||||
|
-e "s/@LABEL_WIDTH@/$label_width/g" \
|
||||||
|
-e "s/@VALUE_WIDTH@/$value_width/g" \
|
||||||
|
-e "s/@LABEL_CENTER@/$label_center/g" \
|
||||||
|
-e "s/@VALUE_CENTER@/$value_center/g" \
|
||||||
|
-e "s/@VERTICAL_CENTER@/$vertical_center/g" \
|
||||||
|
-e "s/@FONT_SIZE@/$font_size/g" \
|
||||||
|
-e "s/@COLOR@/$color/g" \
|
||||||
|
-e "s|@TITLE@|$escaped_title|g" \
|
||||||
|
-e "s|@TEXT@|$escaped_text|g" \
|
||||||
|
-e "s|@VALUE@|$escaped_value|g" \
|
||||||
|
>"$temporary" <<'EOF'
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" width="@BADGE_WIDTH@" height="@HEIGHT@" role="img" aria-label="@TITLE@">
|
||||||
|
<title>@TITLE@</title>
|
||||||
|
<linearGradient id="s" x2="0" y2="100%">
|
||||||
|
<stop offset="0" stop-color="#fff" stop-opacity=".7"/>
|
||||||
|
<stop offset=".1" stop-color="#aaa" stop-opacity=".1"/>
|
||||||
|
<stop offset=".9" stop-opacity=".3"/>
|
||||||
|
<stop offset="1" stop-opacity=".5"/>
|
||||||
|
</linearGradient>
|
||||||
|
<clipPath id="r"><rect width="@BADGE_WIDTH@" height="@HEIGHT@" rx="3"/></clipPath>
|
||||||
|
<g clip-path="url(#r)">
|
||||||
|
<rect width="@LABEL_WIDTH@" height="@HEIGHT@" fill="#555"/>
|
||||||
|
<rect x="@LABEL_WIDTH@" width="@VALUE_WIDTH@" height="@HEIGHT@" fill="@COLOR@"/>
|
||||||
|
<rect width="@BADGE_WIDTH@" height="@HEIGHT@" fill="url(#s)"/>
|
||||||
|
</g>
|
||||||
|
<g fill="#fff" text-anchor="middle" dominant-baseline="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="@FONT_SIZE@">
|
||||||
|
<text x="@LABEL_CENTER@" y="@VERTICAL_CENTER@">@TEXT@</text>
|
||||||
|
<text x="@VALUE_CENTER@" y="@VERTICAL_CENTER@">@VALUE@</text>
|
||||||
|
</g>
|
||||||
|
</svg>
|
||||||
|
EOF
|
||||||
|
|
||||||
|
mv "$temporary" "$output"
|
||||||
|
trap - EXIT HUP INT TERM
|
||||||
|
|
@ -0,0 +1,63 @@
|
||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
repository_dir=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
||||||
|
cd "$repository_dir"
|
||||||
|
|
||||||
|
manifest_version=$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n 1)
|
||||||
|
release_version=${1:-$manifest_version}
|
||||||
|
if [ "$release_version" != "$manifest_version" ]; then
|
||||||
|
echo "release version $release_version does not match Cargo.toml $manifest_version" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
target_triple=$(rustc -vV | sed -n 's/^host: //p')
|
||||||
|
source_date_epoch=$(git log -1 --format=%ct)
|
||||||
|
export SOURCE_DATE_EPOCH="$source_date_epoch"
|
||||||
|
|
||||||
|
bundle_name="john-slop-${release_version}-${target_triple}"
|
||||||
|
bundle_dir="dist/${bundle_name}"
|
||||||
|
archive_path="dist/${bundle_name}.tar.gz"
|
||||||
|
|
||||||
|
if [ -e "$bundle_dir" ] || [ -e "$archive_path" ]; then
|
||||||
|
echo "release output already exists for $bundle_name; move it aside first" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$bundle_dir"
|
||||||
|
cargo build --locked --release --all-features
|
||||||
|
|
||||||
|
for binary in john-slop john-slop-fmt john-slop-lsp; do
|
||||||
|
cp "target/release/$binary" "$bundle_dir/$binary"
|
||||||
|
done
|
||||||
|
cp README.md CHANGELOG.md LICENSE "$bundle_dir/"
|
||||||
|
|
||||||
|
commit=$(git rev-parse HEAD)
|
||||||
|
rust_version=$(rustc --version)
|
||||||
|
cargo_version=$(cargo --version)
|
||||||
|
{
|
||||||
|
echo "version=$release_version"
|
||||||
|
echo "commit=$commit"
|
||||||
|
echo "target=$target_triple"
|
||||||
|
echo "source_date_epoch=$source_date_epoch"
|
||||||
|
echo "rustc=$rust_version"
|
||||||
|
echo "cargo=$cargo_version"
|
||||||
|
} >"$bundle_dir/BUILD-INFO"
|
||||||
|
|
||||||
|
(
|
||||||
|
cd "$bundle_dir"
|
||||||
|
sha256sum john-slop john-slop-fmt john-slop-lsp >SHA256SUMS
|
||||||
|
)
|
||||||
|
|
||||||
|
tar \
|
||||||
|
--sort=name \
|
||||||
|
--mtime="@$source_date_epoch" \
|
||||||
|
--owner=0 \
|
||||||
|
--group=0 \
|
||||||
|
--numeric-owner \
|
||||||
|
-cf - \
|
||||||
|
-C dist \
|
||||||
|
"$bundle_name" | gzip -n >"$archive_path"
|
||||||
|
sha256sum "$archive_path" >"${archive_path}.sha256"
|
||||||
|
|
||||||
|
echo "$archive_path"
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue