Compare commits
3 Commits
main
...
poc-replac
| Author | SHA1 | Date |
|---|---|---|
|
|
f8dca15b90 | |
|
|
c10f2b9559 | |
|
|
d141f4e0c2 |
|
|
@ -4,7 +4,9 @@
|
||||||
debug/
|
debug/
|
||||||
target/
|
target/
|
||||||
|
|
||||||
# Cargo.lock is tracked because this package ships executable binaries.
|
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
|
||||||
|
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
|
||||||
|
Cargo.lock
|
||||||
|
|
||||||
# These are backup files generated by rustfmt
|
# These are backup files generated by rustfmt
|
||||||
**/*.rs.bk
|
**/*.rs.bk
|
||||||
|
|
@ -17,19 +19,4 @@ target/
|
||||||
# Added by cargo
|
# Added by cargo
|
||||||
|
|
||||||
/target
|
/target
|
||||||
/dist/
|
bob/
|
||||||
|
|
||||||
# Local implementation journal (intentionally not committed)
|
|
||||||
/recap.md
|
|
||||||
|
|
||||||
# Generated locally or in CI by Cargo Tarpaulin and the badge scripts
|
|
||||||
/tarpaulin-report.html
|
|
||||||
/coverage/
|
|
||||||
|
|
||||||
# Legacy cargo-cyclonedx intermediates may remain in reused CI workspaces.
|
|
||||||
**/sbom.cdx.json
|
|
||||||
|
|
||||||
# cargo-fuzz runtime output
|
|
||||||
/fuzz/artifacts/
|
|
||||||
/fuzz/coverage/
|
|
||||||
/fuzz/target/
|
|
||||||
|
|
|
||||||
277
AGENTS.md
277
AGENTS.md
|
|
@ -1,277 +0,0 @@
|
||||||
# John Slop contributor guide
|
|
||||||
|
|
||||||
## Mission and product boundaries
|
|
||||||
|
|
||||||
John Slop is a transparent, local-first heuristic linter for low-effort or
|
|
||||||
insufficiently reviewed text and source code. It reports concrete signals that
|
|
||||||
are often associated with careless AI-assisted work. It must not claim to prove
|
|
||||||
that a person used AI, identify a model, or infer authorship from writing style.
|
|
||||||
|
|
||||||
Treat the **low-care evidence score** as the bounded strength of configured
|
|
||||||
signals, not as the probability that content is AI-generated. User-facing language must say
|
|
||||||
"suspicious", "low-care signal", or "needs review" rather than making an
|
|
||||||
accusation. A result must always be explainable from the findings that produced
|
|
||||||
it.
|
|
||||||
|
|
||||||
The formatter scope is deliberately narrow: configured Unicode characters with
|
|
||||||
ASCII replacements. The generated UTS #39 profile is broad and therefore must
|
|
||||||
retain `replace`, non-mutating `review`, and `off` policies. The LSP publishes
|
|
||||||
every applicable detector finding but exposes edits only under replacement
|
|
||||||
policy, reusing the same library
|
|
||||||
implementations as the CLI and formatter. Do not add prose rewriting, generative
|
|
||||||
inference, or destructive archive editing without a separate design decision.
|
|
||||||
|
|
||||||
## Required architecture
|
|
||||||
|
|
||||||
Keep domain logic in a library and binaries as thin adapters. The intended
|
|
||||||
layout is:
|
|
||||||
|
|
||||||
```text
|
|
||||||
src/
|
|
||||||
lib.rs compatibility facade over workspace libraries
|
|
||||||
bin/
|
|
||||||
john-slop.rs command-line detector
|
|
||||||
john-slop-fmt.rs replacement formatter
|
|
||||||
john-slop-lsp.rs stdio language server
|
|
||||||
crates/
|
|
||||||
john-slop-core/
|
|
||||||
assets/ embedded replacement and heuristic catalogs
|
|
||||||
src/ config, input, detection, diagnostics, and edits
|
|
||||||
john-slop-lsp/src/ protocol state, UTF-16 mapping, and code actions
|
|
||||||
tests/
|
|
||||||
fixtures/ small, reviewed input/output fixtures
|
|
||||||
```
|
|
||||||
|
|
||||||
Names can evolve when Rust conventions demand it, but preserve these boundaries:
|
|
||||||
|
|
||||||
- `input` acquires text and never scores or changes it.
|
|
||||||
- `detection` is pure over text plus validated rules.
|
|
||||||
- `replacement` returns non-overlapping edits and does no I/O.
|
|
||||||
- `diagnostic` renders findings but does not discover them.
|
|
||||||
- the CLI, formatter, and LSP call the same config, detector, and replacement APIs.
|
|
||||||
- filesystem writes are confined to explicit adapter methods.
|
|
||||||
|
|
||||||
Do not retain the current all-files-in-memory iterator design as the core API.
|
|
||||||
Prefer a streaming iterator of per-file results so one unreadable or binary file
|
|
||||||
does not erase findings from unrelated files. Preserve the outer archive path and
|
|
||||||
inner member name as separate fields. Reject unsafe archive paths and apply size,
|
|
||||||
entry-count, and nesting limits before extraction or decompression.
|
|
||||||
|
|
||||||
## Core data contracts
|
|
||||||
|
|
||||||
Use typed, presentation-neutral domain values. The exact names may change, but
|
|
||||||
the model must express at least:
|
|
||||||
|
|
||||||
```rust
|
|
||||||
struct Finding {
|
|
||||||
rule_id: RuleId,
|
|
||||||
category: Category,
|
|
||||||
message: String,
|
|
||||||
help: Option<String>,
|
|
||||||
byte_span: Range<usize>,
|
|
||||||
matched_text: String,
|
|
||||||
weight: f64,
|
|
||||||
fix: Option<TextEdit>,
|
|
||||||
}
|
|
||||||
|
|
||||||
struct Analysis {
|
|
||||||
findings: Vec<Finding>,
|
|
||||||
evidence_score: EvidenceScore,
|
|
||||||
}
|
|
||||||
|
|
||||||
struct TextEdit {
|
|
||||||
byte_span: Range<usize>,
|
|
||||||
replacement: String,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
All internal spans are zero-based UTF-8 byte ranges into the exact source
|
|
||||||
snapshot that was analyzed. Validate boundaries before slicing. Convert spans at
|
|
||||||
the edge: `miette::SourceSpan` for terminal output and UTF-16 line/column ranges
|
|
||||||
for LSP. Keep an indexed line map per document so LSP conversions are tested and
|
|
||||||
not repeatedly O(n).
|
|
||||||
|
|
||||||
Rule IDs are stable, unique, lowercase kebab-case identifiers. Findings are
|
|
||||||
sorted by source identity, span start, then rule ID so output is deterministic.
|
|
||||||
Never use a hash-map iteration order as output order.
|
|
||||||
|
|
||||||
## Heuristics and evidence scoring
|
|
||||||
|
|
||||||
Store data-driven heuristic definitions in
|
|
||||||
`crates/john-slop-core/assets/heuristics.toml`, separately
|
|
||||||
from replacement configuration. Compile and validate rules once at startup.
|
|
||||||
Invalid regular expressions, duplicate IDs, negative/non-finite weights, invalid
|
|
||||||
thresholds, and unsafe empty matches are configuration errors—not ignored rules.
|
|
||||||
|
|
||||||
Each rule needs an ID, category, rationale/message, help text, matcher type,
|
|
||||||
weight, and optional applicability filters such as prose/source-code extensions.
|
|
||||||
Start with high-precision, independently testable signals:
|
|
||||||
|
|
||||||
- copied assistant disclaimers or knowledge-cutoff phrases;
|
|
||||||
- prompt/response scaffolding accidentally left in a document;
|
|
||||||
- templated filler and repeated conclusion/transition phrases;
|
|
||||||
- suspicious Unicode/confusable punctuation when the configured corpus expects
|
|
||||||
ASCII;
|
|
||||||
- source-code placeholders, tutorial residue, fabricated references, or comments
|
|
||||||
that merely narrate obvious code, using conservative language-specific rules.
|
|
||||||
|
|
||||||
Avoid weak demographic, dialect, spelling, grammar, vocabulary, or
|
|
||||||
English-fluency proxies. Repetition alone must not trigger a high evidence-band
|
|
||||||
verdict. Give users category and per-rule enablement controls, per-rule weight
|
|
||||||
overrides, and path/language scopes. A rule is active only when both its
|
|
||||||
category gate and individual `enabled` field are active.
|
|
||||||
|
|
||||||
Scoring must be documented and deterministic. Use bounded weighted evidence,
|
|
||||||
apply a per-rule occurrence cap, and damp correlated findings from the same rule
|
|
||||||
or family. Map the aggregate onto `0..=100` and named bands such as `low`,
|
|
||||||
`medium`, and `high`; keep thresholds configurable. Tests must pin the formula,
|
|
||||||
caps, threshold edges, and empty-input result. Never label the numeric result an
|
|
||||||
accuracy or probability unless it has been independently calibrated against a
|
|
||||||
published, representative corpus.
|
|
||||||
|
|
||||||
Every terminal finding must become a `miette::Diagnostic` with named source,
|
|
||||||
precise label, stable diagnostic code (the rule ID), severity, message, and help.
|
|
||||||
A summary score supplements findings; it never replaces their evidence. Machine
|
|
||||||
output must offer a versioned JSON schema and must not include ANSI escapes.
|
|
||||||
|
|
||||||
## Configuration contract
|
|
||||||
|
|
||||||
Use `figment` with its TOML support and Serde for typed layered configuration.
|
|
||||||
Its providers allow embedded serialized defaults and later TOML providers to
|
|
||||||
override individual values while retaining source metadata for useful errors.
|
|
||||||
Use `directories` (or an equivalent platform-aware crate) to resolve the user
|
|
||||||
configuration directory; do not hand-build it from `HOME`.
|
|
||||||
|
|
||||||
The precedence from lowest to highest is:
|
|
||||||
|
|
||||||
1. embedded `crates/john-slop-core/assets/config.toml` and
|
|
||||||
`crates/john-slop-core/assets/heuristics.toml`;
|
|
||||||
2. runtime diagnostic catalogs: the optional user-wide `diagnostics.toml`, then
|
|
||||||
explicitly selected catalogs in command-line order;
|
|
||||||
3. user base config at `$XDG_CONFIG_HOME/john-slop/config.toml`, conventionally
|
|
||||||
`~/.config/john-slop/config.toml` on Linux;
|
|
||||||
4. named user config at `$XDG_CONFIG_HOME/john-slop/<config_name>.toml` when a
|
|
||||||
validated config name is selected;
|
|
||||||
5. an explicit `--config <path>` file;
|
|
||||||
6. explicit CLI flags for the current invocation.
|
|
||||||
|
|
||||||
Diagnostic catalog files may contain only `schema_version` and `[[rules]]`.
|
|
||||||
They can add or patch data-driven rules but cannot define executable matcher
|
|
||||||
algorithms, categories, replacements, thresholds, or input policy. Normal
|
|
||||||
configuration layers intentionally load afterward so they can disable or tune
|
|
||||||
catalog rules by stable ID.
|
|
||||||
|
|
||||||
Missing optional files are allowed; malformed files are fatal and identify their
|
|
||||||
origin. A config name accepts only ASCII letters, digits, `_`, and `-`; it cannot
|
|
||||||
contain separators, `.` segments, or an extension. Decide array behavior per
|
|
||||||
field: replacement/rule tables merge by stable ID, while ignore globs and file
|
|
||||||
extensions replace as complete arrays unless the schema explicitly says
|
|
||||||
otherwise. Do not rely on a library's default array-merge semantics implicitly.
|
|
||||||
|
|
||||||
`crates/john-slop-core/assets/unicode.toml` is the canonical generated Unicode
|
|
||||||
replacement and diagnostic catalog; `config.toml` owns other defaults. Each
|
|
||||||
replacement entry has an ID, Unicode source string/code point, ASCII target,
|
|
||||||
rationale, and enabled flag. Reject duplicate sources, empty sources, non-ASCII
|
|
||||||
targets, and conflicting edits. Longer sources win matching precedence. Provide a command
|
|
||||||
that prints the resolved configuration with secrets/redacted fields if secrets
|
|
||||||
are ever introduced.
|
|
||||||
|
|
||||||
Configuration schema changes require a `schema_version`, migration notes, and
|
|
||||||
tests for precedence. Embed assets with `include_str!` so installed binaries do
|
|
||||||
not depend on the source tree or current working directory.
|
|
||||||
|
|
||||||
## CLI, formatter, and LSP behavior
|
|
||||||
|
|
||||||
Use explicit Cargo binaries named `john-slop`, `john-slop-fmt`, and
|
|
||||||
`john-slop-lsp`.
|
|
||||||
|
|
||||||
- The detector reads files/directories/stdin, reports findings, and exits `0`
|
|
||||||
below the selected threshold, `1` when findings cross it, and `2` for usage or
|
|
||||||
operational/configuration errors.
|
|
||||||
- The formatter supports stdin/stdout, `--check`, and an explicit `--write`.
|
|
||||||
Default behavior must not rewrite a file. Writes must be atomic where the
|
|
||||||
platform permits, preserve original bytes when no edit applies, preserve line
|
|
||||||
endings, and be idempotent.
|
|
||||||
- The LSP communicates only over stdin/stdout; logs go to stderr. It publishes
|
|
||||||
applicable configured heuristic diagnostics and exposes quick-fix/code-action
|
|
||||||
edits only for safe replacements. It must honor document versions, clear
|
|
||||||
diagnostics on close, handle invalid URIs without panicking, and never write
|
|
||||||
editor-owned files.
|
|
||||||
|
|
||||||
Replacement edits are generated from the original snapshot, sorted, checked for
|
|
||||||
overlap, then applied from the end of the buffer toward the beginning. The
|
|
||||||
formatter and LSP must produce byte-for-byte equivalent results for the same
|
|
||||||
source and resolved config.
|
|
||||||
|
|
||||||
## Errors, safety, and observability
|
|
||||||
|
|
||||||
- Libraries return typed errors with `thiserror`; application boundaries attach
|
|
||||||
path/rule context and render with `miette`.
|
|
||||||
- Do not use `unwrap`, `expect`, `panic!`, `todo!`, or `unreachable!` on
|
|
||||||
user-controlled input. An `expect` is acceptable only for an invariant proved
|
|
||||||
during construction and explained in a comment.
|
|
||||||
- Treat invalid UTF-8 and binary content as explicit skipped-input diagnostics
|
|
||||||
according to policy; do not silently truncate or lossily rewrite it.
|
|
||||||
- Do not follow symlinks by default. Respect ignore configuration and establish
|
|
||||||
file-size, archive-size, archive-entry, and recursion limits.
|
|
||||||
- Do not make network calls during analysis. Results should be reproducible and
|
|
||||||
private by default.
|
|
||||||
- Structured logs go to stderr and are disabled or quiet by default. Never mix
|
|
||||||
logs with JSON, formatter stdout, or the LSP protocol stream.
|
|
||||||
|
|
||||||
## Testing and quality gates
|
|
||||||
|
|
||||||
Each behavior change requires tests at the lowest useful layer plus an
|
|
||||||
integration test when it crosses a binary boundary. Include positive,
|
|
||||||
near-miss/negative, Unicode, empty, large-input, malformed-config, and ordering
|
|
||||||
cases. Regression fixtures must state why they should or should not match.
|
|
||||||
|
|
||||||
Before considering a task complete, run:
|
|
||||||
|
|
||||||
```text
|
|
||||||
cargo fmt --all -- --check
|
|
||||||
cargo clippy --all-targets --all-features -- -D warnings
|
|
||||||
cargo test --all-targets --all-features
|
|
||||||
cargo test --all-targets --no-default-features
|
|
||||||
cargo doc --no-deps
|
|
||||||
```
|
|
||||||
|
|
||||||
Run `cargo tarpaulin --config tarpaulin.toml` when tests or executable Rust code
|
|
||||||
change. It enforces the 90% line-coverage floor. Jenkins renders the version,
|
|
||||||
test, and coverage badges from package metadata and CI results; generated badge
|
|
||||||
files and status snapshots do not belong in the repository.
|
|
||||||
|
|
||||||
Also run focused CLI golden tests and LSP protocol tests once those binaries
|
|
||||||
exist. Favor property tests for replacement idempotence/non-overlap and fuzz
|
|
||||||
targets for config parsing, span conversion, archives, and replacement planning.
|
|
||||||
Benchmarks should cover a representative large document and directory; optimize
|
|
||||||
only from measured results.
|
|
||||||
|
|
||||||
## Change discipline
|
|
||||||
|
|
||||||
Keep commits and patches small enough to review. Do not combine a structural
|
|
||||||
move with new detection behavior unless the new behavior requires it. Preserve
|
|
||||||
public behavior during refactors or document the deliberate break. Update sample
|
|
||||||
configuration, schema/version notes, CLI help, and tests together.
|
|
||||||
|
|
||||||
When implementing `tasks.md`, work in dependency order, mark only verified items
|
|
||||||
complete, and add newly discovered work under the relevant milestone. Record a
|
|
||||||
short decision note in the task file when changing configuration precedence,
|
|
||||||
the score formula, a public data contract, or safety limits.
|
|
||||||
|
|
||||||
After completing and verifying each individual task, create a dedicated commit
|
|
||||||
before starting the next task. Stage only the files that belong to that task;
|
|
||||||
never include unrelated or pre-existing user changes. Use a concise conventional
|
|
||||||
commit message that describes the completed behavior. If a task cannot be
|
|
||||||
verified or committed, leave it unchecked and report the blocker instead of
|
|
||||||
combining it with later work.
|
|
||||||
|
|
||||||
After every milestone's acceptance criteria and quality gates pass, create an
|
|
||||||
annotated Git tag named `v0.1.N`, where `N` is the milestone number, on its
|
|
||||||
completion commit. Before that completion commit, set the workspace package
|
|
||||||
version and internal registry dependency requirements to `0.1.N`, refresh
|
|
||||||
`Cargo.lock` and cut the matching changelog entry. Jenkins derives the version
|
|
||||||
badge from that metadata. Verify the manifests, lockfile, changelog, and binary
|
|
||||||
`--version` output agree before tagging. Never move an existing milestone tag;
|
|
||||||
add a corrective commit and a documented successor tag if a published milestone
|
|
||||||
needs amendment.
|
|
||||||
230
CHANGELOG.md
230
CHANGELOG.md
|
|
@ -1,230 +0,0 @@
|
||||||
# Changelog
|
|
||||||
|
|
||||||
All notable changes to John Slop are recorded here. The format follows
|
|
||||||
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and releases use
|
|
||||||
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
||||||
|
|
||||||
## [Unreleased]
|
|
||||||
|
|
||||||
## [0.1.27-rc.8] - 2026-09-30
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Selected current Cross target images so all Linux and Windows release
|
|
||||||
artifacts compile with the declared Rust 1.98.1-or-newer toolchain.
|
|
||||||
|
|
||||||
## [0.1.27-rc.7] - 2026-09-30
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Used the publisher's repository fallback mapping so releases reach the HTTPS
|
|
||||||
proxy while the developer SSH origin remains unchanged.
|
|
||||||
|
|
||||||
## [0.1.27-rc.6] - 2026-09-30
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Published cargo-binstall `.tgz` archives alongside raw binaries so the
|
|
||||||
package metadata added in RC5 resolves to release assets.
|
|
||||||
|
|
||||||
## [0.1.27-rc.5] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added `cargo-binstall` package metadata for downloading versioned target
|
|
||||||
archives from Gitea releases.
|
|
||||||
|
|
||||||
## [0.1.27-rc.4] - 2026-09-30
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Made the release wrapper temporarily select the canonical HTTPS origin for
|
|
||||||
publication and restore the developer SSH origin whether publication succeeds
|
|
||||||
or fails.
|
|
||||||
|
|
||||||
## [0.1.27-rc.3] - 2026-09-30
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Unified the root Cargo package, repository slug, release bundles, runtime
|
|
||||||
configuration directory, documentation, and Git remote around the canonical
|
|
||||||
`john-slop` spelling. Rust source continues to import the package as
|
|
||||||
`john_slop`, because Rust identifiers cannot contain hyphens.
|
|
||||||
|
|
||||||
## [0.1.27-rc.2] - 2026-09-30
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Switched canonical repository and release metadata to the TLS-valid
|
|
||||||
`proxy.dandaedre.com/gitea` endpoint so authenticated Gitea API publication
|
|
||||||
does not fail during connection setup.
|
|
||||||
|
|
||||||
## [0.1.27-rc.1] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added a token-file wrapper around `rust_realease` for publishing tagged raw
|
|
||||||
binary release candidates without placing Gitea credentials in arguments.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Removed unused SBOM generation and unpublished Tarpaulin execution from the
|
|
||||||
Jenkins pipeline while retaining local coverage validation.
|
|
||||||
- Made reused Jenkins workspaces tolerate legacy generated release artifacts.
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Replaced the yanked transitive `yoke-derive 0.8.3` lockfile entry with
|
|
||||||
`0.8.4`.
|
|
||||||
|
|
||||||
## [0.1.26] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added conservative response-residue, editorial-placeholder, source-residue,
|
|
||||||
stock-closing, and clustered excess-vocabulary rules with positive and
|
|
||||||
realistic near-miss characterization fixtures.
|
|
||||||
- Added a research-backed review-signal methodology that separates directly
|
|
||||||
reviewable residue, low-weight population clues, verification targets, and
|
|
||||||
deliberately excluded language-proficiency proxies.
|
|
||||||
- Added LSP `--all-diagnostics` and `--minimum-diagnostic-weight WEIGHT`
|
|
||||||
switches for explicit full-catalog and weighted editor review.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- The LSP now publishes only configured Unicode diagnostics by default. The
|
|
||||||
visibility policy does not alter shared detector results, evidence scores,
|
|
||||||
replacement planning, or formatting.
|
|
||||||
|
|
||||||
## [0.1.25] - 2026-09-30
|
|
||||||
|
|
||||||
### Fixed
|
|
||||||
|
|
||||||
- Fullwidth forms U+FF01 through U+FF5E now width-fold one-to-one to ASCII
|
|
||||||
`!` through `~`, instead of using visually motivated UTS #39 skeletons or
|
|
||||||
remaining unchanged. Ideographic space now maps to ASCII space.
|
|
||||||
- LSP formatting, quick fixes, the standalone formatter, and detector fixes
|
|
||||||
share the corrected mappings while retaining previously shipped stable IDs.
|
|
||||||
|
|
||||||
## [0.1.24] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added restricted runtime `diagnostics.toml` catalogs for defining
|
|
||||||
exact-phrase, regex, and repeated-regex rules without recompilation.
|
|
||||||
- Added repeatable `--diagnostics <FILE>` support across the detector,
|
|
||||||
formatter, and LSP, including LSP configuration reloads.
|
|
||||||
- Added `john-slop config init-diagnostics`, a commented starter catalog, and a
|
|
||||||
complete project example.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- User and explicit diagnostic catalogs load before normal configuration so
|
|
||||||
later base, named, and explicit layers can tune or disable their stable IDs.
|
|
||||||
- Runtime catalogs reject code-backed matcher engines and unrelated settings;
|
|
||||||
executable plugin behavior remains outside the declarative configuration
|
|
||||||
boundary.
|
|
||||||
|
|
||||||
## [0.1.23] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added typed category-level diagnostic switches for `typography`, `style`,
|
|
||||||
`prompt-leakage`, and `source-code`, composed with the existing individual
|
|
||||||
rule switches through every configuration layer.
|
|
||||||
- Added `john-slop config init` and its `config generate` alias to write the
|
|
||||||
commented embedded base configuration into the current directory.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Configuration generation protects an existing `config.toml` unless
|
|
||||||
`--force` is explicitly supplied.
|
|
||||||
|
|
||||||
## [0.1.22] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added a generated, versioned Unicode policy containing 1,829 mappings and
|
|
||||||
their shared diagnostic metadata, plus an exhaustive drift test and
|
|
||||||
deterministic regeneration example.
|
|
||||||
- Added `replace`, `review`, and `off` bulk policies for printable-ASCII UTS #39
|
|
||||||
mappings.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Printable-ASCII Unicode 16.0 UTS #39 lookalikes now receive consistent fixes
|
|
||||||
from CLI diagnostics, the standalone formatter, LSP quick fixes, and LSP
|
|
||||||
document formatting under the default `replace` policy.
|
|
||||||
- Invisible and bidirectional controls remain diagnostic-only.
|
|
||||||
|
|
||||||
## [0.1.21] - 2026-09-30
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Added real LSP document formatting backed by the same resolved replacement
|
|
||||||
plan as `john-slop-fmt`.
|
|
||||||
- Added non-mutating exact-span diagnostics for Unicode 16.0 UTS #39
|
|
||||||
ASCII-lookalikes and pinned Gitea invisible/bidirectional controls.
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Expanded the shared fixtures and expected-failure diagnostic showcase with
|
|
||||||
deliberately non-fixable Unicode review cases.
|
|
||||||
|
|
||||||
## [0.1.20] - 2026-09-29
|
|
||||||
|
|
||||||
### Changed
|
|
||||||
|
|
||||||
- Synchronized workspace package versions, internal dependency requirements,
|
|
||||||
lockfile metadata, binary version output, milestone tags, and the generated
|
|
||||||
version badge, with a regression contract to prevent future drift.
|
|
||||||
- Made graphical Miette output easier to scan with marked rule-code titles,
|
|
||||||
attached diagnostic bodies, and spacing between complete findings.
|
|
||||||
- Expanded the expected-failure diagnostic showcase to include prose and source
|
|
||||||
rules plus the complete embedded replacement-character set.
|
|
||||||
- Documented Cargo's required `--path .` syntax when installing all binaries
|
|
||||||
from a local source checkout.
|
|
||||||
- Restructured the README and documentation index as a progressive path from
|
|
||||||
quickstart to usage guides, with engineering detail in a dedicated advanced
|
|
||||||
guide.
|
|
||||||
- Added an opt-in Rust example that maps fullwidth Unicode forms to all 95
|
|
||||||
printable ASCII targets while leaving production defaults unchanged.
|
|
||||||
- Extended the LSP from replacement-only warnings to every applicable heuristic
|
|
||||||
finding while retaining quick fixes only for safe ASCII replacements.
|
|
||||||
- Adopted the measured three-package Cargo workspace. Shared package metadata,
|
|
||||||
dependency requirements, and lint policy now use workspace inheritance while
|
|
||||||
the reusable core remains isolated from CLI, async, and LSP dependencies.
|
|
||||||
- Renamed the ambiguous confidence metric to low-care evidence score throughout
|
|
||||||
the Rust API, CLI, terminal output, and JSON schema version 2. Deprecated Rust
|
|
||||||
type aliases and `--fail-confidence` remain available for migration; the
|
|
||||||
scoring formula and values are unchanged.
|
|
||||||
|
|
||||||
## [0.1.0] - 2026-09-29
|
|
||||||
|
|
||||||
### Added
|
|
||||||
|
|
||||||
- Local, explainable heuristic analysis with a bounded low-care evidence score.
|
|
||||||
- Layered embedded, user, named, explicit, and CLI configuration.
|
|
||||||
- File, directory, ZIP, office-container, and stdin input adapters with limits.
|
|
||||||
- Human, graphical Miette, concise, and versioned JSON detector output.
|
|
||||||
- Conservative Unicode-to-ASCII formatter with check, diff, and atomic write.
|
|
||||||
- Stdio LSP replacement diagnostics and versioned quick fixes.
|
|
||||||
- Runnable examples, reviewed fixtures, fuzz targets, benchmarks, and CI gates.
|
|
||||||
|
|
||||||
[Unreleased]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.8...HEAD
|
|
||||||
[0.1.27-rc.8]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.7...v0.1.27-rc.8
|
|
||||||
[0.1.27-rc.7]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.6...v0.1.27-rc.7
|
|
||||||
[0.1.27-rc.6]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.5...v0.1.27-rc.6
|
|
||||||
[0.1.27-rc.5]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.4...v0.1.27-rc.5
|
|
||||||
[0.1.27-rc.4]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.3...v0.1.27-rc.4
|
|
||||||
[0.1.27-rc.3]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.2...v0.1.27-rc.3
|
|
||||||
[0.1.27-rc.2]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.27-rc.1...v0.1.27-rc.2
|
|
||||||
[0.1.27-rc.1]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.26...v0.1.27-rc.1
|
|
||||||
[0.1.26]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.25...v0.1.26
|
|
||||||
[0.1.25]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.24...v0.1.25
|
|
||||||
[0.1.24]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.23...v0.1.24
|
|
||||||
[0.1.23]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.22...v0.1.23
|
|
||||||
[0.1.22]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.21...v0.1.22
|
|
||||||
[0.1.21]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.20...v0.1.21
|
|
||||||
[0.1.20]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/compare/v0.1.0...v0.1.20
|
|
||||||
[0.1.0]: https://proxy.dandaedre.com/gitea/idle-org/john-slop/releases/tag/v0.1.0
|
|
||||||
|
|
@ -1,15 +0,0 @@
|
||||||
# Code of Conduct
|
|
||||||
|
|
||||||
John Slop uses the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
|
|
||||||
|
|
||||||
Be respectful, specific, and evidence-led. Harassment, discriminatory language,
|
|
||||||
personal attacks, doxxing, sexualized conduct, and sustained disruption are not
|
|
||||||
acceptable in project spaces. Maintainers may edit, remove, or reject comments,
|
|
||||||
commits, issues, and other contributions that violate these expectations, and
|
|
||||||
may temporarily or permanently restrict participation.
|
|
||||||
|
|
||||||
Report conduct concerns privately to the repository owner through the hosting
|
|
||||||
service profile. Reports will be reviewed promptly and handled with discretion;
|
|
||||||
the maintainer will disclose information only when necessary to investigate or
|
|
||||||
protect participants. This project does not tolerate retaliation against a
|
|
||||||
reporter acting in good faith.
|
|
||||||
|
|
@ -1,22 +0,0 @@
|
||||||
# Contributing
|
|
||||||
|
|
||||||
Contributions should preserve John Slop's explainable, local-first product
|
|
||||||
boundary. Findings must describe observable signals without asserting AI
|
|
||||||
authorship, model identity, or personal characteristics.
|
|
||||||
|
|
||||||
1. Open an issue for behavior or public-contract changes before investing in a
|
|
||||||
large patch.
|
|
||||||
2. Create a focused branch and add tests at the lowest useful layer.
|
|
||||||
3. Update user documentation, samples, and stable output snapshots together.
|
|
||||||
4. Run the quality, supply-chain, fuzz-smoke, or benchmark commands relevant to
|
|
||||||
the change as documented in [Development](docs/development.md).
|
|
||||||
5. Submit small conventional commits and explain rule false-positive tradeoffs.
|
|
||||||
|
|
||||||
New heuristic rules require a stable lowercase kebab-case ID, conservative
|
|
||||||
positive and realistic near-miss fixtures, scopes, a rationale and help text,
|
|
||||||
and evaluation-corpus review. Default weights or thresholds require the
|
|
||||||
documented sign-off process in [Evaluation](docs/evaluation.md).
|
|
||||||
|
|
||||||
By participating, contributors agree to follow the
|
|
||||||
[Code of Conduct](CODE_OF_CONDUCT.md). Report vulnerabilities through
|
|
||||||
[SECURITY.md](SECURITY.md), not a public issue.
|
|
||||||
File diff suppressed because it is too large
Load Diff
116
Cargo.toml
116
Cargo.toml
|
|
@ -1,109 +1,13 @@
|
||||||
[workspace]
|
|
||||||
members = ["crates/john-slop-core", "crates/john-slop-lsp"]
|
|
||||||
default-members = [".", "crates/john-slop-core", "crates/john-slop-lsp"]
|
|
||||||
resolver = "3"
|
|
||||||
|
|
||||||
[workspace.package]
|
|
||||||
version = "0.1.27-rc.8"
|
|
||||||
edition = "2024"
|
|
||||||
rust-version = "1.98.1"
|
|
||||||
license = "MIT"
|
|
||||||
repository = "https://proxy.dandaedre.com/gitea/idle-org/john-slop"
|
|
||||||
|
|
||||||
[workspace.dependencies]
|
|
||||||
clap = { version = "4.6.7", default-features = false, features = ["derive", "error-context", "help", "std", "suggestions", "usage"] }
|
|
||||||
criterion = { version = "0.8.2", default-features = false }
|
|
||||||
directories = "6.0.0"
|
|
||||||
figment = { version = "0.10.19", default-features = false, features = ["toml"] }
|
|
||||||
futures = { version = "0.3.34", default-features = false, features = ["std"] }
|
|
||||||
globset = { version = "0.4.20", default-features = false }
|
|
||||||
insta = { version = "1.48.0", default-features = false }
|
|
||||||
john-slop-core = { version = "0.1.27-rc.8", path = "crates/john-slop-core" }
|
|
||||||
john-slop-lsp-engine = { package = "john-slop-lsp", version = "0.1.27-rc.8", path = "crates/john-slop-lsp" }
|
|
||||||
miette = { version = "7.6.0", default-features = false, features = ["derive"] }
|
|
||||||
regex = "1.13.1"
|
|
||||||
serde = { version = "1.0.229", features = ["derive"] }
|
|
||||||
serde_json = "1.0.151"
|
|
||||||
sha2 = "0.11.0"
|
|
||||||
similar = "2.7.0"
|
|
||||||
tempfile = "3.27.0"
|
|
||||||
thiserror = "2.0.21"
|
|
||||||
tokio = { version = "1.53.1", default-features = false }
|
|
||||||
toml = "1.1.6"
|
|
||||||
tower = { version = "0.5.3", default-features = false, features = ["util"] }
|
|
||||||
tower-lsp-server = { version = "0.23.0", default-features = false, features = ["runtime-tokio"] }
|
|
||||||
unicode-security = "0.1.2"
|
|
||||||
walkdir = "2.5.0"
|
|
||||||
zip = { version = "8.6.0", default-features = false, features = ["deflate-flate2-zlib-rs"] }
|
|
||||||
|
|
||||||
[workspace.lints.rust]
|
|
||||||
unsafe_code = "forbid"
|
|
||||||
|
|
||||||
[package]
|
[package]
|
||||||
name = "john-slop"
|
name = "john_slop"
|
||||||
version.workspace = true
|
version = "0.1.0"
|
||||||
edition.workspace = true
|
edition = "2024"
|
||||||
rust-version.workspace = true
|
|
||||||
license.workspace = true
|
|
||||||
repository.workspace = true
|
|
||||||
description = "Inspect files, directories, or ZIP-based documents for low-care signals"
|
|
||||||
autobins = false
|
|
||||||
|
|
||||||
[package.metadata.binstall]
|
|
||||||
pkg-url = "{ repo }/releases/download/v{ version }/{ name }-{ target }-v{ version }.tgz"
|
|
||||||
pkg-fmt = "tgz"
|
|
||||||
bin-dir = "{ name }-{ target }-v{ version }/{ bin }{ binary-ext }"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "john-slop"
|
|
||||||
path = "src/bin/john-slop.rs"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "john-slop-fmt"
|
|
||||||
path = "src/bin/john-slop-fmt.rs"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "john-slop-lsp"
|
|
||||||
path = "src/bin/john-slop-lsp.rs"
|
|
||||||
required-features = ["lsp"]
|
|
||||||
|
|
||||||
[features]
|
|
||||||
default = ["fancy-diagnostics", "lsp"]
|
|
||||||
fancy-diagnostics = ["miette/fancy"]
|
|
||||||
lsp = ["dep:john-slop-lsp-engine", "dep:tokio", "dep:tower-lsp-server"]
|
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
clap.workspace = true
|
anyhow = "1.0.100"
|
||||||
globset.workspace = true
|
clap = { version = "4.5.53", features = ["derive"] }
|
||||||
john-slop-core.workspace = true
|
rayon = "1.11.0"
|
||||||
john-slop-lsp-engine = { workspace = true, optional = true }
|
regex = "1.12.2"
|
||||||
miette.workspace = true
|
serde = { version = "1.0.228", features = ["derive"] }
|
||||||
serde.workspace = true
|
toml = "0.9.8"
|
||||||
serde_json.workspace = true
|
walkdir = "2.5.0"
|
||||||
similar.workspace = true
|
|
||||||
tempfile.workspace = true
|
|
||||||
thiserror.workspace = true
|
|
||||||
tokio = { workspace = true, features = ["io-std", "macros", "rt-multi-thread", "sync"], optional = true }
|
|
||||||
toml.workspace = true
|
|
||||||
tower-lsp-server = { workspace = true, optional = true }
|
|
||||||
|
|
||||||
[dev-dependencies]
|
|
||||||
criterion.workspace = true
|
|
||||||
futures.workspace = true
|
|
||||||
insta.workspace = true
|
|
||||||
sha2.workspace = true
|
|
||||||
tower.workspace = true
|
|
||||||
zip.workspace = true
|
|
||||||
|
|
||||||
[lints]
|
|
||||||
workspace = true
|
|
||||||
|
|
||||||
[[bench]]
|
|
||||||
name = "core"
|
|
||||||
harness = false
|
|
||||||
|
|
||||||
[profile.release]
|
|
||||||
codegen-units = 1
|
|
||||||
lto = "thin"
|
|
||||||
panic = "abort"
|
|
||||||
strip = "symbols"
|
|
||||||
|
|
|
||||||
13
Cross.toml
13
Cross.toml
|
|
@ -1,13 +0,0 @@
|
||||||
[build]
|
|
||||||
|
|
||||||
[build.env]
|
|
||||||
passthrough = ["RUSTUP_TOOLCHAIN"]
|
|
||||||
|
|
||||||
[target.x86_64-unknown-linux-musl]
|
|
||||||
image = "ghcr.io/cross-rs/x86_64-unknown-linux-musl:main"
|
|
||||||
|
|
||||||
[target.aarch64-unknown-linux-musl]
|
|
||||||
image = "ghcr.io/cross-rs/aarch64-unknown-linux-musl:main"
|
|
||||||
|
|
||||||
[target.x86_64-pc-windows-gnu]
|
|
||||||
image = "ghcr.io/cross-rs/x86_64-pc-windows-gnu:main"
|
|
||||||
|
|
@ -1,363 +0,0 @@
|
||||||
// Add this webhook in Gitea:
|
|
||||||
// http://192.168.1.37:8080/gitea-webhook/post
|
|
||||||
|
|
||||||
pipeline {
|
|
||||||
agent { label 'rust_no_docker' }
|
|
||||||
|
|
||||||
environment {
|
|
||||||
CARGO_TERM_COLOR = 'always'
|
|
||||||
RUST_BACKTRACE = '1'
|
|
||||||
MSRV = '1.98.1'
|
|
||||||
CARGO_TARPAULIN_VERSION = '0.37.5'
|
|
||||||
CARGO_AUDIT_VERSION = '0.22.2'
|
|
||||||
CARGO_DENY_VERSION = '0.20.2'
|
|
||||||
BADGE_PUBLISH_ROOT = '/mnt/assets/coverage'
|
|
||||||
}
|
|
||||||
|
|
||||||
stages {
|
|
||||||
stage('Checkout') {
|
|
||||||
steps {
|
|
||||||
checkout scm
|
|
||||||
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
git submodule sync --recursive
|
|
||||||
git submodule update --init --recursive
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Setup Rust') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
if ! command -v rustup >/dev/null 2>&1; then
|
|
||||||
echo "ERROR: rustup is not installed on this worker." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Ensuring the stable Rust toolchain is available..."
|
|
||||||
|
|
||||||
rustup toolchain install stable \
|
|
||||||
--profile minimal \
|
|
||||||
--component rustfmt \
|
|
||||||
--component clippy
|
|
||||||
|
|
||||||
rustup toolchain install "${MSRV}" \
|
|
||||||
--profile minimal
|
|
||||||
|
|
||||||
rustup default stable
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "Using:"
|
|
||||||
rustc --version
|
|
||||||
cargo --version
|
|
||||||
rustfmt --version
|
|
||||||
cargo clippy --version
|
|
||||||
|
|
||||||
INSTALLED_AUDIT="$(cargo audit --version 2>/dev/null || true)"
|
|
||||||
if [ "${INSTALLED_AUDIT}" != "cargo-audit ${CARGO_AUDIT_VERSION}" ]; then
|
|
||||||
cargo install cargo-audit --locked --version "${CARGO_AUDIT_VERSION}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
INSTALLED_DENY="$(cargo deny --version 2>/dev/null || true)"
|
|
||||||
if [ "${INSTALLED_DENY}" != "cargo-deny ${CARGO_DENY_VERSION}" ]; then
|
|
||||||
cargo install cargo-deny --locked --version "${CARGO_DENY_VERSION}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
INSTALLED_TARPAULIN="$(cargo tarpaulin --version 2>/dev/null || true)"
|
|
||||||
if [ "${INSTALLED_TARPAULIN}" != "cargo-tarpaulin-tarpaulin ${CARGO_TARPAULIN_VERSION}" ]; then
|
|
||||||
cargo install cargo-tarpaulin --locked --version "${CARGO_TARPAULIN_VERSION}"
|
|
||||||
fi
|
|
||||||
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('MSRV') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Checking all targets with Rust ${MSRV}..."
|
|
||||||
|
|
||||||
cargo "+${MSRV}" check \
|
|
||||||
--locked \
|
|
||||||
--all-targets \
|
|
||||||
--all-features
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Format') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Checking Rust formatting..."
|
|
||||||
|
|
||||||
cargo fmt --all -- --check
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Check') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Checking all targets..."
|
|
||||||
|
|
||||||
cargo check \
|
|
||||||
--locked \
|
|
||||||
--all-targets \
|
|
||||||
--all-features
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Test') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Running tests..."
|
|
||||||
|
|
||||||
cargo test \
|
|
||||||
--locked \
|
|
||||||
--all-targets \
|
|
||||||
--all-features
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Test Minimal Features') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Running tests without default features..."
|
|
||||||
|
|
||||||
cargo test \
|
|
||||||
--locked \
|
|
||||||
--all-targets \
|
|
||||||
--no-default-features
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Fixtures and Goldens') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cargo test --locked --test example_fixtures --all-features
|
|
||||||
cargo test --locked --test cli --all-features
|
|
||||||
|
|
||||||
mkdir -p coverage
|
|
||||||
: > "coverage/.tests-passed-${BUILD_NUMBER}"
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Coverage') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cargo tarpaulin \
|
|
||||||
--config tarpaulin.toml \
|
|
||||||
--out Json \
|
|
||||||
--output-dir coverage
|
|
||||||
|
|
||||||
mv coverage/tarpaulin-report.json coverage/summary.json
|
|
||||||
coverage_percent=$(sed -n \
|
|
||||||
's/^.*],"coverage":\\([0-9][0-9.]*\\),"covered":[0-9][0-9]*,"coverable":[0-9][0-9]*}$/\\1/p' \
|
|
||||||
coverage/summary.json)
|
|
||||||
scripts/render-coverage-badge.sh \
|
|
||||||
"${coverage_percent}" \
|
|
||||||
coverage/coverage.svg
|
|
||||||
rm coverage/summary.json
|
|
||||||
: > "coverage/.coverage-ready-${BUILD_NUMBER}"
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Clippy') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Running Clippy..."
|
|
||||||
|
|
||||||
cargo clippy \
|
|
||||||
--locked \
|
|
||||||
--all-targets \
|
|
||||||
--all-features \
|
|
||||||
-- \
|
|
||||||
-D warnings
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Documentation') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cargo doc --locked --no-deps --all-features
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Supply Chain') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cargo audit --deny warnings
|
|
||||||
cargo deny --locked check
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Workspace Packages') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Assembling all publishable workspace packages..."
|
|
||||||
|
|
||||||
scripts/package-workspace.sh --no-verify
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Build Release') {
|
|
||||||
steps {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
echo "Building John Slop release binaries..."
|
|
||||||
|
|
||||||
# Jenkins reuses workspaces. Remove only ignored release
|
|
||||||
# output so rerunning the same commit stays deterministic.
|
|
||||||
rm -rf -- dist
|
|
||||||
|
|
||||||
# The packaging script reads and validates Cargo.toml itself.
|
|
||||||
scripts/package-release.sh
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "Release build completed."
|
|
||||||
|
|
||||||
ls -lh \
|
|
||||||
target/release/john-slop \
|
|
||||||
target/release/john-slop-fmt \
|
|
||||||
target/release/john-slop-lsp \
|
|
||||||
dist/*.tar.gz \
|
|
||||||
dist/*.sha256
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
post {
|
|
||||||
success {
|
|
||||||
publishChecks(
|
|
||||||
conclusion: 'SUCCESS',
|
|
||||||
name: 'Rust CI',
|
|
||||||
summary: 'JohnSlop passed all CI checks.',
|
|
||||||
text: 'The Linux MSRV/stable matrix, formatting, feature tests, fixtures, coverage and badge generation, docs, Clippy, supply-chain policy, and release build completed successfully.',
|
|
||||||
title: 'Build Successful'
|
|
||||||
)
|
|
||||||
|
|
||||||
echo '✅ JohnSlop passed all CI checks.'
|
|
||||||
}
|
|
||||||
|
|
||||||
failure {
|
|
||||||
publishChecks(
|
|
||||||
conclusion: 'FAILURE',
|
|
||||||
name: 'Rust CI',
|
|
||||||
summary: 'JohnSlop failed CI.',
|
|
||||||
text: 'A toolchain, formatting, test, fixture, coverage, badge generation, documentation, Clippy, supply-chain, or release check failed. Check the Jenkins log.',
|
|
||||||
title: 'Build Failed'
|
|
||||||
)
|
|
||||||
|
|
||||||
echo '❌ JohnSlop CI failed.'
|
|
||||||
}
|
|
||||||
|
|
||||||
unstable {
|
|
||||||
publishChecks(
|
|
||||||
conclusion: 'ACTION_REQUIRED',
|
|
||||||
name: 'Rust CI',
|
|
||||||
summary: 'JohnSlop CI is unstable.',
|
|
||||||
text: 'The Jenkins build completed with an unstable status.',
|
|
||||||
title: 'Build Unstable'
|
|
||||||
)
|
|
||||||
|
|
||||||
echo '⚠️ JohnSlop CI is unstable.'
|
|
||||||
}
|
|
||||||
|
|
||||||
cleanup {
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
mkdir -p coverage
|
|
||||||
test_status=failing
|
|
||||||
if [ -f "coverage/.tests-passed-${BUILD_NUMBER}" ]; then
|
|
||||||
test_status=passing
|
|
||||||
fi
|
|
||||||
scripts/render-test-badge.sh \
|
|
||||||
"${test_status}" \
|
|
||||||
coverage/test.svg
|
|
||||||
|
|
||||||
if [ ! -f "coverage/.coverage-ready-${BUILD_NUMBER}" ]; then
|
|
||||||
scripts/render-coverage-badge.sh - coverage/coverage.svg
|
|
||||||
fi
|
|
||||||
|
|
||||||
package_version=$(awk '
|
|
||||||
$0 == "[workspace.package]" { in_package = 1; next }
|
|
||||||
substr($0, 1, 1) == "[" { in_package = 0 }
|
|
||||||
in_package && $1 == "version" && $2 == "=" {
|
|
||||||
gsub(/^"|"$/, "", $3)
|
|
||||||
print $3
|
|
||||||
exit
|
|
||||||
}
|
|
||||||
' Cargo.toml)
|
|
||||||
if [ -z "${package_version}" ]; then
|
|
||||||
echo "could not determine workspace package version" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
scripts/render-version-badge.sh \
|
|
||||||
"${package_version}" \
|
|
||||||
coverage/version.svg
|
|
||||||
|
|
||||||
rm -f \
|
|
||||||
coverage/summary.json \
|
|
||||||
coverage/tarpaulin-report.json \
|
|
||||||
"coverage/.tests-passed-${BUILD_NUMBER}" \
|
|
||||||
"coverage/.coverage-ready-${BUILD_NUMBER}"
|
|
||||||
'''
|
|
||||||
|
|
||||||
archiveArtifacts(
|
|
||||||
allowEmptyArchive: false,
|
|
||||||
artifacts: 'coverage/coverage.svg,coverage/test.svg,coverage/version.svg',
|
|
||||||
fingerprint: true
|
|
||||||
)
|
|
||||||
|
|
||||||
sh '''
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
if [ "${BRANCH_NAME:-}" = "main" ]; then
|
|
||||||
scripts/publish-badges.sh \
|
|
||||||
"${BADGE_PUBLISH_ROOT}" \
|
|
||||||
coverage \
|
|
||||||
test \
|
|
||||||
version
|
|
||||||
fi
|
|
||||||
'''
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
128
README.md
128
README.md
|
|
@ -1,125 +1,9 @@
|
||||||
# John Slop
|
# JohnSlop
|
||||||
|
|
||||||

|
Detect AI slop markers
|
||||||

|
|
||||||

|
|
||||||
|
|
||||||
John Slop is a local-first linter for concrete signs of low-effort or
|
|
||||||
insufficiently reviewed text and source code. It reports explainable evidence
|
|
||||||
for human review; it does not prove AI authorship or identify a model.
|
|
||||||
|
|
||||||
## Quickstart
|
# Links
|
||||||
|
* [https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing]
|
||||||
With `cargo-binstall` available, install all three prebuilt binaries from the
|
* [https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup]
|
||||||
latest release candidate:
|
* [https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion]
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo binstall --git https://proxy.dandaedre.com/gitea/idle-org/john-slop \
|
|
||||||
john-slop
|
|
||||||
```
|
|
||||||
|
|
||||||
To build from source, Rust 1.98.1 or newer is required. Install from the
|
|
||||||
upstream Git repository:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo install --git https://proxy.dandaedre.com/gitea/idle-org/john-slop \
|
|
||||||
--locked --all-features
|
|
||||||
```
|
|
||||||
|
|
||||||
Or install a checkout from its repository root:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo install --path . --locked --all-features
|
|
||||||
```
|
|
||||||
|
|
||||||
Scan a document or source directory:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop path/to/document path/to/source-directory
|
|
||||||
```
|
|
||||||
|
|
||||||
Generate a commented `config.toml` in the current directory, then edit category
|
|
||||||
or individual-rule switches as needed:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config init
|
|
||||||
john-slop config init-diagnostics
|
|
||||||
```
|
|
||||||
|
|
||||||
The second command creates a runtime rule catalog: exact-phrase and regex
|
|
||||||
diagnostics can then be added without changing or recompiling the program.
|
|
||||||
|
|
||||||
Diagnostics are colored Miette reports by default. Make the command fail when a
|
|
||||||
document reaches a chosen low-care evidence score with `--fail-score 0..100`:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop --fail-score 40 path/to/document
|
|
||||||
```
|
|
||||||
|
|
||||||
Preview configured Unicode-to-ASCII replacements, or check them in automation:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop-fmt path/to/document
|
|
||||||
john-slop-fmt --check path/to/document
|
|
||||||
```
|
|
||||||
|
|
||||||
Formatting is read-only unless `--write` is explicit. Start the editor server
|
|
||||||
with `john-slop-lsp`; the [editor guide](docs/editors.md) has ready-to-use Helix
|
|
||||||
and Vim configuration.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
John Slop provides three binaries backed by the same configuration and source
|
|
||||||
spans:
|
|
||||||
|
|
||||||
- `john-slop` detects copied assistant scaffolding, generic style residue,
|
|
||||||
source placeholders, tutorial narration, configured replacements, ASCII-like
|
|
||||||
Unicode, and invisible or bidirectional controls.
|
|
||||||
- `john-slop-fmt` previews, checks, or explicitly applies configured ASCII
|
|
||||||
replacements, including the default printable-ASCII UTS #39 profile.
|
|
||||||
- `john-slop-lsp` publishes Unicode diagnostics over stdio by default, offers
|
|
||||||
explicit weighted or all-diagnostic opt-ins, provides quick fixes for safe
|
|
||||||
replacements, and formats without writing editor-owned files.
|
|
||||||
|
|
||||||
The detector accepts UTF-8 files, directories, ZIP archives, and common
|
|
||||||
ZIP-based office documents. Each finding includes its source span, stable rule
|
|
||||||
ID, severity, message, and remediation help. A bounded evidence score aggregates
|
|
||||||
configured signals; it is not the probability that AI wrote the content.
|
|
||||||
|
|
||||||
Analysis is local and deterministic, with no network requests or telemetry.
|
|
||||||
False positives remain possible—especially in transcripts, tutorials,
|
|
||||||
multilingual text, and intentional typography—so findings are prompts for
|
|
||||||
review, not verdicts. Projects with multilingual text should consider the
|
|
||||||
Unicode profile's non-mutating `review` mode before writing formatted output.
|
|
||||||
|
|
||||||
## Learn progressively
|
|
||||||
|
|
||||||
Start with the guide closest to what you want to do:
|
|
||||||
|
|
||||||
1. [Use the CLI and understand exit statuses](docs/usage.md).
|
|
||||||
2. [Read or integrate the available output formats](docs/output.md).
|
|
||||||
3. [Configure rules, replacements, and layered overrides](docs/configuration.md),
|
|
||||||
with [commented configuration examples](examples/config/).
|
|
||||||
4. [Run the Rust and expected-failure examples](examples/README.md).
|
|
||||||
5. [Connect the formatter and LSP to Helix or Vim](docs/editors.md).
|
|
||||||
6. [Understand the review-signal methodology](docs/review-signals.md), the
|
|
||||||
[heuristics and evidence score](docs/heuristics.md), and then the
|
|
||||||
[interpretation limits](docs/limitations.md).
|
|
||||||
|
|
||||||
The [documentation index](docs/README.md) offers the same guided path with a
|
|
||||||
little more context.
|
|
||||||
|
|
||||||
## Advanced topics
|
|
||||||
|
|
||||||
Architecture, the public library API, evaluation results, performance,
|
|
||||||
dependency policy, design decisions, Unicode research, development, and release
|
|
||||||
engineering are collected in the [advanced guide](docs/advanced.md).
|
|
||||||
|
|
||||||
## Project
|
|
||||||
|
|
||||||
See [CONTRIBUTING.md](CONTRIBUTING.md) to work on the project and
|
|
||||||
[SECURITY.md](SECURITY.md) to report a vulnerability. The ordered implementation
|
|
||||||
plan lives in [tasks.md](tasks.md), and agent contribution rules live in
|
|
||||||
[AGENTS.md](AGENTS.md).
|
|
||||||
|
|
||||||
John Slop is distributed under the [MIT license](LICENSE).
|
|
||||||
|
|
|
||||||
22
SECURITY.md
22
SECURITY.md
|
|
@ -1,22 +0,0 @@
|
||||||
# Security policy
|
|
||||||
|
|
||||||
The current supported line is `0.1.x`. Older snapshots and untagged forks do not
|
|
||||||
receive security fixes.
|
|
||||||
|
|
||||||
Do not open a public issue for a suspected vulnerability. Use the repository
|
|
||||||
host's private security-advisory/reporting interface when available. Otherwise,
|
|
||||||
contact the repository owner privately through the hosting profile and request
|
|
||||||
a secure channel. Include the affected version/commit, operating system, impact,
|
|
||||||
minimal reproduction, and whether details have been shared elsewhere. Do not
|
|
||||||
include private documents processed by John Slop.
|
|
||||||
|
|
||||||
The maintainer will acknowledge a report within seven days, assess scope and
|
|
||||||
severity, coordinate a fix and disclosure date when confirmed, and credit the
|
|
||||||
reporter if requested. This is a best-effort volunteer response target, not a
|
|
||||||
service-level guarantee.
|
|
||||||
|
|
||||||
John Slop processes untrusted text and archives but is not a sandbox. Archive
|
|
||||||
limits and path validation reduce exposure; reports that bypass those limits,
|
|
||||||
cause unintended writes, leak document contents, corrupt protocol output, or
|
|
||||||
execute code are in scope. False-positive disagreements and requests for new
|
|
||||||
heuristics belong in the normal issue tracker.
|
|
||||||
119
benches/core.rs
119
benches/core.rs
|
|
@ -1,119 +0,0 @@
|
||||||
use std::{
|
|
||||||
fs::{self, File},
|
|
||||||
io::Write,
|
|
||||||
};
|
|
||||||
|
|
||||||
use criterion::{Criterion, Throughput, criterion_group, criterion_main};
|
|
||||||
use john_slop::{AnalysisContext, Config, Detector, read_inputs};
|
|
||||||
use tempfile::TempDir;
|
|
||||||
use zip::write::FileOptions;
|
|
||||||
|
|
||||||
fn benchmark_large_document(criterion: &mut Criterion) {
|
|
||||||
let config = Config::embedded();
|
|
||||||
let detector = Detector::new(&config).expect("embedded detector");
|
|
||||||
let source = "Reviewed prose without configured signals.\n".repeat(25_000);
|
|
||||||
let context = AnalysisContext::for_path("large.txt");
|
|
||||||
let mut group = criterion.benchmark_group("analysis");
|
|
||||||
group.throughput(Throughput::Bytes(source.len() as u64));
|
|
||||||
group.bench_function("one-megabyte-document", |bencher| {
|
|
||||||
bencher.iter(|| detector.analyze_with(&source, &context));
|
|
||||||
});
|
|
||||||
group.finish();
|
|
||||||
}
|
|
||||||
|
|
||||||
fn benchmark_many_small_files(criterion: &mut Criterion) {
|
|
||||||
let directory = small_file_tree(250);
|
|
||||||
let config = Config::embedded();
|
|
||||||
let mut group = criterion.benchmark_group("input");
|
|
||||||
group.throughput(Throughput::Elements(250));
|
|
||||||
group.bench_function("250-small-files", |bencher| {
|
|
||||||
bencher.iter(|| {
|
|
||||||
read_inputs(&config, directory.path())
|
|
||||||
.expect("open fixture directory")
|
|
||||||
.collect::<Result<Vec<_>, _>>()
|
|
||||||
.expect("read fixture directory")
|
|
||||||
});
|
|
||||||
});
|
|
||||||
group.finish();
|
|
||||||
}
|
|
||||||
|
|
||||||
fn benchmark_rule_scaling(criterion: &mut Criterion) {
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
let original = config.rules.clone();
|
|
||||||
for generation in 1..10 {
|
|
||||||
config
|
|
||||||
.rules
|
|
||||||
.extend(original.iter().cloned().map(|mut rule| {
|
|
||||||
rule.id = format!("{}-{generation}", rule.id);
|
|
||||||
rule
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
config.validate().expect("scaled config remains valid");
|
|
||||||
let source = "Assistant: Certainly! Here's a result—review it.\n".repeat(100);
|
|
||||||
let context = AnalysisContext::for_path("scaled.txt");
|
|
||||||
|
|
||||||
criterion.bench_function("rules/build-and-analyze-110-rules", |bencher| {
|
|
||||||
bencher.iter(|| {
|
|
||||||
Detector::new(&config)
|
|
||||||
.expect("build scaled detector")
|
|
||||||
.analyze_with(&source, &context)
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
fn benchmark_archive_limits(criterion: &mut Criterion) {
|
|
||||||
let directory = archive_with_members(100);
|
|
||||||
let path = directory.path().join("many.zip");
|
|
||||||
let config = Config::embedded();
|
|
||||||
let mut group = criterion.benchmark_group("input");
|
|
||||||
group.throughput(Throughput::Elements(100));
|
|
||||||
group.bench_function("100-member-archive", |bencher| {
|
|
||||||
bencher.iter(|| {
|
|
||||||
read_inputs(&config, &path)
|
|
||||||
.expect("open fixture archive")
|
|
||||||
.collect::<Result<Vec<_>, _>>()
|
|
||||||
.expect("read fixture archive")
|
|
||||||
});
|
|
||||||
});
|
|
||||||
group.finish();
|
|
||||||
}
|
|
||||||
|
|
||||||
fn small_file_tree(count: usize) -> TempDir {
|
|
||||||
let directory = tempfile::tempdir().expect("create benchmark directory");
|
|
||||||
for index in 0..count {
|
|
||||||
fs::write(
|
|
||||||
directory.path().join(format!("document-{index:04}.txt")),
|
|
||||||
"Reviewed prose without configured signals.\n",
|
|
||||||
)
|
|
||||||
.expect("write benchmark document");
|
|
||||||
}
|
|
||||||
directory
|
|
||||||
}
|
|
||||||
|
|
||||||
fn archive_with_members(count: usize) -> TempDir {
|
|
||||||
let directory = tempfile::tempdir().expect("create benchmark directory");
|
|
||||||
let file = File::create(directory.path().join("many.zip")).expect("create benchmark archive");
|
|
||||||
let mut archive = zip::ZipWriter::new(file);
|
|
||||||
let options: FileOptions<()> = FileOptions::default()
|
|
||||||
.compression_method(zip::CompressionMethod::Stored)
|
|
||||||
.unix_permissions(0o644);
|
|
||||||
for index in 0..count {
|
|
||||||
archive
|
|
||||||
.start_file(format!("document-{index:04}.txt"), options)
|
|
||||||
.expect("start benchmark member");
|
|
||||||
archive
|
|
||||||
.write_all(b"Reviewed prose without configured signals.\n")
|
|
||||||
.expect("write benchmark member");
|
|
||||||
}
|
|
||||||
archive.finish().expect("finish benchmark archive");
|
|
||||||
directory
|
|
||||||
}
|
|
||||||
|
|
||||||
criterion_group!(
|
|
||||||
benches,
|
|
||||||
benchmark_large_document,
|
|
||||||
benchmark_many_small_files,
|
|
||||||
benchmark_rule_scaling,
|
|
||||||
benchmark_archive_limits
|
|
||||||
);
|
|
||||||
criterion_main!(benches);
|
|
||||||
|
|
@ -0,0 +1,7 @@
|
||||||
|
[[patterns]]
|
||||||
|
regex = "\\p{Pd}"
|
||||||
|
replacement = "-"
|
||||||
|
[[patterns]]
|
||||||
|
regex = "[\u2018\u2019\u201A\u201B\u02BC\u02BB\u00B4\uFF40\uFF07]"
|
||||||
|
replacement = "'"
|
||||||
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
[package]
|
|
||||||
name = "john-slop-core"
|
|
||||||
version.workspace = true
|
|
||||||
edition.workspace = true
|
|
||||||
rust-version.workspace = true
|
|
||||||
license.workspace = true
|
|
||||||
repository.workspace = true
|
|
||||||
description = "Reusable heuristic analysis, configuration, input, and replacement engine for John Slop"
|
|
||||||
|
|
||||||
[dependencies]
|
|
||||||
directories.workspace = true
|
|
||||||
figment.workspace = true
|
|
||||||
globset.workspace = true
|
|
||||||
miette.workspace = true
|
|
||||||
regex.workspace = true
|
|
||||||
serde.workspace = true
|
|
||||||
thiserror.workspace = true
|
|
||||||
unicode-security.workspace = true
|
|
||||||
walkdir.workspace = true
|
|
||||||
zip.workspace = true
|
|
||||||
|
|
||||||
[dev-dependencies]
|
|
||||||
tempfile.workspace = true
|
|
||||||
toml.workspace = true
|
|
||||||
|
|
||||||
[lints]
|
|
||||||
workspace = true
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
# Embedded input defaults. Unicode replacements live in the generated
|
|
||||||
# `unicode.toml` policy asset so detection, diagnostics, and fixes stay aligned.
|
|
||||||
schema_version = 1
|
|
||||||
replacements = []
|
|
||||||
|
|
||||||
# Diagnostic category gates are applied in addition to each rule's `enabled`
|
|
||||||
# field. Set a category to false to silence that family without copying every
|
|
||||||
# embedded rule into this file.
|
|
||||||
[diagnostics.categories]
|
|
||||||
typography = true
|
|
||||||
style = true
|
|
||||||
prompt-leakage = true
|
|
||||||
source-code = true
|
|
||||||
|
|
||||||
# Disable or re-enable one diagnostic by its stable ID. Uncomment and change
|
|
||||||
# the ID as needed; `john-slop config show` lists the complete resolved rules.
|
|
||||||
# [[rules]]
|
|
||||||
# id = "templated-conclusion"
|
|
||||||
# enabled = false
|
|
||||||
|
|
||||||
[input]
|
|
||||||
max_document_bytes = 8388608
|
|
||||||
max_archive_bytes = 67108864
|
|
||||||
max_expanded_bytes = 268435456
|
|
||||||
max_archive_entries = 10000
|
|
||||||
max_compression_ratio = 100
|
|
||||||
max_walk_depth = 64
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
# Runtime diagnostic catalog. Copy or generate this as `diagnostics.toml` and
|
|
||||||
# add data-driven rules without rebuilding John Slop.
|
|
||||||
schema_version = 1
|
|
||||||
|
|
||||||
# This starter rule is disabled until its phrase and metadata fit your policy.
|
|
||||||
# Duplicate this complete table for additional diagnostics; every ID must be
|
|
||||||
# unique lowercase kebab-case.
|
|
||||||
[[rules]]
|
|
||||||
id = "custom-review-marker"
|
|
||||||
category = "style"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "REVIEW BEFORE PUBLISHING"
|
|
||||||
weight = 8.0
|
|
||||||
message = "a project review marker remains in the source"
|
|
||||||
help = "Resolve or remove the marker before publishing."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = false
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose", "source-code"]
|
|
||||||
extensions = []
|
|
||||||
include = []
|
|
||||||
exclude = []
|
|
||||||
skip_generated = true
|
|
||||||
skip_vendor = true
|
|
||||||
skip_minified = true
|
|
||||||
|
|
@ -1,280 +0,0 @@
|
||||||
# Embedded detection-rule metadata. Milestone 3 compiles and executes these
|
|
||||||
# matchers; Milestone 2 makes their configuration typed, layered, and validated.
|
|
||||||
schema_version = 1
|
|
||||||
|
|
||||||
[thresholds]
|
|
||||||
low = 20
|
|
||||||
medium = 50
|
|
||||||
high = 75
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-self-identification"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "As an AI language model"
|
|
||||||
weight = 30.0
|
|
||||||
message = "assistant self-identification appears in the source"
|
|
||||||
help = "Review whether an assistant response was copied into the document."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "knowledge-cutoff-disclaimer"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "knowledge cutoff"
|
|
||||||
weight = 20.0
|
|
||||||
message = "a knowledge-cutoff disclaimer appears in the source"
|
|
||||||
help = "Verify the statement and remove assistant boilerplate if it is not relevant."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-capability-disclaimer"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "I cannot browse the internet"
|
|
||||||
weight = 20.0
|
|
||||||
message = "an assistant capability disclaimer appears in the source"
|
|
||||||
help = "Replace generic capability boilerplate with a source-specific limitation if needed."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-answer-preface"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "Certainly! Here's"
|
|
||||||
weight = 15.0
|
|
||||||
message = "a stock assistant answer preface appears in the source"
|
|
||||||
help = "Review whether conversational response scaffolding was copied into the document."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "prompt-response-scaffolding"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?m)^(User|Assistant):[ \\t]"
|
|
||||||
weight = 25.0
|
|
||||||
message = "prompt/response scaffolding appears in the source"
|
|
||||||
help = "Confirm that chat transcript markers are intentional."
|
|
||||||
occurrence_cap = 2
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose", "source-code"]
|
|
||||||
extensions = []
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-follow-up-offer"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?i)\\b(?:let me know|tell me) if you(?:'d| would) like me to\\b"
|
|
||||||
weight = 12.0
|
|
||||||
message = "a conversational assistant follow-up offer appears in the source"
|
|
||||||
help = "Review whether a response-closing offer was copied into the document."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-revision-preface"
|
|
||||||
category = "prompt-leakage"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?i)\\bhere(?:'s| is) (?:a|the) (?:revised|polished|rewritten) (?:version|draft)\\b"
|
|
||||||
weight = 12.0
|
|
||||||
message = "an assistant-style revision preface appears in the source"
|
|
||||||
help = "Remove conversational revision scaffolding if it is not part of the document."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "templated-conclusion"
|
|
||||||
category = "style"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?i)\\bin conclusion,? (it is|we can)"
|
|
||||||
weight = 5.0
|
|
||||||
message = "a templated conclusion phrase appears in the source"
|
|
||||||
help = "Review whether the conclusion adds specific information."
|
|
||||||
occurrence_cap = 2
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "templated-filler"
|
|
||||||
category = "style"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?i)\\bit is (?:important|worth) (?:to note|noting) that\\b"
|
|
||||||
weight = 4.0
|
|
||||||
message = "a generic filler transition appears in the source"
|
|
||||||
help = "Prefer a direct, source-specific statement when the transition adds no meaning."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "repeated-transition"
|
|
||||||
category = "style"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "repeated-regex"
|
|
||||||
pattern = "(?im)^(?:Moreover|Furthermore|Additionally),[ \\t]"
|
|
||||||
weight = 4.0
|
|
||||||
message = "the same family of stock transitions is repeated"
|
|
||||||
help = "Review repeated transitions and keep only those that clarify the argument."
|
|
||||||
occurrence_cap = 3
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "editorial-insertion-placeholder"
|
|
||||||
category = "style"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?i)\\[(?:insert|add) (?:citation|reference|source|example|details?)(?: here)?\\]"
|
|
||||||
weight = 10.0
|
|
||||||
message = "an unresolved editorial insertion placeholder appears in the source"
|
|
||||||
help = "Supply the requested material or remove the placeholder before publishing."
|
|
||||||
occurrence_cap = 2
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "stock-helpful-closing"
|
|
||||||
category = "style"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?im)^[ \\t]*I hope this helps[.!][ \\t]*$"
|
|
||||||
weight = 2.0
|
|
||||||
message = "a stock conversational closing appears in the source"
|
|
||||||
help = "Keep the closing only when the document is intentionally conversational."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "excess-vocabulary-cluster"
|
|
||||||
category = "style"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "repeated-regex"
|
|
||||||
pattern = "(?i)\\b(?:delv(?:e|es|ed|ing)|intricate|multifaceted|pivotal|underscore[sd]?|realm|landscape)\\b"
|
|
||||||
weight = 1.0
|
|
||||||
message = "several terms from a population-level excess-vocabulary cluster appear"
|
|
||||||
help = "This weak corpus-level clue is not authorship evidence; review only for fit and specificity."
|
|
||||||
occurrence_cap = 3
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt", "rst", "adoc"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "source-placeholder-todo"
|
|
||||||
category = "source-code"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?i)\\bTODO:[ \\t]+(implement|add implementation)\\b"
|
|
||||||
weight = 10.0
|
|
||||||
message = "an implementation placeholder remains in source code"
|
|
||||||
help = "Implement the missing behavior or link the placeholder to tracked work."
|
|
||||||
occurrence_cap = 3
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["source-code"]
|
|
||||||
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "source-tutorial-narration"
|
|
||||||
category = "source-code"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?im)^[ \\t]*//[ \\t]*(?:first|next|finally),?[ \\t]+we\\b"
|
|
||||||
weight = 8.0
|
|
||||||
message = "a tutorial-style narration comment appears in source code"
|
|
||||||
help = "Describe why the code exists rather than narrating the next obvious operation."
|
|
||||||
occurrence_cap = 2
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["source-code"]
|
|
||||||
extensions = ["c", "cpp", "go", "java", "js", "rs", "ts"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "source-omitted-code-placeholder"
|
|
||||||
category = "source-code"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?im)^[ \\t]*(?://|#)[ \\t]*(?:\\.\\.\\.[ \\t]*)?(?:existing|remaining|rest of (?:the )?)[ \\t]+code(?:[ \\t]+(?:here|remains?|unchanged))?[ \\t]*$"
|
|
||||||
weight = 10.0
|
|
||||||
message = "an omitted-code placeholder remains in source code"
|
|
||||||
help = "Restore the omitted implementation or remove the placeholder."
|
|
||||||
occurrence_cap = 2
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["source-code"]
|
|
||||||
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "source-markdown-fence"
|
|
||||||
category = "source-code"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "(?m)^[ \\t]*```(?:[A-Za-z0-9_+-]+)?[ \\t]*$"
|
|
||||||
weight = 6.0
|
|
||||||
message = "a Markdown code fence appears in a source-code file"
|
|
||||||
help = "Remove response-formatting fences unless they are intentionally embedded data."
|
|
||||||
occurrence_cap = 2
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["source-code"]
|
|
||||||
extensions = ["c", "cpp", "go", "java", "js", "py", "rs", "ts"]
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,192 +0,0 @@
|
||||||
//! Regenerate `assets/unicode.toml` from reviewed mappings and Unicode Security.
|
|
||||||
|
|
||||||
use std::collections::BTreeSet;
|
|
||||||
|
|
||||||
use serde::Serialize;
|
|
||||||
|
|
||||||
const CURATED: &[(&str, &str, &str, &str)] = &[
|
|
||||||
("hyphen", "‐", "-", "Unicode hyphen"),
|
|
||||||
("non-breaking-hyphen", "‑", "-", "non-breaking hyphen"),
|
|
||||||
("figure-dash", "‒", "-", "figure dash"),
|
|
||||||
("en-dash", "–", "-", "en dash"),
|
|
||||||
("em-dash", "—", "-", "em dash"),
|
|
||||||
("horizontal-bar", "―", "-", "horizontal bar"),
|
|
||||||
("minus-sign", "−", "-", "mathematical minus sign"),
|
|
||||||
("small-em-dash", "﹘", "-", "small em dash"),
|
|
||||||
("small-hyphen-minus", "﹣", "-", "small hyphen-minus"),
|
|
||||||
];
|
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct Catalog {
|
|
||||||
schema_version: u32,
|
|
||||||
unicode: UnicodePolicy,
|
|
||||||
replacements: Vec<Replacement>,
|
|
||||||
rules: Vec<Rule>,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct UnicodePolicy {
|
|
||||||
confusables: &'static str,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct Replacement {
|
|
||||||
id: String,
|
|
||||||
source: String,
|
|
||||||
replacement: String,
|
|
||||||
rationale: String,
|
|
||||||
enabled: bool,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct Rule {
|
|
||||||
id: &'static str,
|
|
||||||
category: &'static str,
|
|
||||||
severity: &'static str,
|
|
||||||
matcher: &'static str,
|
|
||||||
pattern: &'static str,
|
|
||||||
weight: f64,
|
|
||||||
message: &'static str,
|
|
||||||
help: &'static str,
|
|
||||||
occurrence_cap: usize,
|
|
||||||
enabled: bool,
|
|
||||||
scope: Scope,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Serialize)]
|
|
||||||
struct Scope {
|
|
||||||
content: [&'static str; 2],
|
|
||||||
extensions: [String; 0],
|
|
||||||
}
|
|
||||||
|
|
||||||
fn main() {
|
|
||||||
let mut replacements = CURATED
|
|
||||||
.iter()
|
|
||||||
.map(|(id, source, replacement, name)| Replacement {
|
|
||||||
id: (*id).to_owned(),
|
|
||||||
source: (*source).to_owned(),
|
|
||||||
replacement: (*replacement).to_owned(),
|
|
||||||
rationale: format!("Replace the {name} with reviewed ASCII `{replacement}`."),
|
|
||||||
enabled: true,
|
|
||||||
})
|
|
||||||
.collect::<Vec<_>>();
|
|
||||||
|
|
||||||
replacements.push(Replacement {
|
|
||||||
id: "ideographic-space".to_owned(),
|
|
||||||
source: '\u{3000}'.to_string(),
|
|
||||||
replacement: " ".to_owned(),
|
|
||||||
rationale: "Width-fold the ideographic space to ASCII space.".to_owned(),
|
|
||||||
enabled: true,
|
|
||||||
});
|
|
||||||
for ascii in b'!'..=b'~' {
|
|
||||||
// U+FF01..=U+FF5E is a contiguous block corresponding to ASCII !..=~.
|
|
||||||
let value = u32::from(ascii) + 0xfee0;
|
|
||||||
let Some(fullwidth) = char::from_u32(value) else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
let id = if ascii == b'-' {
|
|
||||||
// Preserve the public ID shipped before the complete width-folding set.
|
|
||||||
"fullwidth-hyphen-minus".to_owned()
|
|
||||||
} else if printable_ascii_skeleton(fullwidth).is_some() {
|
|
||||||
// Preserve IDs already shipped as generated UTS #39 mappings.
|
|
||||||
format!("uts39-u{value:04x}")
|
|
||||||
} else {
|
|
||||||
format!("fullwidth-u{value:04x}")
|
|
||||||
};
|
|
||||||
let replacement = char::from(ascii).to_string();
|
|
||||||
replacements.push(Replacement {
|
|
||||||
id,
|
|
||||||
source: fullwidth.to_string(),
|
|
||||||
rationale: format!(
|
|
||||||
"Width-fold Unicode fullwidth form U+{value:04X} to ASCII {replacement:?}."
|
|
||||||
),
|
|
||||||
replacement,
|
|
||||||
enabled: true,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let reviewed_sources = replacements
|
|
||||||
.iter()
|
|
||||||
.filter_map(|rule| rule.source.chars().next())
|
|
||||||
.collect::<BTreeSet<_>>();
|
|
||||||
|
|
||||||
for value in 0..=char::MAX as u32 {
|
|
||||||
let Some(character) = char::from_u32(value) else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
if character.is_ascii() || reviewed_sources.contains(&character) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
let source = character.to_string();
|
|
||||||
let Some(skeleton) = printable_ascii_skeleton(character) else {
|
|
||||||
continue;
|
|
||||||
};
|
|
||||||
replacements.push(Replacement {
|
|
||||||
id: format!("uts39-u{value:04x}"),
|
|
||||||
source,
|
|
||||||
replacement: skeleton.clone(),
|
|
||||||
rationale: format!(
|
|
||||||
"Replace Unicode 16.0 UTS #39 lookalike U+{value:04X} with its printable ASCII skeleton {skeleton:?}."
|
|
||||||
),
|
|
||||||
enabled: true,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let catalog = Catalog {
|
|
||||||
schema_version: 1,
|
|
||||||
unicode: UnicodePolicy {
|
|
||||||
confusables: "replace",
|
|
||||||
},
|
|
||||||
replacements,
|
|
||||||
rules: vec![
|
|
||||||
Rule {
|
|
||||||
id: "unicode-ascii-replacement",
|
|
||||||
category: "typography",
|
|
||||||
severity: "advice",
|
|
||||||
matcher: "replacement",
|
|
||||||
pattern: "configured-replacements",
|
|
||||||
weight: 10.0,
|
|
||||||
message: "a Unicode character or sequence has a configured ASCII replacement",
|
|
||||||
help: "Review the character or apply the configured ASCII replacement.",
|
|
||||||
occurrence_cap: 5,
|
|
||||||
enabled: true,
|
|
||||||
scope: Scope {
|
|
||||||
content: ["prose", "source-code"],
|
|
||||||
extensions: [],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
Rule {
|
|
||||||
id: "unicode-invisible-control",
|
|
||||||
category: "typography",
|
|
||||||
severity: "warning",
|
|
||||||
matcher: "invisible",
|
|
||||||
pattern: "gitea-invisible-9c715c0",
|
|
||||||
weight: 12.0,
|
|
||||||
message: "an invisible or bidirectional Unicode control appears in the source",
|
|
||||||
help: "Review the hidden or direction-changing character; it is not removed automatically.",
|
|
||||||
occurrence_cap: 5,
|
|
||||||
enabled: true,
|
|
||||||
scope: Scope {
|
|
||||||
content: ["prose", "source-code"],
|
|
||||||
extensions: [],
|
|
||||||
},
|
|
||||||
},
|
|
||||||
],
|
|
||||||
};
|
|
||||||
|
|
||||||
println!(
|
|
||||||
"# Generated by `cargo run -p john-slop-core --example generate_unicode_catalog`.\n\
|
|
||||||
# Unicode Security 0.1.2 / Unicode 16.0.0 UTS #39, plus reviewed punctuation and width folding.\n\
|
|
||||||
# Do not edit generated mappings by hand; edit this generator and regenerate.\n{}",
|
|
||||||
toml::to_string_pretty(&catalog).expect("the catalog model must serialize")
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
fn printable_ascii_skeleton(character: char) -> Option<String> {
|
|
||||||
let source = character.to_string();
|
|
||||||
let skeleton = unicode_security::skeleton(&source).collect::<String>();
|
|
||||||
(!skeleton.is_empty()
|
|
||||||
&& skeleton.bytes().all(|byte| (b' '..=b'~').contains(&byte))
|
|
||||||
&& skeleton != source)
|
|
||||||
.then_some(skeleton)
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,943 +0,0 @@
|
||||||
//! Compiled, reusable registry of configured detection rules.
|
|
||||||
|
|
||||||
use std::{
|
|
||||||
collections::BTreeMap,
|
|
||||||
path::{Path, PathBuf},
|
|
||||||
};
|
|
||||||
|
|
||||||
use globset::{Glob, GlobSet, GlobSetBuilder};
|
|
||||||
use regex::Regex;
|
|
||||||
|
|
||||||
use crate::{
|
|
||||||
Analysis, Category, Config, ConfigValidationError, ContentKind, EvidenceBand, EvidenceScore,
|
|
||||||
Finding, HeuristicRule, MatcherType, RuleId, SourceId, replacement, unicode_review,
|
|
||||||
};
|
|
||||||
|
|
||||||
/// A validated heuristic definition paired with its reusable matcher.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct Rule {
|
|
||||||
definition: HeuristicRule,
|
|
||||||
matcher: CompiledMatcher,
|
|
||||||
scope: CompiledScope,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Rule {
|
|
||||||
/// Return this rule's stable identifier.
|
|
||||||
pub fn id(&self) -> &str {
|
|
||||||
&self.definition.id
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return this rule's evidence family.
|
|
||||||
pub const fn category(&self) -> Category {
|
|
||||||
self.definition.category
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return whether this rule is enabled in the resolved configuration.
|
|
||||||
pub const fn is_enabled(&self) -> bool {
|
|
||||||
self.definition.enabled
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the literal, regex source, or replacement-catalog sentinel.
|
|
||||||
pub fn pattern(&self) -> &str {
|
|
||||||
match &self.matcher {
|
|
||||||
CompiledMatcher::ExactPhrase(pattern) => pattern,
|
|
||||||
CompiledMatcher::Regex(regex) => regex.as_str(),
|
|
||||||
CompiledMatcher::RepeatedRegex(regex) => regex.as_str(),
|
|
||||||
CompiledMatcher::Replacement => "configured-replacements",
|
|
||||||
CompiledMatcher::Invisible => "gitea-invisible-9c715c0",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Input metadata used to select applicable rules without reading the source twice.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
||||||
pub struct AnalysisContext {
|
|
||||||
path: Option<PathBuf>,
|
|
||||||
content_kind: ContentKind,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl AnalysisContext {
|
|
||||||
/// Describe prose without a known filesystem path, such as standard input.
|
|
||||||
pub const fn prose() -> Self {
|
|
||||||
Self {
|
|
||||||
path: None,
|
|
||||||
content_kind: ContentKind::Prose,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Describe source code without a known filesystem path.
|
|
||||||
pub const fn source_code() -> Self {
|
|
||||||
Self {
|
|
||||||
path: None,
|
|
||||||
content_kind: ContentKind::SourceCode,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Infer the broad content kind from a path's extension.
|
|
||||||
pub fn for_path(path: impl Into<PathBuf>) -> Self {
|
|
||||||
let path = path.into();
|
|
||||||
let content_kind = infer_content_kind(&path);
|
|
||||||
Self {
|
|
||||||
path: Some(path),
|
|
||||||
content_kind,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Describe a loaded document using its exact source identity.
|
|
||||||
pub fn for_source(source: &SourceId) -> Self {
|
|
||||||
match source {
|
|
||||||
SourceId::Stdin => Self::prose(),
|
|
||||||
SourceId::File(path) => Self::for_path(path),
|
|
||||||
SourceId::ArchiveMember { member, .. } => Self::for_path(member),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the path used by extension and glob filters, when known.
|
|
||||||
pub fn path(&self) -> Option<&Path> {
|
|
||||||
self.path.as_deref()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the broad content kind selected for this input.
|
|
||||||
pub const fn content_kind(&self) -> ContentKind {
|
|
||||||
self.content_kind
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Default for AnalysisContext {
|
|
||||||
fn default() -> Self {
|
|
||||||
Self::prose()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Matcher compiled once when the detector registry is built.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
enum CompiledMatcher {
|
|
||||||
ExactPhrase(String),
|
|
||||||
Regex(Regex),
|
|
||||||
RepeatedRegex(Regex),
|
|
||||||
Replacement,
|
|
||||||
Invisible,
|
|
||||||
}
|
|
||||||
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
struct CompiledScope {
|
|
||||||
include: Option<GlobSet>,
|
|
||||||
exclude: GlobSet,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Reusable detection engine built from one resolved configuration snapshot.
|
|
||||||
#[derive(Debug, Clone)]
|
|
||||||
pub struct Detector {
|
|
||||||
config: Config,
|
|
||||||
rules: Vec<Rule>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Detector {
|
|
||||||
/// Validate a resolved configuration and compile every reusable matcher.
|
|
||||||
pub fn new(config: &Config) -> Result<Self, DetectorBuildError> {
|
|
||||||
config
|
|
||||||
.validate()
|
|
||||||
.map_err(|error| DetectorBuildError::InvalidConfig(Box::new(error)))?;
|
|
||||||
let rules = config
|
|
||||||
.rules
|
|
||||||
.iter()
|
|
||||||
.cloned()
|
|
||||||
.map(Rule::compile)
|
|
||||||
.collect::<Result<Vec<_>, _>>()?;
|
|
||||||
Ok(Self {
|
|
||||||
config: config.clone(),
|
|
||||||
rules,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the compiled registry in deterministic configuration order.
|
|
||||||
pub fn rules(&self) -> &[Rule] {
|
|
||||||
&self.rules
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Analyze text with the compiled registry.
|
|
||||||
pub fn analyze(&self, source: &str) -> Analysis {
|
|
||||||
self.analyze_with(source, &AnalysisContext::default())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Analyze text with explicit content and path applicability metadata.
|
|
||||||
pub fn analyze_with(&self, source: &str, context: &AnalysisContext) -> Analysis {
|
|
||||||
let mut findings = self
|
|
||||||
.rules
|
|
||||||
.iter()
|
|
||||||
.filter(|rule| {
|
|
||||||
rule.is_enabled()
|
|
||||||
&& self.config.diagnostics.category_enabled(rule.category())
|
|
||||||
&& rule.applies(context)
|
|
||||||
})
|
|
||||||
.flat_map(|rule| rule.findings(&self.config, source))
|
|
||||||
.collect::<Vec<_>>();
|
|
||||||
normalize_findings(&mut findings);
|
|
||||||
let evidence_score = aggregate_evidence_score(&self.config, &findings);
|
|
||||||
let evidence_band = (!findings.is_empty())
|
|
||||||
.then(|| band_for_score(self.config.thresholds, evidence_score.score()));
|
|
||||||
|
|
||||||
Analysis {
|
|
||||||
findings,
|
|
||||||
evidence_score,
|
|
||||||
evidence_band,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn normalize_findings(findings: &mut Vec<Finding>) {
|
|
||||||
findings.sort_by(|left, right| {
|
|
||||||
(left.byte_span.start, left.byte_span.end, &left.rule_id).cmp(&(
|
|
||||||
right.byte_span.start,
|
|
||||||
right.byte_span.end,
|
|
||||||
&right.rule_id,
|
|
||||||
))
|
|
||||||
});
|
|
||||||
findings
|
|
||||||
.dedup_by(|right, left| right.rule_id == left.rule_id && right.byte_span == left.byte_span);
|
|
||||||
}
|
|
||||||
|
|
||||||
fn aggregate_evidence_score(config: &Config, findings: &[Finding]) -> EvidenceScore {
|
|
||||||
let mut family_rule_counts = BTreeMap::<Category, usize>::new();
|
|
||||||
let mut evidence = 0.0;
|
|
||||||
|
|
||||||
for rule in &config.rules {
|
|
||||||
let occurrences = findings
|
|
||||||
.iter()
|
|
||||||
.filter(|finding| finding.rule_id.as_str() == rule.id)
|
|
||||||
.count()
|
|
||||||
.min(rule.occurrence_cap);
|
|
||||||
if occurrences == 0 {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
let occurrence_factor = (0..occurrences)
|
|
||||||
.map(|index| 0.5_f64.powi(i32::try_from(index).unwrap_or(i32::MAX)))
|
|
||||||
.sum::<f64>();
|
|
||||||
let family_index = family_rule_counts.entry(rule.category).or_default();
|
|
||||||
let family_factor =
|
|
||||||
family_decay(rule.category).powi(i32::try_from(*family_index).unwrap_or(i32::MAX));
|
|
||||||
evidence += rule.weight * occurrence_factor * family_factor;
|
|
||||||
*family_index += 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
let normalized = if evidence == 0.0 {
|
|
||||||
0.0
|
|
||||||
} else {
|
|
||||||
100.0 * evidence / (evidence + 50.0)
|
|
||||||
};
|
|
||||||
EvidenceScore::from_score(normalized.round() as u8)
|
|
||||||
}
|
|
||||||
|
|
||||||
const fn family_decay(category: Category) -> f64 {
|
|
||||||
match category {
|
|
||||||
Category::PromptLeakage => 0.90,
|
|
||||||
Category::Typography | Category::Style | Category::SourceCode => 0.75,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn band_for_score(thresholds: crate::EvidenceThresholds, score: u8) -> EvidenceBand {
|
|
||||||
if score >= thresholds.high {
|
|
||||||
EvidenceBand::High
|
|
||||||
} else if score >= thresholds.medium {
|
|
||||||
EvidenceBand::Medium
|
|
||||||
} else if score >= thresholds.low {
|
|
||||||
EvidenceBand::Low
|
|
||||||
} else {
|
|
||||||
EvidenceBand::BelowThreshold
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Rule {
|
|
||||||
fn compile(definition: HeuristicRule) -> Result<Self, DetectorBuildError> {
|
|
||||||
let matcher = match definition.matcher {
|
|
||||||
MatcherType::ExactPhrase => CompiledMatcher::ExactPhrase(definition.pattern.clone()),
|
|
||||||
MatcherType::Regex => {
|
|
||||||
let regex = Regex::new(&definition.pattern).map_err(|source| {
|
|
||||||
DetectorBuildError::InvalidRegex {
|
|
||||||
id: definition.id.clone(),
|
|
||||||
source,
|
|
||||||
}
|
|
||||||
})?;
|
|
||||||
CompiledMatcher::Regex(regex)
|
|
||||||
}
|
|
||||||
MatcherType::RepeatedRegex => {
|
|
||||||
let regex = Regex::new(&definition.pattern).map_err(|source| {
|
|
||||||
DetectorBuildError::InvalidRegex {
|
|
||||||
id: definition.id.clone(),
|
|
||||||
source,
|
|
||||||
}
|
|
||||||
})?;
|
|
||||||
CompiledMatcher::RepeatedRegex(regex)
|
|
||||||
}
|
|
||||||
MatcherType::Replacement => CompiledMatcher::Replacement,
|
|
||||||
MatcherType::Invisible => CompiledMatcher::Invisible,
|
|
||||||
};
|
|
||||||
let scope = CompiledScope::compile(&definition)?;
|
|
||||||
Ok(Self {
|
|
||||||
definition,
|
|
||||||
matcher,
|
|
||||||
scope,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn applies(&self, context: &AnalysisContext) -> bool {
|
|
||||||
if !self
|
|
||||||
.definition
|
|
||||||
.scope
|
|
||||||
.content
|
|
||||||
.contains(&context.content_kind)
|
|
||||||
{
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
let Some(path) = context.path() else {
|
|
||||||
return self.scope.include.is_none();
|
|
||||||
};
|
|
||||||
if !self.definition.scope.extensions.is_empty()
|
|
||||||
&& !path
|
|
||||||
.extension()
|
|
||||||
.and_then(|extension| extension.to_str())
|
|
||||||
.is_some_and(|extension| {
|
|
||||||
self.definition
|
|
||||||
.scope
|
|
||||||
.extensions
|
|
||||||
.iter()
|
|
||||||
.any(|allowed| extension.eq_ignore_ascii_case(allowed))
|
|
||||||
})
|
|
||||||
{
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
if self
|
|
||||||
.scope
|
|
||||||
.include
|
|
||||||
.as_ref()
|
|
||||||
.is_some_and(|include| !include.is_match(path))
|
|
||||||
|| self.scope.exclude.is_match(path)
|
|
||||||
{
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
!(self.definition.scope.skip_generated && is_generated(path)
|
|
||||||
|| self.definition.scope.skip_vendor && is_vendor(path)
|
|
||||||
|| self.definition.scope.skip_minified && is_minified(path))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn findings(&self, config: &Config, source: &str) -> Vec<Finding> {
|
|
||||||
match &self.matcher {
|
|
||||||
CompiledMatcher::ExactPhrase(pattern) => source
|
|
||||||
.match_indices(pattern)
|
|
||||||
.map(|(start, matched)| self.finding(source, start..start + matched.len(), None))
|
|
||||||
.collect(),
|
|
||||||
CompiledMatcher::Regex(regex) => regex
|
|
||||||
.find_iter(source)
|
|
||||||
.map(|matched| self.finding(source, matched.range(), None))
|
|
||||||
.collect(),
|
|
||||||
CompiledMatcher::RepeatedRegex(regex) => {
|
|
||||||
let matches = regex.find_iter(source).collect::<Vec<_>>();
|
|
||||||
if matches.len() < 2 {
|
|
||||||
Vec::new()
|
|
||||||
} else {
|
|
||||||
matches
|
|
||||||
.into_iter()
|
|
||||||
.map(|matched| self.finding(source, matched.range(), None))
|
|
||||||
.collect()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
CompiledMatcher::Replacement => {
|
|
||||||
let mut findings = replacement::plan(config, source)
|
|
||||||
.into_iter()
|
|
||||||
.map(|edit| self.finding(source, edit.byte_span.clone(), Some(edit)))
|
|
||||||
.collect::<Vec<_>>();
|
|
||||||
findings.extend(
|
|
||||||
replacement::plan_confusables_for_review(config, source)
|
|
||||||
.into_iter()
|
|
||||||
.map(|edit| self.finding(source, edit.byte_span, None)),
|
|
||||||
);
|
|
||||||
findings
|
|
||||||
}
|
|
||||||
CompiledMatcher::Invisible => source
|
|
||||||
.char_indices()
|
|
||||||
.filter(|(start, character)| {
|
|
||||||
unicode_review::is_gitea_invisible(*character)
|
|
||||||
&& !(*character == '\r'
|
|
||||||
&& source.as_bytes().get(start + 1) == Some(&b'\n'))
|
|
||||||
})
|
|
||||||
.map(|(start, character)| {
|
|
||||||
let mut finding =
|
|
||||||
self.finding(source, start..start + character.len_utf8(), None);
|
|
||||||
finding.help = Some(format!(
|
|
||||||
"Review the hidden or direction-changing character U+{:04X}. It is never removed automatically.",
|
|
||||||
character as u32
|
|
||||||
));
|
|
||||||
finding
|
|
||||||
})
|
|
||||||
.collect(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn finding(
|
|
||||||
&self,
|
|
||||||
source: &str,
|
|
||||||
byte_span: std::ops::Range<usize>,
|
|
||||||
fix: Option<crate::TextEdit>,
|
|
||||||
) -> Finding {
|
|
||||||
Finding {
|
|
||||||
rule_id: RuleId::new(self.id()),
|
|
||||||
category: self.category(),
|
|
||||||
severity: self.definition.severity,
|
|
||||||
message: self.definition.message.clone(),
|
|
||||||
help: Some(self.definition.help.clone()),
|
|
||||||
matched_text: source[byte_span.clone()].to_owned(),
|
|
||||||
byte_span,
|
|
||||||
weight: self.definition.weight,
|
|
||||||
fix,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl CompiledScope {
|
|
||||||
fn compile(rule: &HeuristicRule) -> Result<Self, DetectorBuildError> {
|
|
||||||
Ok(Self {
|
|
||||||
include: if rule.scope.include.is_empty() {
|
|
||||||
None
|
|
||||||
} else {
|
|
||||||
Some(compile_globs(rule, "include", &rule.scope.include)?)
|
|
||||||
},
|
|
||||||
exclude: compile_globs(rule, "exclude", &rule.scope.exclude)?,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn compile_globs(
|
|
||||||
rule: &HeuristicRule,
|
|
||||||
field: &'static str,
|
|
||||||
patterns: &[String],
|
|
||||||
) -> Result<GlobSet, DetectorBuildError> {
|
|
||||||
let mut builder = GlobSetBuilder::new();
|
|
||||||
for pattern in patterns {
|
|
||||||
builder.add(
|
|
||||||
Glob::new(pattern).map_err(|source| DetectorBuildError::InvalidGlob {
|
|
||||||
id: rule.id.clone(),
|
|
||||||
field,
|
|
||||||
pattern: pattern.clone(),
|
|
||||||
source,
|
|
||||||
})?,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
builder
|
|
||||||
.build()
|
|
||||||
.map_err(|source| DetectorBuildError::InvalidGlob {
|
|
||||||
id: rule.id.clone(),
|
|
||||||
field,
|
|
||||||
pattern: patterns.join(", "),
|
|
||||||
source,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn infer_content_kind(path: &Path) -> ContentKind {
|
|
||||||
const SOURCE_EXTENSIONS: &[&str] = &[
|
|
||||||
"c", "cc", "cpp", "cs", "go", "h", "hpp", "java", "js", "jsx", "kt", "php", "py", "rb",
|
|
||||||
"rs", "swift", "ts", "tsx",
|
|
||||||
];
|
|
||||||
if path
|
|
||||||
.extension()
|
|
||||||
.and_then(|extension| extension.to_str())
|
|
||||||
.is_some_and(|extension| {
|
|
||||||
SOURCE_EXTENSIONS
|
|
||||||
.iter()
|
|
||||||
.any(|candidate| extension.eq_ignore_ascii_case(candidate))
|
|
||||||
})
|
|
||||||
{
|
|
||||||
ContentKind::SourceCode
|
|
||||||
} else {
|
|
||||||
ContentKind::Prose
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn has_component(path: &Path, candidates: &[&str]) -> bool {
|
|
||||||
path.components().any(|component| {
|
|
||||||
component.as_os_str().to_str().is_some_and(|value| {
|
|
||||||
candidates
|
|
||||||
.iter()
|
|
||||||
.any(|candidate| value.eq_ignore_ascii_case(candidate))
|
|
||||||
})
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn is_generated(path: &Path) -> bool {
|
|
||||||
has_component(path, &["generated", "gen", "dist", "build", "target"])
|
|
||||||
}
|
|
||||||
|
|
||||||
fn is_vendor(path: &Path) -> bool {
|
|
||||||
has_component(
|
|
||||||
path,
|
|
||||||
&[
|
|
||||||
"vendor",
|
|
||||||
"vendors",
|
|
||||||
"node_modules",
|
|
||||||
"third_party",
|
|
||||||
"third-party",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn is_minified(path: &Path) -> bool {
|
|
||||||
path.file_name()
|
|
||||||
.and_then(|name| name.to_str())
|
|
||||||
.is_some_and(|name| {
|
|
||||||
let name = name.to_ascii_lowercase();
|
|
||||||
name.contains(".min.") || name.ends_with("-min.js") || name.ends_with("-min.css")
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Failure while constructing a reusable detector registry.
|
|
||||||
#[derive(Debug, thiserror::Error)]
|
|
||||||
pub enum DetectorBuildError {
|
|
||||||
/// The resolved configuration was not semantically valid.
|
|
||||||
#[error("cannot build detector from invalid configuration")]
|
|
||||||
InvalidConfig(#[source] Box<ConfigValidationError>),
|
|
||||||
/// A configured regular expression could not be compiled.
|
|
||||||
#[error("failed to compile matcher for rule `{id}`")]
|
|
||||||
InvalidRegex {
|
|
||||||
/// Stable rule identifier.
|
|
||||||
id: String,
|
|
||||||
/// Regex compiler failure.
|
|
||||||
#[source]
|
|
||||||
source: regex::Error,
|
|
||||||
},
|
|
||||||
/// A configured path applicability glob could not be compiled.
|
|
||||||
#[error("failed to compile {field} glob `{pattern}` for rule `{id}`")]
|
|
||||||
InvalidGlob {
|
|
||||||
/// Stable rule identifier.
|
|
||||||
id: String,
|
|
||||||
/// Scope field containing the glob.
|
|
||||||
field: &'static str,
|
|
||||||
/// Invalid glob source.
|
|
||||||
pattern: String,
|
|
||||||
/// Glob compiler failure.
|
|
||||||
#[source]
|
|
||||||
source: globset::Error,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn registry_is_built_from_the_validated_configuration() {
|
|
||||||
let config = Config::embedded();
|
|
||||||
let detector = Detector::new(&config).unwrap();
|
|
||||||
|
|
||||||
assert_eq!(detector.rules().len(), config.rules.len());
|
|
||||||
assert_eq!(detector.rules()[0].id(), config.rules[0].id);
|
|
||||||
assert_eq!(detector.rules()[0].pattern(), config.rules[0].pattern);
|
|
||||||
assert_eq!(
|
|
||||||
detector
|
|
||||||
.rules()
|
|
||||||
.iter()
|
|
||||||
.filter(|rule| rule.is_enabled())
|
|
||||||
.count(),
|
|
||||||
config.rules.iter().filter(|rule| rule.enabled).count()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn replacement_rule_preserves_explainable_findings() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let analysis = detector.analyze("ordinary-word and suspicious—dash");
|
|
||||||
|
|
||||||
assert_eq!(analysis.findings.len(), 1);
|
|
||||||
assert_eq!(
|
|
||||||
analysis.findings[0].rule_id.as_str(),
|
|
||||||
"unicode-ascii-replacement"
|
|
||||||
);
|
|
||||||
assert_eq!(analysis.findings[0].matched_text, "—");
|
|
||||||
assert_eq!(analysis.findings[0].fix.as_ref().unwrap().replacement, "-");
|
|
||||||
assert_eq!(analysis.evidence_score.score(), 17);
|
|
||||||
assert_eq!(analysis.evidence_band, Some(EvidenceBand::BelowThreshold));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn category_and_individual_switches_are_both_required() {
|
|
||||||
let source = "In conclusion, it is complete.";
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
config.diagnostics.categories.style = false;
|
|
||||||
assert!(
|
|
||||||
Detector::new(&config)
|
|
||||||
.unwrap()
|
|
||||||
.analyze(source)
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
|
|
||||||
config.diagnostics.categories.style = true;
|
|
||||||
config
|
|
||||||
.rules
|
|
||||||
.iter_mut()
|
|
||||||
.find(|rule| rule.id == "templated-conclusion")
|
|
||||||
.unwrap()
|
|
||||||
.enabled = false;
|
|
||||||
assert!(
|
|
||||||
Detector::new(&config)
|
|
||||||
.unwrap()
|
|
||||||
.analyze(source)
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
|
|
||||||
config
|
|
||||||
.rules
|
|
||||||
.iter_mut()
|
|
||||||
.find(|rule| rule.id == "templated-conclusion")
|
|
||||||
.unwrap()
|
|
||||||
.enabled = true;
|
|
||||||
assert_eq!(
|
|
||||||
Detector::new(&config)
|
|
||||||
.unwrap()
|
|
||||||
.analyze(source)
|
|
||||||
.findings
|
|
||||||
.len(),
|
|
||||||
1
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn unicode_review_findings_have_exact_spans_and_safe_fixes() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let source = "é pаss\u{200b}word\u{202e}\r\nnext";
|
|
||||||
let findings = detector.analyze(source).findings;
|
|
||||||
|
|
||||||
assert_eq!(findings.len(), 3);
|
|
||||||
assert_eq!(findings[0].rule_id.as_str(), "unicode-ascii-replacement");
|
|
||||||
assert_eq!(findings[0].matched_text, "а");
|
|
||||||
assert_eq!(&source[findings[0].byte_span.clone()], "а");
|
|
||||||
assert_eq!(findings[0].fix.as_ref().unwrap().replacement, "a");
|
|
||||||
assert_eq!(
|
|
||||||
findings
|
|
||||||
.iter()
|
|
||||||
.filter(|finding| finding.rule_id.as_str() == "unicode-invisible-control")
|
|
||||||
.count(),
|
|
||||||
2
|
|
||||||
);
|
|
||||||
assert!(findings[1..].iter().all(|finding| finding.fix.is_none()));
|
|
||||||
assert!(findings.iter().all(|finding| finding.matched_text != "\r"));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn confusable_policy_switches_between_fix_review_and_off() {
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
|
|
||||||
config.unicode.confusables = crate::ConfusablePolicy::Review;
|
|
||||||
let review = Detector::new(&config).unwrap().analyze("pаss");
|
|
||||||
assert_eq!(review.findings.len(), 1);
|
|
||||||
assert!(review.findings[0].fix.is_none());
|
|
||||||
|
|
||||||
config.unicode.confusables = crate::ConfusablePolicy::Off;
|
|
||||||
assert!(
|
|
||||||
Detector::new(&config)
|
|
||||||
.unwrap()
|
|
||||||
.analyze("pаss")
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn safe_replacement_characters_are_not_duplicated_as_confusable_findings() {
|
|
||||||
let findings = Detector::new(&Config::embedded())
|
|
||||||
.unwrap()
|
|
||||||
.analyze("one—two")
|
|
||||||
.findings;
|
|
||||||
|
|
||||||
assert_eq!(findings.len(), 1);
|
|
||||||
assert_eq!(findings[0].rule_id.as_str(), "unicode-ascii-replacement");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn compiled_regex_uses_utf8_byte_offsets() {
|
|
||||||
let regex = Regex::new(r"\u{1D306}").unwrap();
|
|
||||||
let matched = regex.find("a𝌆b").unwrap();
|
|
||||||
|
|
||||||
assert_eq!(matched.range(), 1..5);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn exact_phrase_findings_use_configured_metadata_and_byte_spans() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let source = "préface: As an AI language model, I cannot verify this.";
|
|
||||||
|
|
||||||
let finding = detector
|
|
||||||
.analyze(source)
|
|
||||||
.findings
|
|
||||||
.into_iter()
|
|
||||||
.find(|finding| finding.rule_id.as_str() == "assistant-self-identification")
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
let start = source.find("As an AI language model").unwrap();
|
|
||||||
assert_eq!(finding.byte_span, start..start + finding.matched_text.len());
|
|
||||||
assert_eq!(finding.matched_text, "As an AI language model");
|
|
||||||
assert_eq!(finding.category, Category::PromptLeakage);
|
|
||||||
assert_eq!(finding.weight, 30.0);
|
|
||||||
assert!(finding.message.contains("self-identification"));
|
|
||||||
assert!(finding.help.unwrap().contains("copied"));
|
|
||||||
assert!(finding.fix.is_none());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn regex_findings_cover_the_exact_utf8_source_slice() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let source = "préface\nAssistant: generated response";
|
|
||||||
|
|
||||||
let finding = detector
|
|
||||||
.analyze(source)
|
|
||||||
.findings
|
|
||||||
.into_iter()
|
|
||||||
.find(|finding| finding.rule_id.as_str() == "prompt-response-scaffolding")
|
|
||||||
.unwrap();
|
|
||||||
|
|
||||||
assert_eq!(&source[finding.byte_span.clone()], finding.matched_text);
|
|
||||||
assert_eq!(finding.matched_text, "Assistant: ");
|
|
||||||
assert_eq!(finding.category, Category::PromptLeakage);
|
|
||||||
assert_eq!(finding.weight, 25.0);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn embedded_catalog_covers_each_conservative_signal_family() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let ids = detector
|
|
||||||
.rules()
|
|
||||||
.iter()
|
|
||||||
.map(Rule::id)
|
|
||||||
.collect::<std::collections::BTreeSet<_>>();
|
|
||||||
|
|
||||||
for expected in [
|
|
||||||
"assistant-self-identification",
|
|
||||||
"assistant-capability-disclaimer",
|
|
||||||
"assistant-answer-preface",
|
|
||||||
"knowledge-cutoff-disclaimer",
|
|
||||||
"prompt-response-scaffolding",
|
|
||||||
"templated-filler",
|
|
||||||
"templated-conclusion",
|
|
||||||
"repeated-transition",
|
|
||||||
"source-placeholder-todo",
|
|
||||||
"source-tutorial-narration",
|
|
||||||
"unicode-ascii-replacement",
|
|
||||||
] {
|
|
||||||
assert!(ids.contains(expected), "missing {expected}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn repeated_transition_requires_multiple_occurrences() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let one = detector.analyze("Moreover, one transition is ordinary.");
|
|
||||||
let repeated =
|
|
||||||
detector.analyze("Moreover, the first claim.\nAdditionally, the second claim.");
|
|
||||||
|
|
||||||
assert!(
|
|
||||||
!one.findings
|
|
||||||
.iter()
|
|
||||||
.any(|finding| finding.rule_id.as_str() == "repeated-transition")
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
repeated
|
|
||||||
.findings
|
|
||||||
.iter()
|
|
||||||
.filter(|finding| finding.rule_id.as_str() == "repeated-transition")
|
|
||||||
.count(),
|
|
||||||
2
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn content_and_extension_scopes_select_the_expected_rules() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let text = "TODO: implement the parser";
|
|
||||||
|
|
||||||
assert!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::prose())
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::for_path("src/parser.rs"))
|
|
||||||
.findings
|
|
||||||
.len(),
|
|
||||||
1
|
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::for_path("src/parser.md"))
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn include_and_exclude_globs_are_compiled_and_applied() {
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
let rule = config
|
|
||||||
.rules
|
|
||||||
.iter_mut()
|
|
||||||
.find(|rule| rule.id == "assistant-self-identification")
|
|
||||||
.unwrap();
|
|
||||||
rule.scope.include = vec!["docs/**".to_owned()];
|
|
||||||
rule.scope.exclude = vec!["docs/archive/**".to_owned()];
|
|
||||||
let detector = Detector::new(&config).unwrap();
|
|
||||||
let text = "As an AI language model, I can help.";
|
|
||||||
|
|
||||||
assert_eq!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::for_path("docs/guide.md"))
|
|
||||||
.findings
|
|
||||||
.len(),
|
|
||||||
1
|
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::for_path("notes/guide.md"))
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::for_path("docs/archive/guide.md"))
|
|
||||||
.findings
|
|
||||||
.is_empty()
|
|
||||||
);
|
|
||||||
assert!(detector.analyze(text).findings.is_empty());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn generated_vendor_and_minified_paths_are_skipped_by_default() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let text = "TODO: implement the parser";
|
|
||||||
|
|
||||||
for path in [
|
|
||||||
"generated/parser.rs",
|
|
||||||
"vendor/parser.rs",
|
|
||||||
"node_modules/parser.js",
|
|
||||||
"src/parser.min.js",
|
|
||||||
] {
|
|
||||||
assert!(
|
|
||||||
detector
|
|
||||||
.analyze_with(text, &AnalysisContext::for_path(path))
|
|
||||||
.findings
|
|
||||||
.is_empty(),
|
|
||||||
"unexpected finding for {path}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn archive_members_use_the_inner_path_for_applicability() {
|
|
||||||
let context = AnalysisContext::for_source(&SourceId::ArchiveMember {
|
|
||||||
archive: PathBuf::from("bundle.zip"),
|
|
||||||
member: PathBuf::from("src/lib.rs"),
|
|
||||||
});
|
|
||||||
|
|
||||||
assert_eq!(context.path(), Some(Path::new("src/lib.rs")));
|
|
||||||
assert_eq!(context.content_kind(), ContentKind::SourceCode);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn evidence_score_caps_occurrences_and_applies_diminishing_returns() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let five = detector.analyze(&"—".repeat(5));
|
|
||||||
let twenty = detector.analyze(&"—".repeat(20));
|
|
||||||
|
|
||||||
assert_eq!(five.evidence_score, twenty.evidence_score);
|
|
||||||
assert_eq!(five.evidence_score.score(), 28);
|
|
||||||
assert_eq!(twenty.findings.len(), 20);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn evidence_score_damps_distinct_rules_in_the_same_family() {
|
|
||||||
let source = "As an AI language model, I cannot browse the internet.";
|
|
||||||
let correlated = Detector::new(&Config::embedded()).unwrap().analyze(source);
|
|
||||||
let mut independent_config = Config::embedded();
|
|
||||||
independent_config
|
|
||||||
.rules
|
|
||||||
.iter_mut()
|
|
||||||
.find(|rule| rule.id == "assistant-capability-disclaimer")
|
|
||||||
.unwrap()
|
|
||||||
.category = Category::Style;
|
|
||||||
let independent = Detector::new(&independent_config).unwrap().analyze(source);
|
|
||||||
|
|
||||||
assert!(correlated.evidence_score < independent.evidence_score);
|
|
||||||
assert_eq!(correlated.findings.len(), independent.findings.len());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn prompt_leakage_uses_lighter_family_damping_than_other_categories() {
|
|
||||||
assert_eq!(family_decay(Category::PromptLeakage), 0.90);
|
|
||||||
assert_eq!(family_decay(Category::Style), 0.75);
|
|
||||||
assert_eq!(family_decay(Category::Typography), 0.75);
|
|
||||||
assert_eq!(family_decay(Category::SourceCode), 0.75);
|
|
||||||
|
|
||||||
let source = concat!(
|
|
||||||
"User: request\n",
|
|
||||||
"Assistant: Certainly! Here's a response.\n",
|
|
||||||
"As an AI language model, I cannot browse the internet. ",
|
|
||||||
"My knowledge cutoff applies."
|
|
||||||
);
|
|
||||||
let analysis = Detector::new(&Config::embedded()).unwrap().analyze(source);
|
|
||||||
|
|
||||||
assert_eq!(analysis.findings.len(), 6);
|
|
||||||
assert_eq!(analysis.evidence_score.score(), 67);
|
|
||||||
assert_eq!(analysis.evidence_band, Some(EvidenceBand::Medium));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn configured_thresholds_define_named_band_boundaries() {
|
|
||||||
let thresholds = crate::EvidenceThresholds {
|
|
||||||
low: 20,
|
|
||||||
medium: 50,
|
|
||||||
high: 75,
|
|
||||||
};
|
|
||||||
|
|
||||||
assert_eq!(band_for_score(thresholds, 19), EvidenceBand::BelowThreshold);
|
|
||||||
assert_eq!(band_for_score(thresholds, 20), EvidenceBand::Low);
|
|
||||||
assert_eq!(band_for_score(thresholds, 50), EvidenceBand::Medium);
|
|
||||||
assert_eq!(band_for_score(thresholds, 75), EvidenceBand::High);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn one_weak_repeated_signal_cannot_reach_high_evidence() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let source = "Moreover, first.\nAdditionally, second.\nFurthermore, third.";
|
|
||||||
let analysis = detector.analyze(source);
|
|
||||||
|
|
||||||
assert_eq!(analysis.findings.len(), 3);
|
|
||||||
assert_ne!(analysis.evidence_band, Some(EvidenceBand::High));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn empty_and_clean_text_have_zero_evidence_and_no_verdict() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
|
|
||||||
for source in ["", "A specific, carefully sourced statement."] {
|
|
||||||
let analysis = detector.analyze(source);
|
|
||||||
assert!(analysis.findings.is_empty());
|
|
||||||
assert_eq!(analysis.evidence_score.score(), 0);
|
|
||||||
assert_eq!(analysis.evidence_band, None);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn findings_are_sorted_and_identical_rule_spans_are_deduplicated() {
|
|
||||||
let detector = Detector::new(&Config::embedded()).unwrap();
|
|
||||||
let mut findings = detector.analyze("As an AI language model—").findings;
|
|
||||||
findings.reverse();
|
|
||||||
findings.push(findings[0].clone());
|
|
||||||
|
|
||||||
normalize_findings(&mut findings);
|
|
||||||
|
|
||||||
assert_eq!(findings.len(), 2);
|
|
||||||
assert!(findings[0].byte_span.start < findings[1].byte_span.start);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,156 +0,0 @@
|
||||||
//! Rich source diagnostics derived from presentation-neutral findings.
|
|
||||||
|
|
||||||
use std::{error::Error, fmt};
|
|
||||||
|
|
||||||
use miette::{Diagnostic, LabeledSpan, NamedSource, Severity, SourceCode, SourceSpan};
|
|
||||||
|
|
||||||
use crate::{Finding, FindingSeverity, RuleId, SourceId};
|
|
||||||
|
|
||||||
/// Owned `miette` diagnostic for one finding against an exact source snapshot.
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub struct FindingDiagnostic {
|
|
||||||
rule_id: RuleId,
|
|
||||||
severity: FindingSeverity,
|
|
||||||
message: String,
|
|
||||||
help: Option<String>,
|
|
||||||
matched_text: String,
|
|
||||||
source: NamedSource<String>,
|
|
||||||
span: SourceSpan,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl FindingDiagnostic {
|
|
||||||
/// Create a diagnostic by binding a finding to its named source snapshot.
|
|
||||||
pub fn new(source_id: &SourceId, source: &str, finding: &Finding) -> Self {
|
|
||||||
Self {
|
|
||||||
rule_id: finding.rule_id.clone(),
|
|
||||||
severity: finding.severity,
|
|
||||||
message: finding.message.clone(),
|
|
||||||
help: finding.help.clone(),
|
|
||||||
matched_text: finding.matched_text.clone(),
|
|
||||||
source: NamedSource::new(source_id.to_string(), source.to_owned()),
|
|
||||||
span: finding.byte_span.clone().into(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the stable rule identifier used as the diagnostic code.
|
|
||||||
pub fn rule_id(&self) -> &RuleId {
|
|
||||||
&self.rule_id
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the configured finding severity.
|
|
||||||
pub const fn finding_severity(&self) -> FindingSeverity {
|
|
||||||
self.severity
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the exact source span labeled by this diagnostic.
|
|
||||||
pub const fn span(&self) -> SourceSpan {
|
|
||||||
self.span
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the named source snapshot used for rendering.
|
|
||||||
pub fn named_source(&self) -> &NamedSource<String> {
|
|
||||||
&self.source
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for FindingDiagnostic {
|
|
||||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
self.message.fmt(formatter)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Error for FindingDiagnostic {}
|
|
||||||
|
|
||||||
impl Diagnostic for FindingDiagnostic {
|
|
||||||
fn code(&self) -> Option<Box<dyn fmt::Display + '_>> {
|
|
||||||
Some(Box::new(&self.rule_id))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn severity(&self) -> Option<Severity> {
|
|
||||||
Some(match self.severity {
|
|
||||||
FindingSeverity::Advice => Severity::Advice,
|
|
||||||
FindingSeverity::Warning => Severity::Warning,
|
|
||||||
FindingSeverity::Error => Severity::Error,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn help(&self) -> Option<Box<dyn fmt::Display + '_>> {
|
|
||||||
self.help
|
|
||||||
.as_deref()
|
|
||||||
.map(|help| Box::new(help) as Box<dyn fmt::Display>)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn source_code(&self) -> Option<&dyn SourceCode> {
|
|
||||||
Some(&self.source)
|
|
||||||
}
|
|
||||||
|
|
||||||
fn labels(&self) -> Option<Box<dyn Iterator<Item = LabeledSpan> + '_>> {
|
|
||||||
Some(Box::new(std::iter::once(
|
|
||||||
LabeledSpan::new_primary_with_span(Some(self.matched_text.clone()), self.span),
|
|
||||||
)))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use crate::{AnalysisContext, Config, Detector};
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn finding_diagnostic_preserves_code_severity_source_span_and_help() {
|
|
||||||
let source = "préface: As an AI language model, I can help.";
|
|
||||||
let finding = Detector::new(&Config::embedded())
|
|
||||||
.unwrap()
|
|
||||||
.analyze_with(source, &AnalysisContext::for_path("article.md"))
|
|
||||||
.findings
|
|
||||||
.into_iter()
|
|
||||||
.find(|finding| finding.rule_id.as_str() == "assistant-self-identification")
|
|
||||||
.unwrap();
|
|
||||||
let diagnostic =
|
|
||||||
FindingDiagnostic::new(&SourceId::File("article.md".into()), source, &finding);
|
|
||||||
|
|
||||||
assert_eq!(
|
|
||||||
diagnostic.code().unwrap().to_string(),
|
|
||||||
finding.rule_id.as_str()
|
|
||||||
);
|
|
||||||
assert_eq!(diagnostic.severity(), Some(Severity::Warning));
|
|
||||||
assert!(diagnostic.help().unwrap().to_string().contains("copied"));
|
|
||||||
assert_eq!(diagnostic.named_source().name(), "article.md");
|
|
||||||
let label = diagnostic.labels().unwrap().next().unwrap();
|
|
||||||
assert_eq!(label.offset(), finding.byte_span.start);
|
|
||||||
assert_eq!(label.len(), finding.byte_span.len());
|
|
||||||
assert_eq!(
|
|
||||||
&source[label.offset()..label.offset() + label.len()],
|
|
||||||
finding.matched_text
|
|
||||||
);
|
|
||||||
assert_eq!(diagnostic.rule_id(), &finding.rule_id);
|
|
||||||
assert_eq!(diagnostic.finding_severity(), FindingSeverity::Warning);
|
|
||||||
assert_eq!(diagnostic.span(), finding.byte_span.clone().into());
|
|
||||||
assert_eq!(diagnostic.to_string(), finding.message);
|
|
||||||
assert!(diagnostic.source_code().is_some());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn every_finding_severity_maps_to_miette_and_help_remains_optional() {
|
|
||||||
for (finding_severity, miette_severity) in [
|
|
||||||
(FindingSeverity::Advice, Severity::Advice),
|
|
||||||
(FindingSeverity::Warning, Severity::Warning),
|
|
||||||
(FindingSeverity::Error, Severity::Error),
|
|
||||||
] {
|
|
||||||
let finding = Finding {
|
|
||||||
rule_id: RuleId::from_static("severity-test"),
|
|
||||||
category: crate::Category::Style,
|
|
||||||
severity: finding_severity,
|
|
||||||
message: "review this".into(),
|
|
||||||
help: None,
|
|
||||||
byte_span: 0..1,
|
|
||||||
matched_text: "x".into(),
|
|
||||||
weight: 1.0,
|
|
||||||
fix: None,
|
|
||||||
};
|
|
||||||
let diagnostic = FindingDiagnostic::new(&SourceId::Stdin, "x", &finding);
|
|
||||||
assert_eq!(diagnostic.severity(), Some(miette_severity));
|
|
||||||
assert!(diagnostic.help().is_none());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,171 +0,0 @@
|
||||||
//! Typed failures returned by the input layer.
|
|
||||||
|
|
||||||
use std::{io, path::PathBuf, string::FromUtf8Error};
|
|
||||||
|
|
||||||
use thiserror::Error;
|
|
||||||
|
|
||||||
use crate::SourceId;
|
|
||||||
|
|
||||||
/// Failure while discovering or reading one input document.
|
|
||||||
#[derive(Debug, Error)]
|
|
||||||
pub enum InputError {
|
|
||||||
/// Reading standard input failed.
|
|
||||||
#[error("failed to read standard input")]
|
|
||||||
StdinIo {
|
|
||||||
/// Underlying operating-system error.
|
|
||||||
#[source]
|
|
||||||
source: io::Error,
|
|
||||||
},
|
|
||||||
/// The requested path does not identify a file or directory.
|
|
||||||
#[error("path `{path}` is neither a file nor a directory")]
|
|
||||||
MissingPath {
|
|
||||||
/// Requested path.
|
|
||||||
path: PathBuf,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// A filesystem operation failed.
|
|
||||||
#[error("failed to {action} `{path}`")]
|
|
||||||
Io {
|
|
||||||
/// Operation being attempted.
|
|
||||||
action: &'static str,
|
|
||||||
/// Affected filesystem path.
|
|
||||||
path: PathBuf,
|
|
||||||
/// Underlying operating-system error.
|
|
||||||
#[source]
|
|
||||||
source: io::Error,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// Recursive directory traversal failed for one entry.
|
|
||||||
#[error("failed while walking an input directory")]
|
|
||||||
Walk {
|
|
||||||
/// Underlying traversal error.
|
|
||||||
#[source]
|
|
||||||
source: walkdir::Error,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// A ZIP container or member could not be decoded.
|
|
||||||
#[error("failed to read ZIP data from `{archive}`")]
|
|
||||||
Zip {
|
|
||||||
/// Outer archive path.
|
|
||||||
archive: PathBuf,
|
|
||||||
/// Underlying ZIP format error.
|
|
||||||
#[source]
|
|
||||||
source: zip::result::ZipError,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// A source exceeded a configured byte or count limit.
|
|
||||||
#[error("{source_id} uses {actual} {resource}, exceeding the limit of {limit}")]
|
|
||||||
LimitExceeded {
|
|
||||||
/// Source that violated the limit.
|
|
||||||
source_id: SourceId,
|
|
||||||
/// Human-readable resource unit.
|
|
||||||
resource: &'static str,
|
|
||||||
/// Observed value, or `u64::MAX` after arithmetic overflow.
|
|
||||||
actual: u64,
|
|
||||||
/// Configured maximum.
|
|
||||||
limit: u64,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// Archive expansion is disproportionate to compressed size.
|
|
||||||
#[error("archive member {source_id} exceeds the configured {limit}:1 compression ratio")]
|
|
||||||
CompressionRatio {
|
|
||||||
/// Archive member identity.
|
|
||||||
source_id: SourceId,
|
|
||||||
/// Maximum allowed ratio.
|
|
||||||
limit: u64,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// An archive member name is unsafe to treat as a relative identity.
|
|
||||||
#[error("unsafe archive member path `{member}` in `{archive}`")]
|
|
||||||
UnsafeArchiveMember {
|
|
||||||
/// Outer archive path.
|
|
||||||
archive: PathBuf,
|
|
||||||
/// Untrusted member name from the archive.
|
|
||||||
member: String,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// Encrypted members are unsupported.
|
|
||||||
#[error("archive member {source_id} is encrypted")]
|
|
||||||
EncryptedMember {
|
|
||||||
/// Encrypted archive member identity.
|
|
||||||
source_id: SourceId,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// Input contained a NUL byte and is treated as binary.
|
|
||||||
#[error("input {source_id} contains a NUL byte")]
|
|
||||||
BinaryContent {
|
|
||||||
/// Source treated as binary.
|
|
||||||
source_id: SourceId,
|
|
||||||
},
|
|
||||||
|
|
||||||
/// Input was not strict UTF-8.
|
|
||||||
#[error("input {source_id} is not valid UTF-8")]
|
|
||||||
InvalidUtf8 {
|
|
||||||
/// Source that failed decoding.
|
|
||||||
source_id: SourceId,
|
|
||||||
/// Underlying UTF-8 conversion error.
|
|
||||||
#[source]
|
|
||||||
source: FromUtf8Error,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
impl InputError {
|
|
||||||
/// Return the exact document identity when the failure occurred after discovery.
|
|
||||||
pub const fn source_id(&self) -> Option<&SourceId> {
|
|
||||||
match self {
|
|
||||||
Self::LimitExceeded { source_id, .. }
|
|
||||||
| Self::CompressionRatio { source_id, .. }
|
|
||||||
| Self::EncryptedMember { source_id }
|
|
||||||
| Self::BinaryContent { source_id }
|
|
||||||
| Self::InvalidUtf8 { source_id, .. } => Some(source_id),
|
|
||||||
Self::StdinIo { .. }
|
|
||||||
| Self::MissingPath { .. }
|
|
||||||
| Self::Io { .. }
|
|
||||||
| Self::Walk { .. }
|
|
||||||
| Self::Zip { .. }
|
|
||||||
| Self::UnsafeArchiveMember { .. } => None,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Result type used by the input layer.
|
|
||||||
pub type InputResult<T> = Result<T, InputError>;
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn source_identity_is_available_only_for_document_specific_failures() {
|
|
||||||
let source_id = SourceId::Stdin;
|
|
||||||
for error in [
|
|
||||||
InputError::LimitExceeded {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
resource: "bytes",
|
|
||||||
actual: 2,
|
|
||||||
limit: 1,
|
|
||||||
},
|
|
||||||
InputError::CompressionRatio {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
limit: 2,
|
|
||||||
},
|
|
||||||
InputError::EncryptedMember {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
},
|
|
||||||
InputError::BinaryContent {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
},
|
|
||||||
InputError::InvalidUtf8 {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
source: String::from_utf8(vec![0xff]).unwrap_err(),
|
|
||||||
},
|
|
||||||
] {
|
|
||||||
assert_eq!(error.source_id(), Some(&source_id));
|
|
||||||
}
|
|
||||||
|
|
||||||
let error = InputError::MissingPath {
|
|
||||||
path: PathBuf::from("missing"),
|
|
||||||
};
|
|
||||||
assert_eq!(error.source_id(), None);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,426 +0,0 @@
|
||||||
//! Streaming ZIP-based archive input.
|
|
||||||
|
|
||||||
use std::{
|
|
||||||
fs::{self, File},
|
|
||||||
io::Read,
|
|
||||||
path::{Component, Path, PathBuf},
|
|
||||||
};
|
|
||||||
|
|
||||||
use zip::ZipArchive;
|
|
||||||
|
|
||||||
use crate::{
|
|
||||||
SourceId,
|
|
||||||
config::InputLimits,
|
|
||||||
error::{InputError, InputResult},
|
|
||||||
input::Document,
|
|
||||||
};
|
|
||||||
|
|
||||||
/// Iterator over text members in one ZIP archive.
|
|
||||||
pub(super) struct ArchiveDocuments {
|
|
||||||
/// Outer archive identity.
|
|
||||||
outer: PathBuf,
|
|
||||||
/// Open ZIP reader retained between iterator calls.
|
|
||||||
archive: ZipArchive<File>,
|
|
||||||
/// Index of the next member to inspect.
|
|
||||||
index: usize,
|
|
||||||
/// Cumulative declared expanded bytes accepted so far.
|
|
||||||
expanded_bytes: u64,
|
|
||||||
/// Resource policy copied from resolved configuration.
|
|
||||||
limits: InputLimits,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl std::fmt::Debug for ArchiveDocuments {
|
|
||||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
|
||||||
formatter
|
|
||||||
.debug_struct("ArchiveDocuments")
|
|
||||||
.field("outer", &self.outer)
|
|
||||||
.field("members", &self.archive.len())
|
|
||||||
.field("index", &self.index)
|
|
||||||
.finish()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl ArchiveDocuments {
|
|
||||||
/// Open an archive for streaming member reads.
|
|
||||||
pub(super) fn open(outer: PathBuf, limits: &InputLimits) -> InputResult<Self> {
|
|
||||||
let archive_id = SourceId::File(outer.clone());
|
|
||||||
let archive_size = fs::metadata(&outer)
|
|
||||||
.map_err(|source| InputError::Io {
|
|
||||||
action: "inspect",
|
|
||||||
path: outer.clone(),
|
|
||||||
source,
|
|
||||||
})?
|
|
||||||
.len();
|
|
||||||
if archive_size > limits.max_archive_bytes {
|
|
||||||
return Err(InputError::LimitExceeded {
|
|
||||||
source_id: archive_id.clone(),
|
|
||||||
resource: "bytes",
|
|
||||||
actual: archive_size,
|
|
||||||
limit: limits.max_archive_bytes,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let file = File::open(&outer).map_err(|source| InputError::Io {
|
|
||||||
action: "open",
|
|
||||||
path: outer.clone(),
|
|
||||||
source,
|
|
||||||
})?;
|
|
||||||
let archive = ZipArchive::new(file).map_err(|source| InputError::Zip {
|
|
||||||
archive: outer.clone(),
|
|
||||||
source,
|
|
||||||
})?;
|
|
||||||
if archive.len() > limits.max_archive_entries {
|
|
||||||
return Err(InputError::LimitExceeded {
|
|
||||||
source_id: archive_id,
|
|
||||||
resource: "entries",
|
|
||||||
actual: u64::try_from(archive.len()).unwrap_or(u64::MAX),
|
|
||||||
limit: u64::try_from(limits.max_archive_entries).unwrap_or(u64::MAX),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(Self {
|
|
||||||
outer,
|
|
||||||
archive,
|
|
||||||
index: 0,
|
|
||||||
expanded_bytes: 0,
|
|
||||||
limits: limits.clone(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Iterator for ArchiveDocuments {
|
|
||||||
type Item = InputResult<Document>;
|
|
||||||
|
|
||||||
fn next(&mut self) -> Option<Self::Item> {
|
|
||||||
while self.index < self.archive.len() {
|
|
||||||
let index = self.index;
|
|
||||||
self.index += 1;
|
|
||||||
let mut member = match self
|
|
||||||
.archive
|
|
||||||
.by_index(index)
|
|
||||||
.map_err(|source| InputError::Zip {
|
|
||||||
archive: self.outer.clone(),
|
|
||||||
source,
|
|
||||||
}) {
|
|
||||||
Ok(member) => member,
|
|
||||||
Err(error) => return Some(Err(error)),
|
|
||||||
};
|
|
||||||
|
|
||||||
if member.is_dir() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
let name = match safe_member_path(&self.outer, member.name()) {
|
|
||||||
Ok(name) => name,
|
|
||||||
Err(error) => return Some(Err(error)),
|
|
||||||
};
|
|
||||||
let source_id = SourceId::ArchiveMember {
|
|
||||||
archive: self.outer.clone(),
|
|
||||||
member: name.clone(),
|
|
||||||
};
|
|
||||||
if member.encrypted() {
|
|
||||||
return Some(Err(InputError::EncryptedMember { source_id }));
|
|
||||||
}
|
|
||||||
|
|
||||||
let expanded_size = member.size();
|
|
||||||
if expanded_size > self.limits.max_document_bytes {
|
|
||||||
return Some(Err(InputError::LimitExceeded {
|
|
||||||
source_id,
|
|
||||||
resource: "bytes",
|
|
||||||
actual: expanded_size,
|
|
||||||
limit: self.limits.max_document_bytes,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
let compressed_size = member.compressed_size();
|
|
||||||
let allowed_expanded =
|
|
||||||
compressed_size.saturating_mul(self.limits.max_compression_ratio);
|
|
||||||
if expanded_size > 0 && expanded_size > allowed_expanded {
|
|
||||||
return Some(Err(InputError::CompressionRatio {
|
|
||||||
source_id,
|
|
||||||
limit: self.limits.max_compression_ratio,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
self.expanded_bytes = match self.expanded_bytes.checked_add(expanded_size) {
|
|
||||||
Some(total) if total <= self.limits.max_expanded_bytes => total,
|
|
||||||
_ => {
|
|
||||||
return Some(Err(InputError::LimitExceeded {
|
|
||||||
source_id: SourceId::File(self.outer.clone()),
|
|
||||||
resource: "expanded bytes",
|
|
||||||
actual: u64::MAX,
|
|
||||||
limit: self.limits.max_expanded_bytes,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let read_limit = self.limits.max_document_bytes.saturating_add(1);
|
|
||||||
let mut bytes = Vec::new();
|
|
||||||
if let Err(source) = member.by_ref().take(read_limit).read_to_end(&mut bytes) {
|
|
||||||
return Some(Err(InputError::Io {
|
|
||||||
action: "read archive member from",
|
|
||||||
path: self.outer.clone(),
|
|
||||||
source,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > self.limits.max_document_bytes {
|
|
||||||
return Some(Err(InputError::LimitExceeded {
|
|
||||||
source_id,
|
|
||||||
resource: "bytes",
|
|
||||||
actual: u64::try_from(bytes.len()).unwrap_or(u64::MAX),
|
|
||||||
limit: self.limits.max_document_bytes,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
if bytes.contains(&0) {
|
|
||||||
return Some(Err(InputError::BinaryContent { source_id }));
|
|
||||||
}
|
|
||||||
let content = match String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
source,
|
|
||||||
}) {
|
|
||||||
Ok(content) => content,
|
|
||||||
Err(error) => return Some(Err(error)),
|
|
||||||
};
|
|
||||||
|
|
||||||
return Some(Ok(Document {
|
|
||||||
source: SourceId::ArchiveMember {
|
|
||||||
archive: self.outer.clone(),
|
|
||||||
member: name,
|
|
||||||
},
|
|
||||||
content,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
|
|
||||||
None
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Parse an archive name into a safe relative member identity.
|
|
||||||
fn safe_member_path(archive: &Path, name: &str) -> InputResult<PathBuf> {
|
|
||||||
let normalized = name.replace('\\', "/");
|
|
||||||
let path = Path::new(&normalized);
|
|
||||||
if normalized.is_empty()
|
|
||||||
|| path
|
|
||||||
.components()
|
|
||||||
.any(|component| !matches!(component, Component::Normal(_)))
|
|
||||||
{
|
|
||||||
return Err(InputError::UnsafeArchiveMember {
|
|
||||||
archive: archive.to_path_buf(),
|
|
||||||
member: name.to_owned(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(path.to_path_buf())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use std::io::Write;
|
|
||||||
|
|
||||||
use tempfile::tempdir;
|
|
||||||
use zip::write::FileOptions;
|
|
||||||
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
fn create_archive(path: &Path, entries: &[(&str, &[u8])], compression: zip::CompressionMethod) {
|
|
||||||
let file = File::create(path).unwrap();
|
|
||||||
let mut writer = zip::ZipWriter::new(file);
|
|
||||||
let options: FileOptions<()> = FileOptions::default().compression_method(compression);
|
|
||||||
for (name, bytes) in entries {
|
|
||||||
writer.start_file(*name, options).unwrap();
|
|
||||||
writer.write_all(bytes).unwrap();
|
|
||||||
}
|
|
||||||
writer.finish().unwrap();
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn rejects_absolute_and_traversing_member_paths() {
|
|
||||||
for path in [
|
|
||||||
"../escape.txt",
|
|
||||||
"/absolute.txt",
|
|
||||||
"a/../../escape",
|
|
||||||
r"a\..\escape",
|
|
||||||
] {
|
|
||||||
assert!(
|
|
||||||
safe_member_path(Path::new("archive.zip"), path).is_err(),
|
|
||||||
"accepted {path}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
assert_eq!(
|
|
||||||
safe_member_path(Path::new("archive.zip"), "safe/member.txt").unwrap(),
|
|
||||||
Path::new("safe/member.txt")
|
|
||||||
);
|
|
||||||
assert!(safe_member_path(Path::new("archive.zip"), "").is_err());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn opening_archives_reports_filesystem_format_size_and_entry_limits() {
|
|
||||||
let directory = tempdir().unwrap();
|
|
||||||
let missing = directory.path().join("missing.zip");
|
|
||||||
assert!(matches!(
|
|
||||||
ArchiveDocuments::open(missing.clone(), &InputLimits::default()),
|
|
||||||
Err(InputError::Io { action: "inspect", path, .. }) if path == missing
|
|
||||||
));
|
|
||||||
|
|
||||||
let malformed = directory.path().join("malformed.zip");
|
|
||||||
fs::write(&malformed, b"not a zip").unwrap();
|
|
||||||
assert!(matches!(
|
|
||||||
ArchiveDocuments::open(malformed.clone(), &InputLimits::default()),
|
|
||||||
Err(InputError::Zip { archive, .. }) if archive == malformed
|
|
||||||
));
|
|
||||||
|
|
||||||
let path = directory.path().join("limits.zip");
|
|
||||||
create_archive(
|
|
||||||
&path,
|
|
||||||
&[("one.txt", b"one"), ("two.txt", b"two")],
|
|
||||||
zip::CompressionMethod::Stored,
|
|
||||||
);
|
|
||||||
let limits = InputLimits {
|
|
||||||
max_archive_bytes: 1,
|
|
||||||
..InputLimits::default()
|
|
||||||
};
|
|
||||||
assert!(matches!(
|
|
||||||
ArchiveDocuments::open(path.clone(), &limits),
|
|
||||||
Err(InputError::LimitExceeded {
|
|
||||||
resource: "bytes",
|
|
||||||
..
|
|
||||||
})
|
|
||||||
));
|
|
||||||
|
|
||||||
let limits = InputLimits {
|
|
||||||
max_archive_bytes: u64::MAX,
|
|
||||||
max_archive_entries: 1,
|
|
||||||
..InputLimits::default()
|
|
||||||
};
|
|
||||||
assert!(matches!(
|
|
||||||
ArchiveDocuments::open(path, &limits),
|
|
||||||
Err(InputError::LimitExceeded {
|
|
||||||
resource: "entries",
|
|
||||||
actual: 2,
|
|
||||||
limit: 1,
|
|
||||||
..
|
|
||||||
})
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn iterator_skips_directories_and_preserves_safe_member_identity() {
|
|
||||||
let directory = tempdir().unwrap();
|
|
||||||
let path = directory.path().join("documents.zip");
|
|
||||||
let file = File::create(&path).unwrap();
|
|
||||||
let mut writer = zip::ZipWriter::new(file);
|
|
||||||
let options: FileOptions<()> = FileOptions::default();
|
|
||||||
writer.add_directory("docs/", options).unwrap();
|
|
||||||
writer.start_file("docs/article.txt", options).unwrap();
|
|
||||||
writer.write_all("héllo".as_bytes()).unwrap();
|
|
||||||
writer.finish().unwrap();
|
|
||||||
|
|
||||||
let mut archive = ArchiveDocuments::open(path.clone(), &InputLimits::default()).unwrap();
|
|
||||||
let debug = format!("{archive:?}");
|
|
||||||
assert!(debug.contains("members: 2"));
|
|
||||||
let document = archive.next().unwrap().unwrap();
|
|
||||||
assert_eq!(document.content, "héllo");
|
|
||||||
assert_eq!(
|
|
||||||
document.source,
|
|
||||||
SourceId::ArchiveMember {
|
|
||||||
archive: path,
|
|
||||||
member: PathBuf::from("docs/article.txt"),
|
|
||||||
}
|
|
||||||
);
|
|
||||||
assert!(archive.next().is_none());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn iterator_rejects_unsafe_binary_and_non_utf8_members_independently() {
|
|
||||||
for (name, bytes, expected) in [
|
|
||||||
("../escape.txt", b"text".as_slice(), "unsafe"),
|
|
||||||
("binary.txt", b"a\0b".as_slice(), "binary"),
|
|
||||||
("invalid.txt", [0xff].as_slice(), "utf8"),
|
|
||||||
] {
|
|
||||||
let directory = tempdir().unwrap();
|
|
||||||
let path = directory.path().join("invalid-member.zip");
|
|
||||||
create_archive(&path, &[(name, bytes)], zip::CompressionMethod::Stored);
|
|
||||||
let error = ArchiveDocuments::open(path, &InputLimits::default())
|
|
||||||
.unwrap()
|
|
||||||
.next()
|
|
||||||
.unwrap()
|
|
||||||
.unwrap_err();
|
|
||||||
assert!(
|
|
||||||
matches!(
|
|
||||||
(expected, error),
|
|
||||||
("unsafe", InputError::UnsafeArchiveMember { .. })
|
|
||||||
| ("binary", InputError::BinaryContent { .. })
|
|
||||||
| ("utf8", InputError::InvalidUtf8 { .. })
|
|
||||||
),
|
|
||||||
"wrong error for {name}"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn iterator_enforces_member_expansion_and_compression_limits() {
|
|
||||||
let directory = tempdir().unwrap();
|
|
||||||
let oversized = directory.path().join("oversized.zip");
|
|
||||||
create_archive(
|
|
||||||
&oversized,
|
|
||||||
&[("large.txt", b"12345")],
|
|
||||||
zip::CompressionMethod::Stored,
|
|
||||||
);
|
|
||||||
let limits = InputLimits {
|
|
||||||
max_document_bytes: 4,
|
|
||||||
..InputLimits::default()
|
|
||||||
};
|
|
||||||
let error = ArchiveDocuments::open(oversized, &limits)
|
|
||||||
.unwrap()
|
|
||||||
.next()
|
|
||||||
.unwrap()
|
|
||||||
.unwrap_err();
|
|
||||||
assert!(matches!(
|
|
||||||
error,
|
|
||||||
InputError::LimitExceeded {
|
|
||||||
resource: "bytes",
|
|
||||||
actual: 5,
|
|
||||||
limit: 4,
|
|
||||||
..
|
|
||||||
}
|
|
||||||
));
|
|
||||||
|
|
||||||
let compressed = directory.path().join("compressed.zip");
|
|
||||||
create_archive(
|
|
||||||
&compressed,
|
|
||||||
&[("repeated.txt", &[b'a'; 4096])],
|
|
||||||
zip::CompressionMethod::Deflated,
|
|
||||||
);
|
|
||||||
let limits = InputLimits {
|
|
||||||
max_compression_ratio: 1,
|
|
||||||
..InputLimits::default()
|
|
||||||
};
|
|
||||||
assert!(matches!(
|
|
||||||
ArchiveDocuments::open(compressed, &limits)
|
|
||||||
.unwrap()
|
|
||||||
.next()
|
|
||||||
.unwrap(),
|
|
||||||
Err(InputError::CompressionRatio { limit: 1, .. })
|
|
||||||
));
|
|
||||||
|
|
||||||
let cumulative = directory.path().join("cumulative.zip");
|
|
||||||
create_archive(
|
|
||||||
&cumulative,
|
|
||||||
&[("one.txt", b"1234"), ("two.txt", b"5678")],
|
|
||||||
zip::CompressionMethod::Stored,
|
|
||||||
);
|
|
||||||
let limits = InputLimits {
|
|
||||||
max_expanded_bytes: 6,
|
|
||||||
..InputLimits::default()
|
|
||||||
};
|
|
||||||
let mut archive = ArchiveDocuments::open(cumulative, &limits).unwrap();
|
|
||||||
assert!(archive.next().unwrap().is_ok());
|
|
||||||
assert!(matches!(
|
|
||||||
archive.next().unwrap(),
|
|
||||||
Err(InputError::LimitExceeded {
|
|
||||||
resource: "expanded bytes",
|
|
||||||
actual: u64::MAX,
|
|
||||||
limit: 6,
|
|
||||||
..
|
|
||||||
})
|
|
||||||
));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
//! Regular-file input support.
|
|
||||||
|
|
||||||
use std::{fs, path::Path};
|
|
||||||
|
|
||||||
use crate::{
|
|
||||||
SourceId,
|
|
||||||
config::InputLimits,
|
|
||||||
error::{InputError, InputResult},
|
|
||||||
};
|
|
||||||
|
|
||||||
/// Read a regular file as strict UTF-8 text.
|
|
||||||
pub(super) fn read_text(path: &Path, limits: &InputLimits) -> InputResult<String> {
|
|
||||||
let source_id = SourceId::File(path.to_path_buf());
|
|
||||||
let metadata = fs::metadata(path).map_err(|source| InputError::Io {
|
|
||||||
action: "inspect",
|
|
||||||
path: path.to_path_buf(),
|
|
||||||
source,
|
|
||||||
})?;
|
|
||||||
if metadata.len() > limits.max_document_bytes {
|
|
||||||
return Err(InputError::LimitExceeded {
|
|
||||||
source_id,
|
|
||||||
resource: "bytes",
|
|
||||||
actual: metadata.len(),
|
|
||||||
limit: limits.max_document_bytes,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let bytes = fs::read(path).map_err(|source| InputError::Io {
|
|
||||||
action: "read",
|
|
||||||
path: path.to_path_buf(),
|
|
||||||
source,
|
|
||||||
})?;
|
|
||||||
if bytes.contains(&0) {
|
|
||||||
return Err(InputError::BinaryContent { source_id });
|
|
||||||
}
|
|
||||||
String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 { source_id, source })
|
|
||||||
}
|
|
||||||
|
|
@ -1,127 +0,0 @@
|
||||||
//! Streaming input acquisition for files, directories, and ZIP containers.
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests;
|
|
||||||
|
|
||||||
mod archive;
|
|
||||||
mod file;
|
|
||||||
mod walk;
|
|
||||||
|
|
||||||
use std::{io::Read, path::Path, vec};
|
|
||||||
|
|
||||||
use crate::{
|
|
||||||
SourceId,
|
|
||||||
config::InputLimits,
|
|
||||||
error::{InputError, InputResult},
|
|
||||||
};
|
|
||||||
|
|
||||||
use self::{archive::ArchiveDocuments, walk::WalkDocuments};
|
|
||||||
|
|
||||||
/// One strict UTF-8 source document.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
||||||
pub struct Document {
|
|
||||||
/// Stable identity of the file or archive member.
|
|
||||||
pub source: SourceId,
|
|
||||||
/// Complete UTF-8 content of this document.
|
|
||||||
pub content: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Streaming documents produced from one requested path.
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub struct Documents {
|
|
||||||
/// Input-specific iterator implementation.
|
|
||||||
inner: DocumentsInner,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Private iterator variants hidden behind the stable public wrapper.
|
|
||||||
#[derive(Debug)]
|
|
||||||
enum DocumentsInner {
|
|
||||||
/// One regular-file result.
|
|
||||||
File(vec::IntoIter<InputResult<Document>>),
|
|
||||||
/// Recursive directory results.
|
|
||||||
Directory(WalkDocuments),
|
|
||||||
/// ZIP member results.
|
|
||||||
Archive(ArchiveDocuments),
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Iterator for Documents {
|
|
||||||
type Item = InputResult<Document>;
|
|
||||||
|
|
||||||
fn next(&mut self) -> Option<Self::Item> {
|
|
||||||
match &mut self.inner {
|
|
||||||
DocumentsInner::File(document) => document.next(),
|
|
||||||
DocumentsInner::Directory(documents) => documents.next(),
|
|
||||||
DocumentsInner::Archive(documents) => documents.next(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Open a file, directory, or recognized ZIP-based container for streaming.
|
|
||||||
pub fn read(path: &Path, limits: &InputLimits) -> InputResult<Documents> {
|
|
||||||
if path.is_file() {
|
|
||||||
if has_zip_extension(path) {
|
|
||||||
return ArchiveDocuments::open(path.to_path_buf(), limits).map(|archive| Documents {
|
|
||||||
inner: DocumentsInner::Archive(archive),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
let result = file::read_text(path, limits).map(|content| Document {
|
|
||||||
source: SourceId::File(path.to_path_buf()),
|
|
||||||
content,
|
|
||||||
});
|
|
||||||
return Ok(Documents {
|
|
||||||
inner: DocumentsInner::File(vec![result].into_iter()),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
if path.is_dir() {
|
|
||||||
return Ok(Documents {
|
|
||||||
inner: DocumentsInner::Directory(WalkDocuments::new(path, limits)),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
Err(InputError::MissingPath {
|
|
||||||
path: path.to_path_buf(),
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read standard input as one bounded strict UTF-8 document.
|
|
||||||
pub fn read_stdin(reader: impl Read, limits: &InputLimits) -> InputResult<Document> {
|
|
||||||
let source_id = SourceId::Stdin;
|
|
||||||
let mut bytes = Vec::new();
|
|
||||||
reader
|
|
||||||
.take(limits.max_document_bytes.saturating_add(1))
|
|
||||||
.read_to_end(&mut bytes)
|
|
||||||
.map_err(|source| InputError::StdinIo { source })?;
|
|
||||||
if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > limits.max_document_bytes {
|
|
||||||
return Err(InputError::LimitExceeded {
|
|
||||||
source_id,
|
|
||||||
resource: "bytes",
|
|
||||||
actual: u64::try_from(bytes.len()).unwrap_or(u64::MAX),
|
|
||||||
limit: limits.max_document_bytes,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
if bytes.contains(&0) {
|
|
||||||
return Err(InputError::BinaryContent { source_id });
|
|
||||||
}
|
|
||||||
let content = String::from_utf8(bytes).map_err(|source| InputError::InvalidUtf8 {
|
|
||||||
source_id: source_id.clone(),
|
|
||||||
source,
|
|
||||||
})?;
|
|
||||||
Ok(Document {
|
|
||||||
source: source_id,
|
|
||||||
content,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Whether a path has an extension treated as a ZIP container.
|
|
||||||
fn has_zip_extension(path: &Path) -> bool {
|
|
||||||
path.extension()
|
|
||||||
.and_then(|extension| extension.to_str())
|
|
||||||
.is_some_and(|extension| {
|
|
||||||
matches!(
|
|
||||||
extension.to_ascii_lowercase().as_str(),
|
|
||||||
"zip" | "docx" | "xlsx" | "pptx"
|
|
||||||
)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
@ -1,197 +0,0 @@
|
||||||
use std::{
|
|
||||||
fs::{self, File},
|
|
||||||
io::Write,
|
|
||||||
path::Path,
|
|
||||||
};
|
|
||||||
|
|
||||||
use tempfile::tempdir;
|
|
||||||
use zip::write::FileOptions;
|
|
||||||
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
struct FailingReader;
|
|
||||||
|
|
||||||
impl std::io::Read for FailingReader {
|
|
||||||
fn read(&mut self, _: &mut [u8]) -> std::io::Result<usize> {
|
|
||||||
Err(std::io::Error::other("fixture read failure"))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
fn limits() -> InputLimits {
|
|
||||||
InputLimits::default()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn write_file(path: &Path, content: &str) {
|
|
||||||
let mut file = File::create(path).expect("create fixture");
|
|
||||||
file.write_all(content.as_bytes()).expect("write fixture");
|
|
||||||
}
|
|
||||||
|
|
||||||
fn create_zip(path: &Path) -> std::io::Result<()> {
|
|
||||||
let file = File::create(path)?;
|
|
||||||
let mut zip = zip::ZipWriter::new(file);
|
|
||||||
let options: FileOptions<()> = FileOptions::default()
|
|
||||||
.compression_method(zip::CompressionMethod::Deflated)
|
|
||||||
.unix_permissions(0o644);
|
|
||||||
|
|
||||||
zip.start_file("foo.txt", options)?;
|
|
||||||
zip.write_all(b"hello foo")?;
|
|
||||||
zip.start_file("bar.txt", options)?;
|
|
||||||
zip.write_all(b"hello bar")?;
|
|
||||||
zip.finish()?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn reads_one_file() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let path = directory.path().join("test.txt");
|
|
||||||
write_file(&path, "line1\nline2");
|
|
||||||
|
|
||||||
let documents = read(&path, &limits())
|
|
||||||
.expect("open input")
|
|
||||||
.collect::<InputResult<Vec<_>>>()
|
|
||||||
.expect("read input");
|
|
||||||
|
|
||||||
assert_eq!(documents.len(), 1);
|
|
||||||
assert_eq!(documents[0].source, SourceId::File(path));
|
|
||||||
assert_eq!(documents[0].content, "line1\nline2");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn yields_a_directory_error_without_discarding_other_files() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
write_file(&directory.path().join("a.txt"), "a");
|
|
||||||
fs::write(directory.path().join("b.txt"), [0xff]).expect("write invalid fixture");
|
|
||||||
write_file(&directory.path().join("c.txt"), "c");
|
|
||||||
|
|
||||||
let results = read(directory.path(), &limits())
|
|
||||||
.expect("open directory")
|
|
||||||
.collect::<Vec<_>>();
|
|
||||||
|
|
||||||
assert_eq!(results.len(), 3);
|
|
||||||
assert_eq!(results.iter().filter(|result| result.is_ok()).count(), 2);
|
|
||||||
assert_eq!(results.iter().filter(|result| result.is_err()).count(), 1);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn streams_archive_members_with_outer_identity() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let path = directory.path().join("archive.zip");
|
|
||||||
create_zip(&path).expect("create zip");
|
|
||||||
|
|
||||||
let documents = read(&path, &limits())
|
|
||||||
.expect("open archive")
|
|
||||||
.collect::<InputResult<Vec<_>>>()
|
|
||||||
.expect("read archive");
|
|
||||||
|
|
||||||
assert_eq!(documents.len(), 2);
|
|
||||||
assert!(documents.iter().all(|document| matches!(
|
|
||||||
&document.source,
|
|
||||||
SourceId::ArchiveMember { archive, .. } if archive == &path
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn invalid_utf8_is_a_per_document_error() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let path = directory.path().join("invalid.txt");
|
|
||||||
fs::write(&path, [0xff, 0xfe]).expect("write fixture");
|
|
||||||
|
|
||||||
let mut documents = read(&path, &limits()).expect("open input");
|
|
||||||
assert!(documents.next().expect("one result").is_err());
|
|
||||||
assert!(documents.next().is_none());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn binary_content_is_an_input_error_not_a_clean_document() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let path = directory.path().join("binary.txt");
|
|
||||||
fs::write(&path, b"clean prefix\0binary suffix").expect("write fixture");
|
|
||||||
|
|
||||||
let mut documents = read(&path, &limits()).expect("open input");
|
|
||||||
let error = documents.next().expect("one result").unwrap_err();
|
|
||||||
|
|
||||||
assert!(matches!(
|
|
||||||
error,
|
|
||||||
InputError::BinaryContent {
|
|
||||||
source_id: SourceId::File(error_path)
|
|
||||||
} if error_path == path
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn missing_path_is_rejected() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let path = directory.path().join("missing.txt");
|
|
||||||
|
|
||||||
let error = read(&path, &limits()).expect_err("missing input should fail");
|
|
||||||
assert!(matches!(error, InputError::MissingPath { path: error_path } if error_path == path));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn enforces_document_size_limit_before_reading() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let path = directory.path().join("large.txt");
|
|
||||||
write_file(&path, "too large");
|
|
||||||
let mut limits = limits();
|
|
||||||
limits.max_document_bytes = 3;
|
|
||||||
|
|
||||||
let mut documents = read(&path, &limits).expect("open input");
|
|
||||||
let error = documents
|
|
||||||
.next()
|
|
||||||
.expect("one result")
|
|
||||||
.expect_err("oversized input");
|
|
||||||
|
|
||||||
assert!(matches!(
|
|
||||||
error,
|
|
||||||
InputError::LimitExceeded {
|
|
||||||
actual: 9,
|
|
||||||
limit: 3,
|
|
||||||
..
|
|
||||||
}
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn stdin_is_bounded_and_decoded_as_strict_utf8() {
|
|
||||||
let document = read_stdin("hello".as_bytes(), &limits()).unwrap();
|
|
||||||
assert_eq!(document.source, SourceId::Stdin);
|
|
||||||
assert_eq!(document.content, "hello");
|
|
||||||
|
|
||||||
let mut small = limits();
|
|
||||||
small.max_document_bytes = 3;
|
|
||||||
assert!(matches!(
|
|
||||||
read_stdin("hello".as_bytes(), &small),
|
|
||||||
Err(InputError::LimitExceeded {
|
|
||||||
source_id: SourceId::Stdin,
|
|
||||||
..
|
|
||||||
})
|
|
||||||
));
|
|
||||||
assert!(matches!(
|
|
||||||
read_stdin([0xff].as_slice(), &limits()),
|
|
||||||
Err(InputError::InvalidUtf8 {
|
|
||||||
source_id: SourceId::Stdin,
|
|
||||||
..
|
|
||||||
})
|
|
||||||
));
|
|
||||||
assert!(matches!(
|
|
||||||
read_stdin(b"nul\0byte".as_slice(), &limits()),
|
|
||||||
Err(InputError::BinaryContent {
|
|
||||||
source_id: SourceId::Stdin
|
|
||||||
})
|
|
||||||
));
|
|
||||||
assert!(matches!(
|
|
||||||
read_stdin(FailingReader, &limits()),
|
|
||||||
Err(InputError::StdinIo { .. })
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn regular_file_metadata_failures_retain_operation_context() {
|
|
||||||
let directory = tempdir().expect("temp dir");
|
|
||||||
let missing = directory.path().join("missing.txt");
|
|
||||||
assert!(matches!(
|
|
||||||
file::read_text(&missing, &limits()),
|
|
||||||
Err(InputError::Io { action: "inspect", path, .. }) if path == missing
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
@ -1,62 +0,0 @@
|
||||||
//! Streaming directory traversal.
|
|
||||||
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
use walkdir::{IntoIter, WalkDir};
|
|
||||||
|
|
||||||
use crate::{
|
|
||||||
SourceId,
|
|
||||||
config::InputLimits,
|
|
||||||
error::{InputError, InputResult},
|
|
||||||
input::Document,
|
|
||||||
};
|
|
||||||
|
|
||||||
/// Iterator over readable regular files below a directory.
|
|
||||||
#[derive(Debug)]
|
|
||||||
pub(super) struct WalkDocuments {
|
|
||||||
/// Underlying streaming directory iterator.
|
|
||||||
entries: IntoIter,
|
|
||||||
/// Limits applied to every yielded file.
|
|
||||||
limits: InputLimits,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl WalkDocuments {
|
|
||||||
/// Begin a recursive walk without following symbolic links.
|
|
||||||
pub(super) fn new(root: &Path, limits: &InputLimits) -> Self {
|
|
||||||
Self {
|
|
||||||
entries: WalkDir::new(root)
|
|
||||||
.follow_links(false)
|
|
||||||
.max_depth(limits.max_walk_depth)
|
|
||||||
.sort_by_file_name()
|
|
||||||
.into_iter(),
|
|
||||||
limits: limits.clone(),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Iterator for WalkDocuments {
|
|
||||||
type Item = InputResult<Document>;
|
|
||||||
|
|
||||||
fn next(&mut self) -> Option<Self::Item> {
|
|
||||||
loop {
|
|
||||||
let entry = match self.entries.next()? {
|
|
||||||
Ok(entry) => entry,
|
|
||||||
Err(source) => return Some(Err(InputError::Walk { source })),
|
|
||||||
};
|
|
||||||
if !entry.file_type().is_file() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
let path = entry.into_path();
|
|
||||||
let content = match super::file::read_text(&path, &self.limits) {
|
|
||||||
Ok(content) => content,
|
|
||||||
Err(error) => return Some(Err(error)),
|
|
||||||
};
|
|
||||||
|
|
||||||
return Some(Ok(Document {
|
|
||||||
source: SourceId::File(path),
|
|
||||||
content,
|
|
||||||
}));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,106 +0,0 @@
|
||||||
//! Shared analysis library for John Slop's command-line, formatter, and LSP
|
|
||||||
//! adapters.
|
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
|
||||||
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
mod config;
|
|
||||||
mod detector;
|
|
||||||
mod diagnostic;
|
|
||||||
mod error;
|
|
||||||
mod input;
|
|
||||||
mod model;
|
|
||||||
mod replacement;
|
|
||||||
mod unicode_review;
|
|
||||||
|
|
||||||
pub use config::{
|
|
||||||
Config, ConfigError, ConfigLayerError, ConfigLoader, ConfigName, ConfigNameError,
|
|
||||||
ConfigOverrides, ConfigPaths, ConfigValidationError, ConfusablePolicy, ContentKind,
|
|
||||||
DiagnosticCategories, DiagnosticPolicy, EvidenceThresholds, HeuristicRule, InputLimits,
|
|
||||||
InputOverrides, MatcherType, ReplacementRule, RuleScope, ThresholdOverrides, UnicodePolicy,
|
|
||||||
default_config_template, default_diagnostics_template,
|
|
||||||
};
|
|
||||||
pub use detector::{AnalysisContext, Detector, DetectorBuildError, Rule};
|
|
||||||
pub use diagnostic::FindingDiagnostic;
|
|
||||||
pub use error::{InputError, InputResult};
|
|
||||||
pub use input::{Document, Documents, read_stdin};
|
|
||||||
pub use model::{
|
|
||||||
Analysis, AnalysisSummary, Category, EvidenceBand, EvidenceScore, Finding, FindingSeverity,
|
|
||||||
RuleId, SeverityCounts, SourceId, TextEdit,
|
|
||||||
};
|
|
||||||
|
|
||||||
/// Deprecated name for [`EvidenceScore`].
|
|
||||||
#[deprecated(since = "0.1.0", note = "use EvidenceScore")]
|
|
||||||
pub type Confidence = EvidenceScore;
|
|
||||||
|
|
||||||
/// Deprecated name for [`EvidenceBand`].
|
|
||||||
#[deprecated(since = "0.1.0", note = "use EvidenceBand")]
|
|
||||||
pub type ConfidenceBand = EvidenceBand;
|
|
||||||
|
|
||||||
/// Deprecated name for [`EvidenceThresholds`].
|
|
||||||
#[deprecated(since = "0.1.0", note = "use EvidenceThresholds")]
|
|
||||||
pub type ConfidenceThresholds = EvidenceThresholds;
|
|
||||||
pub use replacement::ReplacementError;
|
|
||||||
|
|
||||||
/// Load embedded defaults plus optional platform user configuration.
|
|
||||||
pub fn load_config() -> Result<Config, ConfigError> {
|
|
||||||
ConfigLoader::discover(None, None)?.load()
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Analyze a UTF-8 source snapshot with the resolved configuration.
|
|
||||||
pub fn analyze_source(config: &Config, source: &str) -> Analysis {
|
|
||||||
Detector::new(config)
|
|
||||||
.expect("validated configuration must build a detector")
|
|
||||||
.analyze(source)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Plan safe ASCII replacements without modifying the source.
|
|
||||||
pub fn plan_replacements(config: &Config, source: &str) -> Vec<TextEdit> {
|
|
||||||
replacement::plan(config, source)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Validate and apply text edits against their original UTF-8 source snapshot.
|
|
||||||
pub fn apply_replacements<'source>(
|
|
||||||
source: &'source str,
|
|
||||||
edits: &[TextEdit],
|
|
||||||
) -> Result<std::borrow::Cow<'source, str>, ReplacementError> {
|
|
||||||
replacement::apply(source, edits)
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Read documents from a file, directory, or supported ZIP-based container.
|
|
||||||
pub fn read_inputs(config: &Config, path: &Path) -> InputResult<Documents> {
|
|
||||||
input::read(path, &config.input)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn facade_uses_one_configuration_for_analysis_and_edits() {
|
|
||||||
let config = load_config().unwrap();
|
|
||||||
let source = "review—this";
|
|
||||||
let analysis = analyze_source(&config, source);
|
|
||||||
let edits = plan_replacements(&config, source);
|
|
||||||
|
|
||||||
assert_eq!(analysis.findings.len(), 1);
|
|
||||||
assert_eq!(analysis.findings[0].fix.as_ref(), edits.first());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
#[allow(deprecated)]
|
|
||||||
fn legacy_score_type_aliases_remain_source_compatible() {
|
|
||||||
let score: Confidence = EvidenceScore::from_score(42);
|
|
||||||
let band: ConfidenceBand = EvidenceBand::Medium;
|
|
||||||
let thresholds: ConfidenceThresholds = EvidenceThresholds {
|
|
||||||
low: 20,
|
|
||||||
medium: 50,
|
|
||||||
high: 75,
|
|
||||||
};
|
|
||||||
|
|
||||||
assert_eq!(score.score(), 42);
|
|
||||||
assert_eq!(band, EvidenceBand::Medium);
|
|
||||||
assert_eq!(thresholds.low, 20);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,377 +0,0 @@
|
||||||
//! Presentation-neutral domain values shared by analysis adapters.
|
|
||||||
|
|
||||||
use std::{fmt, ops::Range, path::PathBuf};
|
|
||||||
|
|
||||||
use serde::{Deserialize, Serialize};
|
|
||||||
|
|
||||||
/// Stable identifier for a detection rule.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
|
||||||
pub struct RuleId(String);
|
|
||||||
|
|
||||||
impl RuleId {
|
|
||||||
/// Construct a rule identifier from validated configuration text.
|
|
||||||
pub fn new(value: impl Into<String>) -> Self {
|
|
||||||
Self(value.into())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Construct a rule identifier from a static, trusted value.
|
|
||||||
pub fn from_static(value: &'static str) -> Self {
|
|
||||||
Self(value.to_owned())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the identifier as text.
|
|
||||||
pub fn as_str(&self) -> &str {
|
|
||||||
&self.0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for RuleId {
|
|
||||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
self.0.fmt(formatter)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Broad family of evidence reported by a rule.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
|
||||||
#[serde(rename_all = "kebab-case")]
|
|
||||||
pub enum Category {
|
|
||||||
/// Suspicious typography or character usage.
|
|
||||||
Typography,
|
|
||||||
/// Repeated or templated writing style.
|
|
||||||
Style,
|
|
||||||
/// Prompt, response, or assistant scaffolding left in the source.
|
|
||||||
PromptLeakage,
|
|
||||||
/// Low-care patterns specific to source code.
|
|
||||||
SourceCode,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Category {
|
|
||||||
/// Return the stable kebab-case presentation name.
|
|
||||||
pub const fn as_str(self) -> &'static str {
|
|
||||||
match self {
|
|
||||||
Self::Typography => "typography",
|
|
||||||
Self::Style => "style",
|
|
||||||
Self::PromptLeakage => "prompt-leakage",
|
|
||||||
Self::SourceCode => "source-code",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for Category {
|
|
||||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
formatter.write_str(self.as_str())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Review severity attached to an individual configured finding.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
|
||||||
#[serde(rename_all = "lowercase")]
|
|
||||||
pub enum FindingSeverity {
|
|
||||||
/// Informational signal worth considering during review.
|
|
||||||
Advice,
|
|
||||||
/// Stronger signal that should be reviewed before acceptance.
|
|
||||||
Warning,
|
|
||||||
/// Definite policy violation under the selected configuration.
|
|
||||||
Error,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl FindingSeverity {
|
|
||||||
/// Return the stable lowercase presentation name.
|
|
||||||
pub const fn as_str(self) -> &'static str {
|
|
||||||
match self {
|
|
||||||
Self::Advice => "advice",
|
|
||||||
Self::Warning => "warning",
|
|
||||||
Self::Error => "error",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for FindingSeverity {
|
|
||||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
formatter.write_str(self.as_str())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Bounded strength of the configured low-care evidence found in a document.
|
|
||||||
///
|
|
||||||
/// This is deliberately not an AI-authorship probability.
|
|
||||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord)]
|
|
||||||
pub struct EvidenceScore(u8);
|
|
||||||
|
|
||||||
impl EvidenceScore {
|
|
||||||
/// Construct an evidence score, clamping values above 100.
|
|
||||||
pub const fn from_score(score: u8) -> Self {
|
|
||||||
Self(if score > 100 { 100 } else { score })
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Return the normalized score in the inclusive range `0..=100`.
|
|
||||||
pub const fn score(self) -> u8 {
|
|
||||||
self.0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Named interpretation of a non-empty analysis score.
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
|
|
||||||
#[serde(rename_all = "kebab-case")]
|
|
||||||
pub enum EvidenceBand {
|
|
||||||
/// Evidence exists but remains below the configured low threshold.
|
|
||||||
BelowThreshold,
|
|
||||||
/// Score meets the configured low threshold.
|
|
||||||
Low,
|
|
||||||
/// Score meets the configured medium threshold.
|
|
||||||
Medium,
|
|
||||||
/// Score meets the configured high threshold.
|
|
||||||
High,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl EvidenceBand {
|
|
||||||
/// Return the stable kebab-case presentation name.
|
|
||||||
pub const fn as_str(self) -> &'static str {
|
|
||||||
match self {
|
|
||||||
Self::BelowThreshold => "below-threshold",
|
|
||||||
Self::Low => "low",
|
|
||||||
Self::Medium => "medium",
|
|
||||||
Self::High => "high",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for EvidenceBand {
|
|
||||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
formatter.write_str(self.as_str())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Identity of the exact source snapshot being analyzed.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
||||||
pub enum SourceId {
|
|
||||||
/// Content supplied through standard input.
|
|
||||||
Stdin,
|
|
||||||
/// A regular file on disk.
|
|
||||||
File(PathBuf),
|
|
||||||
/// A member contained in an archive file.
|
|
||||||
ArchiveMember {
|
|
||||||
/// Path of the outer archive.
|
|
||||||
archive: PathBuf,
|
|
||||||
/// Path-like member name stored in the archive.
|
|
||||||
member: PathBuf,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
impl fmt::Display for SourceId {
|
|
||||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
||||||
match self {
|
|
||||||
Self::Stdin => formatter.write_str("<stdin>"),
|
|
||||||
Self::File(path) => path.display().fmt(formatter),
|
|
||||||
Self::ArchiveMember { archive, member } => {
|
|
||||||
write!(formatter, "{}::{}", archive.display(), member.display())
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// A non-overlapping replacement against an analyzed source snapshot.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
||||||
pub struct TextEdit {
|
|
||||||
/// Zero-based, half-open UTF-8 byte range to replace.
|
|
||||||
pub byte_span: Range<usize>,
|
|
||||||
/// Text to insert in place of the range.
|
|
||||||
pub replacement: String,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// One explainable piece of evidence produced by a detection rule.
|
|
||||||
#[derive(Debug, Clone, PartialEq)]
|
|
||||||
pub struct Finding {
|
|
||||||
/// Stable rule identifier.
|
|
||||||
pub rule_id: RuleId,
|
|
||||||
/// Evidence family.
|
|
||||||
pub category: Category,
|
|
||||||
/// Configured review severity.
|
|
||||||
pub severity: FindingSeverity,
|
|
||||||
/// Concise description of the observed signal.
|
|
||||||
pub message: String,
|
|
||||||
/// Optional remediation or review guidance.
|
|
||||||
pub help: Option<String>,
|
|
||||||
/// Zero-based, half-open UTF-8 byte range in the analyzed source.
|
|
||||||
pub byte_span: Range<usize>,
|
|
||||||
/// Exact source text covered by `byte_span`.
|
|
||||||
pub matched_text: String,
|
|
||||||
/// Configured evidence weight before aggregate normalization.
|
|
||||||
pub weight: f64,
|
|
||||||
/// Optional safe replacement for this finding.
|
|
||||||
pub fix: Option<TextEdit>,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Complete deterministic analysis of one source snapshot.
|
|
||||||
#[derive(Debug, Clone, Default, PartialEq)]
|
|
||||||
pub struct Analysis {
|
|
||||||
/// Findings sorted by source offset and rule identifier.
|
|
||||||
pub findings: Vec<Finding>,
|
|
||||||
/// Aggregate strength of the configured evidence found.
|
|
||||||
pub evidence_score: EvidenceScore,
|
|
||||||
/// Named score band, or no verdict when no evidence was found.
|
|
||||||
pub evidence_band: Option<EvidenceBand>,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl Analysis {
|
|
||||||
/// Summarize score, named band, and severity counts without consuming findings.
|
|
||||||
pub fn summary(&self) -> AnalysisSummary {
|
|
||||||
AnalysisSummary {
|
|
||||||
evidence_score: self.evidence_score.score(),
|
|
||||||
evidence_band: self.evidence_band,
|
|
||||||
findings: SeverityCounts::from_findings(&self.findings),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Compact deterministic summary of one completed analysis.
|
|
||||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
|
||||||
pub struct AnalysisSummary {
|
|
||||||
/// Normalized evidence score in `0..=100`.
|
|
||||||
pub evidence_score: u8,
|
|
||||||
/// Named band, or no verdict when the source has no evidence.
|
|
||||||
pub evidence_band: Option<EvidenceBand>,
|
|
||||||
/// Finding totals grouped by configured severity.
|
|
||||||
pub findings: SeverityCounts,
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Finding counts grouped by configured presentation severity.
|
|
||||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
|
|
||||||
pub struct SeverityCounts {
|
|
||||||
/// Informational findings.
|
|
||||||
pub advice: usize,
|
|
||||||
/// Warning findings.
|
|
||||||
pub warnings: usize,
|
|
||||||
/// Error findings.
|
|
||||||
pub errors: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl SeverityCounts {
|
|
||||||
/// Return the total number of findings across severities.
|
|
||||||
pub const fn total(self) -> usize {
|
|
||||||
self.advice + self.warnings + self.errors
|
|
||||||
}
|
|
||||||
|
|
||||||
fn from_findings(findings: &[Finding]) -> Self {
|
|
||||||
let mut counts = Self::default();
|
|
||||||
for finding in findings {
|
|
||||||
match finding.severity {
|
|
||||||
FindingSeverity::Advice => counts.advice += 1,
|
|
||||||
FindingSeverity::Warning => counts.warnings += 1,
|
|
||||||
FindingSeverity::Error => counts.errors += 1,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
counts
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn evidence_score_is_bounded() {
|
|
||||||
assert_eq!(EvidenceScore::from_score(0).score(), 0);
|
|
||||||
assert_eq!(EvidenceScore::from_score(73).score(), 73);
|
|
||||||
assert_eq!(EvidenceScore::from_score(255).score(), 100);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn source_id_display_distinguishes_all_source_kinds() {
|
|
||||||
assert_eq!(SourceId::Stdin.to_string(), "<stdin>");
|
|
||||||
assert_eq!(
|
|
||||||
SourceId::File(PathBuf::from("article.md")).to_string(),
|
|
||||||
"article.md"
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
SourceId::ArchiveMember {
|
|
||||||
archive: PathBuf::from("bundle.zip"),
|
|
||||||
member: PathBuf::from("docs/article.md"),
|
|
||||||
}
|
|
||||||
.to_string(),
|
|
||||||
"bundle.zip::docs/article.md"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn rule_id_has_stable_text_and_ordering() {
|
|
||||||
let first = RuleId::new("alpha-rule");
|
|
||||||
let second = RuleId::from_static("beta-rule");
|
|
||||||
|
|
||||||
assert_eq!(first.as_str(), "alpha-rule");
|
|
||||||
assert_eq!(first.to_string(), "alpha-rule");
|
|
||||||
assert!(first < second);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn analysis_summary_counts_severity_without_removing_findings() {
|
|
||||||
let analysis = Analysis {
|
|
||||||
findings: vec![
|
|
||||||
Finding {
|
|
||||||
rule_id: RuleId::from_static("advice-rule"),
|
|
||||||
category: Category::Style,
|
|
||||||
severity: FindingSeverity::Advice,
|
|
||||||
message: "advice".to_owned(),
|
|
||||||
help: None,
|
|
||||||
byte_span: 0..1,
|
|
||||||
matched_text: "a".to_owned(),
|
|
||||||
weight: 1.0,
|
|
||||||
fix: None,
|
|
||||||
},
|
|
||||||
Finding {
|
|
||||||
rule_id: RuleId::from_static("warning-rule"),
|
|
||||||
category: Category::PromptLeakage,
|
|
||||||
severity: FindingSeverity::Warning,
|
|
||||||
message: "warning".to_owned(),
|
|
||||||
help: None,
|
|
||||||
byte_span: 2..3,
|
|
||||||
matched_text: "b".to_owned(),
|
|
||||||
weight: 10.0,
|
|
||||||
fix: None,
|
|
||||||
},
|
|
||||||
],
|
|
||||||
evidence_score: EvidenceScore::from_score(32),
|
|
||||||
evidence_band: Some(EvidenceBand::Low),
|
|
||||||
};
|
|
||||||
|
|
||||||
let summary = analysis.summary();
|
|
||||||
|
|
||||||
assert_eq!(analysis.findings.len(), 2);
|
|
||||||
assert_eq!(summary.evidence_score, 32);
|
|
||||||
assert_eq!(summary.evidence_band, Some(EvidenceBand::Low));
|
|
||||||
assert_eq!(summary.findings.advice, 1);
|
|
||||||
assert_eq!(summary.findings.warnings, 1);
|
|
||||||
assert_eq!(summary.findings.errors, 0);
|
|
||||||
assert_eq!(summary.findings.total(), analysis.findings.len());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn enum_names_are_stable_for_display_and_serialization() {
|
|
||||||
for (category, expected) in [
|
|
||||||
(Category::Typography, "typography"),
|
|
||||||
(Category::Style, "style"),
|
|
||||||
(Category::PromptLeakage, "prompt-leakage"),
|
|
||||||
(Category::SourceCode, "source-code"),
|
|
||||||
] {
|
|
||||||
assert_eq!(category.as_str(), expected);
|
|
||||||
assert_eq!(category.to_string(), expected);
|
|
||||||
}
|
|
||||||
for (severity, expected) in [
|
|
||||||
(FindingSeverity::Advice, "advice"),
|
|
||||||
(FindingSeverity::Warning, "warning"),
|
|
||||||
(FindingSeverity::Error, "error"),
|
|
||||||
] {
|
|
||||||
assert_eq!(severity.as_str(), expected);
|
|
||||||
assert_eq!(severity.to_string(), expected);
|
|
||||||
}
|
|
||||||
for (band, expected) in [
|
|
||||||
(EvidenceBand::BelowThreshold, "below-threshold"),
|
|
||||||
(EvidenceBand::Low, "low"),
|
|
||||||
(EvidenceBand::Medium, "medium"),
|
|
||||||
(EvidenceBand::High, "high"),
|
|
||||||
] {
|
|
||||||
assert_eq!(band.as_str(), expected);
|
|
||||||
assert_eq!(band.to_string(), expected);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,367 +0,0 @@
|
||||||
//! Pure planning and application of configured text replacements.
|
|
||||||
|
|
||||||
use std::{borrow::Cow, collections::BTreeMap, ops::Range};
|
|
||||||
|
|
||||||
use miette::Diagnostic;
|
|
||||||
|
|
||||||
use crate::{Config, ConfusablePolicy, ReplacementRule, TextEdit};
|
|
||||||
|
|
||||||
/// An edit set cannot be applied safely to its source snapshot.
|
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error, Diagnostic)]
|
|
||||||
#[diagnostic(code(john_slop::replacement::invalid_edit))]
|
|
||||||
pub enum ReplacementError {
|
|
||||||
/// A range is empty, reversed, or extends beyond the source.
|
|
||||||
#[error("replacement byte range {start}..{end} is invalid for a {source_len}-byte source")]
|
|
||||||
InvalidRange {
|
|
||||||
/// Inclusive start byte supplied by the edit.
|
|
||||||
start: usize,
|
|
||||||
/// Exclusive end byte supplied by the edit.
|
|
||||||
end: usize,
|
|
||||||
/// Source snapshot length in bytes.
|
|
||||||
source_len: usize,
|
|
||||||
},
|
|
||||||
/// A range endpoint does not lie on a UTF-8 character boundary.
|
|
||||||
#[error("replacement byte offset {offset} is not a UTF-8 character boundary")]
|
|
||||||
InvalidCharacterBoundary {
|
|
||||||
/// Invalid endpoint.
|
|
||||||
offset: usize,
|
|
||||||
},
|
|
||||||
/// Two edits cover at least one common source byte.
|
|
||||||
#[error("replacement ranges overlap at byte {offset}")]
|
|
||||||
OverlappingEdits {
|
|
||||||
/// Start of the later edit inside an earlier edit.
|
|
||||||
offset: usize,
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Plan deterministic non-overlapping edits without modifying the source.
|
|
||||||
pub fn plan(config: &Config, source: &str) -> Vec<TextEdit> {
|
|
||||||
plan_matching(config, source, |rule| {
|
|
||||||
rule.enabled
|
|
||||||
&& (!is_bulk_confusable(rule)
|
|
||||||
|| config.unicode.confusables == ConfusablePolicy::Replace)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
pub(crate) fn plan_confusables_for_review(config: &Config, source: &str) -> Vec<TextEdit> {
|
|
||||||
if config.unicode.confusables != ConfusablePolicy::Review {
|
|
||||||
return Vec::new();
|
|
||||||
}
|
|
||||||
plan_matching(config, source, |rule| {
|
|
||||||
rule.enabled && is_bulk_confusable(rule)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
fn is_bulk_confusable(rule: &ReplacementRule) -> bool {
|
|
||||||
let mut characters = rule.source.chars();
|
|
||||||
let is_fullwidth_scalar =
|
|
||||||
matches!(characters.next(), Some('\u{ff01}'..='\u{ff5e}')) && characters.next().is_none();
|
|
||||||
rule.id.starts_with("uts39-") && !is_fullwidth_scalar
|
|
||||||
}
|
|
||||||
|
|
||||||
fn plan_matching(
|
|
||||||
config: &Config,
|
|
||||||
source: &str,
|
|
||||||
include: impl Fn(&ReplacementRule) -> bool,
|
|
||||||
) -> Vec<TextEdit> {
|
|
||||||
let mut single_character_rules = BTreeMap::new();
|
|
||||||
let mut sequence_rules = Vec::new();
|
|
||||||
for rule in config.replacements.iter().filter(|rule| include(rule)) {
|
|
||||||
let mut characters = rule.source.chars();
|
|
||||||
let first = characters.next();
|
|
||||||
if let (Some(character), None) = (first, characters.next()) {
|
|
||||||
single_character_rules.insert(character, rule);
|
|
||||||
} else {
|
|
||||||
sequence_rules.push(rule);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
sequence_rules.sort_by(|left, right| {
|
|
||||||
right
|
|
||||||
.source
|
|
||||||
.len()
|
|
||||||
.cmp(&left.source.len())
|
|
||||||
.then_with(|| left.id.cmp(&right.id))
|
|
||||||
});
|
|
||||||
|
|
||||||
let mut edits = Vec::new();
|
|
||||||
let mut offset = 0;
|
|
||||||
while offset < source.len() {
|
|
||||||
if let Some(rule) = sequence_rules
|
|
||||||
.iter()
|
|
||||||
.find(|rule| source[offset..].starts_with(&rule.source))
|
|
||||||
{
|
|
||||||
let end = offset + rule.source.len();
|
|
||||||
edits.push(TextEdit {
|
|
||||||
byte_span: offset..end,
|
|
||||||
replacement: rule.replacement.clone(),
|
|
||||||
});
|
|
||||||
offset = end;
|
|
||||||
} else {
|
|
||||||
let Some(character) = source[offset..].chars().next() else {
|
|
||||||
break;
|
|
||||||
};
|
|
||||||
let end = offset + character.len_utf8();
|
|
||||||
if let Some(rule) = single_character_rules.get(&character) {
|
|
||||||
edits.push(TextEdit {
|
|
||||||
byte_span: offset..end,
|
|
||||||
replacement: rule.replacement.clone(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
offset = end;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
edits
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Validate and apply edits from the end of the source toward the beginning.
|
|
||||||
///
|
|
||||||
/// With no edits, the returned [`Cow`] borrows the original source without an
|
|
||||||
/// allocation. All ranges refer to the original source snapshot.
|
|
||||||
pub fn apply<'source>(
|
|
||||||
source: &'source str,
|
|
||||||
edits: &[TextEdit],
|
|
||||||
) -> Result<Cow<'source, str>, ReplacementError> {
|
|
||||||
if edits.is_empty() {
|
|
||||||
return Ok(Cow::Borrowed(source));
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut ordered = edits.iter().collect::<Vec<_>>();
|
|
||||||
ordered.sort_by(|left, right| range_order(&left.byte_span, &right.byte_span));
|
|
||||||
let mut previous_end = 0;
|
|
||||||
for edit in &ordered {
|
|
||||||
let range = &edit.byte_span;
|
|
||||||
if range.start >= range.end || range.end > source.len() {
|
|
||||||
return Err(ReplacementError::InvalidRange {
|
|
||||||
start: range.start,
|
|
||||||
end: range.end,
|
|
||||||
source_len: source.len(),
|
|
||||||
});
|
|
||||||
}
|
|
||||||
for offset in [range.start, range.end] {
|
|
||||||
if !source.is_char_boundary(offset) {
|
|
||||||
return Err(ReplacementError::InvalidCharacterBoundary { offset });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if range.start < previous_end {
|
|
||||||
return Err(ReplacementError::OverlappingEdits {
|
|
||||||
offset: range.start,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
previous_end = range.end;
|
|
||||||
}
|
|
||||||
|
|
||||||
let mut output = source.to_owned();
|
|
||||||
for edit in ordered.into_iter().rev() {
|
|
||||||
output.replace_range(edit.byte_span.clone(), &edit.replacement);
|
|
||||||
}
|
|
||||||
Ok(Cow::Owned(output))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn range_order(left: &Range<usize>, right: &Range<usize>) -> std::cmp::Ordering {
|
|
||||||
(left.start, left.end).cmp(&(right.start, right.end))
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn plans_only_configured_non_ascii_characters() {
|
|
||||||
let edits = plan(&Config::embedded(), "plain-word—fixed");
|
|
||||||
|
|
||||||
assert_eq!(edits.len(), 1);
|
|
||||||
assert_eq!(edits[0].byte_span, 10..13);
|
|
||||||
assert_eq!(edits[0].replacement, "-");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn plans_multiple_edits_in_source_order_with_utf8_ranges() {
|
|
||||||
let edits = plan(&Config::embedded(), "a—b−c");
|
|
||||||
|
|
||||||
assert_eq!(
|
|
||||||
edits,
|
|
||||||
vec![
|
|
||||||
TextEdit {
|
|
||||||
byte_span: 1..4,
|
|
||||||
replacement: "-".to_owned(),
|
|
||||||
},
|
|
||||||
TextEdit {
|
|
||||||
byte_span: 5..8,
|
|
||||||
replacement: "-".to_owned(),
|
|
||||||
},
|
|
||||||
]
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn generated_catalog_replaces_letters_digits_punctuation_and_multi_byte_skeletons() {
|
|
||||||
let source = "а𝐀!𝟘ⅿ🯹";
|
|
||||||
let edits = plan(&Config::embedded(), source);
|
|
||||||
|
|
||||||
assert_eq!(edits.len(), 6);
|
|
||||||
assert_eq!(apply(source, &edits).unwrap(), "aA!Orn9");
|
|
||||||
assert_eq!(edits[4].replacement, "rn");
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn generated_catalog_width_folds_every_printable_fullwidth_ascii_character() {
|
|
||||||
let source = std::iter::once('\u{3000}')
|
|
||||||
.chain((b'!'..=b'~').filter_map(|ascii| char::from_u32(u32::from(ascii) + 0xfee0)))
|
|
||||||
.collect::<String>();
|
|
||||||
let expected = (b' '..=b'~').map(char::from).collect::<String>();
|
|
||||||
let edits = plan(&Config::embedded(), &source);
|
|
||||||
|
|
||||||
assert_eq!(edits.len(), 95);
|
|
||||||
assert_eq!(apply(&source, &edits).unwrap(), expected);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn bulk_confusable_policy_preserves_curated_replacements() {
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
config.unicode.confusables = ConfusablePolicy::Review;
|
|
||||||
assert_eq!(apply("а—I ", &plan(&config, "а—I ")).unwrap(), "а-I ");
|
|
||||||
assert_eq!(plan_confusables_for_review(&config, "а—I ").len(), 1);
|
|
||||||
|
|
||||||
config.unicode.confusables = ConfusablePolicy::Off;
|
|
||||||
assert_eq!(apply("а—I ", &plan(&config, "а—I ")).unwrap(), "а-I ");
|
|
||||||
assert!(plan_confusables_for_review(&config, "а—I ").is_empty());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn empty_or_ascii_sources_need_no_edits() {
|
|
||||||
let config = Config::embedded();
|
|
||||||
|
|
||||||
assert!(plan(&config, "").is_empty());
|
|
||||||
assert!(plan(&config, "ordinary-ascii").is_empty());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn longest_source_wins_independently_of_configuration_order() {
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
config.replacements.insert(
|
|
||||||
0,
|
|
||||||
crate::ReplacementRule {
|
|
||||||
id: "double-em-dash".to_owned(),
|
|
||||||
source: "——".to_owned(),
|
|
||||||
replacement: "--".to_owned(),
|
|
||||||
rationale: "Preserve a doubled dash.".to_owned(),
|
|
||||||
enabled: true,
|
|
||||||
},
|
|
||||||
);
|
|
||||||
config.validate().unwrap();
|
|
||||||
|
|
||||||
let edits = plan(&config, "a——b—c");
|
|
||||||
|
|
||||||
assert_eq!(edits.len(), 2);
|
|
||||||
assert_eq!(edits[0].byte_span, 1..7);
|
|
||||||
assert_eq!(edits[0].replacement, "--");
|
|
||||||
assert_eq!(edits[1].byte_span, 8..11);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn applies_edits_in_reverse_and_preserves_line_endings() {
|
|
||||||
let source = "a—b\r\nc−d\n";
|
|
||||||
let edits = plan(&Config::embedded(), source);
|
|
||||||
|
|
||||||
assert_eq!(apply(source, &edits).unwrap(), "a-b\r\nc-d\n");
|
|
||||||
assert_eq!(
|
|
||||||
apply(&apply(source, &edits).unwrap(), &[]).unwrap(),
|
|
||||||
"a-b\r\nc-d\n"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn no_edits_borrow_the_original_source() {
|
|
||||||
let source = "ordinary ASCII";
|
|
||||||
|
|
||||||
assert!(matches!(apply(source, &[]).unwrap(), Cow::Borrowed(value) if value == source));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn rejects_invalid_boundaries_ranges_and_overlaps() {
|
|
||||||
let invalid_boundary = [TextEdit {
|
|
||||||
byte_span: 1..2,
|
|
||||||
replacement: "-".to_owned(),
|
|
||||||
}];
|
|
||||||
assert!(matches!(
|
|
||||||
apply("é", &invalid_boundary),
|
|
||||||
Err(ReplacementError::InvalidCharacterBoundary { offset: 1 })
|
|
||||||
));
|
|
||||||
|
|
||||||
let invalid_range = [TextEdit {
|
|
||||||
byte_span: 0..3,
|
|
||||||
replacement: "-".to_owned(),
|
|
||||||
}];
|
|
||||||
assert!(matches!(
|
|
||||||
apply("ab", &invalid_range),
|
|
||||||
Err(ReplacementError::InvalidRange { .. })
|
|
||||||
));
|
|
||||||
|
|
||||||
let overlapping = [
|
|
||||||
TextEdit {
|
|
||||||
byte_span: 0..2,
|
|
||||||
replacement: "x".to_owned(),
|
|
||||||
},
|
|
||||||
TextEdit {
|
|
||||||
byte_span: 1..3,
|
|
||||||
replacement: "y".to_owned(),
|
|
||||||
},
|
|
||||||
];
|
|
||||||
assert!(matches!(
|
|
||||||
apply("abc", &overlapping),
|
|
||||||
Err(ReplacementError::OverlappingEdits { offset: 1 })
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn generated_sequences_are_stable_non_overlapping_and_idempotent() {
|
|
||||||
let config = Config::embedded();
|
|
||||||
let alphabet = ["a", "é", "—", "−", "\r", "\n"];
|
|
||||||
|
|
||||||
for first in alphabet {
|
|
||||||
for second in alphabet {
|
|
||||||
for third in alphabet {
|
|
||||||
for fourth in alphabet {
|
|
||||||
let source = [first, second, third, fourth].concat();
|
|
||||||
let edits = plan(&config, &source);
|
|
||||||
assert_eq!(edits, plan(&config, &source));
|
|
||||||
for edit in &edits {
|
|
||||||
assert!(source.is_char_boundary(edit.byte_span.start));
|
|
||||||
assert!(source.is_char_boundary(edit.byte_span.end));
|
|
||||||
}
|
|
||||||
assert!(
|
|
||||||
edits
|
|
||||||
.windows(2)
|
|
||||||
.all(|pair| { pair[0].byte_span.end <= pair[1].byte_span.start })
|
|
||||||
);
|
|
||||||
|
|
||||||
let formatted = apply(&source, &edits).unwrap();
|
|
||||||
assert_eq!(
|
|
||||||
source
|
|
||||||
.chars()
|
|
||||||
.filter(|character| *character == '\r')
|
|
||||||
.count(),
|
|
||||||
formatted
|
|
||||||
.chars()
|
|
||||||
.filter(|character| *character == '\r')
|
|
||||||
.count()
|
|
||||||
);
|
|
||||||
assert_eq!(
|
|
||||||
source
|
|
||||||
.chars()
|
|
||||||
.filter(|character| *character == '\n')
|
|
||||||
.count(),
|
|
||||||
formatted
|
|
||||||
.chars()
|
|
||||||
.filter(|character| *character == '\n')
|
|
||||||
.count()
|
|
||||||
);
|
|
||||||
let second_edits = plan(&config, &formatted);
|
|
||||||
assert!(second_edits.is_empty());
|
|
||||||
assert_eq!(apply(&formatted, &second_edits).unwrap(), formatted);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,55 +0,0 @@
|
||||||
//! Pinned invisible and bidirectional-control review profile.
|
|
||||||
|
|
||||||
/// Match the invisible/control profile generated by Gitea at commit 9c715c0.
|
|
||||||
///
|
|
||||||
/// Gitea removes ordinary space, tab, and line feed before generating its
|
|
||||||
/// table. Callers remain responsible for treating CRLF as one ordinary line
|
|
||||||
/// ending rather than a suspicious standalone carriage return.
|
|
||||||
pub(crate) const fn is_gitea_invisible(character: char) -> bool {
|
|
||||||
matches!(
|
|
||||||
character as u32,
|
|
||||||
11..=13
|
|
||||||
| 127
|
|
||||||
| 160
|
|
||||||
| 173
|
|
||||||
| 847
|
|
||||||
| 1564
|
|
||||||
| 4447
|
|
||||||
| 4448
|
|
||||||
| 6068
|
|
||||||
| 6069
|
|
||||||
| 6155..=6158
|
|
||||||
| 7355..=7356
|
|
||||||
| 8192..=8207
|
|
||||||
| 8234..=8239
|
|
||||||
| 8287..=8303
|
|
||||||
| 10240
|
|
||||||
| 12288
|
|
||||||
| 12644
|
|
||||||
| 65024..=65039
|
|
||||||
| 65279
|
|
||||||
| 65440
|
|
||||||
| 65520..=65528
|
|
||||||
| 65532
|
|
||||||
| 78844
|
|
||||||
| 119155..=119162
|
|
||||||
| 917504..=917631
|
|
||||||
| 917760..=917999
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn gitea_profile_covers_invisible_and_bidi_ranges_without_normal_whitespace() {
|
|
||||||
assert!(is_gitea_invisible('\u{200b}'));
|
|
||||||
assert!(is_gitea_invisible('\u{202e}'));
|
|
||||||
assert!(is_gitea_invisible('\u{2066}'));
|
|
||||||
assert!(is_gitea_invisible('\u{e0001}'));
|
|
||||||
assert!(!is_gitea_invisible(' '));
|
|
||||||
assert!(!is_gitea_invisible('\t'));
|
|
||||||
assert!(!is_gitea_invisible('\n'));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
[package]
|
|
||||||
name = "john-slop-lsp"
|
|
||||||
version.workspace = true
|
|
||||||
edition.workspace = true
|
|
||||||
rust-version.workspace = true
|
|
||||||
license.workspace = true
|
|
||||||
repository.workspace = true
|
|
||||||
description = "Language Server Protocol engine for John Slop"
|
|
||||||
|
|
||||||
[lib]
|
|
||||||
name = "john_slop_lsp_engine"
|
|
||||||
|
|
||||||
[dependencies]
|
|
||||||
john-slop-core.workspace = true
|
|
||||||
serde_json.workspace = true
|
|
||||||
thiserror.workspace = true
|
|
||||||
tokio = { workspace = true, features = ["sync"] }
|
|
||||||
tower-lsp-server.workspace = true
|
|
||||||
|
|
||||||
[lints]
|
|
||||||
workspace = true
|
|
||||||
File diff suppressed because it is too large
Load Diff
32
deny.toml
32
deny.toml
|
|
@ -1,32 +0,0 @@
|
||||||
[graph]
|
|
||||||
all-features = true
|
|
||||||
|
|
||||||
[advisories]
|
|
||||||
version = 2
|
|
||||||
yanked = "deny"
|
|
||||||
|
|
||||||
[licenses]
|
|
||||||
version = 2
|
|
||||||
confidence-threshold = 0.8
|
|
||||||
allow = [
|
|
||||||
"0BSD",
|
|
||||||
"Apache-2.0",
|
|
||||||
"Apache-2.0 WITH LLVM-exception",
|
|
||||||
"ISC",
|
|
||||||
"MIT",
|
|
||||||
"MIT-0",
|
|
||||||
"MPL-2.0",
|
|
||||||
"Unicode-3.0",
|
|
||||||
"Unlicense",
|
|
||||||
"Zlib",
|
|
||||||
]
|
|
||||||
|
|
||||||
[bans]
|
|
||||||
multiple-versions = "warn"
|
|
||||||
wildcards = "deny"
|
|
||||||
highlight = "all"
|
|
||||||
|
|
||||||
[sources]
|
|
||||||
unknown-registry = "deny"
|
|
||||||
unknown-git = "deny"
|
|
||||||
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
# John Slop documentation
|
|
||||||
|
|
||||||
The root [README](../README.md) gets you from installation to a first scan. Move
|
|
||||||
through these guides as your needs grow; you do not need to understand the
|
|
||||||
internals to use the tools.
|
|
||||||
|
|
||||||
## Start using it
|
|
||||||
|
|
||||||
1. [Usage](usage.md) covers accepted inputs, detector options, stable exit
|
|
||||||
statuses, formatter modes, and the LSP command.
|
|
||||||
2. [Output](output.md) explains graphical Miette reports, plain human output,
|
|
||||||
concise records, versioned JSON Lines, and partial input errors.
|
|
||||||
3. [Configuration](configuration.md) documents source precedence, named
|
|
||||||
profiles, stable-ID merges, validation, and inspection commands. Start from
|
|
||||||
the [commented examples](../examples/config/).
|
|
||||||
|
|
||||||
## Put it into practice
|
|
||||||
|
|
||||||
4. [Runnable examples](../examples/README.md) demonstrate in-memory analysis,
|
|
||||||
streaming path input, shared replacement planning, and an expected-failure
|
|
||||||
diagnostic showcase.
|
|
||||||
5. [Editor setup](editors.md) provides working Helix (`hx`) and Vim formatter
|
|
||||||
and LSP configurations with validation steps.
|
|
||||||
6. [Review signals and evidence methodology](review-signals.md) explains the
|
|
||||||
research boundary, evidence tiers, and deliberately excluded proxies.
|
|
||||||
7. [Heuristics and evidence scoring](heuristics.md) explains what each rule
|
|
||||||
contributes and how the bounded score is calculated.
|
|
||||||
8. [Limitations](limitations.md) covers false positives, evasion, domain shift,
|
|
||||||
score semantics, and safe interpretation.
|
|
||||||
|
|
||||||
## Go deeper
|
|
||||||
|
|
||||||
The [advanced guide](advanced.md) is the single entry point for architecture,
|
|
||||||
the public library API, evaluation, benchmarks, dependencies, design records,
|
|
||||||
research references, development, security, and releases.
|
|
||||||
|
|
||||||
Project work is tracked in [tasks.md](../tasks.md), while contribution policy is
|
|
||||||
defined in [CONTRIBUTING.md](../CONTRIBUTING.md) and
|
|
||||||
[AGENTS.md](../AGENTS.md).
|
|
||||||
|
|
@ -1,58 +0,0 @@
|
||||||
# Advanced concepts and project internals
|
|
||||||
|
|
||||||
This guide collects material that is useful after the normal CLI,
|
|
||||||
configuration, output, and editor workflows are familiar. The root
|
|
||||||
[README](../README.md) and [documentation path](README.md) remain the best entry
|
|
||||||
points for day-to-day use.
|
|
||||||
|
|
||||||
## Detection model and evaluation
|
|
||||||
|
|
||||||
- [Heuristics and evidence scoring](heuristics.md) describes applicability,
|
|
||||||
occurrence damping, normalization, and why the score is not authorship
|
|
||||||
confidence.
|
|
||||||
- [Review signals and evidence methodology](review-signals.md) records the
|
|
||||||
research basis, evidence tiers, and signals deliberately kept out.
|
|
||||||
- [Evaluation](evaluation.md) records corpus provenance, per-rule measurements,
|
|
||||||
aggregate false positives, and the limits of the current metrics.
|
|
||||||
- [Limitations](limitations.md) covers evasion, domain shift, interpretation,
|
|
||||||
and operational boundaries.
|
|
||||||
|
|
||||||
## Architecture and embedding
|
|
||||||
|
|
||||||
- [Architecture and public API](architecture.md) describes the Cargo workspace,
|
|
||||||
module boundaries, shared domain values, streaming input, and reusable facade.
|
|
||||||
- [Output contracts](output.md) specifies human presentation and the versioned
|
|
||||||
JSON Lines schema for downstream integrations.
|
|
||||||
- [Decision records](decisions/) preserve the reasoning behind core contracts,
|
|
||||||
input limits, the LSP protocol, and the supported workspace.
|
|
||||||
- [Crate-boundary experiment](experiments/crate-boundary.md) contains the
|
|
||||||
measurements that preceded the workspace decision.
|
|
||||||
|
|
||||||
## Performance, dependencies, and safety
|
|
||||||
|
|
||||||
- [Performance benchmarks](benchmarks.md) defines representative workloads and
|
|
||||||
the evidence required before adding parallel execution.
|
|
||||||
- [Dependency and binary review](dependencies.md) records feature choices,
|
|
||||||
advisory and license policy, unsafe boundaries, and release-size measurements.
|
|
||||||
- [Security policy](../SECURITY.md) explains supported versions and private
|
|
||||||
vulnerability reporting.
|
|
||||||
|
|
||||||
## Research references
|
|
||||||
|
|
||||||
- [Background](background.md) collects external context on observable signs of
|
|
||||||
low-care AI-assisted writing.
|
|
||||||
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
|
|
||||||
Gitea's locale-aware catalog, its upstream source, and safe adoption
|
|
||||||
boundaries.
|
|
||||||
|
|
||||||
## Development and release engineering
|
|
||||||
|
|
||||||
- [Development](development.md) covers the Rust toolchain, feature matrix,
|
|
||||||
tests, snapshots, coverage, badges, and CI gates.
|
|
||||||
- [Contributing](../CONTRIBUTING.md) explains the patch workflow and review
|
|
||||||
expectations; [AGENTS.md](../AGENTS.md) defines repository-specific rules for
|
|
||||||
coding agents.
|
|
||||||
- [Release procedure](releasing.md) defines tagged reproducible packaging,
|
|
||||||
checksums, and provenance metadata.
|
|
||||||
- [Implementation plan](../tasks.md) records completed milestones and deferred
|
|
||||||
product work.
|
|
||||||
|
|
@ -1,109 +0,0 @@
|
||||||
# Architecture
|
|
||||||
|
|
||||||
John Slop is a workspace with a compatibility facade and two reusable libraries.
|
|
||||||
Executables remain thin adapters responsible for argument parsing, presentation,
|
|
||||||
and process exit statuses.
|
|
||||||
|
|
||||||
The current flow is:
|
|
||||||
|
|
||||||
```text
|
|
||||||
path -> input stream -> Document -> analysis -> Analysis + Finding
|
|
||||||
-> diagnostic/output adapters
|
|
||||||
-> replacement planner -> TextEdit -> applier
|
|
||||||
-> formatter I/O
|
|
||||||
-> LSP UTF-16 edit
|
|
||||||
```
|
|
||||||
|
|
||||||
The crate boundary separates dependency families:
|
|
||||||
|
|
||||||
- `john-slop-core` owns configuration, input, detection, diagnostics,
|
|
||||||
replacement planning, and presentation-neutral domain values;
|
|
||||||
- `john-slop-lsp` owns asynchronous editor state and protocol conversion while
|
|
||||||
depending only on `john-slop-core`;
|
|
||||||
- the root `john-slop` package re-exports both libraries to preserve its public
|
|
||||||
API and distributes the detector, formatter, and LSP binaries.
|
|
||||||
|
|
||||||
The root manifest is intentionally both a package and the workspace root. A
|
|
||||||
virtual manifest would relocate the compatibility facade, binaries, examples,
|
|
||||||
and fixtures without improving dependency isolation. `[workspace.package]`,
|
|
||||||
`[workspace.dependencies]`, and `[workspace.lints]` provide one source of truth
|
|
||||||
for shared release metadata, dependency requirements, and first-party unsafe
|
|
||||||
code policy. Member manifests retain only package-specific descriptions,
|
|
||||||
dependency selections, feature activation, and optionality. In particular, the
|
|
||||||
workspace Tokio requirement enables no features: the LSP library selects
|
|
||||||
`sync`, and the root LSP adapter selects its runtime and stdio features.
|
|
||||||
|
|
||||||
The modules within those crates retain deliberately narrow responsibilities:
|
|
||||||
|
|
||||||
- `config` owns validated embedded defaults, restricted runtime diagnostic
|
|
||||||
catalogs, the generated Unicode policy, and input limits;
|
|
||||||
- `input` streams files, directory entries, and archive members as documents;
|
|
||||||
- `detector` compiles configured matchers and path scopes once, then produces
|
|
||||||
explainable findings and a bounded low-care evidence score;
|
|
||||||
- `diagnostic` binds findings to exact named snapshots as typed `miette`
|
|
||||||
diagnostics;
|
|
||||||
- `replacement` produces non-overlapping edits and validates/applies arbitrary
|
|
||||||
edit sets without performing I/O;
|
|
||||||
- `unicode_review` pins the UTS #39 ASCII-skeleton version and Gitea-compatible
|
|
||||||
invisible/control ranges; generated printable-ASCII mappings feed the shared
|
|
||||||
replacement planner while controls remain non-mutating;
|
|
||||||
- `lsp` indexes open snapshots, runs the compiled detector, converts canonical
|
|
||||||
finding and replacement spans to UTF-16 coordinates, and implements
|
|
||||||
versioned quick fixes plus whole-document formatting;
|
|
||||||
- `model` contains presentation-neutral domain values;
|
|
||||||
- `error` exposes typed input failures for library consumers;
|
|
||||||
- `src/bin` contains the detector presentation/exit policy and formatter/LSP
|
|
||||||
adapters.
|
|
||||||
|
|
||||||
All source spans are zero-based UTF-8 byte ranges into the exact analyzed source
|
|
||||||
snapshot. Presentation layers will convert those canonical spans to terminal or
|
|
||||||
UTF-16 LSP coordinates at their boundaries. Archive sources retain both their
|
|
||||||
outer path and inner member identity. Rule applicability uses the member path
|
|
||||||
so language and path filters behave the same inside and outside archives.
|
|
||||||
|
|
||||||
## Public facade
|
|
||||||
|
|
||||||
The root compatibility facade and `john-slop-core` expose the same core entry
|
|
||||||
points and presentation-neutral types:
|
|
||||||
|
|
||||||
- `load_config` resolves the configuration embedded in the current build;
|
|
||||||
- `read_inputs` returns a streaming iterator of per-document results;
|
|
||||||
- `read_stdin` returns one bounded strict-UTF-8 standard-input document;
|
|
||||||
- `analyze_source` returns findings and an evidence score for UTF-8 text;
|
|
||||||
- `plan_replacements` returns safe edits against the same source snapshot;
|
|
||||||
- `apply_replacements` validates boundaries and overlap, then applies edits in
|
|
||||||
descending byte order while borrowing no-op input unchanged.
|
|
||||||
|
|
||||||
`FindingDiagnostic` is the library bridge to `miette`; CLI-specific concise and
|
|
||||||
JSON rendering stays in the detector adapter so it cannot alter analysis.
|
|
||||||
|
|
||||||
`ConfigLoader` applies embedded catalogs, runtime diagnostic catalogs, user
|
|
||||||
base, named user, explicit file, and CLI layers in order. Stable-ID merging and
|
|
||||||
semantic validation happen before a configuration reaches analysis. Keeping
|
|
||||||
these operations behind the facade avoids duplicating policy across the three
|
|
||||||
binaries. Runtime catalogs are declarative: they can add rules using existing
|
|
||||||
matcher implementations but cannot load executable code.
|
|
||||||
|
|
||||||
The formatter adapter owns filesystem mutation. It resolves configuration and
|
|
||||||
preflights every requested document before writing, then uses same-directory
|
|
||||||
temporary files and preserves permissions. Archive mutation remains outside the
|
|
||||||
library and adapter contract.
|
|
||||||
|
|
||||||
The LSP adapter owns no filesystem mutation. `tower-lsp-server` supplies stdio
|
|
||||||
JSON-RPC framing while `LspBackend` tracks authoritative open-document versions
|
|
||||||
behind asynchronous locks. Every publication is checked against the current
|
|
||||||
version after analysis so a replaced snapshot cannot overwrite newer results.
|
|
||||||
Configuration reload swaps only a fully validated model and republishes open
|
|
||||||
documents in deterministic URI order. The detector is compiled once per
|
|
||||||
resolved configuration rather than once per edit. File URI paths drive normal
|
|
||||||
scope applicability; language IDs supply source/prose context for untitled or
|
|
||||||
extensionless buffers. The default publication filter retains only Unicode
|
|
||||||
findings; explicit all-diagnostic and minimum-weight modes expose broader
|
|
||||||
findings, while only findings with core `TextEdit`s become code actions. Document formatting calls the same
|
|
||||||
replacement planner directly and never derives mutations from diagnostics. See
|
|
||||||
[ADR 0003](decisions/0003-lsp-framework-and-protocol.md) for framework and
|
|
||||||
protocol choices.
|
|
||||||
|
|
||||||
See [ADR 0001](decisions/0001-core-analysis-contracts.md) for the foundational
|
|
||||||
contracts and [ADR 0002](decisions/0002-input-policy-and-limits.md) for traversal
|
|
||||||
and archive safety decisions.
|
|
||||||
|
|
@ -1,14 +0,0 @@
|
||||||
# Background reading
|
|
||||||
|
|
||||||
John Slop focuses on observable low-care signals and avoids making unsupported
|
|
||||||
authorship claims. These resources provide context for that product boundary and
|
|
||||||
the kinds of review problems the project aims to help surface:
|
|
||||||
|
|
||||||
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
|
|
||||||
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
|
|
||||||
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
|
|
||||||
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
|
|
||||||
|
|
||||||
Project-specific decisions take precedence over external guidance. See
|
|
||||||
[ADR 0001](decisions/0001-core-analysis-contracts.md) for evidence scoring and
|
|
||||||
authorship semantics.
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
# Performance benchmarks
|
|
||||||
|
|
||||||
Criterion benchmarks cover the scaling boundaries most likely to change
|
|
||||||
architectural decisions:
|
|
||||||
|
|
||||||
- analysis of an approximately 1 MiB clean prose document;
|
|
||||||
- streaming and decoding 250 small files in deterministic path order;
|
|
||||||
- compiling and analyzing with 110 rules (10× the embedded catalog); and
|
|
||||||
- opening, validating, and reading a 100-member stored ZIP archive.
|
|
||||||
|
|
||||||
Run them on an otherwise idle machine:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo bench --locked --bench core
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `-- --quick` for a smoke measurement. Criterion output under `target/` is
|
|
||||||
not a portable performance artifact: record the CPU, OS, Rust version, commit,
|
|
||||||
and exact command when comparing results.
|
|
||||||
|
|
||||||
The Milestone 7 smoke baseline was measured on 2026-09-29 with
|
|
||||||
`cargo bench --locked --bench core -- --quick`, Rust 1.100.0-nightly, Linux
|
|
||||||
7.2.7, and an AMD Ryzen 9 9950X3D:
|
|
||||||
|
|
||||||
| Workload | Criterion estimate |
|
|
||||||
| --- | ---: |
|
|
||||||
| approximately 1 MiB document | 15.686 ms / 65.36 MiB/s |
|
|
||||||
| 250 small files | 286.54 µs / 872.48 Kfiles/s |
|
|
||||||
| build and analyze 110 rules | 9.474 ms |
|
|
||||||
| 100-member stored archive | 116.37 µs / 859.34 Kmembers/s |
|
|
||||||
|
|
||||||
These quick-mode figures are orientation data, not a cross-machine performance
|
|
||||||
guarantee. A release decision should use Criterion's normal sampling mode.
|
|
||||||
|
|
||||||
Milestone 7 retains the ordered single-threaded implementation. The baseline
|
|
||||||
workloads complete in milliseconds on the development machine, and directory
|
|
||||||
and archive results already stream rather than accumulating the entire input
|
|
||||||
tree before analysis. Parallel traversal would add scheduling and output-order
|
|
||||||
coordination without an established user-visible bottleneck. Reconsider it only
|
|
||||||
with a representative workload, a reproducible before/after benchmark, and
|
|
||||||
tests proving byte-for-byte output order remains unchanged.
|
|
||||||
|
|
@ -1,213 +0,0 @@
|
||||||
# Configuration
|
|
||||||
|
|
||||||
John Slop embeds schema-versioned base, heuristic, and Unicode catalogs in every
|
|
||||||
binary. Installed binaries therefore work without a source checkout or external
|
|
||||||
default files.
|
|
||||||
|
|
||||||
## Precedence
|
|
||||||
|
|
||||||
Configuration layers are applied from lowest to highest priority:
|
|
||||||
|
|
||||||
1. embedded `assets/config.toml` and `assets/heuristics.toml`;
|
|
||||||
2. the optional platform user diagnostic catalog `diagnostics.toml`, followed
|
|
||||||
by explicitly selected diagnostic catalogs in declaration order;
|
|
||||||
3. the platform user file `config.toml`;
|
|
||||||
4. a selected user `<config_name>.toml`;
|
|
||||||
5. an explicit `--config <FILE>`;
|
|
||||||
6. configuration flags for the current invocation.
|
|
||||||
|
|
||||||
Diagnostic catalogs load before normal configuration so a base, named, or
|
|
||||||
explicit configuration can tune or disable a catalog rule by stable ID. A
|
|
||||||
catalog accepts only `schema_version` and `[[rules]]`; unrelated settings are a
|
|
||||||
configuration error rather than being silently applied from the wrong file.
|
|
||||||
|
|
||||||
On Linux, the user directory conventionally resolves to
|
|
||||||
`~/.config/john-slop`. John Slop uses operating-system APIs rather than reading
|
|
||||||
`HOME` directly, so Windows and macOS receive their native application config
|
|
||||||
directories. Run this to inspect the exact paths on the current machine:
|
|
||||||
|
|
||||||
The RC3 naming migration changes the former `john_slop` application directory
|
|
||||||
to `john-slop`. Move existing `config.toml`, named configurations, and
|
|
||||||
`diagnostics.toml` into the new directory before upgrading; the old directory
|
|
||||||
is no longer loaded automatically.
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config paths
|
|
||||||
```
|
|
||||||
|
|
||||||
Named configurations accept only ASCII letters, digits, `_`, and `-`. Do not
|
|
||||||
include a directory, dot component, or `.toml` extension. Missing user and named
|
|
||||||
files are optional; a selected explicit file must exist.
|
|
||||||
|
|
||||||
## Generate a project configuration
|
|
||||||
|
|
||||||
Create the embedded, commented base configuration as `./config.toml`:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config init
|
|
||||||
```
|
|
||||||
|
|
||||||
The command refuses to overwrite an existing file. Use
|
|
||||||
`john-slop config init --force` only when replacing that file is intentional.
|
|
||||||
`config generate` is an alias for `config init`. Load a project-local file with
|
|
||||||
`john-slop --config config.toml path/to/input` or inspect it with the command
|
|
||||||
shown below.
|
|
||||||
|
|
||||||
Generate a dedicated runtime diagnostic catalog separately:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config init-diagnostics
|
|
||||||
```
|
|
||||||
|
|
||||||
Place the result at the `user_diagnostics` path printed by `config paths` for
|
|
||||||
automatic loading, or select one or more project/team catalogs explicitly:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop --diagnostics team.toml --diagnostics project.toml article.md
|
|
||||||
john-slop-lsp --diagnostics project.toml
|
|
||||||
```
|
|
||||||
|
|
||||||
Explicit catalogs are applied in command-line order. All catalogs load before
|
|
||||||
`config.toml`, named configuration, and `--config`, allowing those normal
|
|
||||||
layers to override their rules.
|
|
||||||
|
|
||||||
## Schema
|
|
||||||
|
|
||||||
The current `schema_version` is `1`. The complete defaults live in
|
|
||||||
[`config.toml`](../crates/john-slop-core/assets/config.toml),
|
|
||||||
[`heuristics.toml`](../crates/john-slop-core/assets/heuristics.toml), and the
|
|
||||||
generated [`unicode.toml`](../crates/john-slop-core/assets/unicode.toml).
|
|
||||||
Ready-to-copy, commented partial layers live in
|
|
||||||
[`examples/config`](../examples/config/): `config.toml` demonstrates a user-wide
|
|
||||||
base and `editorial.toml` demonstrates a named overlay.
|
|
||||||
|
|
||||||
Replacement and heuristic arrays merge by stable `id`. A partial record changes
|
|
||||||
only the supplied fields:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[diagnostics.categories]
|
|
||||||
# Disable every style diagnostic as one family.
|
|
||||||
style = false
|
|
||||||
|
|
||||||
[[replacements]]
|
|
||||||
id = "em-dash"
|
|
||||||
enabled = false
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "templated-conclusion"
|
|
||||||
# An individual rule switch is independent of the category switch.
|
|
||||||
enabled = false
|
|
||||||
```
|
|
||||||
|
|
||||||
The category keys are `typography`, `style`, `prompt-leakage`, and
|
|
||||||
`source-code`. A diagnostic runs only when its category switch and its
|
|
||||||
individual `[[rules]]` `enabled` field are both `true`. Consequently, enabling
|
|
||||||
a category does not silently re-enable a rule that another layer disabled
|
|
||||||
individually; set that rule's `enabled = true` explicitly when that is intended.
|
|
||||||
Later files override earlier category values using the normal precedence above.
|
|
||||||
|
|
||||||
A new ID must provide every field. Scope arrays replace the prior array as a
|
|
||||||
whole. A rule scope can select `prose` or `source-code`, restrict lowercase
|
|
||||||
extensions, and use `include`/`exclude` path globs. The `skip_generated`,
|
|
||||||
`skip_vendor`, and `skip_minified` switches default to `true`; set one to
|
|
||||||
`false` only when those paths are intentionally in scope. Archive rules match
|
|
||||||
the inner member path. Unknown keys, malformed TOML, duplicate IDs within one
|
|
||||||
layer, invalid regular expressions or globs, unsafe matchers, non-ASCII
|
|
||||||
replacement targets, and invalid thresholds are fatal configuration
|
|
||||||
diagnostics.
|
|
||||||
|
|
||||||
The built-in matcher names are `exact-phrase`, `regex`, `repeated-regex`,
|
|
||||||
`replacement`, and `invisible`. The last two use validated sentinel patterns
|
|
||||||
because their implementations are data-backed rather than free-form strings:
|
|
||||||
`configured-replacements` and `gitea-invisible-9c715c0`. They remain ordinary
|
|
||||||
rule records, so later layers
|
|
||||||
can disable them or adjust severity, weight, occurrence cap, and scope without
|
|
||||||
changing code. Invisible findings never authorize edits; generated confusable
|
|
||||||
mappings follow the bulk policy below.
|
|
||||||
|
|
||||||
## Runtime diagnostic catalog
|
|
||||||
|
|
||||||
`diagnostics.toml` supports new data-driven diagnostics without recompilation.
|
|
||||||
Every new rule must provide the complete rule shape:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
schema_version = 1
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "custom-review-marker"
|
|
||||||
category = "style"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "REVIEW BEFORE PUBLISHING"
|
|
||||||
weight = 8.0
|
|
||||||
message = "a project review marker remains"
|
|
||||||
help = "Resolve or remove the marker before publishing."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
scope = { content = ["prose", "source-code"], extensions = [] }
|
|
||||||
```
|
|
||||||
|
|
||||||
Runtime rules may select `exact-phrase`, `regex`, or `repeated-regex`.
|
|
||||||
`replacement` and `invisible` are code-backed catalog sentinels and cannot
|
|
||||||
define new matching behavior. Categories remain the four stable values listed
|
|
||||||
above. Unknown fields, invalid or empty-match regular expressions, duplicate
|
|
||||||
IDs within one catalog, incomplete new rules, and missing explicit catalogs are
|
|
||||||
fatal. These constraints keep startup deterministic and prevent a configuration
|
|
||||||
file from executing code.
|
|
||||||
|
|
||||||
The generated UTS #39 mappings have one bulk policy:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[unicode]
|
|
||||||
confusables = "review" # replace (default), review, or off
|
|
||||||
```
|
|
||||||
|
|
||||||
`replace` enables diagnostics and edits, `review` retains diagnostics without
|
|
||||||
formatter/LSP edits, and `off` suppresses generated lookalikes. The reviewed
|
|
||||||
punctuation and width-folding replacements and invisible/control diagnostics
|
|
||||||
are unaffected.
|
|
||||||
|
|
||||||
Set `enabled = false` on a known rule or replacement ID to disable it. Because
|
|
||||||
records merge by ID, do not copy the full embedded table merely to change one
|
|
||||||
field; a later partial record preserves every unspecified value.
|
|
||||||
|
|
||||||
Replacement `source` values are non-empty Unicode strings containing at least
|
|
||||||
one non-ASCII byte; `replacement` values are non-empty ASCII strings. Sources
|
|
||||||
must be unique. When enabled sources share a prefix, the longest matching source
|
|
||||||
wins at each byte offset, with stable rule ID ordering as the deterministic
|
|
||||||
tie-break. Edits always refer to the original UTF-8 snapshot.
|
|
||||||
|
|
||||||
Generated confusable IDs use `uts39-u<lowercase-codepoint>`. New fullwidth
|
|
||||||
mappings use `fullwidth-u<lowercase-codepoint>`; mappings shipped previously
|
|
||||||
retain their `uts39-*` or `fullwidth-hyphen-minus` IDs for compatibility, while
|
|
||||||
ideographic space uses `ideographic-space`. Every entry can be disabled or
|
|
||||||
overridden through normal merge-by-ID behavior.
|
|
||||||
Maintain the embedded list by editing
|
|
||||||
`crates/john-slop-core/examples/generate_unicode_catalog.rs` and running:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run -p john-slop-core --example generate_unicode_catalog \
|
|
||||||
> crates/john-slop-core/assets/unicode.toml
|
|
||||||
```
|
|
||||||
|
|
||||||
The exhaustive catalog test walks every Unicode scalar and rejects drift from
|
|
||||||
the pinned Unicode 16.0 tables.
|
|
||||||
|
|
||||||
Each heuristic also configures `severity` as `advice`, `warning`, or `error`.
|
|
||||||
Severity controls CLI presentation, LSP severity, and summary counts; the
|
|
||||||
evidence score still comes from the independently configured weight and
|
|
||||||
aggregation formula. In the LSP, advice is information while warnings and errors
|
|
||||||
retain their corresponding levels.
|
|
||||||
|
|
||||||
## Inspection
|
|
||||||
|
|
||||||
Print the fully layered and validated model as deterministic TOML:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config show
|
|
||||||
john-slop config show --config project.toml
|
|
||||||
john-slop config show --config-name strict
|
|
||||||
```
|
|
||||||
|
|
||||||
The output contains no secrets in schema version 1. If secret-bearing fields are
|
|
||||||
introduced later, inspection output must redact them by default.
|
|
||||||
|
|
@ -1,55 +0,0 @@
|
||||||
# ADR 0001: Core analysis contracts
|
|
||||||
|
|
||||||
- Status: accepted
|
|
||||||
- Date: 2026-09-29
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
John Slop needs stable semantics before its proof-of-concept modules are split
|
|
||||||
into a library, CLI, formatter, and language server. In particular, consumers
|
|
||||||
need to know what the aggregate evidence score means, how source locations are represented,
|
|
||||||
whether formatting is destructive by default, and whether archive members can
|
|
||||||
be rewritten.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
### The low-care evidence score represents evidence, not authorship
|
|
||||||
|
|
||||||
The detector reports the bounded strength of configured, observable low-care
|
|
||||||
signals. It does not report the probability that content is AI-generated and
|
|
||||||
must not present a score as an authorship determination. Every score must be
|
|
||||||
decomposable into source-labelled findings from stable rule IDs.
|
|
||||||
|
|
||||||
### UTF-8 byte spans are canonical
|
|
||||||
|
|
||||||
Internal source spans are zero-based, half-open UTF-8 byte ranges into the exact
|
|
||||||
source snapshot analyzed. This matches Rust string and regex offsets and avoids
|
|
||||||
lossy conversions inside the detection engine. Presentation adapters validate
|
|
||||||
boundaries and convert to their required representation: `miette::SourceSpan`
|
|
||||||
for terminal diagnostics and UTF-16 line/character positions for LSP.
|
|
||||||
|
|
||||||
### Formatter writes are opt-in
|
|
||||||
|
|
||||||
The formatter's default file behavior is read-only. It may print transformed
|
|
||||||
text or a diff, while an explicit `--write` is required to mutate files.
|
|
||||||
`--check` only reports whether an edit would be made. A failed analysis,
|
|
||||||
configuration load, or edit validation must leave the original file untouched.
|
|
||||||
|
|
||||||
### Archive mutation is deferred
|
|
||||||
|
|
||||||
The first formatter release may inspect text members in supported ZIP-based
|
|
||||||
containers, but it will not rewrite archives, DOCX, XLSX, or PPTX files. In-place
|
|
||||||
archive editing has additional integrity, metadata-preservation, atomicity, and
|
|
||||||
decompression-safety concerns. It requires a separate decision record and test
|
|
||||||
plan before implementation.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
- User-facing terminology must describe evidence and request review rather than
|
|
||||||
accuse an author.
|
|
||||||
- The domain model stores byte ranges, while the LSP owns a tested UTF-16 mapping
|
|
||||||
layer tied to document versions.
|
|
||||||
- Replacement planning remains a pure operation; filesystem mutation lives in
|
|
||||||
an explicit formatter adapter.
|
|
||||||
- Archive inputs can produce findings but an attempted archive write produces a
|
|
||||||
clear unsupported-operation diagnostic.
|
|
||||||
|
|
@ -1,65 +0,0 @@
|
||||||
# ADR 0002: Input policy and resource limits
|
|
||||||
|
|
||||||
- Status: accepted
|
|
||||||
- Date: 2026-09-29
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
The detector accepts individual files, directory trees, and ZIP-based
|
|
||||||
containers. Unbounded traversal or decompression can exhaust memory, CPU, file
|
|
||||||
descriptors, or disk. Extension-only detection can also misclassify binary files
|
|
||||||
as text. The input layer needs deterministic defaults before it is redesigned as
|
|
||||||
a streaming API.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
Limits apply per invocation and can be lowered by configuration or CLI flags.
|
|
||||||
Raising them requires an explicit option; there is no unlimited sentinel.
|
|
||||||
|
|
||||||
| Resource | Default limit |
|
|
||||||
| --- | ---: |
|
|
||||||
| Plain file or archive member | 8 MiB |
|
|
||||||
| Input archive file | 64 MiB |
|
|
||||||
| Total expanded archive bytes | 256 MiB |
|
|
||||||
| Archive entries inspected | 10,000 |
|
|
||||||
| Expanded/compressed ratio per member | 100:1 |
|
|
||||||
| Directory traversal depth | 64 levels |
|
|
||||||
| Nested archive traversal | disabled (depth 0) |
|
|
||||||
|
|
||||||
An exact file passed by the user is considered regardless of extension. During
|
|
||||||
directory traversal, configurable extension allowlists select likely prose,
|
|
||||||
markup, configuration, and source-code files. Content still has to decode as
|
|
||||||
strict UTF-8 and must not contain a NUL byte. Unsupported/binary content produces
|
|
||||||
a skipped-input result rather than a clean analysis or a lossy conversion.
|
|
||||||
|
|
||||||
ZIP-based containers are recognized from both a supported extension (`zip`,
|
|
||||||
`docx`, `xlsx`, or `pptx`) and ZIP magic bytes. An extension/magic mismatch is a
|
|
||||||
typed input error. Members are streamed and counted before allocation. Directory
|
|
||||||
members and encrypted members are skipped with a reason. Absolute paths,
|
|
||||||
platform prefixes, and any path containing a parent (`..`) component are
|
|
||||||
rejected. Archive member names are identities only; members are never extracted
|
|
||||||
to the filesystem by the analysis path.
|
|
||||||
|
|
||||||
Directory walking does not follow symbolic links. Hidden paths are skipped by
|
|
||||||
default. The default ignore set includes `.git`, `.hg`, `.svn`, `target`,
|
|
||||||
`node_modules`, `vendor`, common generated directories, source maps, and minified
|
|
||||||
assets. Repository ignore files are respected when the eventual walker supports
|
|
||||||
them. An explicitly named file can bypass path/extension ignores, but cannot
|
|
||||||
bypass content or resource-safety limits.
|
|
||||||
|
|
||||||
One unreadable, invalid, or oversized path yields a per-document input error and
|
|
||||||
does not discard successful documents from the same directory or archive.
|
|
||||||
Results remain ordered by normalized source identity even if reading is later
|
|
||||||
parallelized.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
- The input API must stream per-document results and track cumulative archive
|
|
||||||
counters.
|
|
||||||
- `Document` needs distinct outer-path and optional archive-member identities.
|
|
||||||
- Size checks must use checked arithmetic and occur before allocating from
|
|
||||||
metadata supplied by an archive.
|
|
||||||
- Tests need boundary cases immediately below, at, and above each configurable
|
|
||||||
limit, plus malicious archive names and misleading extensions.
|
|
||||||
- The README and CLI help must distinguish skipped inputs, operational errors,
|
|
||||||
and clean analyzed documents.
|
|
||||||
|
|
@ -1,101 +0,0 @@
|
||||||
# ADR 0003: LSP framework and protocol contract
|
|
||||||
|
|
||||||
- Status: accepted, amended by Milestone 19
|
|
||||||
- Date: 2026-09-29
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
John Slop needs a small stdio language server that reuses the validated
|
|
||||||
replacement configuration and exact `TextEdit`s from the library. Implementing
|
|
||||||
JSON-RPC framing and the Language Server Protocol state machine locally would
|
|
||||||
add protocol risk without improving detection. The original `tower-lsp` crate's
|
|
||||||
latest release is 0.20.0, while its maintained community continuation publishes
|
|
||||||
the same Tower-oriented API as `tower-lsp-server`.
|
|
||||||
|
|
||||||
The dependency review compared the crates' published metadata, documentation,
|
|
||||||
runtime features, and minimum Rust versions. `tower-lsp-server` 0.23.0 is the
|
|
||||||
latest stable release; 0.24.0-rc.1 is a prerelease. Version 0.23.0 requires Rust
|
|
||||||
1.85, below this project's Rust 1.98.1 baseline, and provides the Tokio stdio
|
|
||||||
transport, generated LSP types, client notifications, and in-process service
|
|
||||||
interface needed by the milestone. Its MIT OR Apache-2.0 licensing is compatible
|
|
||||||
with this project.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
Use `tower-lsp-server` 0.23 and its Tokio runtime feature. Use Tokio directly
|
|
||||||
only for the stdio runtime and synchronized in-memory document state. Do not
|
|
||||||
enable proposed LSP extensions or an alternate URI implementation.
|
|
||||||
|
|
||||||
The initial server targets stable LSP 3.17 behavior with these assumptions:
|
|
||||||
|
|
||||||
- transport is stdio only, with protocol bytes reserved for stdout;
|
|
||||||
- document positions are zero-based UTF-16 code-unit offsets;
|
|
||||||
- clients send full-document synchronization, simplifying authoritative version
|
|
||||||
handling while replacement behavior is established;
|
|
||||||
- the server owns snapshots of open documents only and never writes their URIs;
|
|
||||||
- diagnostics and quick fixes cover configured ASCII replacements, not heuristic
|
|
||||||
authorship or aggregate heuristic findings;
|
|
||||||
- quick fixes use versioned workspace edits and are rejected when the request
|
|
||||||
refers to a stale or unknown document version;
|
|
||||||
- `workspace/didChangeConfiguration` reloads the same layered configuration used
|
|
||||||
by the CLI and formatter, then republishes diagnostics for open snapshots;
|
|
||||||
- malformed messages and invalid URIs are protocol/input errors, never panics.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
- The replacement engine remains the single source of edit ranges and text.
|
|
||||||
- A tested line index converts canonical UTF-8 byte offsets to LSP UTF-16
|
|
||||||
positions at the adapter boundary.
|
|
||||||
- Protocol tests can drive `LspService` in process without launching a child
|
|
||||||
process, while a small stdio smoke test protects stream purity.
|
|
||||||
- Incremental synchronization may be added later, but must preserve version and
|
|
||||||
stale-result guarantees before it is advertised.
|
|
||||||
- Upgrading to the 0.24 line waits for a stable release and a separate API and
|
|
||||||
minimum-Rust review.
|
|
||||||
|
|
||||||
## Amendment: full heuristic diagnostics
|
|
||||||
|
|
||||||
On 2026-09-29, Milestone 19 superseded the initial replacement-only diagnostic
|
|
||||||
limit after the core detector and configuration contracts had stabilized. The
|
|
||||||
server now compiles the same detector as the CLI once per resolved configuration
|
|
||||||
and publishes every applicable finding for each open snapshot.
|
|
||||||
|
|
||||||
This amendment does not broaden mutation behavior:
|
|
||||||
|
|
||||||
- safe configured replacements remain the only findings with quick fixes;
|
|
||||||
- style, prompt-leakage, and source-code review findings are diagnostic-only;
|
|
||||||
- stable heuristic IDs replace replacement-entry IDs as published diagnostic
|
|
||||||
codes, avoiding duplicate typography diagnostics;
|
|
||||||
- configured advice, warning, and error severities map to LSP information,
|
|
||||||
warning, and error respectively;
|
|
||||||
- file URI paths retain extension and path-scope behavior, while editor language
|
|
||||||
IDs determine prose/source applicability for untitled or extensionless
|
|
||||||
buffers; and
|
|
||||||
- configuration reload atomically swaps the validated configuration and compiled
|
|
||||||
detector before open documents are republished.
|
|
||||||
|
|
||||||
The original full-snapshot synchronization, UTF-16 conversion, version checks,
|
|
||||||
stdio purity, no-write boundary, and stale-action guarantees remain unchanged.
|
|
||||||
|
|
||||||
## Amendment: quiet diagnostic publication
|
|
||||||
|
|
||||||
On 2026-09-30, Milestone 26 changed the presentation default without changing
|
|
||||||
the shared detector. The LSP now publishes only Unicode findings unless it is
|
|
||||||
started with `--all-diagnostics` or `--minimum-diagnostic-weight WEIGHT`.
|
|
||||||
Minimum-weight mode always retains Unicode findings and filters only the other
|
|
||||||
categories. This makes exact, fixable problems the quiet editor default while
|
|
||||||
keeping the full catalog easy to exercise in tests. Filtering occurs after
|
|
||||||
analysis, so CLI results, evidence scores, formatting, configuration reload,
|
|
||||||
and the core finding contract remain unchanged.
|
|
||||||
|
|
||||||
## Amendment: document formatting parity
|
|
||||||
|
|
||||||
Milestone 21 adds the standard `textDocument/formatting` capability. The server
|
|
||||||
plans edits from its current open-document snapshot with the same resolved
|
|
||||||
configuration and core replacement planner used by `john-slop-fmt`, then
|
|
||||||
converts only the edit coordinates from UTF-8 bytes to LSP UTF-16 positions.
|
|
||||||
Formatting remains limited to configured safe ASCII replacements; the server
|
|
||||||
does not write files, infer formatting from diagnostics, or apply broader
|
|
||||||
Unicode review findings. Protocol tests compare the result of an actual
|
|
||||||
formatting request with the standalone formatter contract, including Unicode,
|
|
||||||
CRLF, no-op, and configuration-reload cases.
|
|
||||||
|
|
@ -1,55 +0,0 @@
|
||||||
# ADR 0004: Supported Cargo workspace
|
|
||||||
|
|
||||||
- Status: accepted
|
|
||||||
- Date: 2026-09-29
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
The crate-boundary experiment extracted reusable analysis/configuration/input
|
|
||||||
code into `john-slop-core` and LSP protocol state into `john-slop-lsp`, leaving
|
|
||||||
the root `john-slop` package as a compatibility facade and binary distribution.
|
|
||||||
Cold core and no-LSP builds used roughly 35–55% less CPU and disk, while LSP and
|
|
||||||
whole-project builds were effectively unchanged. The experiment initially
|
|
||||||
recommended waiting because three publishable packages require synchronized
|
|
||||||
versions, ordered publication, workspace-aware CI, and clear asset ownership.
|
|
||||||
|
|
||||||
The project has now chosen to offer the reusable package boundary and accepts
|
|
||||||
that release overhead. Repeating package metadata and dependency versions in
|
|
||||||
three manifests would make that overhead unnecessarily error-prone.
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
Adopt the three-package layout as the supported architecture:
|
|
||||||
|
|
||||||
- `john-slop-core` owns embedded catalogs and reusable synchronous domain logic;
|
|
||||||
- `john-slop-lsp` depends on core and owns protocol state/conversion;
|
|
||||||
- root `john-slop` remains the compatibility facade and owns all three binaries.
|
|
||||||
|
|
||||||
Keep the root manifest as both a package and the workspace root. Moving its
|
|
||||||
sources under another directory solely to create a virtual manifest adds path,
|
|
||||||
fixture, and packaging churn without changing the dependency graph.
|
|
||||||
|
|
||||||
Centralize shared version, edition, MSRV, license, repository, dependency
|
|
||||||
requirements, and lint policy in the root workspace tables. Every member
|
|
||||||
inherits them. Dependency feature activation and optionality remain in member
|
|
||||||
manifests: workspace inheritance must not cause the core package to compile
|
|
||||||
Tokio/LSP dependencies or make the facade's LSP stack unconditional.
|
|
||||||
|
|
||||||
Keep all three packages as default members so existing root quality commands
|
|
||||||
cover the whole workspace. Enforce membership, synchronized resolved versions,
|
|
||||||
inheritance, and feature isolation with an integration test. Assemble packages
|
|
||||||
with Cargo's workspace packaging command and publish in dependency order: core,
|
|
||||||
LSP, then facade.
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
- Core/no-LSP consumers retain the measured lean dependency graph.
|
|
||||||
- Existing `john_slop` API paths and binary names remain compatible by default.
|
|
||||||
- Shared requirements have one update point, while member feature choices stay
|
|
||||||
reviewable at their point of use.
|
|
||||||
- Full builds gain no material speedup from the boundary.
|
|
||||||
- A release coordinates three matching package versions. Before matching
|
|
||||||
dependency versions exist in the registry, package assembly uses
|
|
||||||
`--no-verify`; registry-form verification follows publication.
|
|
||||||
- Embedded catalog changes belong to `john-slop-core`, and workspace coverage,
|
|
||||||
CI, dependency review, and packaging remain permanent maintenance costs.
|
|
||||||
|
|
@ -1,62 +0,0 @@
|
||||||
# Dependency and binary review
|
|
||||||
|
|
||||||
Milestone 7 reviewed the locked graph on 2026-09-29 with Rust 1.100.0-nightly
|
|
||||||
on x86-64 Linux.
|
|
||||||
|
|
||||||
## Features and scope
|
|
||||||
|
|
||||||
The normal dependency graph contains 135 distinct package names. Direct crates
|
|
||||||
serve narrow boundaries: Clap parses adapters, Figment/Serde/TOML load typed
|
|
||||||
configuration, Regex/Globset compile rules and scopes, Miette renders
|
|
||||||
diagnostics, Walkdir/Zip acquire inputs, Tokio/tower-lsp-server implement the
|
|
||||||
stdio protocol, Unicode Security supplies the pinned UTS #39 skeleton data, and
|
|
||||||
Tempfile/Similar support atomic writes and diffs.
|
|
||||||
|
|
||||||
Default features are disabled where they would add unrelated behavior. In
|
|
||||||
particular, Zip previously enabled AES, Bzip2, Deflate64, LZMA, PPMd, Zstandard,
|
|
||||||
time, and multiple encoders. John Slop reads unencrypted office/ZIP inputs and
|
|
||||||
now enables only the zlib-rs Deflate path required by common containers. A
|
|
||||||
Deflate archive regression test pins that support. This removed 394 lockfile
|
|
||||||
lines. Regex retains Unicode and performance defaults because configured rules
|
|
||||||
use Unicode text and case matching; Miette's graphical stack remains isolated
|
|
||||||
behind `fancy-diagnostics`.
|
|
||||||
|
|
||||||
All publishable packages inherit dependency requirements from the root
|
|
||||||
`[workspace.dependencies]` table. This centralizes versions and baseline feature
|
|
||||||
policy without globally enabling member-specific features: `john-slop-core`
|
|
||||||
does not select Clap, Tokio, or the LSP stack; `john-slop-lsp` selects only
|
|
||||||
Tokio synchronization; and the root adapter enables runtime/stdio support only
|
|
||||||
behind its optional `lsp` feature. `tests/workspace_contract.rs` pins this
|
|
||||||
boundary and the synchronized package metadata.
|
|
||||||
|
|
||||||
## Security, licenses, and unsafe code
|
|
||||||
|
|
||||||
`cargo audit --deny warnings` scanned 193 locked packages against 1,277 RustSec
|
|
||||||
advisories without a finding. `cargo deny --locked check` passed advisories,
|
|
||||||
licenses, and sources; it reports seven transitive duplicate-version families
|
|
||||||
as warnings for ongoing review. [`deny.toml`](../deny.toml) contains the reviewed
|
|
||||||
license allow-list and rejects wildcard versions, yanked packages, and unknown
|
|
||||||
registries or Git sources.
|
|
||||||
|
|
||||||
A workspace lint forbids first-party Rust `unsafe` code, and the source scan
|
|
||||||
found no existing blocks. Dependencies may use
|
|
||||||
unsafe code internally, notably for compression, operating-system integration,
|
|
||||||
and async/runtime primitives; this review does not claim otherwise. Lockfile,
|
|
||||||
advisory, source, and license checks remain CI gates because this result changes
|
|
||||||
when dependencies change.
|
|
||||||
|
|
||||||
## Release size baseline
|
|
||||||
|
|
||||||
Before release-profile stripping, the all-feature Linux binaries measured 6.21,
|
|
||||||
5.84, and 9.38 MB. The finalized thin-LTO, single-codegen-unit,
|
|
||||||
abort-on-panic, symbol-stripped profile measures:
|
|
||||||
|
|
||||||
| Binary | Bytes |
|
|
||||||
| --- | ---: |
|
|
||||||
| `john-slop` | 3,542,544 |
|
|
||||||
| `john-slop-fmt` | 3,322,944 |
|
|
||||||
| `john-slop-lsp` | 4,756,688 |
|
|
||||||
|
|
||||||
These are build-specific engineering measurements, not promised download sizes.
|
|
||||||
The release procedure records final sizes and emits checksums beside any
|
|
||||||
distributed binaries.
|
|
||||||
|
|
@ -1,119 +0,0 @@
|
||||||
# Development
|
|
||||||
|
|
||||||
The project uses Rust edition 2024 and requires Rust 1.98.1 or newer. The
|
|
||||||
committed lockfile makes application builds reproducible. Linux is the supported
|
|
||||||
and exercised platform; other platforms are not declared supported until CI
|
|
||||||
covers them. Jenkins checks Linux on both Rust 1.98.1 (the MSRV) and current
|
|
||||||
stable, with all features and without default features.
|
|
||||||
|
|
||||||
## Quality gates
|
|
||||||
|
|
||||||
Run the same core checks required by the contributor guide:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo fmt --all -- --check
|
|
||||||
cargo clippy --locked --all-targets --all-features -- -D warnings
|
|
||||||
cargo test --locked --all-targets --all-features
|
|
||||||
cargo test --locked --all-targets --no-default-features
|
|
||||||
cargo doc --locked --no-deps --all-features
|
|
||||||
cargo audit --deny warnings
|
|
||||||
cargo deny --locked check
|
|
||||||
```
|
|
||||||
|
|
||||||
The root package is also the workspace root, and all three packages are default
|
|
||||||
members. These unqualified commands therefore cover the compatibility/binary
|
|
||||||
package, `john-slop-core`, and `john-slop-lsp`. Use `-p john-slop-core` for the
|
|
||||||
isolated reusable dependency graph or `-p john-slop-lsp` for protocol-only work.
|
|
||||||
The workspace contract test validates membership, version synchronization,
|
|
||||||
manifest inheritance, and feature isolation. Before a release, assemble every
|
|
||||||
publishable crate with `scripts/package-workspace.sh --no-verify`; see the
|
|
||||||
[release procedure](releasing.md) for the registry verification boundary.
|
|
||||||
|
|
||||||
To smoke-test installation from the current source checkout, identify the local
|
|
||||||
package explicitly (recent Cargo versions no longer infer it from an omitted
|
|
||||||
package argument):
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo install --path . --locked --all-features
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `--root <temporary-directory>` when verifying this command without changing
|
|
||||||
the normal Cargo installation root.
|
|
||||||
|
|
||||||
The default `fancy-diagnostics` feature enables the detector's colored Miette
|
|
||||||
renderer and default output mode. The default `lsp` feature exposes the
|
|
||||||
compatibility facade's protocol module and builds the LSP binary. The
|
|
||||||
no-default-features test run exercises the core, detector, and formatter without
|
|
||||||
the async protocol dependency graph and with the plain diagnostic fallback. Focused
|
|
||||||
unit and integration tests cover domain invariants, input failures, archives,
|
|
||||||
CLI exit behavior, and feature configurations.
|
|
||||||
|
|
||||||
CI runs the executable fixture manifest and Insta CLI snapshots explicitly in
|
|
||||||
addition to the full suite. `cargo-audit` rejects RustSec advisories and yanked
|
|
||||||
packages. [`deny.toml`](../deny.toml) rejects unknown dependency sources,
|
|
||||||
wildcard versions, and licenses outside the reviewed allow-list; duplicate
|
|
||||||
versions are reported for review while upstream compatibility is assessed.
|
|
||||||
|
|
||||||
Insta snapshots are reserved for stable user-facing text such as CLI help and
|
|
||||||
usage errors. Structured domain values use direct assertions so failures remain
|
|
||||||
precise and easy to review. Accept snapshot changes only after inspecting the
|
|
||||||
generated diff.
|
|
||||||
|
|
||||||
## Coverage and badges
|
|
||||||
|
|
||||||
Run the same line-coverage command used by Jenkins:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo tarpaulin --config tarpaulin.toml --out Json --output-dir coverage
|
|
||||||
```
|
|
||||||
|
|
||||||
The configuration enforces a 90% floor. The current suite measured 92.72%
|
|
||||||
(1,604/1,730 lines) on 2026-09-30. The remaining lines are dominated by
|
|
||||||
platform/I/O failure injection, malformed archive internals, coordinate
|
|
||||||
overflow, LSP logging/reload failures, and subprocess attribution. Do not add
|
|
||||||
brittle tests or production seams solely to make Tarpaulin count those paths.
|
|
||||||
|
|
||||||
`Cargo.toml` is the authoritative version source. Jenkins temporarily renames
|
|
||||||
the Tarpaulin JSON output to `coverage/summary.json`, extracts and validates its
|
|
||||||
overall percentage, renders the badges locally, and then removes the summary.
|
|
||||||
It archives only `coverage/version.svg`, `coverage/test.svg`, and
|
|
||||||
`coverage/coverage.svg`, with fingerprinting. Successful `main` builds publish
|
|
||||||
those files atomically under
|
|
||||||
`/mnt/assets/coverage/idle-org/john-slop/main/`, which serves these mutable URLs:
|
|
||||||
|
|
||||||
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/version.svg`
|
|
||||||
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/test.svg`
|
|
||||||
- `https://assets.dandaedre.com/coverage/idle-org/john-slop/main/coverage.svg`
|
|
||||||
|
|
||||||
The asset server is deployment-owned: it must mount assets read-only, disable
|
|
||||||
directory listings, serve SVG with a restrictive content security policy, and
|
|
||||||
revalidate these URLs or apply a short cache lifetime.
|
|
||||||
|
|
||||||
## Test scope
|
|
||||||
|
|
||||||
Add behavior tests at the lowest useful layer and an integration test when a
|
|
||||||
change crosses a binary boundary. Important edge cases include malformed input,
|
|
||||||
invalid UTF-8, empty text, Unicode byte spans, deterministic ordering, archive
|
|
||||||
limits, and near misses for detection rules. The detailed expectations and
|
|
||||||
task-level commit discipline are in [AGENTS.md](../AGENTS.md).
|
|
||||||
|
|
||||||
Each completed milestone receives an immutable annotated `v0.1.N` Git tag,
|
|
||||||
where `N` is the milestone number, after its acceptance criteria and quality
|
|
||||||
gates pass. Its completion commit must also use package version `0.1.N` in the
|
|
||||||
workspace manifest, internal registry requirements, lockfile, changelog, binary
|
|
||||||
version output. Jenkins derives the current version badge from that metadata.
|
|
||||||
|
|
||||||
## Fuzzing
|
|
||||||
|
|
||||||
The standalone [`fuzz`](../fuzz/) package contains five `cargo-fuzz` targets and
|
|
||||||
reviewed seeds for configuration merging, rule validation/compilation, UTF-8
|
|
||||||
span conversion, replacement invariants, and hostile archive bytes. See its
|
|
||||||
README for invocation. Fuzz artifacts remain ignored until a failure has been
|
|
||||||
minimized and promoted to a normal regression test.
|
|
||||||
|
|
||||||
## Benchmarks
|
|
||||||
|
|
||||||
Run `cargo bench --locked --bench core` for the large-document, many-file,
|
|
||||||
rule-scaling, and archive-limit workloads described in
|
|
||||||
[Performance benchmarks](benchmarks.md). Benchmark reports are local artifacts;
|
|
||||||
only methodology and contextualized measurements belong in version control.
|
|
||||||
141
docs/editors.md
141
docs/editors.md
|
|
@ -1,141 +0,0 @@
|
||||||
# Editor setup
|
|
||||||
|
|
||||||
Both `john-slop-fmt` and `john-slop-lsp` are operational. The LSP publishes only
|
|
||||||
configured Unicode replacement and control findings by default, keeping noisy
|
|
||||||
or error-prone heuristics out of the normal editor loop. Safe ASCII-replacement
|
|
||||||
findings additionally provide versioned quick fixes, and the server implements
|
|
||||||
whole-document formatting with the exact same replacement planner as
|
|
||||||
`john-slop-fmt`.
|
|
||||||
|
|
||||||
Install all three binaries somewhere on `PATH`, then confirm the editor-facing
|
|
||||||
help is available:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop-fmt --help
|
|
||||||
john-slop-lsp --help
|
|
||||||
```
|
|
||||||
|
|
||||||
## Helix (`hx` / `helix-term`)
|
|
||||||
|
|
||||||
Helix configures formatters and language servers in `languages.toml`. Put this
|
|
||||||
in `~/.config/helix/languages.toml` on Linux, or in `.helix/languages.toml` for
|
|
||||||
project-local configuration:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[language-server.john-slop]
|
|
||||||
command = "john-slop-lsp"
|
|
||||||
|
|
||||||
[[language]]
|
|
||||||
name = "john-slop-text"
|
|
||||||
language-id = "plaintext"
|
|
||||||
scope = "text.john-slop"
|
|
||||||
file-types = ["txt"]
|
|
||||||
formatter = { command = "john-slop-fmt" }
|
|
||||||
language-servers = ["john-slop"]
|
|
||||||
auto-format = false
|
|
||||||
```
|
|
||||||
|
|
||||||
This complete custom language entry makes the repository's `example.txt`
|
|
||||||
fixtures useful without depending on a built-in plain-text grammar. It selects
|
|
||||||
the external `john-slop-fmt` adapter. To exercise formatting through the LSP
|
|
||||||
instead, omit the `formatter = ...` line; `:format` then requests
|
|
||||||
`textDocument/formatting` from `john-slop-lsp`. Both routes resolve the same
|
|
||||||
configuration and produce the same text. Set `auto-format = true` only if
|
|
||||||
save-time replacement is wanted.
|
|
||||||
|
|
||||||
For an existing language, merge John Slop into that language's entry rather
|
|
||||||
than replacing its useful defaults. For example, Rust should retain
|
|
||||||
`rust-analyzer`:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
[[language]]
|
|
||||||
name = "rust"
|
|
||||||
formatter = { command = "john-slop-fmt" }
|
|
||||||
language-servers = ["rust-analyzer", "john-slop"]
|
|
||||||
```
|
|
||||||
|
|
||||||
When several language servers advertise formatting, retain an explicit Helix
|
|
||||||
`formatter` or select the intended server interactively; John Slop must not
|
|
||||||
silently replace language-specific formatting from `rust-analyzer` or another
|
|
||||||
primary server.
|
|
||||||
|
|
||||||
Run `hx --health rust` (or the selected language name) to confirm both binaries
|
|
||||||
are visible. Helix documents that a formatter reads the buffer from stdin and
|
|
||||||
writes it to stdout, while an LSP command must be on `PATH`; those are the
|
|
||||||
contracts the two John Slop milestones target. See the official
|
|
||||||
[Helix language configuration reference](https://docs.helix-editor.com/languages.html).
|
|
||||||
|
|
||||||
## Vim
|
|
||||||
|
|
||||||
Vim's native `formatprg` option is enough for formatting. Add this to
|
|
||||||
`~/.vimrc`:
|
|
||||||
|
|
||||||
```vim
|
|
||||||
augroup john-slop-format
|
|
||||||
autocmd!
|
|
||||||
autocmd BufRead,BufNewFile *.txt setfiletype text
|
|
||||||
autocmd FileType text,markdown,rust setlocal formatprg=john-slop-fmt
|
|
||||||
augroup END
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `gggqG` to filter the whole buffer through the formatter, or select a range
|
|
||||||
and press `gq`. Vim requires `formatprg` programs to read stdin and write stdout;
|
|
||||||
see the official [`formatprg` and `gq` help](https://vimhelp.org/options.txt.html#%27formatprg%27).
|
|
||||||
|
|
||||||
Classic Vim needs an LSP client plugin. With
|
|
||||||
[`prabirshrestha/vim-lsp`](https://github.com/prabirshrestha/vim-lsp), register
|
|
||||||
John Slop in `~/.vimrc`:
|
|
||||||
|
|
||||||
```vim
|
|
||||||
if executable('john-slop-lsp')
|
|
||||||
au User lsp_setup call lsp#register_server({
|
|
||||||
\ 'name': 'john-slop',
|
|
||||||
\ 'cmd': {server_info->['john-slop-lsp']},
|
|
||||||
\ 'allowlist': ['text', 'markdown', 'rust'],
|
|
||||||
\ })
|
|
||||||
endif
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the plugin's `:LspStatus`, `:LspNextDiagnostic`, and `:LspCodeAction`
|
|
||||||
commands to inspect server state, move through heuristic diagnostics, and apply
|
|
||||||
available ASCII quick fixes. The registration shape and `allowlist` follow the
|
|
||||||
plugin's current upstream documentation.
|
|
||||||
|
|
||||||
Clients that expose LSP document formatting can request it directly from John
|
|
||||||
Slop instead of configuring `formatprg`; both paths use the same replacement
|
|
||||||
catalog. Keep `formatprg=john-slop-fmt` when the Vim client cannot select one
|
|
||||||
formatter among multiple attached language servers.
|
|
||||||
|
|
||||||
The server uses the same configuration layers as the other binaries. Add
|
|
||||||
`args = ["--config-name", "editorial"]` to the Helix language-server table, or
|
|
||||||
append those arguments to the Vim `cmd` list, when an editor-specific named
|
|
||||||
configuration is wanted. Use `args = ["--diagnostics", "diagnostics.toml"]` (and
|
|
||||||
the equivalent Vim command argument) for a project runtime diagnostic catalog.
|
|
||||||
Add `--all-diagnostics` to exercise every configured rule during testing, or
|
|
||||||
`--minimum-diagnostic-weight 10` to retain Unicode findings and opt into only
|
|
||||||
non-Unicode findings with weight 10 or greater. This visibility filter does not
|
|
||||||
change detector scoring or formatting.
|
|
||||||
`--log-json` is available for troubleshooting and writes lifecycle metadata to
|
|
||||||
stderr without document contents.
|
|
||||||
|
|
||||||
## Try the shared fixtures
|
|
||||||
|
|
||||||
Open `examples/fixtures/typography/example.txt` in either editor. Its expected
|
|
||||||
formatter result is the adjacent `formatted.txt`; the shared manifest records
|
|
||||||
three replacements and one typography rule. Compare detector and formatter
|
|
||||||
output with:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run --bin john-slop -- --output concise \
|
|
||||||
examples/fixtures/typography/example.txt
|
|
||||||
cargo run --bin john-slop-fmt -- \
|
|
||||||
examples/fixtures/typography/example.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
After starting the server with `--all-diagnostics`, open
|
|
||||||
`examples/fixtures/style/example.txt` to see `repeated-transition` and other
|
|
||||||
style findings, or `examples/fixtures/source/example.rs` to see source-only
|
|
||||||
placeholder and narration findings alongside typography. Advice maps to LSP
|
|
||||||
information, warnings to warnings, and configured errors to errors. Every
|
|
||||||
diagnostic carries its stable heuristic rule ID as the code and includes the
|
|
||||||
configured remediation text.
|
|
||||||
|
|
@ -1,83 +0,0 @@
|
||||||
# Evaluation corpus
|
|
||||||
|
|
||||||
Milestone 3 ships a small, reviewed characterization corpus under
|
|
||||||
`tests/fixtures/evaluation`. Its text was authored for this repository and is
|
|
||||||
covered by the MIT license. Labels name observable configured signals, never a
|
|
||||||
presumed author or generation method.
|
|
||||||
|
|
||||||
With the embedded schema-version 1 catalog, the corpus currently reports:
|
|
||||||
|
|
||||||
| Rule | True positive | False positive | False negative | Precision | Recall |
|
|
||||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
||||||
| assistant-self-identification | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| assistant-capability-disclaimer | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| assistant-answer-preface | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| knowledge-cutoff-disclaimer | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| prompt-response-scaffolding | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| assistant-follow-up-offer | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| assistant-revision-preface | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| templated-conclusion | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| templated-filler | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| repeated-transition | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| editorial-insertion-placeholder | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| stock-helpful-closing | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| excess-vocabulary-cluster | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| source-placeholder-todo | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| source-tutorial-narration | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| source-omitted-code-placeholder | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| source-markdown-fence | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
| unicode-ascii-replacement | 2 | 0 | 0 | 100% | 100% |
|
|
||||||
| unicode-invisible-control | 1 | 0 | 0 | 100% | 100% |
|
|
||||||
|
|
||||||
Aggregate labeled-rule counts are 20 true positives, 0 false positives, and 0
|
|
||||||
false negatives. The clean-document false-positive rate is 0/5 (0%). The test
|
|
||||||
suite recomputes these values from the manifest and fails on drift.
|
|
||||||
|
|
||||||
The same manifest pins evidence scores for regression characterization:
|
|
||||||
|
|
||||||
| Corpus case | Score |
|
|
||||||
| --- | ---: |
|
|
||||||
| six assistant disclaimer/preface rules | 64 |
|
|
||||||
| one prompt/response marker | 33 |
|
|
||||||
| style signals | 25 |
|
|
||||||
| source-code signals | 33 |
|
|
||||||
| typography signal | 17 |
|
|
||||||
| one ASCII lookalike and one invisible character | 28 |
|
|
||||||
| each of five negative documents | 0 |
|
|
||||||
|
|
||||||
These perfect values describe a tiny purpose-built regression set, not expected
|
|
||||||
performance on real documents. They must not be used as an accuracy or
|
|
||||||
authorship claim. Broader, independently sourced data and domain-specific review
|
|
||||||
are required before tuning default weights or thresholds.
|
|
||||||
|
|
||||||
## Default-policy sign-off
|
|
||||||
|
|
||||||
Each shipped rule has at least one explicitly labeled realistic near miss. The
|
|
||||||
initial 0.1.0 review kept all schema-version 1 rules, weights, and thresholds
|
|
||||||
unchanged. The prompt-leakage review accepted lighter `0.90ⁿ` correlation
|
|
||||||
damping for distinct prompt-leakage rules while retaining `0.75ⁿ` for other
|
|
||||||
categories. The Milestone 22 review makes printable-ASCII UTS #39 mappings
|
|
||||||
replaceable by default, retains a non-mutating `review` policy, and keeps the
|
|
||||||
Gitea-compatible invisible/control profile diagnostic-only, with explicit
|
|
||||||
multilingual limitations. The Milestone 25 review gives one-to-one fullwidth
|
|
||||||
ASCII compatibility folds precedence over visual skeletons under every policy.
|
|
||||||
The Milestone 26 review adds narrow response and unfinished-editing residue,
|
|
||||||
keeps population-level vocabulary and stock closings deliberately low-weight,
|
|
||||||
and excludes language-proficiency proxies.
|
|
||||||
The checked-in `signoff.toml` records those decisions
|
|
||||||
and contains the SHA-256 digest of `config.toml`, `heuristics.toml`, and
|
|
||||||
`unicode.toml`. This binds the reviewed scoring behavior to both rule catalogs
|
|
||||||
and the default category policy.
|
|
||||||
|
|
||||||
Any default rule, matcher, pattern, weight, occurrence cap, scope, or threshold
|
|
||||||
change invalidates the sign-off test. Renewal requires reviewing positive and
|
|
||||||
near-miss cases, reporting changed per-rule/aggregate metrics, explaining the
|
|
||||||
false-positive tradeoff, updating the digest and decision text, and obtaining
|
|
||||||
explicit maintainer approval in the change review. A green test alone is not
|
|
||||||
approval to tune defaults.
|
|
||||||
|
|
||||||
Runtime diagnostic catalogs are deployment policy and are not part of the
|
|
||||||
embedded-catalog digest. Their rules use the same validation and scoring
|
|
||||||
formula, but teams must evaluate their own added patterns and near misses for
|
|
||||||
their corpus rather than treating the embedded sign-off as validation of custom
|
|
||||||
rules.
|
|
||||||
|
|
@ -1,99 +0,0 @@
|
||||||
# Crate-boundary experiment
|
|
||||||
|
|
||||||
Status: **adopted for Milestone 13**. The experiment originally recommended
|
|
||||||
waiting because full builds do not improve and three publishable packages add
|
|
||||||
release coordination. The project subsequently chose to support the reusable
|
|
||||||
package boundary and accepted those measured costs; the durable policy is in
|
|
||||||
[ADR 0004](../decisions/0004-supported-cargo-workspace.md).
|
|
||||||
|
|
||||||
## Candidate design
|
|
||||||
|
|
||||||
The experiment keeps the root `john-slop` package as a compatibility facade and
|
|
||||||
binary distribution. It moves reusable analysis into `john-slop-core`, moves
|
|
||||||
protocol state into `john-slop-lsp`, and makes the facade's LSP dependency
|
|
||||||
optional behind the default-enabled `lsp` feature. Existing default API paths
|
|
||||||
and executable names remain intact.
|
|
||||||
|
|
||||||
The embedded catalogs live with `john-slop-core`, which makes that package
|
|
||||||
self-contained. Workspace default members include all three packages so the
|
|
||||||
existing unqualified quality commands continue to cover them.
|
|
||||||
|
|
||||||
## Method
|
|
||||||
|
|
||||||
Baseline commit `c12b6b8` (`v0.1.11`) and the experimental branch were
|
|
||||||
checked in separate worktrees. Each cold command used an empty, dedicated
|
|
||||||
`CARGO_TARGET_DIR`; dependency sources and the operating-system file cache were
|
|
||||||
warm. Timings are one directional sample rather than a statistical benchmark.
|
|
||||||
Artifact size is the resulting target directory size reported by `du -sh`.
|
|
||||||
|
|
||||||
Environment: Rust/Cargo 1.100.0-nightly from 2026-09-15, x86-64 Linux, 32 logical
|
|
||||||
CPUs. Commands used `--locked`; the full checks used all targets and features.
|
|
||||||
|
|
||||||
## Results
|
|
||||||
|
|
||||||
| Scenario | Baseline wall / CPU | Candidate wall / CPU | Baseline / candidate artifacts |
|
|
||||||
|---|---:|---:|---:|
|
|
||||||
| Reusable core | 7.51s / 29.96s¹ | 3.82s / 13.56s | 256 / 116 MiB¹ |
|
|
||||||
| Root library, no defaults | 7.51s / 29.96s | 4.09s / 19.04s | 256 / 158 MiB |
|
|
||||||
| Formatter, no defaults | 6.98s / 29.50s | 4.28s / 19.19s | 244 / 160 MiB |
|
|
||||||
| LSP | 7.92s / 30.13s | 6.97s / 29.66s | 243 / 244 MiB |
|
|
||||||
| Whole project | 7.90s / 49.68s | 7.73s / 50.69s | 415 / 416 MiB |
|
|
||||||
| No-change library check | 0.043s wall | 0.036s wall | not material |
|
|
||||||
|
|
||||||
¹ The single-crate baseline has no isolated core target, so its library result
|
|
||||||
is the nearest comparison.
|
|
||||||
|
|
||||||
The useful changes are therefore concentrated in non-LSP builds:
|
|
||||||
|
|
||||||
- the directly consumable core uses about 55% less CPU and disk and 49% less
|
|
||||||
wall time than the former monolithic library check;
|
|
||||||
- the compatibility library without default features uses about 36% less CPU,
|
|
||||||
38% less disk, and 46% less wall time;
|
|
||||||
- the formatter without default features uses about 35% less CPU and disk and
|
|
||||||
39% less wall time;
|
|
||||||
- LSP, full-project, and no-change incremental costs are effectively unchanged.
|
|
||||||
|
|
||||||
Tarpaulin still reports 92.19% (1,334/1,447 lines) with workspace coverage, so
|
|
||||||
the boundary neither gains nor loses measured test coverage.
|
|
||||||
|
|
||||||
## Gains
|
|
||||||
|
|
||||||
- Async protocol dependencies no longer enter core or no-LSP builds.
|
|
||||||
- The module ownership matches the conceptual detector/LSP boundary.
|
|
||||||
- A future library consumer can depend on `john-slop-core` without CLI, JSON,
|
|
||||||
diff, Tokio, or LSP dependencies.
|
|
||||||
- The root facade avoids a forced source migration for default-feature users.
|
|
||||||
|
|
||||||
## Costs and risks
|
|
||||||
|
|
||||||
- One package and manifest become three, with synchronized versions and the
|
|
||||||
publish order core → LSP → facade.
|
|
||||||
- The embedded catalogs must move into the core package, changing repository
|
|
||||||
paths and release ownership.
|
|
||||||
- Coverage and CI must remain workspace-aware.
|
|
||||||
- `john_slop::lsp` is unavailable when default features are disabled; this is a
|
|
||||||
feature-contract change even though default builds are compatible.
|
|
||||||
- Ordinary default and full-project builds receive no material speedup, and
|
|
||||||
runtime behavior and performance do not improve.
|
|
||||||
|
|
||||||
## Original recommendation and adoption
|
|
||||||
|
|
||||||
Do not merge this branch while John Slop is primarily distributed as one set of
|
|
||||||
three binaries and has no demonstrated external core-library consumers. The
|
|
||||||
measured savings are real but apply only when deliberately excluding LSP or
|
|
||||||
depending directly on the new core package; the normal full build is unchanged.
|
|
||||||
|
|
||||||
Reconsider the branch if `john-slop-core` becomes a supported published API, if
|
|
||||||
the adapters gain independent release schedules, or if the full build grows
|
|
||||||
enough that the package boundaries enable additional parallelism. Until then,
|
|
||||||
optionalizing the LSP module in the existing single crate would capture most of
|
|
||||||
the no-default-feature compile benefit with substantially less release overhead.
|
|
||||||
|
|
||||||
Milestone 13 deliberately chose the first reconsideration condition: treat
|
|
||||||
`john-slop-core` and `john-slop-lsp` as supported reusable packages. The measured
|
|
||||||
full-build result has not changed. Adoption instead values the enforceable
|
|
||||||
dependency boundary, lean core consumers, and clearer ownership enough to
|
|
||||||
accept synchronized releases and workspace-aware CI/package checks. Shared
|
|
||||||
metadata and dependency requirements now use Cargo workspace inheritance, while
|
|
||||||
member-specific feature activation remains local so centralization does not
|
|
||||||
erase the isolation this experiment measured.
|
|
||||||
|
|
@ -1,64 +0,0 @@
|
||||||
# Heuristics and evidence scoring
|
|
||||||
|
|
||||||
John Slop reports observable low-care signals. It does not classify authorship.
|
|
||||||
Every contribution to the score is represented by a finding with an exact
|
|
||||||
source span, stable rule ID, category, message, help, and configured weight.
|
|
||||||
|
|
||||||
The detector compiles the resolved rule catalog once. Exact phrases, regular
|
|
||||||
expressions, repeated-pattern rules, configured Unicode replacements, and
|
|
||||||
invisible/control characters share the same finding
|
|
||||||
model. Content kind, extension, path globs, and conventional generated/vendor/
|
|
||||||
minified paths decide whether each rule applies. Findings are deduplicated by
|
|
||||||
rule and byte span, then sorted by start offset, end offset, and rule ID so
|
|
||||||
every adapter observes the same order.
|
|
||||||
|
|
||||||
Additional exact-phrase, regex, and repeated-regex rules can be supplied at
|
|
||||||
runtime through `diagnostics.toml` or repeatable `--diagnostics` arguments.
|
|
||||||
They receive the same validation, scoping, scoring, ordering, CLI presentation,
|
|
||||||
and LSP publication as embedded rules. New matcher algorithms and categories
|
|
||||||
remain code changes rather than executable configuration.
|
|
||||||
|
|
||||||
The generated Unicode 16.0 profile maps every non-ASCII scalar with a printable
|
|
||||||
ASCII UTS #39 skeleton after explicitly reviewed compatibility-width and
|
|
||||||
punctuation mappings.
|
|
||||||
The default `replace` policy gives each of these findings a formatter/LSP edit;
|
|
||||||
`review` keeps the same exact diagnostics without edits, and `off` suppresses
|
|
||||||
the generated lookalikes. Invisible and direction-changing controls remain
|
|
||||||
diagnostic-only because deleting them silently is not a generally safe edit.
|
|
||||||
|
|
||||||
## Evidence-score formula
|
|
||||||
|
|
||||||
For each rule in deterministic catalog order, only the configured
|
|
||||||
`occurrence_cap` contributes to the evidence score. Its first occurrence receives the
|
|
||||||
full weight, then later occurrences receive one half, one quarter, and so on.
|
|
||||||
Findings beyond the cap remain visible but do not increase the score.
|
|
||||||
|
|
||||||
Rules are grouped by category as correlated evidence families. The first
|
|
||||||
matching rule in a category receives its full occurrence-adjusted contribution;
|
|
||||||
later distinct prompt-leakage rules receive successive factors of `0.90`, while
|
|
||||||
style, typography, and source-code rules receive successive factors of `0.75`.
|
|
||||||
Prompt/response markers and copied assistant disclaimers are stronger concrete
|
|
||||||
artifacts than generic stylistic similarities, but the remaining damping still
|
|
||||||
prevents a cluster of related phrases from being treated as fully independent
|
|
||||||
proof.
|
|
||||||
|
|
||||||
If the resulting evidence sum is `E`, the displayed score is
|
|
||||||
`round(100 × E / (E + 50))`, bounded to `0..=100`. The configured `low`,
|
|
||||||
`medium`, and `high` thresholds name the resulting band. Evidence below `low`
|
|
||||||
is explicitly reported as below threshold. Empty input and text with no
|
|
||||||
findings produce score zero and no evidence-band verdict. Binary, invalid UTF-8,
|
|
||||||
oversized, and unsupported inputs remain typed input errors; they are never
|
|
||||||
reported as clean analyses.
|
|
||||||
|
|
||||||
Every `Analysis` retains its complete finding list and can produce a compact
|
|
||||||
summary containing the numeric score, optional named band, and separate advice,
|
|
||||||
warning, and error counts. Summary generation never filters the evidence used
|
|
||||||
to explain the score.
|
|
||||||
|
|
||||||
Weights and thresholds are review policy, not calibrated probabilities. A
|
|
||||||
score of 75 does not mean a 75% chance that AI wrote the input.
|
|
||||||
|
|
||||||
The diagnostic showcase exercises five distinct prompt-leakage rules, including
|
|
||||||
two prompt/response markers. Together with its style and typography findings it
|
|
||||||
produces a low-care evidence score of 71 (medium). This is a deterministic policy score, not
|
|
||||||
an estimated probability.
|
|
||||||
|
|
@ -1,94 +0,0 @@
|
||||||
# Limitations
|
|
||||||
|
|
||||||
John Slop is a review aid for observable low-care signals. It is not an
|
|
||||||
AI-authorship detector, plagiarism detector, fact checker, security scanner, or
|
|
||||||
quality oracle. A finding says only that configured text matched at a particular
|
|
||||||
source span. Human context remains decisive.
|
|
||||||
|
|
||||||
## False positives
|
|
||||||
|
|
||||||
Legitimate material can contain every shipped signal. Documentation may quote a
|
|
||||||
chat transcript, an article may discuss knowledge cutoffs, a tutorial may use
|
|
||||||
step-by-step comments deliberately, and a tracked source-code TODO may be valid.
|
|
||||||
Unicode punctuation can be correct typography even when a downstream format
|
|
||||||
requires ASCII. The UTS #39 ASCII-skeleton profile intentionally flags many
|
|
||||||
legitimate Greek, Cyrillic, mathematical, and other non-Latin characters for
|
|
||||||
review because it has no locale or identifier context. Generic transitions are
|
|
||||||
weak style evidence, not defects by themselves.
|
|
||||||
|
|
||||||
Use path scopes, rule disabling, and project-specific weights to reflect local
|
|
||||||
policy. Do not automatically reject, shame, or attribute a document based on a
|
|
||||||
score. Review the visible findings and original context.
|
|
||||||
|
|
||||||
## False negatives and evasion
|
|
||||||
|
|
||||||
Exact and regular-expression rules are inherently easy to evade through
|
|
||||||
paraphrasing, case changes, spelling variation, inserted characters, or removal
|
|
||||||
of obvious scaffolding. Careless work need not contain any known signal, while
|
|
||||||
well-reviewed AI-assisted work may contain none. John Slop deliberately avoids
|
|
||||||
opaque statistical authorship guesses, so a clean result is not evidence of
|
|
||||||
human authorship or high quality.
|
|
||||||
|
|
||||||
Adversarial inputs can also target applicability assumptions: misleading file
|
|
||||||
extensions, unusual generated-directory names, or stdin without a filename may
|
|
||||||
select a different scope. Archive analysis uses the inner member path but does
|
|
||||||
not infer semantic language from content.
|
|
||||||
|
|
||||||
## Domain and language shift
|
|
||||||
|
|
||||||
The starter prose rules primarily encode English phrases. Other languages,
|
|
||||||
specialized editorial styles, historical corpora, transcripts, generated API
|
|
||||||
documentation, notebooks, and uncommon programming languages are
|
|
||||||
underrepresented. File-extension inference covers a conservative common set and
|
|
||||||
defaults unknown paths to prose. Teams should validate rules against their own
|
|
||||||
reviewed material before changing enforcement thresholds. Disable or narrow
|
|
||||||
set `[unicode] confusables = "review"` to keep findings without edits, or use
|
|
||||||
`"off"` when ordinary multilingual text makes character-level review too noisy.
|
|
||||||
Locale-aware suppression is not implemented.
|
|
||||||
|
|
||||||
## Evidence score is not probability
|
|
||||||
|
|
||||||
The score is a deterministic compression of configured weights, occurrence
|
|
||||||
caps, and category damping. Those values are policy choices and have not been
|
|
||||||
calibrated against population-level authorship data. A score of 75 does not mean
|
|
||||||
there is a 75% probability of AI authorship, low care, or an error. Thresholds
|
|
||||||
name review bands only, and changing the catalog changes their meaning.
|
|
||||||
|
|
||||||
Every score is traceable to findings, but that explainability does not make the
|
|
||||||
underlying heuristic universally valid. Related signals can remain correlated
|
|
||||||
even after category damping.
|
|
||||||
|
|
||||||
## Evaluation limits
|
|
||||||
|
|
||||||
The checked-in evaluation corpus is small, purpose-built, and intended to catch
|
|
||||||
behavioral regressions. Its current perfect precision and recall reflect fixture
|
|
||||||
construction, not real-world accuracy. It contains only one clean document and
|
|
||||||
must not justify deployment claims or default-weight tuning. Representative,
|
|
||||||
licensed, domain-specific samples and independent human review are required for
|
|
||||||
such decisions.
|
|
||||||
|
|
||||||
## Input and operational boundaries
|
|
||||||
|
|
||||||
Only strict UTF-8 text is analyzed. NUL-containing binary data, invalid UTF-8,
|
|
||||||
encrypted archives, unsafe archive members, and configured resource-limit
|
|
||||||
violations are errors rather than clean results. Generated, vendor, dependency,
|
|
||||||
and minified paths are skipped by default. The formatter handles strict UTF-8
|
|
||||||
files and configured ASCII replacements only; it does not rewrite ZIP-based
|
|
||||||
containers, prose, or source structure. Recursive writes require explicit
|
|
||||||
`--write --recursive` and preflight the complete input set.
|
|
||||||
|
|
||||||
The LSP intentionally handles open editor snapshots only and uses full-document
|
|
||||||
synchronization. It publishes the configured heuristic catalog but does not
|
|
||||||
publish a document-level evidence score; editor diagnostics remain individual
|
|
||||||
review signals. It does not read or write a document URI, mutate archives,
|
|
||||||
provide prose rewrites, or implement incremental range changes. Only configured
|
|
||||||
ASCII replacements receive quick fixes or formatting edits. Under the default
|
|
||||||
policy this includes printable-ASCII UTS #39 skeletons, which can alter valid
|
|
||||||
Greek, Cyrillic, mathematical, or other text; use `review` for non-mutating
|
|
||||||
diagnostics. Invisible and bidirectional-control findings are diagnostic-only, and
|
|
||||||
configuration reload errors retain the last valid configuration and compiled
|
|
||||||
detector.
|
|
||||||
|
|
||||||
Analysis is local and performs no network lookup. Consequently, it cannot check
|
|
||||||
citations, current facts, provenance, licensing, or similarity to external
|
|
||||||
sources.
|
|
||||||
|
|
@ -1,93 +0,0 @@
|
||||||
# Detector output contracts
|
|
||||||
|
|
||||||
The detector exposes four formats through `--output`. All formats preserve the
|
|
||||||
same deterministic document and finding order. Requested paths retain command
|
|
||||||
line order, directory entries are sorted by filename, archive order follows the
|
|
||||||
container, and findings are sorted by byte span and rule ID.
|
|
||||||
|
|
||||||
## Human
|
|
||||||
|
|
||||||
`--output human` renders every finding as a plain `miette` diagnostic with the
|
|
||||||
source name, exact excerpt and primary span, severity, stable rule code,
|
|
||||||
message, and remediation help. A final line per document reports the low-care
|
|
||||||
evidence score/band and advice, warning, and error counts. The narratable renderer
|
|
||||||
deliberately avoids ANSI escapes.
|
|
||||||
|
|
||||||
## Miette
|
|
||||||
|
|
||||||
`--output miette` is the default. It renders the same typed diagnostics with
|
|
||||||
Miette's graphical Unicode source annotations and ANSI severity colors,
|
|
||||||
followed by the same summary. Each diagnostic starts with a `◆`-marked rule-code
|
|
||||||
title immediately followed by its severity/message, and ends with an empty line
|
|
||||||
that separates it from the next diagnostic or summary. This mode requires the
|
|
||||||
default `fancy-diagnostics` Cargo feature; builds made with
|
|
||||||
`--no-default-features`
|
|
||||||
accept the option but fall back to the plain human renderer. Select `human`,
|
|
||||||
`concise`, or `json` when ANSI escapes are not appropriate.
|
|
||||||
|
|
||||||
## Concise
|
|
||||||
|
|
||||||
`--output concise` emits one line per finding followed by one summary line. A
|
|
||||||
finding line contains the source identity, half-open UTF-8 byte range, severity,
|
|
||||||
rule ID, and message. Recoverable input errors go to stderr in human and concise
|
|
||||||
modes.
|
|
||||||
|
|
||||||
## JSON Lines schema version 2
|
|
||||||
|
|
||||||
`--output json` emits one compact JSON object per line. An analyzed document has
|
|
||||||
`kind: "analysis"` and this shape:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": 2,
|
|
||||||
"kind": "analysis",
|
|
||||||
"source": "article.md",
|
|
||||||
"summary": {
|
|
||||||
"evidence_score": 17,
|
|
||||||
"evidence_band": "below-threshold",
|
|
||||||
"advice": 1,
|
|
||||||
"warnings": 0,
|
|
||||||
"errors": 0,
|
|
||||||
"total": 1
|
|
||||||
},
|
|
||||||
"findings": [
|
|
||||||
{
|
|
||||||
"rule_id": "unicode-ascii-replacement",
|
|
||||||
"category": "typography",
|
|
||||||
"severity": "advice",
|
|
||||||
"message": "a Unicode character or sequence has a configured ASCII replacement",
|
|
||||||
"help": "Review the character or apply the configured ASCII replacement.",
|
|
||||||
"byte_span": { "start": 8, "end": 11 },
|
|
||||||
"matched_text": "—",
|
|
||||||
"weight": 10.0,
|
|
||||||
"fix": {
|
|
||||||
"byte_span": { "start": 8, "end": 11 },
|
|
||||||
"replacement": "-"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`evidence_band` is `null` when there is no evidence. Spans are zero-based,
|
|
||||||
half-open UTF-8 byte offsets into the exact source snapshot, not character or
|
|
||||||
UTF-16 positions. `fix` is `null` for findings without a safe configured edit.
|
|
||||||
|
|
||||||
A recoverable input failure has `kind: "input-error"`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema_version": 2,
|
|
||||||
"kind": "input-error",
|
|
||||||
"source": "broken.txt",
|
|
||||||
"message": "input broken.txt is not valid UTF-8"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
JSON records are the only stdout content in JSON mode and never contain ANSI
|
|
||||||
escapes or logs. Fatal argument/configuration/setup failures can occur before
|
|
||||||
the JSON stream starts; they return status 2, leave stdout empty, and report on
|
|
||||||
stderr. Additive fields may appear within schema version 2; incompatible changes
|
|
||||||
require a new schema version. Schema version 2 renamed the version 1 summary
|
|
||||||
fields `confidence_score` and `confidence_band` to `evidence_score` and
|
|
||||||
`evidence_band`; version 1 records are no longer emitted.
|
|
||||||
|
|
@ -1,63 +0,0 @@
|
||||||
# Release procedure
|
|
||||||
|
|
||||||
Releases are built from a clean, reviewed Linux checkout with the exact Rust
|
|
||||||
toolchain recorded in `BUILD-INFO`.
|
|
||||||
|
|
||||||
If publishing Rust packages, publish and verify `john-slop-core` first,
|
|
||||||
`john-slop-lsp` second, and the root `john-slop` facade last. All three versions
|
|
||||||
must remain synchronized; path dependencies carry matching registry versions.
|
|
||||||
Before publication, assemble all three crates in Cargo's dependency order with
|
|
||||||
`scripts/package-workspace.sh --no-verify`. The flag is required until the
|
|
||||||
matching core and LSP versions exist in the registry; after publication, rerun
|
|
||||||
the script without it to verify the registry-form dependency graph.
|
|
||||||
|
|
||||||
1. Ensure `CHANGELOG.md`, `Cargo.toml`, `Cargo.lock`, documentation, and the
|
|
||||||
version badge agree; run every command in `docs/development.md`. Milestone
|
|
||||||
`v0.1.N` completion commits use package version `0.1.N`; regenerate the badge
|
|
||||||
only after updating the manifest and lockfile.
|
|
||||||
2. Run normal Criterion sampling and the fuzz targets appropriate to changes.
|
|
||||||
3. Commit the release preparation and create signed or annotated tag `vX.Y.Z`.
|
|
||||||
4. Check out that tag and build the release archive:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
scripts/package-release.sh X.Y.Z
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Run the packaging command a second time in a fresh checkout with the same
|
|
||||||
Rust version and compare the archive SHA-256 digest.
|
|
||||||
6. Inspect `BUILD-INFO`, `SHA256SUMS`, binary help, archive contents, and recorded
|
|
||||||
sizes. Sign the archive and checksum if a signing identity is available.
|
|
||||||
7. Publish the archive, archive checksum, changelog excerpt, and signature
|
|
||||||
together. Never publish an untagged or dirty build as a release.
|
|
||||||
|
|
||||||
`SOURCE_DATE_EPOCH` comes from the tagged commit. The release profile uses thin
|
|
||||||
LTO, one codegen unit, abort-on-panic, and symbol stripping; GNU tar normalizes
|
|
||||||
entry order, timestamps, ownership, and gzip metadata. Reproducibility still
|
|
||||||
depends on the same target, Rust toolchain, linker, and system libraries, all of
|
|
||||||
which must be recorded or controlled by the release environment.
|
|
||||||
|
|
||||||
The packaging smoke test produced byte-identical archives in two consecutive
|
|
||||||
runs on the Milestone 7 development host and validated both archive and internal
|
|
||||||
checksums.
|
|
||||||
|
|
||||||
## Publish with rust_realease
|
|
||||||
|
|
||||||
Install [`rust_realease`](https://proxy.dandaedre.com/gitea/idle-org/rust_realease),
|
|
||||||
then pass the path of a readable Gitea token file to the repository wrapper:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
./release.sh /path/to/token.txt --dry-run
|
|
||||||
./release.sh /path/to/token.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
The wrapper keeps the token out of command-line arguments, exports it as
|
|
||||||
`GITEA_TOKEN` only to `rust_realease`, selects the `john-slop` workspace package,
|
|
||||||
and publishes cargo-binstall archives alongside raw binary artifacts. The
|
|
||||||
project-local `rust-release.toml` maps the SSH development origin to the
|
|
||||||
canonical HTTPS proxy for release API and download operations without modifying
|
|
||||||
the Git remote. A hyphenated Cargo version such as
|
|
||||||
`0.1.27-rc.8` automatically marks the Gitea release as a prerelease. Additional
|
|
||||||
arguments are forwarded, so use `--dry-run` before publication and
|
|
||||||
`--replace-assets` only after reviewing an existing release collision. The
|
|
||||||
checkout must be clean and its local and remote `vX.Y.Z` tag must match the
|
|
||||||
package version as required by `rust_realease`.
|
|
||||||
|
|
@ -1,106 +0,0 @@
|
||||||
# Review signals and evidence methodology
|
|
||||||
|
|
||||||
John Slop finds concrete text that may deserve review. It does **not** determine
|
|
||||||
whether AI wrote a document, identify a model, or assign an authorship
|
|
||||||
probability. A finding names the exact matched text and the configured reason
|
|
||||||
for reviewing it; the evidence score is only a bounded aggregation of those
|
|
||||||
findings.
|
|
||||||
|
|
||||||
This boundary matters because general-purpose AI-text detection is not reliable
|
|
||||||
enough to support an authorship claim. Sadasivan et al. show both practical
|
|
||||||
paraphrasing attacks and a theoretical trade-off between detection and false
|
|
||||||
positives as human and model distributions become closer [1]. A comparative
|
|
||||||
evaluation of 14 detection tools likewise found them neither accurate nor
|
|
||||||
reliable, with performance worsening after obfuscation [2]. Liang et al. found
|
|
||||||
that perplexity-based detectors disproportionately classified non-native English
|
|
||||||
writing as AI-generated [3]. John Slop therefore does not use perplexity,
|
|
||||||
burstiness, grammar, spelling, vocabulary richness, dialect, or English fluency
|
|
||||||
as evidence.
|
|
||||||
|
|
||||||
## Signal tiers
|
|
||||||
|
|
||||||
### Direct response residue
|
|
||||||
|
|
||||||
These are relatively strong, locally explainable signs that conversational
|
|
||||||
material may have been pasted into a document:
|
|
||||||
|
|
||||||
- assistant self-identification, capability, or knowledge-cutoff disclaimers;
|
|
||||||
- `User:` / `Assistant:` transcript scaffolding;
|
|
||||||
- answer or revision prefaces; and
|
|
||||||
- follow-up offers such as asking whether the reader wants an expansion.
|
|
||||||
|
|
||||||
They still do not prove authorship. A quotation, article about assistants, or
|
|
||||||
intentional transcript can contain the same text, so applicability scopes and
|
|
||||||
near-miss fixtures are part of every rule's review.
|
|
||||||
|
|
||||||
### Unfinished editing and source residue
|
|
||||||
|
|
||||||
Unresolved insertion notes, implementation placeholders, omitted-code comments,
|
|
||||||
tutorial narration, and stray Markdown fences are useful low-care signals
|
|
||||||
regardless of how they were produced. These rules ask whether work is complete;
|
|
||||||
they do not infer who or what produced it.
|
|
||||||
|
|
||||||
### Subtle population-level clues
|
|
||||||
|
|
||||||
Some patterns are meaningful across a large corpus but weak in one document.
|
|
||||||
Kobak et al. measured abrupt, population-level increases in selected style words
|
|
||||||
across more than 15 million biomedical abstracts [4]. John Slop represents this
|
|
||||||
only as `excess-vocabulary-cluster`: it requires multiple matches, has weight
|
|
||||||
`1.0`, is occurrence-capped, and says explicitly that it is not authorship
|
|
||||||
evidence. A stock closing has similarly low weight. These clues should prompt a
|
|
||||||
specificity edit at most, never an accusation.
|
|
||||||
|
|
||||||
Repeated transitions and templated phrases sit between the tiers. They activate
|
|
||||||
only on an exact conservative pattern; repetition from one rule is damped and
|
|
||||||
cannot alone create a high evidence band.
|
|
||||||
|
|
||||||
### Verification targets, not offline shape rules
|
|
||||||
|
|
||||||
Generated references and package names can be wrong. Walters and Wilder found
|
|
||||||
fabricated and substantially erroneous citations in a controlled set of
|
|
||||||
generated bibliographies [5]. Spracklen et al. measured package hallucinations
|
|
||||||
in a large code-generation study [6]. But a citation's typography or a package
|
|
||||||
name's shape cannot establish that it is fictitious. Reliable checks require an
|
|
||||||
authoritative catalog and contextual validation, while John Slop analysis is
|
|
||||||
local and makes no network calls. These remain human review targets rather than
|
|
||||||
shipped heuristic rules.
|
|
||||||
|
|
||||||
## How a rule is admitted
|
|
||||||
|
|
||||||
A shipped rule must have a stable ID, a narrow pattern, a rationale, review
|
|
||||||
help, an applicability scope, a finite non-negative weight, and an occurrence
|
|
||||||
cap. It also needs:
|
|
||||||
|
|
||||||
1. a positive fixture showing the intended match;
|
|
||||||
2. a realistic near miss showing what must not match;
|
|
||||||
3. deterministic byte spans and ordering; and
|
|
||||||
4. review of likely harm, especially language-proficiency and demographic
|
|
||||||
proxies.
|
|
||||||
|
|
||||||
Exact residue and fixable Unicode receive higher confidence than broad stylistic
|
|
||||||
clues. New research is used to propose testable review signals, not to import a
|
|
||||||
paper's population-level conclusion into an individual-document verdict.
|
|
||||||
|
|
||||||
## Scoring and editor visibility
|
|
||||||
|
|
||||||
Rule weight contributes bounded evidence to the shared score. Per-rule caps and
|
|
||||||
correlation damping prevent repeated copies of one pattern from dominating it.
|
|
||||||
The number is neither accuracy nor probability; see
|
|
||||||
[Heuristics and evidence scoring](heuristics.md) for the formula.
|
|
||||||
|
|
||||||
LSP visibility is a separate presentation choice. By default the editor shows
|
|
||||||
only configured Unicode findings, which are exact and may have safe replacement
|
|
||||||
edits. `--all-diagnostics` enables the complete detector catalog for testing or
|
|
||||||
deliberate review. `--minimum-diagnostic-weight <WEIGHT>` is the quieter opt-in:
|
|
||||||
it keeps Unicode findings and adds only non-Unicode findings whose configured
|
|
||||||
weight meets the threshold. Filtering affects editor diagnostics, not CLI
|
|
||||||
analysis or scoring.
|
|
||||||
|
|
||||||
## References
|
|
||||||
|
|
||||||
1. Sadasivan et al., [Can AI-Generated Text be Reliably Detected?](https://arxiv.org/abs/2303.11156)
|
|
||||||
2. Weber-Wulff et al., [Testing of detection tools for AI-generated text](https://link.springer.com/article/10.1007/s40979-023-00146-z)
|
|
||||||
3. Liang et al., [GPT detectors are biased against non-native English writers](https://pmc.ncbi.nlm.nih.gov/articles/PMC10382961/)
|
|
||||||
4. Kobak et al., [Delving into LLM-assisted writing in biomedical publications through excess vocabulary](https://arxiv.org/abs/2406.07016)
|
|
||||||
5. Walters and Wilder, [Fabrication and errors in the bibliographic citations generated by ChatGPT](https://pmc.ncbi.nlm.nih.gov/articles/PMC10484980/)
|
|
||||||
6. Spracklen et al., [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://www.usenix.org/conference/usenixsecurity25/presentation/spracklen)
|
|
||||||
|
|
@ -1,72 +0,0 @@
|
||||||
# Ambiguous and invisible Unicode references
|
|
||||||
|
|
||||||
Gitea maintains the concrete list that most closely matches the requested
|
|
||||||
"ambiguous Unicode characters" catalog:
|
|
||||||
|
|
||||||
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
|
||||||
is the generator input;
|
|
||||||
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
|
||||||
are easier to browse as code;
|
|
||||||
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
|
||||||
combines a locale-specific table with `_common` at runtime; and
|
|
||||||
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
|
||||||
records the upstream source and a separate invisible-rune catalog.
|
|
||||||
|
|
||||||
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
|
||||||
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
|
||||||
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
|
||||||
The array values alternate between a confusable code point and the code point
|
|
||||||
it resembles. Counts across tables are not necessarily unique global mappings.
|
|
||||||
|
|
||||||
Gitea's generator identifies
|
|
||||||
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
|
||||||
as its upstream. For standards-based identifier spoofing, the canonical broader
|
|
||||||
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
|
||||||
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
|
||||||
data file. GitHub separately documents a warning for hidden bidirectional text;
|
|
||||||
that protection is about bidi controls rather than a published Gitea-style
|
|
||||||
locale-aware lookalike table.
|
|
||||||
|
|
||||||
## Implemented John Slop profile
|
|
||||||
|
|
||||||
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
|
|
||||||
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
|
|
||||||
scalar whose single-character skeleton is non-empty printable ASCII, plus nine
|
|
||||||
reviewed punctuation mappings and 95 compatibility width folds. Width folding
|
|
||||||
takes precedence when a UTS #39 visual skeleton would produce a different
|
|
||||||
target, such as fullwidth `I` mapping to `I` rather than `l`. That is 1,884
|
|
||||||
mappings in the current profile.
|
|
||||||
Each entry has a stable ID, source, target, rationale, and enabled switch, and
|
|
||||||
the same asset owns the diagnostic metadata.
|
|
||||||
|
|
||||||
The default `[unicode] confusables = "replace"` policy emits
|
|
||||||
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
|
|
||||||
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
|
|
||||||
`review` for identical findings without edits, or `off` to suppress generated
|
|
||||||
lookalikes. The reviewed punctuation and width-folding mappings remain active
|
|
||||||
under all three settings and can still be disabled individually.
|
|
||||||
|
|
||||||
This broad mapping is intentionally more aggressive than Gitea's locale-aware
|
|
||||||
display warning. Greek, Cyrillic, mathematical, and other characters may be
|
|
||||||
legitimate text, so projects handling multilingual content should prefer
|
|
||||||
`review` unless ASCII normalization is their explicit policy.
|
|
||||||
|
|
||||||
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
|
|
||||||
pinned commit above, including bidi embedding/override and isolate controls.
|
|
||||||
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
|
|
||||||
reported. Each other match receives an exact UTF-8 source span that adapters
|
|
||||||
convert normally. Controls never feed the formatter or LSP quick fixes because
|
|
||||||
silent deletion has no generally safe meaning.
|
|
||||||
|
|
||||||
Regenerate the checked-in policy deterministically with:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run -p john-slop-core --example generate_unicode_catalog \
|
|
||||||
> crates/john-slop-core/assets/unicode.toml
|
|
||||||
```
|
|
||||||
|
|
||||||
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
|
|
||||||
|
|
||||||
Future profile updates must pin their upstream Unicode/data version, retain
|
|
||||||
provenance and licensing, test ordinary multilingual near misses, and renew the
|
|
||||||
hash-bound default-policy sign-off.
|
|
||||||
184
docs/usage.md
184
docs/usage.md
|
|
@ -1,184 +0,0 @@
|
||||||
# Usage
|
|
||||||
|
|
||||||
## Detector
|
|
||||||
|
|
||||||
Run the detector over one or more paths:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run --bin john-slop -- [--fail-score 0..100] <PATH>...
|
|
||||||
cargo run --bin john-slop -- --stdin [--stdin-filename article.md]
|
|
||||||
```
|
|
||||||
|
|
||||||
Each path may be a UTF-8 text file, a directory, a ZIP archive, or a supported
|
|
||||||
ZIP-based office document (`.docx`, `.xlsx`, or `.pptx`). Directory traversal and
|
|
||||||
archive reading stream one result at a time, so callers do not need to retain an
|
|
||||||
entire input tree in memory.
|
|
||||||
|
|
||||||
`--stdin` reads one bounded strict UTF-8 document from standard input. Supply
|
|
||||||
`--stdin-filename PATH` when extension, language, and path scopes should use a
|
|
||||||
virtual filename. Standard input can be analyzed alongside filesystem paths.
|
|
||||||
Repeat `--ignore GLOB` to skip matching file or archive-member paths. Resource
|
|
||||||
limits can be overridden with `--max-document-bytes`, `--max-archive-bytes`,
|
|
||||||
`--max-expanded-bytes`, `--max-archive-entries`,
|
|
||||||
`--max-compression-ratio`, and `--max-walk-depth`.
|
|
||||||
|
|
||||||
Select output with `--output human`, `--output miette`, `--output concise`, or
|
|
||||||
`--output json`. Miette is the default and uses colored Unicode graphical source
|
|
||||||
annotations when the default `fancy-diagnostics` feature is enabled. Its marked
|
|
||||||
rule-code titles stay attached to their messages, while an empty line separates
|
|
||||||
complete diagnostics; it falls
|
|
||||||
back to human mode without that feature. Human mode renders uncolored plain
|
|
||||||
source excerpts, exact labels, remediation help, and a summary. Concise mode
|
|
||||||
emits stable single-line finding and summary records. JSON mode emits one object
|
|
||||||
per analyzed document with `schema_version: 2`, summary fields, exact byte spans,
|
|
||||||
and every finding. Machine modes never emit ANSI escapes or logs on stdout.
|
|
||||||
|
|
||||||
Findings cover the embedded or overridden heuristic catalog and retain exact
|
|
||||||
UTF-8 byte spans through every output mode. See [Output contracts](output.md)
|
|
||||||
for the human/concise formats and complete JSON Lines schema.
|
|
||||||
|
|
||||||
Exit statuses are stable:
|
|
||||||
|
|
||||||
- `0`: every analyzed document is below the requested evidence-score threshold;
|
|
||||||
- `1`: at least one document reaches the threshold;
|
|
||||||
- `2`: arguments are invalid or an operational/input error prevents completion.
|
|
||||||
|
|
||||||
`--fail-score SCORE` selects an exact inclusive score threshold and
|
|
||||||
defaults to 100. `--fail-level low|medium|high` instead uses the corresponding
|
|
||||||
threshold from the fully resolved configuration. The two options are mutually
|
|
||||||
exclusive. The old `--fail-confidence` spelling remains a visible alias for
|
|
||||||
migration.
|
|
||||||
|
|
||||||
Input failures are isolated per document whenever iteration can continue. Valid
|
|
||||||
documents still emit all findings, while any partial input failure makes the
|
|
||||||
final status `2` (taking precedence over threshold status `1`). Human and
|
|
||||||
concise input errors go to stderr. JSON mode emits versioned `input-error`
|
|
||||||
records on stdout alongside `analysis` records and leaves stderr empty unless a
|
|
||||||
fatal setup failure prevents the output contract from starting.
|
|
||||||
|
|
||||||
The low-care evidence score summarizes observable configured evidence. It is
|
|
||||||
not an AI-authorship probability. See [Heuristics and evidence scoring](heuristics.md) for
|
|
||||||
the formula and [Limitations](limitations.md) before using thresholds in review
|
|
||||||
workflows.
|
|
||||||
|
|
||||||
Configuration layers can be selected with `--config-name NAME` and `--config
|
|
||||||
FILE`; repeat `--diagnostics FILE` to load runtime rule catalogs without
|
|
||||||
recompilation. Command-line resource limits are the highest-precedence input overrides.
|
|
||||||
Use `john-slop config show` to print the resolved TOML and `john-slop config
|
|
||||||
paths` to inspect source precedence. See [Configuration](configuration.md) for
|
|
||||||
the schema and merge rules.
|
|
||||||
|
|
||||||
The LSP publishes Unicode findings by default. Start `john-slop-lsp` with
|
|
||||||
`--all-diagnostics` to publish the complete configured detector catalog, or use
|
|
||||||
`--minimum-diagnostic-weight WEIGHT` to retain Unicode findings and add only
|
|
||||||
non-Unicode findings at or above that weight. This editor-only filter does not
|
|
||||||
change CLI analysis, evidence scores, replacement planning, or formatting.
|
|
||||||
|
|
||||||
## Rust examples
|
|
||||||
|
|
||||||
The [`examples` directory](../examples/README.md) contains runnable Cargo
|
|
||||||
examples for direct text analysis, streaming path input, and replacement edit
|
|
||||||
planning. Run all example targets as part of the normal test matrix with
|
|
||||||
`cargo test --examples --all-features`.
|
|
||||||
|
|
||||||
Its [shared fixture manifest](../examples/fixtures/manifest.toml) pairs several
|
|
||||||
`example.txt` inputs with expected detector rules, replacement counts, and
|
|
||||||
formatted output. The executable fixture and protocol tests validate the
|
|
||||||
library, detector, formatter, and LSP against the shared replacement contract.
|
|
||||||
The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase)
|
|
||||||
prints every current prose finding in graphical Miette form and intentionally
|
|
||||||
exits `1` when its low evidence threshold is reached.
|
|
||||||
|
|
||||||
## Supported input policy
|
|
||||||
|
|
||||||
Input reads use strict UTF-8 and reject NUL-containing text. They do not follow
|
|
||||||
symlinks by default. Limits on files, walk depth, archive entries, expanded
|
|
||||||
bytes, compression ratio, and member paths protect analysis from unbounded or
|
|
||||||
unsafe input. The exact policy is recorded in
|
|
||||||
[ADR 0002](decisions/0002-input-policy-and-limits.md).
|
|
||||||
|
|
||||||
## Formatter
|
|
||||||
|
|
||||||
With no path, `john-slop-fmt` reads one bounded strict-UTF-8 document from stdin
|
|
||||||
and writes only formatted bytes to stdout. With a file path, the default remains
|
|
||||||
read-only and prints the transformed file without changing it:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop-fmt < article.txt > article.ascii.txt
|
|
||||||
john-slop-fmt article.txt
|
|
||||||
john-slop-fmt --check article.txt
|
|
||||||
john-slop-fmt --diff article.txt
|
|
||||||
john-slop-fmt --write article.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
The modes are mutually exclusive:
|
|
||||||
|
|
||||||
- default mode emits formatted text and exits `0`;
|
|
||||||
- `--check` emits nothing and exits `1` when any replacement is needed;
|
|
||||||
- `--diff` emits a unified diff and exits `1` when any replacement is needed;
|
|
||||||
- `--write` emits nothing and atomically replaces changed files, exiting `0`;
|
|
||||||
- argument, configuration, UTF-8, input, or write failures exit `2`.
|
|
||||||
|
|
||||||
All modes use the same embedded/user/named/explicit configuration precedence as
|
|
||||||
the detector. `--config-name NAME`, `--config FILE`, and repeatable
|
|
||||||
`--diagnostics FILE` select those layers. Diagnostic rules do not change the
|
|
||||||
formatter's replacement plan, but catalog validation stays consistent across
|
|
||||||
all adapters.
|
|
||||||
Replacement sources use deterministic longest-match semantics, and formatting
|
|
||||||
twice is byte-for-byte identical to formatting once.
|
|
||||||
|
|
||||||
Directories require `--recursive`, and recursive operation is deliberately
|
|
||||||
limited to `--check --recursive` or `--write --recursive`. Before a recursive
|
|
||||||
write, every input is read, decoded, planned, and edit-validated; a failure
|
|
||||||
prevents all writes. Changed files use a temporary file in the same directory,
|
|
||||||
copy original permissions, sync content, and replace by rename where the
|
|
||||||
platform supports it. Files needing no edit are never rewritten.
|
|
||||||
|
|
||||||
ZIP, DOCX, XLSX, and PPTX mutation is not supported. Extract and format an
|
|
||||||
individual UTF-8 file instead.
|
|
||||||
|
|
||||||
## LSP
|
|
||||||
|
|
||||||
`john-slop-lsp` is a stdio Language Server Protocol 3.17 adapter. Editors launch
|
|
||||||
it without positional arguments:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop-lsp
|
|
||||||
john-slop-lsp --config-name editorial
|
|
||||||
john-slop-lsp --config ./john-slop.toml
|
|
||||||
john-slop-lsp --diagnostics ./diagnostics.toml
|
|
||||||
john-slop-lsp --all-diagnostics
|
|
||||||
john-slop-lsp --minimum-diagnostic-weight 10
|
|
||||||
```
|
|
||||||
|
|
||||||
The server advertises full-document synchronization and quick-fix code actions.
|
|
||||||
It stores only currently open UTF-8 snapshots, their URI, language ID, and
|
|
||||||
monotonically increasing version. On open or an accepted newer full-document
|
|
||||||
change it runs the same configured detector as the CLI and publishes every
|
|
||||||
applicable Unicode finding. `--all-diagnostics` publishes the complete result;
|
|
||||||
`--minimum-diagnostic-weight WEIGHT` adds only non-Unicode findings meeting the
|
|
||||||
threshold. File URI extensions select prose/source and path scopes; editor
|
|
||||||
language IDs provide the prose/source fallback for untitled or extensionless
|
|
||||||
buffers.
|
|
||||||
|
|
||||||
Each diagnostic has source `john-slop`, the stable heuristic rule ID as its
|
|
||||||
code, the configured severity and remediation text, document-version metadata,
|
|
||||||
and an exact UTF-16 range. Advice maps to LSP information, while warning and
|
|
||||||
error retain their corresponding LSP severities.
|
|
||||||
|
|
||||||
Runtime catalog rules are reloaded by
|
|
||||||
`workspace/didChangeConfiguration` together with normal configuration. Quick
|
|
||||||
fixes remain intentionally narrower than diagnostics: they are recomputed
|
|
||||||
from the current snapshot with the shared replacement planner and returned only
|
|
||||||
for findings carrying safe configured ASCII edits. Style, prompt-leakage, and
|
|
||||||
source-review findings are diagnostic-only. A request carrying stale John Slop
|
|
||||||
diagnostic metadata is rejected. Closing a document removes its snapshot and
|
|
||||||
publishes an empty diagnostic set. `workspace/didChangeConfiguration` reloads
|
|
||||||
the normal embedded, user, named, and explicit layers, recompiles the detector,
|
|
||||||
and republishes every open snapshot under the unchanged visibility filter; a failed reload keeps the last valid
|
|
||||||
configuration and detector.
|
|
||||||
|
|
||||||
Stdout is reserved for JSON-RPC framing. The server is quiet by default; pass
|
|
||||||
`--log-json` to emit lifecycle-only JSON Lines to stderr. Those records contain
|
|
||||||
event metadata, URI, and version but never document contents. Complete Helix and
|
|
||||||
Vim setup and validation steps live in [Editor setup](editors.md).
|
|
||||||
|
|
@ -1,100 +0,0 @@
|
||||||
# Runnable Rust examples
|
|
||||||
|
|
||||||
These Cargo examples exercise the same library boundaries used by the three
|
|
||||||
John Slop binaries:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run --example analyze_text
|
|
||||||
cargo run --example analyze_path -- examples/fixtures
|
|
||||||
cargo run --example plan_replacements
|
|
||||||
cargo run --example ascii_replacements
|
|
||||||
```
|
|
||||||
|
|
||||||
- `analyze_text` analyzes an in-memory prose snapshot and prints its explainable
|
|
||||||
findings.
|
|
||||||
- `analyze_path` streams a file, directory, or supported ZIP container. Its
|
|
||||||
optional path defaults to `examples/fixtures`.
|
|
||||||
- `plan_replacements` prints the exact UTF-8 byte edits shared by the formatter
|
|
||||||
and LSP quick fixes.
|
|
||||||
- `ascii_replacements` demonstrates the embedded width-folding catalog across
|
|
||||||
all 95 printable ASCII targets.
|
|
||||||
|
|
||||||
These examples use embedded defaults so their behavior does not depend on a
|
|
||||||
developer's user configuration. Applications that want normal configuration
|
|
||||||
layering should call `john_slop::load_config()` instead.
|
|
||||||
|
|
||||||
## Shared fixture corpus
|
|
||||||
|
|
||||||
[`fixtures/manifest.toml`](fixtures/manifest.toml) describes small inputs shared
|
|
||||||
by detector CLI, formatter, and LSP testing. Several directories deliberately
|
|
||||||
contain an `example.txt`, covering clean prose, copied assistant residue, stock
|
|
||||||
style, safe Unicode punctuation replacements, and diagnostic-only Unicode
|
|
||||||
lookalikes/invisibles. A Rust source example covers source-only rules. Each
|
|
||||||
input has a checked-in expected formatted result; the Unicode review fixture's
|
|
||||||
expected result is intentionally byte-identical because those findings do not
|
|
||||||
authorize rewriting.
|
|
||||||
|
|
||||||
Exercise the currently implemented boundaries with:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run --bin john-slop -- --output concise examples/fixtures/assistant/example.txt
|
|
||||||
cargo run --bin john-slop-fmt -- examples/fixtures/typography/example.txt
|
|
||||||
cargo run --bin john-slop-fmt -- --check examples/fixtures/clean/example.txt
|
|
||||||
cargo run --example analyze_path -- examples/fixtures
|
|
||||||
cargo test --test example_fixtures
|
|
||||||
cargo test --test lsp
|
|
||||||
```
|
|
||||||
|
|
||||||
The formatter integration test executes every case and compares stdout with its
|
|
||||||
expected ASCII file. The manifest also names `lsp` as a consumer, and the
|
|
||||||
in-process protocol suite proves that applying LSP edits produces the same bytes
|
|
||||||
as the formatter.
|
|
||||||
|
|
||||||
## Printable ASCII replacement showcase
|
|
||||||
|
|
||||||
[`ascii_replacements.rs`](ascii_replacements.rs) applies the embedded fullwidth
|
|
||||||
mappings for ASCII `!` through `~` and maps ideographic space to ASCII space.
|
|
||||||
Its [Unicode input](ascii-replacement-showcase/fullwidth.txt)
|
|
||||||
and [expected output](ascii-replacement-showcase/ascii.txt) display all 95
|
|
||||||
printable ASCII targets in one place:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run --example ascii_replacements
|
|
||||||
```
|
|
||||||
|
|
||||||
The example reports 95 planned edits on stderr and writes only the transformed
|
|
||||||
fixture to stdout. These compatibility width folds are production defaults;
|
|
||||||
broader script-confusable replacements remain controlled separately by the
|
|
||||||
`[unicode] confusables` policy.
|
|
||||||
|
|
||||||
## Expected-failure diagnostic showcase
|
|
||||||
|
|
||||||
[`diagnostic-showcase.txt`](diagnostic-showcase.txt) contains every currently
|
|
||||||
shipped prose signal, while
|
|
||||||
[`diagnostic-showcase/source.rs`](diagnostic-showcase/source.rs) adds the
|
|
||||||
source-placeholder and tutorial-narration rules plus the reviewed
|
|
||||||
punctuation replacements, a fixable Cyrillic ASCII lookalike, a zero-width space, and a
|
|
||||||
right-to-left override. The adjacent example configuration promotes assistant
|
|
||||||
self-identification from warning to error and lifts the typography occurrence
|
|
||||||
cap for this display, making one run show Miette advice, warning, error,
|
|
||||||
fixable, and deliberately non-fixable control diagnostics together:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo run --bin john-slop -- \
|
|
||||||
--config examples/diagnostic-showcase.toml \
|
|
||||||
--output miette --fail-level low \
|
|
||||||
examples/diagnostic-showcase.txt \
|
|
||||||
examples/diagnostic-showcase/source.rs
|
|
||||||
```
|
|
||||||
|
|
||||||
That command is expected to exit with status `1`: reaching the configured
|
|
||||||
threshold is the behavior under demonstration. The repository test asserts the
|
|
||||||
nonzero result and representative prose/source categories rather than requiring
|
|
||||||
exact equality with an evolving catalog. Keeping new diagnostic kinds visible
|
|
||||||
here is nevertheless part of the intended maintenance workflow. Run the test
|
|
||||||
with capture disabled to print the complete report:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo test --test example_fixtures \
|
|
||||||
diagnostic_showcase_spans_prose_source_and_severities -- --nocapture
|
|
||||||
```
|
|
||||||
|
|
@ -1,31 +0,0 @@
|
||||||
//! Stream and analyze every supported document under a path.
|
|
||||||
|
|
||||||
use std::{env, error::Error, path::PathBuf};
|
|
||||||
|
|
||||||
use john_slop::{AnalysisContext, Config, Detector, read_inputs};
|
|
||||||
|
|
||||||
fn main() -> Result<(), Box<dyn Error>> {
|
|
||||||
let path = env::args_os()
|
|
||||||
.nth(1)
|
|
||||||
.map_or_else(|| PathBuf::from("examples/fixtures"), PathBuf::from);
|
|
||||||
let config = Config::embedded();
|
|
||||||
let detector = Detector::new(&config)?;
|
|
||||||
|
|
||||||
for document in read_inputs(&config, &path)? {
|
|
||||||
match document {
|
|
||||||
Ok(document) => {
|
|
||||||
let context = AnalysisContext::for_source(&document.source);
|
|
||||||
let analysis = detector.analyze_with(&document.content, &context);
|
|
||||||
println!(
|
|
||||||
"{}: evidence_score={}, findings={}",
|
|
||||||
document.source,
|
|
||||||
analysis.evidence_score.score(),
|
|
||||||
analysis.findings.len()
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Err(error) => eprintln!("input error: {error}"),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
//! Analyze an in-memory document with the embedded heuristic catalog.
|
|
||||||
|
|
||||||
use std::error::Error;
|
|
||||||
|
|
||||||
use john_slop::{AnalysisContext, Config, Detector};
|
|
||||||
|
|
||||||
fn main() -> Result<(), Box<dyn Error>> {
|
|
||||||
let source = "As an AI language model, I can help—but this draft needs review.";
|
|
||||||
let detector = Detector::new(&Config::embedded())?;
|
|
||||||
let analysis = detector.analyze_with(source, &AnalysisContext::for_path("article.md"));
|
|
||||||
let summary = analysis.summary();
|
|
||||||
|
|
||||||
println!(
|
|
||||||
"evidence_score={} band={} findings={}",
|
|
||||||
summary.evidence_score,
|
|
||||||
summary
|
|
||||||
.evidence_band
|
|
||||||
.map_or("none", john_slop::EvidenceBand::as_str),
|
|
||||||
summary.findings.total()
|
|
||||||
);
|
|
||||||
|
|
||||||
for finding in analysis.findings {
|
|
||||||
println!(
|
|
||||||
"{} {}..{} {:?}",
|
|
||||||
finding.rule_id, finding.byte_span.start, finding.byte_span.end, finding.matched_text
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
Fullwidth ASCII showcase:
|
|
||||||
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
Fullwidth ASCII showcase:
|
|
||||||
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
|
|
||||||
|
|
@ -1,50 +0,0 @@
|
||||||
//! Showcase embedded width folding for every printable ASCII target.
|
|
||||||
|
|
||||||
use std::io::Write;
|
|
||||||
|
|
||||||
use john_slop::{Config, apply_replacements, plan_replacements};
|
|
||||||
|
|
||||||
const SOURCE: &str = include_str!("ascii-replacement-showcase/fullwidth.txt");
|
|
||||||
|
|
||||||
fn showcase_config() -> Config {
|
|
||||||
Config::embedded()
|
|
||||||
}
|
|
||||||
|
|
||||||
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
|
||||||
let config = showcase_config();
|
|
||||||
let edits = plan_replacements(&config, SOURCE);
|
|
||||||
let formatted = apply_replacements(SOURCE, &edits)?;
|
|
||||||
|
|
||||||
eprintln!(
|
|
||||||
"planned {} embedded replacements covering ASCII space through tilde",
|
|
||||||
edits.len()
|
|
||||||
);
|
|
||||||
std::io::stdout().write_all(formatted.as_bytes())?;
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use std::collections::BTreeSet;
|
|
||||||
|
|
||||||
use super::*;
|
|
||||||
|
|
||||||
const EXPECTED: &str = include_str!("ascii-replacement-showcase/ascii.txt");
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn showcase_maps_each_printable_ascii_target_once() {
|
|
||||||
let config = showcase_config();
|
|
||||||
let edits = plan_replacements(&config, SOURCE);
|
|
||||||
let targets = edits
|
|
||||||
.iter()
|
|
||||||
.map(|edit| edit.replacement.as_bytes()[0])
|
|
||||||
.collect::<BTreeSet<_>>();
|
|
||||||
|
|
||||||
assert_eq!(edits.len(), 95);
|
|
||||||
assert_eq!(targets, (b' '..=b'~').collect());
|
|
||||||
assert_eq!(
|
|
||||||
apply_replacements(SOURCE, &edits).expect("apply non-overlapping showcase edits"),
|
|
||||||
EXPECTED
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,35 +0,0 @@
|
||||||
# Commented configuration examples
|
|
||||||
|
|
||||||
Copy [`config.toml`](config.toml) to the platform path printed by
|
|
||||||
`john-slop config paths` for user-wide policy. Copy [`editorial.toml`](editorial.toml)
|
|
||||||
beside it and select it with `--config-name editorial` for a named overlay.
|
|
||||||
|
|
||||||
Both are deliberately partial. Embedded defaults remain active for fields and
|
|
||||||
stable IDs they do not mention. Inspect the exact merged result before enforcing
|
|
||||||
it:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config show --config-name editorial
|
|
||||||
```
|
|
||||||
|
|
||||||
Project-local policy can use either file with `--config path/to/file.toml`.
|
|
||||||
The CLI can also create the embedded commented base in the current directory:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop config init
|
|
||||||
```
|
|
||||||
|
|
||||||
It preserves an existing `config.toml` unless `--force` is explicit.
|
|
||||||
|
|
||||||
[`diagnostics.toml`](diagnostics.toml) is a complete runtime rule definition.
|
|
||||||
Place it at the `user_diagnostics` path reported by `john-slop config paths` for
|
|
||||||
automatic loading, or keep it with a project and select it explicitly:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
john-slop --diagnostics examples/config/diagnostics.toml article.md
|
|
||||||
john-slop-lsp --diagnostics examples/config/diagnostics.toml
|
|
||||||
```
|
|
||||||
|
|
||||||
Generate the commented starter with `john-slop config init-diagnostics`. The
|
|
||||||
starter rule is disabled until its phrase and metadata are intentionally
|
|
||||||
customized.
|
|
||||||
|
|
@ -1,36 +0,0 @@
|
||||||
# User-wide base layer. `schema_version` is optional in overlays, but spelling
|
|
||||||
# it out makes compatibility expectations explicit.
|
|
||||||
schema_version = 1
|
|
||||||
|
|
||||||
[diagnostics.categories]
|
|
||||||
# Category gates can silence a complete diagnostic family. All four categories
|
|
||||||
# (`typography`, `style`, `prompt-leakage`, and `source-code`) default to true.
|
|
||||||
style = true
|
|
||||||
prompt-leakage = true
|
|
||||||
|
|
||||||
[input]
|
|
||||||
# Bound each decoded document to 4 MiB for this installation.
|
|
||||||
max_document_bytes = 4194304
|
|
||||||
|
|
||||||
[thresholds]
|
|
||||||
# Evidence bands affect labels and `--fail-level`; they are not probabilities.
|
|
||||||
low = 20
|
|
||||||
medium = 50
|
|
||||||
high = 75
|
|
||||||
|
|
||||||
# Arrays merge by stable ID. A known ID needs only the fields being changed.
|
|
||||||
[[replacements]]
|
|
||||||
id = "em-dash"
|
|
||||||
# Keep typographic em dashes instead of offering an ASCII replacement.
|
|
||||||
enabled = false
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "templated-conclusion"
|
|
||||||
# Disable one rule even while its `style` category remains enabled.
|
|
||||||
enabled = false
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "source-placeholder-todo"
|
|
||||||
# Severity changes presentation; weight changes evidence-score contribution.
|
|
||||||
severity = "error"
|
|
||||||
weight = 15.0
|
|
||||||
|
|
@ -1,24 +0,0 @@
|
||||||
# Runtime diagnostic catalog. Add data-driven rules here without rebuilding
|
|
||||||
# John Slop, then pass this file with `--diagnostics`.
|
|
||||||
schema_version = 1
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "custom-review-marker"
|
|
||||||
category = "style"
|
|
||||||
severity = "warning"
|
|
||||||
matcher = "exact-phrase"
|
|
||||||
pattern = "REVIEW BEFORE PUBLISHING"
|
|
||||||
weight = 8.0
|
|
||||||
message = "a project review marker remains in the source"
|
|
||||||
help = "Resolve or remove the marker before publishing."
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = false
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose", "source-code"]
|
|
||||||
extensions = []
|
|
||||||
include = []
|
|
||||||
exclude = []
|
|
||||||
skip_generated = true
|
|
||||||
skip_vendor = true
|
|
||||||
skip_minified = true
|
|
||||||
|
|
@ -1,28 +0,0 @@
|
||||||
# Named overlay selected with `--config-name editorial`. It is applied after the
|
|
||||||
# user-wide config.toml and before an explicit `--config` file.
|
|
||||||
|
|
||||||
[thresholds]
|
|
||||||
low = 15
|
|
||||||
medium = 40
|
|
||||||
high = 70
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "templated-filler"
|
|
||||||
weight = 2.0
|
|
||||||
|
|
||||||
# Scope is one value, so supplied arrays replace the embedded scope arrays.
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-self-identification"
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["md", "txt"]
|
|
||||||
include = ["articles/**", "docs/**"]
|
|
||||||
exclude = ["docs/quoted-transcripts/**"]
|
|
||||||
skip_generated = true
|
|
||||||
skip_vendor = true
|
|
||||||
skip_minified = true
|
|
||||||
|
|
||||||
[[replacements]]
|
|
||||||
id = "en-dash"
|
|
||||||
enabled = false
|
|
||||||
|
|
@ -1,11 +0,0 @@
|
||||||
# This example-only override demonstrates Miette's error presentation. The
|
|
||||||
# embedded catalog otherwise classifies this finding as a warning.
|
|
||||||
[[rules]]
|
|
||||||
id = "assistant-self-identification"
|
|
||||||
severity = "error"
|
|
||||||
|
|
||||||
# Let the companion source fixture display every embedded replacement character
|
|
||||||
# instead of stopping at the production catalog's normal per-document cap.
|
|
||||||
[[rules]]
|
|
||||||
id = "unicode-ascii-replacement"
|
|
||||||
occurrence_cap = 10
|
|
||||||
|
|
@ -1,12 +0,0 @@
|
||||||
User: Write a polished summary without checking any sources.
|
|
||||||
Assistant: Certainly! Here's a comprehensive answer.
|
|
||||||
|
|
||||||
As an AI language model, I cannot browse the internet. My knowledge cutoff may
|
|
||||||
therefore make this inaccurate. It is important to note that the result is
|
|
||||||
seamless—production-ready, even.
|
|
||||||
|
|
||||||
Moreover, the design is robust.
|
|
||||||
Furthermore, the implementation is scalable.
|
|
||||||
Additionally, the solution is intuitive.
|
|
||||||
|
|
||||||
In conclusion, it is important to use this result as-is.
|
|
||||||
|
|
@ -1,7 +0,0 @@
|
||||||
fn main() {
|
|
||||||
// First, we declare an intentionally incomplete showcase value.
|
|
||||||
// Next, we leave the implementation placeholder visible for review.
|
|
||||||
// TODO: implement
|
|
||||||
let _replacement_characters = "‐ ‑ ‒ – — ― − ﹘ ﹣ -";
|
|
||||||
let _unicode_review_only = "pаssword";
|
|
||||||
}
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
Certainly! Here's a polished summary—ready to publish.
|
|
||||||
As an AI language model, I cannot browse the internet.
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
Certainly! Here's a polished summary-ready to publish.
|
|
||||||
As an AI language model, I cannot browse the internet.
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
The release checklist names an owner, a deadline, and a verification command.
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
The release checklist names an owner, a deadline, and a verification command.
|
|
||||||
|
|
@ -1,51 +0,0 @@
|
||||||
schema_version = 1
|
|
||||||
targets = ["cli", "fmt", "lsp"]
|
|
||||||
|
|
||||||
[[cases]]
|
|
||||||
path = "clean/example.txt"
|
|
||||||
formatted = "clean/formatted.txt"
|
|
||||||
expected_rules = []
|
|
||||||
expected_replacements = 0
|
|
||||||
|
|
||||||
[[cases]]
|
|
||||||
path = "assistant/example.txt"
|
|
||||||
formatted = "assistant/formatted.txt"
|
|
||||||
expected_rules = [
|
|
||||||
"assistant-answer-preface",
|
|
||||||
"assistant-capability-disclaimer",
|
|
||||||
"assistant-self-identification",
|
|
||||||
"unicode-ascii-replacement",
|
|
||||||
]
|
|
||||||
expected_replacements = 1
|
|
||||||
|
|
||||||
[[cases]]
|
|
||||||
path = "style/example.txt"
|
|
||||||
formatted = "style/formatted.txt"
|
|
||||||
expected_rules = [
|
|
||||||
"repeated-transition",
|
|
||||||
"templated-conclusion",
|
|
||||||
"templated-filler",
|
|
||||||
]
|
|
||||||
expected_replacements = 0
|
|
||||||
|
|
||||||
[[cases]]
|
|
||||||
path = "typography/example.txt"
|
|
||||||
formatted = "typography/formatted.txt"
|
|
||||||
expected_rules = ["unicode-ascii-replacement"]
|
|
||||||
expected_replacements = 3
|
|
||||||
|
|
||||||
[[cases]]
|
|
||||||
path = "unicode-review/example.txt"
|
|
||||||
formatted = "unicode-review/formatted.txt"
|
|
||||||
expected_rules = ["unicode-ascii-replacement", "unicode-invisible-control"]
|
|
||||||
expected_replacements = 7
|
|
||||||
|
|
||||||
[[cases]]
|
|
||||||
path = "source/example.rs"
|
|
||||||
formatted = "source/formatted.rs"
|
|
||||||
expected_rules = [
|
|
||||||
"source-placeholder-todo",
|
|
||||||
"source-tutorial-narration",
|
|
||||||
"unicode-ascii-replacement",
|
|
||||||
]
|
|
||||||
expected_replacements = 1
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
fn parse() {
|
|
||||||
// First, we create the parser.
|
|
||||||
// TODO: implement the parser
|
|
||||||
let status = "draft—only";
|
|
||||||
}
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
fn parse() {
|
|
||||||
// First, we create the parser.
|
|
||||||
// TODO: implement the parser
|
|
||||||
let status = "draft-only";
|
|
||||||
}
|
|
||||||
|
|
@ -1,4 +0,0 @@
|
||||||
Moreover, the draft repeats a transition.
|
|
||||||
Furthermore, it repeats the same pattern.
|
|
||||||
It is important to note that the claim still needs a source.
|
|
||||||
In conclusion, we can publish it without adding specifics.
|
|
||||||
|
|
@ -1,4 +0,0 @@
|
||||||
Moreover, the draft repeats a transition.
|
|
||||||
Furthermore, it repeats the same pattern.
|
|
||||||
It is important to note that the claim still needs a source.
|
|
||||||
In conclusion, we can publish it without adding specifics.
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
Résumé — coût 5−2; portée – limitée.
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
Résumé - coût 5-2; portée - limitée.
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
The pаssword contains lookalike and invisible characters.
|
|
||||||
Catalog sample: а𝐀!𝟘ⅿ🯹.
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
The password contains lookalike and invisible characters.
|
|
||||||
Catalog sample: aA!Orn9.
|
|
||||||
|
|
@ -1,18 +0,0 @@
|
||||||
//! Inspect the edits shared by detector, formatter, and LSP adapters.
|
|
||||||
|
|
||||||
use john_slop::{Config, apply_replacements, plan_replacements};
|
|
||||||
|
|
||||||
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
|
||||||
let source = "Review—then subtract 5−2.";
|
|
||||||
let edits = plan_replacements(&Config::embedded(), source);
|
|
||||||
|
|
||||||
println!("source: {source}");
|
|
||||||
for edit in &edits {
|
|
||||||
println!(
|
|
||||||
"replace bytes {}..{} with {:?}",
|
|
||||||
edit.byte_span.start, edit.byte_span.end, edit.replacement
|
|
||||||
);
|
|
||||||
}
|
|
||||||
println!("result: {}", apply_replacements(source, &edits)?);
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,52 +0,0 @@
|
||||||
[package]
|
|
||||||
name = "john-slop-fuzz"
|
|
||||||
version = "0.0.0"
|
|
||||||
publish = false
|
|
||||||
edition = "2024"
|
|
||||||
|
|
||||||
[package.metadata]
|
|
||||||
cargo-fuzz = true
|
|
||||||
|
|
||||||
[dependencies]
|
|
||||||
john-slop = { path = "..", default-features = false, features = ["lsp"] }
|
|
||||||
libfuzzer-sys = "0.4.13"
|
|
||||||
tempfile = "3.27.0"
|
|
||||||
toml = "1.1.6"
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "config-merge"
|
|
||||||
path = "fuzz_targets/config_merge.rs"
|
|
||||||
test = false
|
|
||||||
doc = false
|
|
||||||
bench = false
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "rule-loading"
|
|
||||||
path = "fuzz_targets/rule_loading.rs"
|
|
||||||
test = false
|
|
||||||
doc = false
|
|
||||||
bench = false
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "utf8-span-mapping"
|
|
||||||
path = "fuzz_targets/utf8_span_mapping.rs"
|
|
||||||
test = false
|
|
||||||
doc = false
|
|
||||||
bench = false
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "replacement-planning"
|
|
||||||
path = "fuzz_targets/replacement_planning.rs"
|
|
||||||
test = false
|
|
||||||
doc = false
|
|
||||||
bench = false
|
|
||||||
|
|
||||||
[[bin]]
|
|
||||||
name = "archive-metadata"
|
|
||||||
path = "fuzz_targets/archive_metadata.rs"
|
|
||||||
test = false
|
|
||||||
doc = false
|
|
||||||
bench = false
|
|
||||||
|
|
||||||
[workspace]
|
|
||||||
members = ["."]
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
# Fuzzing
|
|
||||||
|
|
||||||
The five harnesses exercise configuration layering, complete rule loading,
|
|
||||||
UTF-8-to-UTF-16 span conversion, replacement planning/application, and hostile
|
|
||||||
ZIP bytes under deliberately small resource limits.
|
|
||||||
|
|
||||||
Install `cargo-fuzz` and run one target from the repository root:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
cargo install cargo-fuzz --locked
|
|
||||||
cargo fuzz run config-merge
|
|
||||||
cargo fuzz run rule-loading
|
|
||||||
cargo fuzz run utf8-span-mapping
|
|
||||||
cargo fuzz run replacement-planning
|
|
||||||
cargo fuzz run archive-metadata
|
|
||||||
```
|
|
||||||
|
|
||||||
Checked-in seeds cover a valid partial override, a complete rule catalog,
|
|
||||||
multibyte/CRLF text, overlapping replacement opportunities, and malformed ZIP
|
|
||||||
metadata. Crashes and minimized artifacts are local build output and must not be
|
|
||||||
committed until converted into a reviewed regression test.
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
PKtruncated-central-directory
|
|
||||||
|
|
@ -1,3 +0,0 @@
|
||||||
[[rules]]
|
|
||||||
id = "templated-filler"
|
|
||||||
enabled = false
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
one—two–three−four‐five
|
|
||||||
|
|
@ -1,31 +0,0 @@
|
||||||
schema_version = 1
|
|
||||||
replacements = []
|
|
||||||
|
|
||||||
[thresholds]
|
|
||||||
low = 20
|
|
||||||
medium = 50
|
|
||||||
high = 75
|
|
||||||
|
|
||||||
[input]
|
|
||||||
max_document_bytes = 1024
|
|
||||||
max_archive_bytes = 2048
|
|
||||||
max_expanded_bytes = 4096
|
|
||||||
max_archive_entries = 4
|
|
||||||
max_compression_ratio = 10
|
|
||||||
max_walk_depth = 4
|
|
||||||
|
|
||||||
[[rules]]
|
|
||||||
id = "seed-rule"
|
|
||||||
category = "style"
|
|
||||||
severity = "advice"
|
|
||||||
matcher = "regex"
|
|
||||||
pattern = "seed+"
|
|
||||||
weight = 1.0
|
|
||||||
message = "seed match"
|
|
||||||
help = "review the seed"
|
|
||||||
occurrence_cap = 1
|
|
||||||
enabled = true
|
|
||||||
|
|
||||||
[rules.scope]
|
|
||||||
content = ["prose"]
|
|
||||||
extensions = ["txt"]
|
|
||||||
|
|
@ -1,2 +0,0 @@
|
||||||
alpha
|
|
||||||
é😀é
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
#![no_main]
|
|
||||||
|
|
||||||
use std::fs;
|
|
||||||
|
|
||||||
use john_slop::{Config, read_inputs};
|
|
||||||
use libfuzzer_sys::fuzz_target;
|
|
||||||
|
|
||||||
fuzz_target!(|data: &[u8]| {
|
|
||||||
let Ok(directory) = tempfile::tempdir() else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let path = directory.path().join("input.zip");
|
|
||||||
if fs::write(&path, data).is_err() {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
let mut config = Config::embedded();
|
|
||||||
config.input.max_archive_bytes = 64 * 1024;
|
|
||||||
config.input.max_expanded_bytes = 128 * 1024;
|
|
||||||
config.input.max_document_bytes = 16 * 1024;
|
|
||||||
config.input.max_archive_entries = 32;
|
|
||||||
config.input.max_compression_ratio = 20;
|
|
||||||
if let Ok(documents) = read_inputs(&config, &path) {
|
|
||||||
for document in documents.take(33) {
|
|
||||||
let _ = document;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
@ -1,23 +0,0 @@
|
||||||
#![no_main]
|
|
||||||
|
|
||||||
use std::fs;
|
|
||||||
|
|
||||||
use john_slop::{ConfigLoader, ConfigPaths};
|
|
||||||
use libfuzzer_sys::fuzz_target;
|
|
||||||
|
|
||||||
fuzz_target!(|data: &[u8]| {
|
|
||||||
let Ok(directory) = tempfile::tempdir() else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let explicit = directory.path().join("fuzz.toml");
|
|
||||||
if fs::write(&explicit, data).is_err() {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
let loader = ConfigLoader::from_paths(ConfigPaths {
|
|
||||||
directory: directory.path().to_path_buf(),
|
|
||||||
base: directory.path().join("absent-base.toml"),
|
|
||||||
named: Some(directory.path().join("absent-named.toml")),
|
|
||||||
explicit: Some(explicit),
|
|
||||||
});
|
|
||||||
let _ = loader.load();
|
|
||||||
});
|
|
||||||
|
|
@ -1,17 +0,0 @@
|
||||||
#![no_main]
|
|
||||||
|
|
||||||
use john_slop::{Config, apply_replacements, plan_replacements};
|
|
||||||
use libfuzzer_sys::fuzz_target;
|
|
||||||
|
|
||||||
fuzz_target!(|data: &[u8]| {
|
|
||||||
let Ok(source) = std::str::from_utf8(data) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let config = Config::embedded();
|
|
||||||
let edits = plan_replacements(&config, source);
|
|
||||||
let Ok(formatted) = apply_replacements(source, &edits) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let second = plan_replacements(&config, &formatted);
|
|
||||||
let _ = apply_replacements(&formatted, &second);
|
|
||||||
});
|
|
||||||
|
|
@ -1,16 +0,0 @@
|
||||||
#![no_main]
|
|
||||||
|
|
||||||
use john_slop::{Config, Detector};
|
|
||||||
use libfuzzer_sys::fuzz_target;
|
|
||||||
|
|
||||||
fuzz_target!(|data: &[u8]| {
|
|
||||||
let Ok(source) = std::str::from_utf8(data) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let Ok(config) = toml::from_str::<Config>(source) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
if config.validate().is_ok() {
|
|
||||||
let _ = Detector::new(&config);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
@ -1,19 +0,0 @@
|
||||||
#![no_main]
|
|
||||||
|
|
||||||
use john_slop::lsp::LineIndex;
|
|
||||||
use libfuzzer_sys::fuzz_target;
|
|
||||||
|
|
||||||
fuzz_target!(|data: &[u8]| {
|
|
||||||
let Ok(source) = std::str::from_utf8(data) else {
|
|
||||||
return;
|
|
||||||
};
|
|
||||||
let index = LineIndex::new(source);
|
|
||||||
for offset in 0..=source.len().saturating_add(1) {
|
|
||||||
let _ = index.position(offset);
|
|
||||||
}
|
|
||||||
if !source.is_empty() {
|
|
||||||
let midpoint = source.len() / 2;
|
|
||||||
let _ = index.range(midpoint..source.len());
|
|
||||||
let _ = index.range(source.len()..midpoint);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
43
release.sh
43
release.sh
|
|
@ -1,43 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
echo "usage: $0 <gitea-token-file> [rust_realease options...]" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
|
|
||||||
[ "$#" -ge 1 ] || usage
|
|
||||||
token_file=$1
|
|
||||||
shift
|
|
||||||
|
|
||||||
if [ ! -f "$token_file" ] || [ ! -r "$token_file" ]; then
|
|
||||||
echo "token file is not a readable regular file: $token_file" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
if ! command -v rust_realease >/dev/null 2>&1; then
|
|
||||||
echo "rust_realease is not installed or not available in PATH" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
GITEA_TOKEN=
|
|
||||||
IFS= read -r GITEA_TOKEN <"$token_file" || :
|
|
||||||
if [ -z "$GITEA_TOKEN" ]; then
|
|
||||||
echo "token file is empty: $token_file" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
export GITEA_TOKEN
|
|
||||||
|
|
||||||
repository_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
|
||||||
manifest_version=$(sed -n 's/^version = "\([^"]*\)"/\1/p' "$repository_dir/Cargo.toml" | head -n 1)
|
|
||||||
|
|
||||||
set -- \
|
|
||||||
"$repository_dir" \
|
|
||||||
--package john-slop \
|
|
||||||
--artifact-format both \
|
|
||||||
"$@"
|
|
||||||
case "$manifest_version" in
|
|
||||||
*-*) set -- "$@" --prerelease ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
exec rust_realease "$@"
|
|
||||||
|
|
@ -1,3 +0,0 @@
|
||||||
[[repository.fallbacks]]
|
|
||||||
origin = "git@git.dandaedre.com:idle-org/john-slop.git"
|
|
||||||
fallback-url = "https://proxy.dandaedre.com/gitea/idle-org/john-slop"
|
|
||||||
|
|
@ -1,270 +0,0 @@
|
||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
usage() {
|
|
||||||
cat >&2 <<'EOF'
|
|
||||||
usage: $0 <version|test|coverage> [options]
|
|
||||||
|
|
||||||
Render one deterministic SVG badge. Defaults:
|
|
||||||
version --text version --value unknown --color #007ec6
|
|
||||||
test --text tests --value passing --color #4c1
|
|
||||||
coverage --text coverage --value - --color #9f9f9f --suffix %
|
|
||||||
|
|
||||||
Options:
|
|
||||||
--text TEXT left-hand label
|
|
||||||
--value VALUE right-hand value
|
|
||||||
--color COLOR value background (#RGB or #RRGGBB)
|
|
||||||
--size PX font size, from 6 through 24 (default: 11)
|
|
||||||
--height PX badge height, from 14 through 64 (default: 20)
|
|
||||||
--label-width PX fixed label width instead of automatic sizing
|
|
||||||
--value-width PX fixed value width instead of automatic sizing
|
|
||||||
--suffix TEXT text appended to the displayed value
|
|
||||||
--output PATH output file (default: coverage/<kind>.svg)
|
|
||||||
-h, --help show this help
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
./ci-coverage-badges.sh version --value 1.2.3
|
|
||||||
./ci-coverage-badges.sh test --value failing
|
|
||||||
./ci-coverage-badges.sh coverage --value 87.4
|
|
||||||
./ci-coverage-badges.sh version --text release --value rc.12 --size 12 --color '#8a2be2'
|
|
||||||
EOF
|
|
||||||
exit "${1:-2}"
|
|
||||||
}
|
|
||||||
|
|
||||||
[ "$#" -ge 1 ] || usage
|
|
||||||
kind=$1
|
|
||||||
shift
|
|
||||||
|
|
||||||
case "$kind" in
|
|
||||||
version)
|
|
||||||
text=version
|
|
||||||
value=unknown
|
|
||||||
color='#007ec6'
|
|
||||||
suffix=
|
|
||||||
output=coverage/version.svg
|
|
||||||
;;
|
|
||||||
test)
|
|
||||||
text=tests
|
|
||||||
value=passing
|
|
||||||
color=
|
|
||||||
suffix=
|
|
||||||
output=coverage/test.svg
|
|
||||||
;;
|
|
||||||
coverage)
|
|
||||||
text=coverage
|
|
||||||
value=-
|
|
||||||
color=
|
|
||||||
suffix=%
|
|
||||||
output=coverage/coverage.svg
|
|
||||||
;;
|
|
||||||
-h|--help) usage 0 ;;
|
|
||||||
*)
|
|
||||||
echo "unsupported badge kind: $kind" >&2
|
|
||||||
usage
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
font_size=11
|
|
||||||
height=20
|
|
||||||
label_width=
|
|
||||||
value_width=
|
|
||||||
color_set=false
|
|
||||||
|
|
||||||
while [ "$#" -gt 0 ]; do
|
|
||||||
option=$1
|
|
||||||
case "$option" in
|
|
||||||
--text|--value|--color|--size|--height|--label-width|--value-width|--suffix|--output)
|
|
||||||
[ "$#" -ge 2 ] || {
|
|
||||||
echo "missing value for $option" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
option_value=$2
|
|
||||||
shift 2
|
|
||||||
case "$option" in
|
|
||||||
--text) text=$option_value ;;
|
|
||||||
--value) value=$option_value ;;
|
|
||||||
--color) color=$option_value; color_set=true ;;
|
|
||||||
--size) font_size=$option_value ;;
|
|
||||||
--height) height=$option_value ;;
|
|
||||||
--label-width) label_width=$option_value ;;
|
|
||||||
--value-width) value_width=$option_value ;;
|
|
||||||
--suffix) suffix=$option_value ;;
|
|
||||||
--output) output=$option_value ;;
|
|
||||||
esac
|
|
||||||
;;
|
|
||||||
-h|--help) usage 0 ;;
|
|
||||||
*)
|
|
||||||
echo "unknown option: $option" >&2
|
|
||||||
usage
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
validate_text() {
|
|
||||||
field_name=$1
|
|
||||||
field_value=$2
|
|
||||||
if ! printf '%sX\n' "$field_value" | LC_ALL=C awk '
|
|
||||||
BEGIN { valid = 1 }
|
|
||||||
NR != 1 || $0 !~ /^[ -~]+X$/ || length($0) <= 1 { valid = 0 }
|
|
||||||
END { exit(valid == 0 ? 1 : 0) }
|
|
||||||
'; then
|
|
||||||
echo "$field_name must contain printable ASCII characters: $field_value" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
validate_integer() {
|
|
||||||
field_name=$1
|
|
||||||
field_value=$2
|
|
||||||
minimum=$3
|
|
||||||
maximum=$4
|
|
||||||
case "$field_value" in
|
|
||||||
''|*[!0-9]*)
|
|
||||||
echo "$field_name must be an integer from $minimum through $maximum: $field_value" >&2
|
|
||||||
exit 2
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
if [ "$field_value" -lt "$minimum" ] || [ "$field_value" -gt "$maximum" ]; then
|
|
||||||
echo "$field_name must be an integer from $minimum through $maximum: $field_value" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
}
|
|
||||||
|
|
||||||
validate_text text "$text"
|
|
||||||
validate_text value "$value"
|
|
||||||
if [ -n "$suffix" ]; then
|
|
||||||
validate_text suffix "$suffix"
|
|
||||||
fi
|
|
||||||
validate_integer size "$font_size" 6 24
|
|
||||||
validate_integer height "$height" 14 64
|
|
||||||
if [ -n "$label_width" ]; then
|
|
||||||
validate_integer label-width "$label_width" 20 1000
|
|
||||||
fi
|
|
||||||
if [ -n "$value_width" ]; then
|
|
||||||
validate_integer value-width "$value_width" 20 1000
|
|
||||||
fi
|
|
||||||
case "$output" in
|
|
||||||
''|*/) echo "output must name an SVG file: $output" >&2; exit 2 ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
case "$kind" in
|
|
||||||
test)
|
|
||||||
if [ "$color_set" = false ]; then
|
|
||||||
case "$value" in
|
|
||||||
passing) color='#4c1' ;;
|
|
||||||
failing) color='#e05d44' ;;
|
|
||||||
*) color='#9f9f9f' ;;
|
|
||||||
esac
|
|
||||||
fi
|
|
||||||
;;
|
|
||||||
coverage)
|
|
||||||
if [ "$value" = '-' ]; then
|
|
||||||
[ "$color_set" = true ] || color='#9f9f9f'
|
|
||||||
else
|
|
||||||
if ! awk -v candidate="$value" 'BEGIN {
|
|
||||||
valid = candidate ~ /^[0-9]+([.][0-9]+)?$/ && candidate >= 0 && candidate <= 100
|
|
||||||
exit(valid ? 0 : 1)
|
|
||||||
}'; then
|
|
||||||
echo "coverage value must be - or a number from 0 through 100: $value" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
value=$(awk -v candidate="$value" 'BEGIN { printf "%.0f", candidate }')
|
|
||||||
if [ "$color_set" = false ]; then
|
|
||||||
if [ "$value" -ge 90 ]; then
|
|
||||||
color='#4c1'
|
|
||||||
elif [ "$value" -ge 70 ]; then
|
|
||||||
color='#dfb317'
|
|
||||||
else
|
|
||||||
color='#e05d44'
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
case "$color" in
|
|
||||||
\#[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]|\#[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]) ;;
|
|
||||||
*) echo "color must be #RGB or #RRGGBB: $color" >&2; exit 2 ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
display_value=$value$suffix
|
|
||||||
if [ -z "$label_width" ]; then
|
|
||||||
label_width=$(printf '%s\\n' "$text" | awk -v size="$font_size" '{
|
|
||||||
width = int(length($0) * size * 0.64 + 14)
|
|
||||||
print (width < 35 ? 35 : width)
|
|
||||||
}')
|
|
||||||
fi
|
|
||||||
if [ -z "$value_width" ]; then
|
|
||||||
value_width=$(printf '%s\\n' "$display_value" | awk -v size="$font_size" '{
|
|
||||||
width = int(length($0) * size * 0.64 + 14)
|
|
||||||
print (width < 35 ? 35 : width)
|
|
||||||
}')
|
|
||||||
fi
|
|
||||||
|
|
||||||
badge_width=$((label_width + value_width))
|
|
||||||
label_center=$(awk -v width="$label_width" 'BEGIN { printf "%.1f", width / 2 }')
|
|
||||||
value_center=$(awk -v left="$label_width" -v width="$value_width" 'BEGIN {
|
|
||||||
printf "%.1f", left + width / 2
|
|
||||||
}')
|
|
||||||
vertical_center=$(awk -v badge_height="$height" 'BEGIN { printf "%.1f", badge_height / 2 }')
|
|
||||||
|
|
||||||
xml_escape() {
|
|
||||||
sed \
|
|
||||||
-e 's/&/\&/g' \
|
|
||||||
-e 's/</\</g' \
|
|
||||||
-e 's/>/\>/g' \
|
|
||||||
-e 's/"/\"/g'
|
|
||||||
}
|
|
||||||
|
|
||||||
sed_replacement_escape() {
|
|
||||||
sed -e 's/[\\&|]/\\&/g'
|
|
||||||
}
|
|
||||||
|
|
||||||
escaped_text=$(printf '%s' "$text" | xml_escape | sed_replacement_escape)
|
|
||||||
escaped_value=$(printf '%s' "$display_value" | xml_escape | sed_replacement_escape)
|
|
||||||
escaped_title=$(printf '%s: %s' "$text" "$display_value" | xml_escape | sed_replacement_escape)
|
|
||||||
|
|
||||||
output_dir=${output%/*}
|
|
||||||
if [ "$output_dir" != "$output" ]; then
|
|
||||||
mkdir -p "$output_dir"
|
|
||||||
fi
|
|
||||||
temporary="${output}.tmp.$$"
|
|
||||||
trap 'rm -f "$temporary"' EXIT HUP INT TERM
|
|
||||||
|
|
||||||
sed \
|
|
||||||
-e "s/@BADGE_WIDTH@/$badge_width/g" \
|
|
||||||
-e "s/@HEIGHT@/$height/g" \
|
|
||||||
-e "s/@LABEL_WIDTH@/$label_width/g" \
|
|
||||||
-e "s/@VALUE_WIDTH@/$value_width/g" \
|
|
||||||
-e "s/@LABEL_CENTER@/$label_center/g" \
|
|
||||||
-e "s/@VALUE_CENTER@/$value_center/g" \
|
|
||||||
-e "s/@VERTICAL_CENTER@/$vertical_center/g" \
|
|
||||||
-e "s/@FONT_SIZE@/$font_size/g" \
|
|
||||||
-e "s/@COLOR@/$color/g" \
|
|
||||||
-e "s|@TITLE@|$escaped_title|g" \
|
|
||||||
-e "s|@TEXT@|$escaped_text|g" \
|
|
||||||
-e "s|@VALUE@|$escaped_value|g" \
|
|
||||||
>"$temporary" <<'EOF'
|
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" width="@BADGE_WIDTH@" height="@HEIGHT@" role="img" aria-label="@TITLE@">
|
|
||||||
<title>@TITLE@</title>
|
|
||||||
<linearGradient id="s" x2="0" y2="100%">
|
|
||||||
<stop offset="0" stop-color="#fff" stop-opacity=".7"/>
|
|
||||||
<stop offset=".1" stop-color="#aaa" stop-opacity=".1"/>
|
|
||||||
<stop offset=".9" stop-opacity=".3"/>
|
|
||||||
<stop offset="1" stop-opacity=".5"/>
|
|
||||||
</linearGradient>
|
|
||||||
<clipPath id="r"><rect width="@BADGE_WIDTH@" height="@HEIGHT@" rx="3"/></clipPath>
|
|
||||||
<g clip-path="url(#r)">
|
|
||||||
<rect width="@LABEL_WIDTH@" height="@HEIGHT@" fill="#555"/>
|
|
||||||
<rect x="@LABEL_WIDTH@" width="@VALUE_WIDTH@" height="@HEIGHT@" fill="@COLOR@"/>
|
|
||||||
<rect width="@BADGE_WIDTH@" height="@HEIGHT@" fill="url(#s)"/>
|
|
||||||
</g>
|
|
||||||
<g fill="#fff" text-anchor="middle" dominant-baseline="middle" font-family="Verdana,Geneva,DejaVu Sans,sans-serif" font-size="@FONT_SIZE@">
|
|
||||||
<text x="@LABEL_CENTER@" y="@VERTICAL_CENTER@">@TEXT@</text>
|
|
||||||
<text x="@VALUE_CENTER@" y="@VERTICAL_CENTER@">@VALUE@</text>
|
|
||||||
</g>
|
|
||||||
</svg>
|
|
||||||
EOF
|
|
||||||
|
|
||||||
mv "$temporary" "$output"
|
|
||||||
trap - EXIT HUP INT TERM
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue