test: add expected-failure diagnostic showcase

This commit is contained in:
Dandaedre 2026-09-29 17:21:20 +02:00
parent b40112a724
commit eeabed9010
Signed by: Dandaedre
GPG Key ID: 440C6592E8A26D99
6 changed files with 72 additions and 0 deletions

View File

@ -48,6 +48,7 @@ and Vim setup.
## Documentation
- [Runnable Rust examples](examples/README.md)
- [Expected-failure Miette diagnostic showcase](examples/README.md#expected-failure-diagnostic-showcase)
- [Documentation index](docs/README.md)
- [Command-line usage and current capabilities](docs/usage.md)
- [Helix and Vim formatter/LSP setup](docs/editors.md)

View File

@ -75,6 +75,9 @@ Its [shared fixture manifest](../examples/fixtures/manifest.toml) pairs several
`example.txt` inputs with expected detector rules, replacement counts, and
formatted output. The executable fixture and protocol tests validate the
library, detector, formatter, and LSP against the shared replacement contract.
The separate [expected-failure diagnostic showcase](../examples/README.md#expected-failure-diagnostic-showcase)
prints every current prose finding in graphical Miette form and intentionally
exits `1` when its low confidence threshold is reached.
## Supported input policy

View File

@ -43,3 +43,27 @@ The formatter integration test executes every case and compares stdout with its
expected ASCII file. The manifest also names `lsp` as a consumer, and the
in-process protocol suite proves that applying LSP edits produces the same bytes
as the formatter.
## Expected-failure diagnostic showcase
[`diagnostic-showcase.txt`](diagnostic-showcase.txt) deliberately contains every
kind of currently shipped prose signal. Its adjacent example configuration
promotes assistant self-identification from warning to error, making one run
show Miette advice, warning, and error diagnostics together:
```sh
cargo run --bin john-slop -- \
--config examples/diagnostic-showcase.toml \
--output miette --fail-level low \
examples/diagnostic-showcase.txt
```
That command is expected to exit with status `1`: reaching the configured
threshold is the behavior under demonstration. The repository test asserts the
nonzero result rather than making the suite red. Run it with capture disabled to
print the complete report:
```sh
cargo test --test example_fixtures \
diagnostic_showcase_is_an_expected_threshold_failure -- --nocapture
```

View File

@ -0,0 +1,5 @@
# This example-only override demonstrates Miette's error presentation. The
# embedded catalog otherwise classifies this finding as a warning.
[[rules]]
id = "assistant-self-identification"
severity = "error"

View File

@ -0,0 +1,12 @@
User: Write a polished summary without checking any sources.
Assistant: Certainly! Here's a comprehensive answer.
As an AI language model, I cannot browse the internet. My knowledge cutoff may
therefore make this inaccurate. It is important to note that the result is
seamless—production-ready, even.
Moreover, the design is robust.
Furthermore, the implementation is scalable.
Additionally, the solution is intuitive.
In conclusion, it is important to use this result as-is.

View File

@ -135,3 +135,30 @@ fn formatter_cli_matches_every_shared_expected_file() {
);
}
}
#[test]
fn diagnostic_showcase_is_an_expected_threshold_failure() {
let root = Path::new(env!("CARGO_MANIFEST_DIR"));
let config_home = tempfile::tempdir().expect("temp config home");
let output = Command::new(env!("CARGO_BIN_EXE_john-slop"))
.env("XDG_CONFIG_HOME", config_home.path())
.args(["--output", "miette", "--fail-level", "low", "--config"])
.arg(root.join("examples/diagnostic-showcase.toml"))
.arg(root.join("examples/diagnostic-showcase.txt"))
.output()
.expect("run expected-failure diagnostic showcase");
let stdout = String::from_utf8(output.stdout).expect("UTF-8 diagnostic output");
// `--nocapture` makes this a convenient manual tour of every diagnostic.
print!("{stdout}");
assert_eq!(output.status.code(), Some(1));
assert!(output.stderr.is_empty());
assert!(stdout.contains("assistant-self-identification"));
assert!(stdout.contains("prompt-response-scaffolding"));
assert!(stdout.contains("typography-non-ascii-hyphen"));
assert!(stdout.contains("advice"));
assert!(stdout.contains("warning"));
assert!(stdout.contains("error"));
#[cfg(feature = "fancy-diagnostics")]
assert!(stdout.contains("╭─["));
}