john-slop/docs/unicode-confusables.md

73 lines
4.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested
"ambiguous Unicode characters" catalog:
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
is the generator input;
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
are easier to browse as code;
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
combines a locale-specific table with `_common` at runtime; and
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
records the upstream source and a separate invisible-rune catalog.
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
1,538 pairs; `_default` and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
## Implemented John Slop profile
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
scalar whose single-character skeleton is non-empty printable ASCII, plus nine
reviewed punctuation mappings and 95 compatibility width folds. Width folding
takes precedence when a UTS #39 visual skeleton would produce a different
target, such as fullwidth `I` mapping to `I` rather than `l`. That is 1,884
mappings in the current profile.
Each entry has a stable ID, source, target, rationale, and enabled switch, and
the same asset owns the diagnostic metadata.
The default `[unicode] confusables = "replace"` policy emits
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
`review` for identical findings without edits, or `off` to suppress generated
lookalikes. The reviewed punctuation and width-folding mappings remain active
under all three settings and can still be disabled individually.
This broad mapping is intentionally more aggressive than Gitea's locale-aware
display warning. Greek, Cyrillic, mathematical, and other characters may be
legitimate text, so projects handling multilingual content should prefer
`review` unless ASCII normalization is their explicit policy.
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
pinned commit above, including bidi embedding/override and isolate controls.
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
reported. Each other match receives an exact UTF-8 source span that adapters
convert normally. Controls never feed the formatter or LSP quick fixes because
silent deletion has no generally safe meaning.
Regenerate the checked-in policy deterministically with:
```sh
cargo run -p john-slop-core --example generate_unicode_catalog \
> crates/john-slop-core/assets/unicode.toml
```
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
Future profile updates must pin their upstream Unicode/data version, retain
provenance and licensing, test ordinary multilingual near misses, and renew the
hash-bound default-policy sign-off.