john-slop/docs/unicode-confusables.md

70 lines
3.8 KiB
Markdown

# Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested
"ambiguous Unicode characters" catalog:
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
is the generator input;
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
are easier to browse as code;
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
combines a locale-specific table with `_common` at runtime; and
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
records the upstream source and a separate invisible-rune catalog.
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
1,538 pairs; `_default` and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
## Implemented John Slop profile
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
scalar whose single-character skeleton is non-empty printable ASCII, plus ten
reviewed punctuation mappings. That is 1,829 mappings in the current profile.
Each entry has a stable ID, source, target, rationale, and enabled switch, and
the same asset owns the diagnostic metadata.
The default `[unicode] confusables = "replace"` policy emits
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
`review` for identical findings without edits, or `off` to suppress generated
lookalikes. The ten curated punctuation mappings remain active under all three
settings and can still be disabled individually.
This broad mapping is intentionally more aggressive than Gitea's locale-aware
display warning. Greek, Cyrillic, mathematical, and other characters may be
legitimate text, so projects handling multilingual content should prefer
`review` unless ASCII normalization is their explicit policy.
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
pinned commit above, including bidi embedding/override and isolate controls.
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
reported. Each other match receives an exact UTF-8 source span that adapters
convert normally. Controls never feed the formatter or LSP quick fixes because
silent deletion has no generally safe meaning.
Regenerate the checked-in policy deterministically with:
```sh
cargo run -p john-slop-core --example generate_unicode_catalog \
> crates/john-slop-core/assets/unicode.toml
```
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
Future profile updates must pin their upstream Unicode/data version, retain
provenance and licensing, test ordinary multilingual near misses, and renew the
hash-bound default-policy sign-off.