john-slop/docs/unicode-confusables.md

60 lines
3.5 KiB
Markdown

# Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested
"ambiguous Unicode characters" catalog:
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
is the generator input;
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
are easier to browse as code;
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
combines a locale-specific table with `_common` at runtime; and
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
records the upstream source and a separate invisible-rune catalog.
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
1,538 pairs; `_default` and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
## Implemented John Slop profile
These sources are useful detection references, not safe formatter replacement
catalogs. Many listed characters are legitimate in particular languages and
scripts, and Gitea deliberately interprets them through the reviewer's locale.
Importing every pair as an automatic Unicode-to-ASCII edit would corrupt valid
text and violate the formatter's conservative replacement contract.
John Slop keeps three policies separate:
1. configured, explicitly safe formatter replacements;
2. non-mutating confusable findings for human review; and
3. non-mutating invisible and bidirectional-control findings.
`unicode-ascii-confusable` uses `unicode-security` 0.1.2, whose checked-in UTS
#39 tables target Unicode 16.0. It reports a non-ASCII scalar only when its
single-character skeleton is entirely ASCII. It is intentionally broader than
Gitea's locale selection and can flag legitimate scripts; locale-aware policy
remains deferred. Characters already covered by the safe replacement allowlist
produce only the fixable replacement finding, not a duplicate confusable one.
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
pinned commit above, including bidi embedding/override and isolate controls.
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
reported. Each other match receives an exact UTF-8 source span that adapters
convert normally. Neither rule feeds the formatter or LSP quick fixes.
Future profile updates must pin their upstream Unicode/data version, retain
provenance and licensing, test ordinary multilingual near misses, renew the
hash-bound default-policy sign-off, and preserve the no-silent-rewrite boundary.