47 lines
2.7 KiB
Markdown
47 lines
2.7 KiB
Markdown
# Ambiguous and invisible Unicode references
|
|
|
|
Gitea maintains the concrete list that most closely matches the requested
|
|
"ambiguous Unicode characters" catalog:
|
|
|
|
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
|
is the generator input;
|
|
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
|
are easier to browse as code;
|
|
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
|
combines a locale-specific table with `_common` at runtime; and
|
|
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
|
records the upstream source and a separate invisible-rune catalog.
|
|
|
|
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
|
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
|
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
|
The array values alternate between a confusable code point and the code point
|
|
it resembles. Counts across tables are not necessarily unique global mappings.
|
|
|
|
Gitea's generator identifies
|
|
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
|
as its upstream. For standards-based identifier spoofing, the canonical broader
|
|
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
|
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
|
data file. GitHub separately documents a warning for hidden bidirectional text;
|
|
that protection is about bidi controls rather than a published Gitea-style
|
|
locale-aware lookalike table.
|
|
|
|
## Implication for John Slop
|
|
|
|
These sources are useful detection references, not safe formatter replacement
|
|
catalogs. Many listed characters are legitimate in particular languages and
|
|
scripts, and Gitea deliberately interprets them through the reviewer's locale.
|
|
Importing every pair as an automatic Unicode-to-ASCII edit would corrupt valid
|
|
text and violate the formatter's conservative replacement contract.
|
|
|
|
A future implementation should therefore keep three policies separate:
|
|
|
|
1. configured, explicitly safe formatter replacements;
|
|
2. locale-aware confusable findings for human review; and
|
|
3. invisible and bidirectional-control security findings.
|
|
|
|
Any vendored or generated catalog should pin its upstream Unicode/data version,
|
|
retain provenance and licensing, test locale behavior and ordinary multilingual
|
|
near-misses, and report exact source spans without silently rewriting text.
|