60 lines
3.5 KiB
Markdown
60 lines
3.5 KiB
Markdown
# Ambiguous and invisible Unicode references
|
|
|
|
Gitea maintains the concrete list that most closely matches the requested
|
|
"ambiguous Unicode characters" catalog:
|
|
|
|
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
|
is the generator input;
|
|
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
|
are easier to browse as code;
|
|
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
|
combines a locale-specific table with `_common` at runtime; and
|
|
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
|
records the upstream source and a separate invisible-rune catalog.
|
|
|
|
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
|
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
|
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
|
The array values alternate between a confusable code point and the code point
|
|
it resembles. Counts across tables are not necessarily unique global mappings.
|
|
|
|
Gitea's generator identifies
|
|
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
|
as its upstream. For standards-based identifier spoofing, the canonical broader
|
|
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
|
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
|
data file. GitHub separately documents a warning for hidden bidirectional text;
|
|
that protection is about bidi controls rather than a published Gitea-style
|
|
locale-aware lookalike table.
|
|
|
|
## Implemented John Slop profile
|
|
|
|
These sources are useful detection references, not safe formatter replacement
|
|
catalogs. Many listed characters are legitimate in particular languages and
|
|
scripts, and Gitea deliberately interprets them through the reviewer's locale.
|
|
Importing every pair as an automatic Unicode-to-ASCII edit would corrupt valid
|
|
text and violate the formatter's conservative replacement contract.
|
|
|
|
John Slop keeps three policies separate:
|
|
|
|
1. configured, explicitly safe formatter replacements;
|
|
2. non-mutating confusable findings for human review; and
|
|
3. non-mutating invisible and bidirectional-control findings.
|
|
|
|
`unicode-ascii-confusable` uses `unicode-security` 0.1.2, whose checked-in UTS
|
|
#39 tables target Unicode 16.0. It reports a non-ASCII scalar only when its
|
|
single-character skeleton is entirely ASCII. It is intentionally broader than
|
|
Gitea's locale selection and can flag legitimate scripts; locale-aware policy
|
|
remains deferred. Characters already covered by the safe replacement allowlist
|
|
produce only the fixable replacement finding, not a duplicate confusable one.
|
|
|
|
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
|
|
pinned commit above, including bidi embedding/override and isolate controls.
|
|
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
|
|
reported. Each other match receives an exact UTF-8 source span that adapters
|
|
convert normally. Neither rule feeds the formatter or LSP quick fixes.
|
|
|
|
Future profile updates must pin their upstream Unicode/data version, retain
|
|
provenance and licensing, test ordinary multilingual near misses, renew the
|
|
hash-bound default-policy sign-off, and preserve the no-silent-rewrite boundary.
|