4.0 KiB
Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested "ambiguous Unicode characters" catalog:
- vendored
ambiguous.jsonis the generator input; - generated Go tables are easier to browse as code;
- locale selection
combines a locale-specific table with
_commonat runtime; and - the generator records the upstream source and a separate invisible-rune catalog.
At Gitea commit 9c715c0f9d12336e026e89be85f9f711c66e7ff6, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. _common contains
1,538 pairs; _default and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
hediet/vscode-unicode-data
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is Unicode UTS #39 and its
confusables.txt
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
Implemented John Slop profile
John Slop uses unicode-security 0.1.2, whose checked-in UTS #39 tables target
Unicode 16.0. The generated assets/unicode.toml contains every non-ASCII
scalar whose single-character skeleton is non-empty printable ASCII, plus nine
reviewed punctuation mappings and 95 compatibility width folds. Width folding
takes precedence when a UTS #39 visual skeleton would produce a different
target, such as fullwidth I mapping to I rather than l. That is 1,884
mappings in the current profile.
Each entry has a stable ID, source, target, rationale, and enabled switch, and
the same asset owns the diagnostic metadata.
The default [unicode] confusables = "replace" policy emits
unicode-ascii-replacement diagnostics and supplies the same edits to CLI
output, john-slop-fmt, LSP quick fixes, and LSP formatting. Set the value to
review for identical findings without edits, or off to suppress generated
lookalikes. The reviewed punctuation and width-folding mappings remain active
under all three settings and can still be disabled individually.
This broad mapping is intentionally more aggressive than Gitea's locale-aware
display warning. Greek, Cyrillic, mathematical, and other characters may be
legitimate text, so projects handling multilingual content should prefer
review unless ASCII normalization is their explicit policy.
unicode-invisible-control uses the invisible ranges generated by Gitea at the
pinned commit above, including bidi embedding/override and isolate controls.
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
reported. Each other match receives an exact UTF-8 source span that adapters
convert normally. Controls never feed the formatter or LSP quick fixes because
silent deletion has no generally safe meaning.
Regenerate the checked-in policy deterministically with:
cargo run -p john-slop-core --example generate_unicode_catalog \
> crates/john-slop-core/assets/unicode.toml
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
Future profile updates must pin their upstream Unicode/data version, retain provenance and licensing, test ordinary multilingual near misses, and renew the hash-bound default-policy sign-off.