3.8 KiB
Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested "ambiguous Unicode characters" catalog:
- vendored
ambiguous.jsonis the generator input; - generated Go tables are easier to browse as code;
- locale selection
combines a locale-specific table with
_commonat runtime; and - the generator records the upstream source and a separate invisible-rune catalog.
At Gitea commit 9c715c0f9d12336e026e89be85f9f711c66e7ff6, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. _common contains
1,538 pairs; _default and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
hediet/vscode-unicode-data
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is Unicode UTS #39 and its
confusables.txt
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
Implemented John Slop profile
John Slop uses unicode-security 0.1.2, whose checked-in UTS #39 tables target
Unicode 16.0. The generated assets/unicode.toml contains every non-ASCII
scalar whose single-character skeleton is non-empty printable ASCII, plus ten
reviewed punctuation mappings. That is 1,829 mappings in the current profile.
Each entry has a stable ID, source, target, rationale, and enabled switch, and
the same asset owns the diagnostic metadata.
The default [unicode] confusables = "replace" policy emits
unicode-ascii-replacement diagnostics and supplies the same edits to CLI
output, john-slop-fmt, LSP quick fixes, and LSP formatting. Set the value to
review for identical findings without edits, or off to suppress generated
lookalikes. The ten curated punctuation mappings remain active under all three
settings and can still be disabled individually.
This broad mapping is intentionally more aggressive than Gitea's locale-aware
display warning. Greek, Cyrillic, mathematical, and other characters may be
legitimate text, so projects handling multilingual content should prefer
review unless ASCII normalization is their explicit policy.
unicode-invisible-control uses the invisible ranges generated by Gitea at the
pinned commit above, including bidi embedding/override and isolate controls.
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
reported. Each other match receives an exact UTF-8 source span that adapters
convert normally. Controls never feed the formatter or LSP quick fixes because
silent deletion has no generally safe meaning.
Regenerate the checked-in policy deterministically with:
cargo run -p john-slop-core --example generate_unicode_catalog \
> crates/john-slop-core/assets/unicode.toml
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
Future profile updates must pin their upstream Unicode/data version, retain provenance and licensing, test ordinary multilingual near misses, and renew the hash-bound default-policy sign-off.