john-slop/docs/unicode-confusables.md

4.0 KiB
Raw Blame History

Ambiguous and invisible Unicode references

Gitea maintains the concrete list that most closely matches the requested "ambiguous Unicode characters" catalog:

At Gitea commit 9c715c0f9d12336e026e89be85f9f711c66e7ff6, the wrapped JSON contains 16 tables and 2,108 mapping pairs across them. _common contains 1,538 pairs; _default and the language-specific tables supply the remainder. The array values alternate between a confusable code point and the code point it resembles. Counts across tables are not necessarily unique global mappings.

Gitea's generator identifies hediet/vscode-unicode-data as its upstream. For standards-based identifier spoofing, the canonical broader reference is Unicode UTS #39 and its confusables.txt data file. GitHub separately documents a warning for hidden bidirectional text; that protection is about bidi controls rather than a published Gitea-style locale-aware lookalike table.

Implemented John Slop profile

John Slop uses unicode-security 0.1.2, whose checked-in UTS #39 tables target Unicode 16.0. The generated assets/unicode.toml contains every non-ASCII scalar whose single-character skeleton is non-empty printable ASCII, plus nine reviewed punctuation mappings and 95 compatibility width folds. Width folding takes precedence when a UTS #39 visual skeleton would produce a different target, such as fullwidth I mapping to I rather than l. That is 1,884 mappings in the current profile. Each entry has a stable ID, source, target, rationale, and enabled switch, and the same asset owns the diagnostic metadata.

The default [unicode] confusables = "replace" policy emits unicode-ascii-replacement diagnostics and supplies the same edits to CLI output, john-slop-fmt, LSP quick fixes, and LSP formatting. Set the value to review for identical findings without edits, or off to suppress generated lookalikes. The reviewed punctuation and width-folding mappings remain active under all three settings and can still be disabled individually.

This broad mapping is intentionally more aggressive than Gitea's locale-aware display warning. Greek, Cyrillic, mathematical, and other characters may be legitimate text, so projects handling multilingual content should prefer review unless ASCII normalization is their explicit policy.

unicode-invisible-control uses the invisible ranges generated by Gitea at the pinned commit above, including bidi embedding/override and isolate controls. Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not reported. Each other match receives an exact UTF-8 source span that adapters convert normally. Controls never feed the formatter or LSP quick fixes because silent deletion has no generally safe meaning.

Regenerate the checked-in policy deterministically with:

cargo run -p john-slop-core --example generate_unicode_catalog \
  > crates/john-slop-core/assets/unicode.toml

An exhaustive test walks all Unicode scalar values and rejects mapping drift.

Future profile updates must pin their upstream Unicode/data version, retain provenance and licensing, test ordinary multilingual near misses, and renew the hash-bound default-policy sign-off.