john-slop/docs/unicode-confusables.md

3.8 KiB

Ambiguous and invisible Unicode references

Gitea maintains the concrete list that most closely matches the requested "ambiguous Unicode characters" catalog:

At Gitea commit 9c715c0f9d12336e026e89be85f9f711c66e7ff6, the wrapped JSON contains 16 tables and 2,108 mapping pairs across them. _common contains 1,538 pairs; _default and the language-specific tables supply the remainder. The array values alternate between a confusable code point and the code point it resembles. Counts across tables are not necessarily unique global mappings.

Gitea's generator identifies hediet/vscode-unicode-data as its upstream. For standards-based identifier spoofing, the canonical broader reference is Unicode UTS #39 and its confusables.txt data file. GitHub separately documents a warning for hidden bidirectional text; that protection is about bidi controls rather than a published Gitea-style locale-aware lookalike table.

Implemented John Slop profile

John Slop uses unicode-security 0.1.2, whose checked-in UTS #39 tables target Unicode 16.0. The generated assets/unicode.toml contains every non-ASCII scalar whose single-character skeleton is non-empty printable ASCII, plus ten reviewed punctuation mappings. That is 1,829 mappings in the current profile. Each entry has a stable ID, source, target, rationale, and enabled switch, and the same asset owns the diagnostic metadata.

The default [unicode] confusables = "replace" policy emits unicode-ascii-replacement diagnostics and supplies the same edits to CLI output, john-slop-fmt, LSP quick fixes, and LSP formatting. Set the value to review for identical findings without edits, or off to suppress generated lookalikes. The ten curated punctuation mappings remain active under all three settings and can still be disabled individually.

This broad mapping is intentionally more aggressive than Gitea's locale-aware display warning. Greek, Cyrillic, mathematical, and other characters may be legitimate text, so projects handling multilingual content should prefer review unless ASCII normalization is their explicit policy.

unicode-invisible-control uses the invisible ranges generated by Gitea at the pinned commit above, including bidi embedding/override and isolate controls. Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not reported. Each other match receives an exact UTF-8 source span that adapters convert normally. Controls never feed the formatter or LSP quick fixes because silent deletion has no generally safe meaning.

Regenerate the checked-in policy deterministically with:

cargo run -p john-slop-core --example generate_unicode_catalog \
  > crates/john-slop-core/assets/unicode.toml

An exhaustive test walks all Unicode scalar values and rejects mapping drift.

Future profile updates must pin their upstream Unicode/data version, retain provenance and licensing, test ordinary multilingual near misses, and renew the hash-bound default-policy sign-off.