73 lines
4.0 KiB
Markdown
73 lines
4.0 KiB
Markdown
# Ambiguous and invisible Unicode references
|
||
|
||
Gitea maintains the concrete list that most closely matches the requested
|
||
"ambiguous Unicode characters" catalog:
|
||
|
||
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
||
is the generator input;
|
||
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
||
are easier to browse as code;
|
||
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
||
combines a locale-specific table with `_common` at runtime; and
|
||
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
||
records the upstream source and a separate invisible-rune catalog.
|
||
|
||
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
||
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
||
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
||
The array values alternate between a confusable code point and the code point
|
||
it resembles. Counts across tables are not necessarily unique global mappings.
|
||
|
||
Gitea's generator identifies
|
||
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
||
as its upstream. For standards-based identifier spoofing, the canonical broader
|
||
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
||
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
||
data file. GitHub separately documents a warning for hidden bidirectional text;
|
||
that protection is about bidi controls rather than a published Gitea-style
|
||
locale-aware lookalike table.
|
||
|
||
## Implemented John Slop profile
|
||
|
||
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
|
||
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
|
||
scalar whose single-character skeleton is non-empty printable ASCII, plus nine
|
||
reviewed punctuation mappings and 95 compatibility width folds. Width folding
|
||
takes precedence when a UTS #39 visual skeleton would produce a different
|
||
target, such as fullwidth `I` mapping to `I` rather than `l`. That is 1,884
|
||
mappings in the current profile.
|
||
Each entry has a stable ID, source, target, rationale, and enabled switch, and
|
||
the same asset owns the diagnostic metadata.
|
||
|
||
The default `[unicode] confusables = "replace"` policy emits
|
||
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
|
||
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
|
||
`review` for identical findings without edits, or `off` to suppress generated
|
||
lookalikes. The reviewed punctuation and width-folding mappings remain active
|
||
under all three settings and can still be disabled individually.
|
||
|
||
This broad mapping is intentionally more aggressive than Gitea's locale-aware
|
||
display warning. Greek, Cyrillic, mathematical, and other characters may be
|
||
legitimate text, so projects handling multilingual content should prefer
|
||
`review` unless ASCII normalization is their explicit policy.
|
||
|
||
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
|
||
pinned commit above, including bidi embedding/override and isolate controls.
|
||
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
|
||
reported. Each other match receives an exact UTF-8 source span that adapters
|
||
convert normally. Controls never feed the formatter or LSP quick fixes because
|
||
silent deletion has no generally safe meaning.
|
||
|
||
Regenerate the checked-in policy deterministically with:
|
||
|
||
```sh
|
||
cargo run -p john-slop-core --example generate_unicode_catalog \
|
||
> crates/john-slop-core/assets/unicode.toml
|
||
```
|
||
|
||
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
|
||
|
||
Future profile updates must pin their upstream Unicode/data version, retain
|
||
provenance and licensing, test ordinary multilingual near misses, and renew the
|
||
hash-bound default-policy sign-off.
|