70 lines
3.8 KiB
Markdown
70 lines
3.8 KiB
Markdown
# Ambiguous and invisible Unicode references
|
|
|
|
Gitea maintains the concrete list that most closely matches the requested
|
|
"ambiguous Unicode characters" catalog:
|
|
|
|
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
|
is the generator input;
|
|
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
|
are easier to browse as code;
|
|
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
|
combines a locale-specific table with `_common` at runtime; and
|
|
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
|
records the upstream source and a separate invisible-rune catalog.
|
|
|
|
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
|
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
|
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
|
The array values alternate between a confusable code point and the code point
|
|
it resembles. Counts across tables are not necessarily unique global mappings.
|
|
|
|
Gitea's generator identifies
|
|
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
|
as its upstream. For standards-based identifier spoofing, the canonical broader
|
|
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
|
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
|
data file. GitHub separately documents a warning for hidden bidirectional text;
|
|
that protection is about bidi controls rather than a published Gitea-style
|
|
locale-aware lookalike table.
|
|
|
|
## Implemented John Slop profile
|
|
|
|
John Slop uses `unicode-security` 0.1.2, whose checked-in UTS #39 tables target
|
|
Unicode 16.0. The generated `assets/unicode.toml` contains every non-ASCII
|
|
scalar whose single-character skeleton is non-empty printable ASCII, plus ten
|
|
reviewed punctuation mappings. That is 1,829 mappings in the current profile.
|
|
Each entry has a stable ID, source, target, rationale, and enabled switch, and
|
|
the same asset owns the diagnostic metadata.
|
|
|
|
The default `[unicode] confusables = "replace"` policy emits
|
|
`unicode-ascii-replacement` diagnostics and supplies the same edits to CLI
|
|
output, `john-slop-fmt`, LSP quick fixes, and LSP formatting. Set the value to
|
|
`review` for identical findings without edits, or `off` to suppress generated
|
|
lookalikes. The ten curated punctuation mappings remain active under all three
|
|
settings and can still be disabled individually.
|
|
|
|
This broad mapping is intentionally more aggressive than Gitea's locale-aware
|
|
display warning. Greek, Cyrillic, mathematical, and other characters may be
|
|
legitimate text, so projects handling multilingual content should prefer
|
|
`review` unless ASCII normalization is their explicit policy.
|
|
|
|
`unicode-invisible-control` uses the invisible ranges generated by Gitea at the
|
|
pinned commit above, including bidi embedding/override and isolate controls.
|
|
Ordinary space, tab, line feed, and the carriage return in a CRLF pair are not
|
|
reported. Each other match receives an exact UTF-8 source span that adapters
|
|
convert normally. Controls never feed the formatter or LSP quick fixes because
|
|
silent deletion has no generally safe meaning.
|
|
|
|
Regenerate the checked-in policy deterministically with:
|
|
|
|
```sh
|
|
cargo run -p john-slop-core --example generate_unicode_catalog \
|
|
> crates/john-slop-core/assets/unicode.toml
|
|
```
|
|
|
|
An exhaustive test walks all Unicode scalar values and rejects mapping drift.
|
|
|
|
Future profile updates must pin their upstream Unicode/data version, retain
|
|
provenance and licensing, test ordinary multilingual near misses, and renew the
|
|
hash-bound default-policy sign-off.
|