diff --git a/README.md b/README.md index 5211c69..859b4e3 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ and Vim setup. - [Development, tests, features, and coverage](docs/development.md) - [Design decision records](docs/decisions/) - [Background reading](docs/background.md) +- [Ambiguous and invisible Unicode data sources](docs/unicode-confusables.md) The ordered implementation plan lives in [tasks.md](tasks.md), and contribution rules live in [AGENTS.md](AGENTS.md). diff --git a/docs/README.md b/docs/README.md index 2763bd7..abd50a8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -28,6 +28,8 @@ the details needed to use, embed, and develop John Slop: matrix, snapshots, coverage, badges, and CI quality gates. - [Background](background.md) collects external context on observable signs of low-care AI-assisted writing. +- [Ambiguous and invisible Unicode references](unicode-confusables.md) records + Gitea's locale-aware catalog, its upstream, and safe adoption boundaries. - [Decision records](decisions/) preserve durable product and engineering choices, including confidence semantics and input limits. diff --git a/docs/background.md b/docs/background.md index c0d6c69..09c958d 100644 --- a/docs/background.md +++ b/docs/background.md @@ -7,6 +7,7 @@ the kinds of review problems the project aims to help surface: - [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing) - [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup) - [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion) +- [Gitea/Unicode confusable-character source notes](unicode-confusables.md) Project-specific decisions take precedence over external guidance. See [ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and diff --git a/docs/unicode-confusables.md b/docs/unicode-confusables.md new file mode 100644 index 0000000..0e98ac6 --- /dev/null +++ b/docs/unicode-confusables.md @@ -0,0 +1,46 @@ +# Ambiguous and invisible Unicode references + +Gitea maintains the concrete list that most closely matches the requested +"ambiguous Unicode characters" catalog: + +- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json) + is the generator input; +- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go) + are easier to browse as code; +- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go) + combines a locale-specific table with `_common` at runtime; and +- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go) + records the upstream source and a separate invisible-rune catalog. + +At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped +JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains +1,538 pairs; `_default` and the language-specific tables supply the remainder. +The array values alternate between a confusable code point and the code point +it resembles. Counts across tables are not necessarily unique global mappings. + +Gitea's generator identifies +[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json) +as its upstream. For standards-based identifier spoofing, the canonical broader +reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its +[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt) +data file. GitHub separately documents a warning for hidden bidirectional text; +that protection is about bidi controls rather than a published Gitea-style +locale-aware lookalike table. + +## Implication for John Slop + +These sources are useful detection references, not safe formatter replacement +catalogs. Many listed characters are legitimate in particular languages and +scripts, and Gitea deliberately interprets them through the reviewer's locale. +Importing every pair as an automatic Unicode-to-ASCII edit would corrupt valid +text and violate the formatter's conservative replacement contract. + +A future implementation should therefore keep three policies separate: + +1. configured, explicitly safe formatter replacements; +2. locale-aware confusable findings for human review; and +3. invisible and bidirectional-control security findings. + +Any vendored or generated catalog should pin its upstream Unicode/data version, +retain provenance and licensing, test locale behavior and ordinary multilingual +near-misses, and report exact source spans without silently rewriting text. diff --git a/tasks.md b/tasks.md index fba067c..813fac8 100644 --- a/tasks.md +++ b/tasks.md @@ -240,6 +240,10 @@ detector's limitations are prominent. - Statistical or model-based authorship classification. - Automatic prose or source-code rewriting beyond configured ASCII replacements. +- Locale-aware confusable, invisible, and bidirectional Unicode detection. The + researched catalogs and adoption constraints are recorded in + `docs/unicode-confusables.md`; detection must remain separate from automatic + replacement. - In-place modification of archive members or office documents. - Network-backed analysis or telemetry. - Editor-specific extensions; the LSP is the integration boundary.