docs: record Unicode confusable catalogs
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good Details

This commit is contained in:
Dandaedre 2026-09-29 17:22:39 +02:00
parent eeabed9010
commit 0e4f018a61
Signed by: Dandaedre
GPG Key ID: 440C6592E8A26D99
5 changed files with 54 additions and 0 deletions

View File

@ -61,6 +61,7 @@ and Vim setup.
- [Development, tests, features, and coverage](docs/development.md)
- [Design decision records](docs/decisions/)
- [Background reading](docs/background.md)
- [Ambiguous and invisible Unicode data sources](docs/unicode-confusables.md)
The ordered implementation plan lives in [tasks.md](tasks.md), and contribution
rules live in [AGENTS.md](AGENTS.md).

View File

@ -28,6 +28,8 @@ the details needed to use, embed, and develop John Slop:
matrix, snapshots, coverage, badges, and CI quality gates.
- [Background](background.md) collects external context on observable signs of
low-care AI-assisted writing.
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
Gitea's locale-aware catalog, its upstream, and safe adoption boundaries.
- [Decision records](decisions/) preserve durable product and engineering
choices, including confidence semantics and input limits.

View File

@ -7,6 +7,7 @@ the kinds of review problems the project aims to help surface:
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
Project-specific decisions take precedence over external guidance. See
[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and

View File

@ -0,0 +1,46 @@
# Ambiguous and invisible Unicode references
Gitea maintains the concrete list that most closely matches the requested
"ambiguous Unicode characters" catalog:
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
is the generator input;
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
are easier to browse as code;
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
combines a locale-specific table with `_common` at runtime; and
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
records the upstream source and a separate invisible-rune catalog.
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
1,538 pairs; `_default` and the language-specific tables supply the remainder.
The array values alternate between a confusable code point and the code point
it resembles. Counts across tables are not necessarily unique global mappings.
Gitea's generator identifies
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
as its upstream. For standards-based identifier spoofing, the canonical broader
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
data file. GitHub separately documents a warning for hidden bidirectional text;
that protection is about bidi controls rather than a published Gitea-style
locale-aware lookalike table.
## Implication for John Slop
These sources are useful detection references, not safe formatter replacement
catalogs. Many listed characters are legitimate in particular languages and
scripts, and Gitea deliberately interprets them through the reviewer's locale.
Importing every pair as an automatic Unicode-to-ASCII edit would corrupt valid
text and violate the formatter's conservative replacement contract.
A future implementation should therefore keep three policies separate:
1. configured, explicitly safe formatter replacements;
2. locale-aware confusable findings for human review; and
3. invisible and bidirectional-control security findings.
Any vendored or generated catalog should pin its upstream Unicode/data version,
retain provenance and licensing, test locale behavior and ordinary multilingual
near-misses, and report exact source spans without silently rewriting text.

View File

@ -240,6 +240,10 @@ detector's limitations are prominent.
- Statistical or model-based authorship classification.
- Automatic prose or source-code rewriting beyond configured ASCII replacements.
- Locale-aware confusable, invisible, and bidirectional Unicode detection. The
researched catalogs and adoption constraints are recorded in
`docs/unicode-confusables.md`; detection must remain separate from automatic
replacement.
- In-place modification of archive members or office documents.
- Network-backed analysis or telemetry.
- Editor-specific extensions; the LSP is the integration boundary.