docs: record Unicode confusable catalogs
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good
Details
Rust CI JohnSlop passed all CI checks.
Gitea/john_slop/pipeline/head This commit looks good
Details
This commit is contained in:
parent
eeabed9010
commit
0e4f018a61
|
|
@ -61,6 +61,7 @@ and Vim setup.
|
|||
- [Development, tests, features, and coverage](docs/development.md)
|
||||
- [Design decision records](docs/decisions/)
|
||||
- [Background reading](docs/background.md)
|
||||
- [Ambiguous and invisible Unicode data sources](docs/unicode-confusables.md)
|
||||
|
||||
The ordered implementation plan lives in [tasks.md](tasks.md), and contribution
|
||||
rules live in [AGENTS.md](AGENTS.md).
|
||||
|
|
|
|||
|
|
@ -28,6 +28,8 @@ the details needed to use, embed, and develop John Slop:
|
|||
matrix, snapshots, coverage, badges, and CI quality gates.
|
||||
- [Background](background.md) collects external context on observable signs of
|
||||
low-care AI-assisted writing.
|
||||
- [Ambiguous and invisible Unicode references](unicode-confusables.md) records
|
||||
Gitea's locale-aware catalog, its upstream, and safe adoption boundaries.
|
||||
- [Decision records](decisions/) preserve durable product and engineering
|
||||
choices, including confidence semantics and input limits.
|
||||
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ the kinds of review problems the project aims to help surface:
|
|||
- [Wikipedia: Signs of AI writing](https://en.wikipedia.org/wiki/Wikipedia%3ASigns_of_AI_writing)
|
||||
- [Wikipedia: WikiProject AI Cleanup](https://en.wikipedia.org/wiki/Wikipedia:WikiProject_AI_Cleanup)
|
||||
- [The Verge: Wikipedia editors are battling AI slop](https://www.theverge.com/report/756810/wikipedia-ai-slop-policies-community-speedy-deletion)
|
||||
- [Gitea/Unicode confusable-character source notes](unicode-confusables.md)
|
||||
|
||||
Project-specific decisions take precedence over external guidance. See
|
||||
[ADR 0001](decisions/0001-core-analysis-contracts.md) for confidence and
|
||||
|
|
|
|||
|
|
@ -0,0 +1,46 @@
|
|||
# Ambiguous and invisible Unicode references
|
||||
|
||||
Gitea maintains the concrete list that most closely matches the requested
|
||||
"ambiguous Unicode characters" catalog:
|
||||
|
||||
- [vendored `ambiguous.json`](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/ambiguous.json)
|
||||
is the generator input;
|
||||
- [generated Go tables](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous_gen.go)
|
||||
are easier to browse as code;
|
||||
- [locale selection](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/ambiguous.go)
|
||||
combines a locale-specific table with `_common` at runtime; and
|
||||
- [the generator](https://github.com/go-gitea/gitea/blob/9c715c0f9d12336e026e89be85f9f711c66e7ff6/modules/charset/generate/generate.go)
|
||||
records the upstream source and a separate invisible-rune catalog.
|
||||
|
||||
At Gitea commit `9c715c0f9d12336e026e89be85f9f711c66e7ff6`, the wrapped
|
||||
JSON contains 16 tables and 2,108 mapping pairs across them. `_common` contains
|
||||
1,538 pairs; `_default` and the language-specific tables supply the remainder.
|
||||
The array values alternate between a confusable code point and the code point
|
||||
it resembles. Counts across tables are not necessarily unique global mappings.
|
||||
|
||||
Gitea's generator identifies
|
||||
[`hediet/vscode-unicode-data`](https://github.com/hediet/vscode-unicode-data/blob/main/out/ambiguous.json)
|
||||
as its upstream. For standards-based identifier spoofing, the canonical broader
|
||||
reference is [Unicode UTS #39](https://www.unicode.org/reports/tr39/) and its
|
||||
[`confusables.txt`](https://www.unicode.org/Public/security/latest/confusables.txt)
|
||||
data file. GitHub separately documents a warning for hidden bidirectional text;
|
||||
that protection is about bidi controls rather than a published Gitea-style
|
||||
locale-aware lookalike table.
|
||||
|
||||
## Implication for John Slop
|
||||
|
||||
These sources are useful detection references, not safe formatter replacement
|
||||
catalogs. Many listed characters are legitimate in particular languages and
|
||||
scripts, and Gitea deliberately interprets them through the reviewer's locale.
|
||||
Importing every pair as an automatic Unicode-to-ASCII edit would corrupt valid
|
||||
text and violate the formatter's conservative replacement contract.
|
||||
|
||||
A future implementation should therefore keep three policies separate:
|
||||
|
||||
1. configured, explicitly safe formatter replacements;
|
||||
2. locale-aware confusable findings for human review; and
|
||||
3. invisible and bidirectional-control security findings.
|
||||
|
||||
Any vendored or generated catalog should pin its upstream Unicode/data version,
|
||||
retain provenance and licensing, test locale behavior and ordinary multilingual
|
||||
near-misses, and report exact source spans without silently rewriting text.
|
||||
4
tasks.md
4
tasks.md
|
|
@ -240,6 +240,10 @@ detector's limitations are prominent.
|
|||
|
||||
- Statistical or model-based authorship classification.
|
||||
- Automatic prose or source-code rewriting beyond configured ASCII replacements.
|
||||
- Locale-aware confusable, invisible, and bidirectional Unicode detection. The
|
||||
researched catalogs and adoption constraints are recorded in
|
||||
`docs/unicode-confusables.md`; detection must remain separate from automatic
|
||||
replacement.
|
||||
- In-place modification of archive members or office documents.
|
||||
- Network-backed analysis or telemetry.
|
||||
- Editor-specific extensions; the LSP is the integration boundary.
|
||||
|
|
|
|||
Loading…
Reference in New Issue