From 6883130266e1fb9af7fccc394416fc2901ded078 Mon Sep 17 00:00:00 2001 From: Dandaedre Date: Sat, 3 Oct 2026 23:48:11 +0200 Subject: [PATCH] docs: move advanced details out of readme --- README.md | 128 ++++++++++------------------------------------- docs/advanced.md | 95 +++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+), 101 deletions(-) create mode 100644 docs/advanced.md diff --git a/README.md b/README.md index 3ccda1b..9a19112 100644 --- a/README.md +++ b/README.md @@ -4,125 +4,51 @@ ![tests](https://assets.dandaedre.com/coverage/Chado/repo-fix/main/test.svg) ![test coverage](https://assets.dandaedre.com/coverage/Chado/repo-fix/main/coverage.svg) -Fix Dandaedre Gitea submodule URLs after cloning, pulling, or syncing: +Normalize Dandaedre Gitea submodule URLs after cloning, pulling, or syncing: -- `.gitmodules` uses HTTPS for CI: `https://proxy.dandaedre.com/gitea/Beffroi/reportist.git` -- `.git/config` uses SSH locally: `git@git.dandaedre.com:Beffroi/reportist.git` -- `.git/modules/**/config` uses the same SSH format for submodule remotes - (`url` and `pushurl`) and nested submodule URL entries. +- `.gitmodules` uses HTTPS through `proxy.dandaedre.com` for CI. +- Local Git configuration uses SSH through `git.dandaedre.com`. +- Initialized and nested submodule remotes are updated as well. -Requires Python 3.10 or newer and Git on your PATH. No runtime pip packages are needed; the script -uses the Python standard library and Git's config commands. +The script requires Python 3.10 or newer and Git. It has no runtime Python +dependencies and does not fetch repositories or submodules. -Preview the changes, then apply them: +## Usage + +Preview changes first, then apply them: ```bash python3 fix_submodules.py /path/to/repository --dry-run python3 fix_submodules.py /path/to/repository ``` -The project follows semantic versioning and currently reports version `0.1.0`: +The repository path defaults to the current directory and may point anywhere +inside a working tree. Repeated runs are safe. -```bash -python3 fix_submodules.py --version -``` - -The repository defaults to the current directory, and may also be a directory -inside a working tree. Run the script again whenever a pull or `git submodule -sync` changes the URLs. Repeated runs are safe. Dry runs do not write files. - -Recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on `dandaedre.com`, -`git.dandaedre.com`, and `proxy.dandaedre.com`. Old URLs such as -`git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository paths and -`.git` suffixes are preserved; the proxy's `/gitea/` prefix is translated. - -Missing local URL entries are added from `.gitmodules`. Existing local URLs -retain their own repository paths. Other hosts and relative URLs are skipped. -Unrelated configuration, including `active`, is preserved. - -Works with `.git` directories and pointer files, including linked worktrees -(whose local config is shared). Existing configs under Git's `modules` -directories are scanned recursively, including nested and deinitialized -submodules. Their remotes (including `origin`) are repaired; the selected -repository's own remotes are unchanged. Nothing is fetched. Nested working-tree -`.gitmodules` files are not edited; run the script inside those submodules to -normalize their declarations as well. - -Commit `.gitmodules` changes for CI. Local config changes are not committed. -After fixing URLs, initialize submodules with: +After applying the changes, commit `.gitmodules` when it changed. Local Git +configuration is intentionally not committed. You can then initialize the +submodules: ```bash git -C /path/to/repository submodule update --init --recursive ``` -Run tests: +Show the current semantic version with: + +```bash +python3 fix_submodules.py --version +``` + +## Development + +Run the dependency-free unit suite with: ```bash python3 -m unittest discover -s tests -v ``` -The unit suite includes a small deterministic fuzz test and malformed-value -cases for URL parsing. They are intended to catch crashes and unsafe path -shapes, not to replace a dedicated fuzzer. +## Advanced -For the same coverage check as CI, install the pinned tools and run: - -```bash -python3 -m venv .venv -.venv/bin/python -m pip install -r requirements-ci.txt -.venv/bin/coverage run -m unittest discover -s tests -v -.venv/bin/coverage report -``` - -Branch coverage must remain at or above 80%. This deliberately modest floor -keeps the check useful without forcing low-value tests. - -## Jenkins test CI - -The `Jenkinsfile` uses the `rust_no_docker` agent from the Ravenous pipeline. -Configure a Jenkins Pipeline from SCM (or a multibranch pipeline) for this -repository with script path `Jenkinsfile`. The agent needs `python3` and `git` -on PATH and network access to PyPI, vulnerability data, and the Gitea HTTPS -proxy. No uv or Docker is required; CI creates isolated virtual environments -and installs the pinned tools. Use the Gitea webhook URL from the Jenkinsfile -if your Jenkins Gitea integration is configured at that address. - -Every build runs the unit tests with coverage. Before testing, it runs two -small supply-chain gates: `pip-audit` 2.10.1 fails on known vulnerabilities or -dependency-collection errors, and `pip-licenses` 5.5.5 allows only MIT-licensed -installed application packages. The CI environment also pins coverage 7.16.1 -and setuptools 80.9.0 in `requirements-ci.txt`. Run the gates locally with: - -```bash -.venv/bin/pip-audit --strict . -.venv/bin/pip-licenses --python=.venv-app/bin/python -``` - -The audit needs network access for package resolution and current advisory -data. The license gate inspects the isolated application environment; update -the explicit allow-list only after reviewing a new dependency. The project has -no runtime dependencies today. These checks do not analyze the standard -library, Git, malicious packages, provenance, or undisclosed vulnerabilities; -no broader hardening tools are in scope. - -By default, CI also shallow-clones -`Ravenous/ravenous_roll20.git` through the HTTPS proxy into a temporary directory -and verifies dry-run safety, both URL formats, repeatability, recovery after -`git submodule sync`, and correction of the old SSH hostname. The clone is -removed after the test. It does not fetch submodules or run code from Ravenous; -SSH connectivity and recursive submodule downloads are not tested. - -Run the same network test locally: - -```bash -python3 -m unittest discover -s integration -v -``` - -The HTTPS clone URL corresponds to -`git@dandaedre.com:Ravenous/ravenous_roll20.git` and avoids requiring an SSH key -for this CI test. If authentication is required, configure Git credentials on -the agent; prompts are disabled. Clone failures fail the build. Disable the -`RUN_CLONE_TEST` build parameter for an offline unit-test-only run. - -This pipeline only checks out and tests. It does not publish artifacts, send -checks, push commits, or deploy files. Test results appear in the console log. +- [URL handling and Git layout support](docs/advanced.md#url-handling-and-git-layouts) +- [Testing, fuzz cases, and coverage](docs/advanced.md#testing-and-coverage) +- [Jenkins CI and dependency gates](docs/advanced.md#jenkins-ci) diff --git a/docs/advanced.md b/docs/advanced.md new file mode 100644 index 0000000..b505e1f --- /dev/null +++ b/docs/advanced.md @@ -0,0 +1,95 @@ +# Advanced usage + +## URL handling and Git layouts + +`repo-fix` recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on +`dandaedre.com`, `git.dandaedre.com`, and `proxy.dandaedre.com`. Legacy URLs +such as `git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository +paths and `.git` suffixes are preserved, and the proxy's `/gitea/` prefix is +translated when switching between HTTPS and SSH. + +Missing local URL entries are added from `.gitmodules`. Existing local URLs +retain their own repository paths. Other hosts, relative URLs, and malformed +values are skipped. Unrelated configuration, including `active`, is preserved. + +The script supports ordinary `.git` directories, `.git` pointer files, and +linked worktrees whose local configuration is shared. It recursively scans +existing configuration under Git's `modules` directories, including nested or +deinitialized submodules, and repairs their remote `url`, `pushurl`, and nested +submodule URL entries. The selected repository's own remotes are not changed. + +Nested working-tree `.gitmodules` files are not edited. Run the script inside +each nested submodule when those declarations also need normalization. + +## Testing and coverage + +The unit suite includes malformed-value cases and a small deterministic fuzz +test for URL parsing. These checks are intended to catch crashes and unsafe +path shapes, not to replace a dedicated fuzzer. + +To run the same branch-coverage check as CI: + +```bash +python3 -m venv .venv +.venv/bin/python -m pip install -r requirements-ci.txt +.venv/bin/coverage run -m unittest discover -s tests -v +.venv/bin/coverage report +``` + +Coverage must remain at or above 80%. The threshold is intentionally modest so +it catches meaningful regressions without encouraging low-value tests. + +The opt-in integration test shallow-clones +`Ravenous/ravenous_roll20.git` through the HTTPS proxy and exercises dry-run +safety, both URL formats, repeatability, recovery after `git submodule sync`, +and correction of the legacy SSH hostname: + +```bash +python3 -m unittest discover -s integration -v +``` + +The clone is removed after the test. It does not fetch submodules, execute code +from Ravenous, test SSH connectivity, or recursively download submodules. + +## Jenkins CI + +Configure Jenkins as a Pipeline from SCM or a multibranch pipeline with +`Jenkinsfile` as the script path. The pipeline uses the `rust_no_docker` agent +from Ravenous and requires Python 3.10 or newer, Git, and network access to +PyPI, vulnerability data, and the Gitea HTTPS proxy. It creates isolated +virtual environments; uv and Docker are not required. + +Before testing, CI runs two focused dependency gates: + +- `pip-audit` 2.10.1 fails on known vulnerabilities or dependency-collection + errors. +- `pip-licenses` 5.5.5 permits only MIT-licensed packages in the isolated + application environment. + +Coverage 7.16.1 and setuptools 80.9.0 are also pinned in +`requirements-ci.txt`. Create the isolated application environment and run the +gates locally with: + +```bash +python3 -m venv .venv-app +.venv-app/bin/python -m pip install setuptools==80.9.0 +.venv-app/bin/python -m pip install --no-build-isolation --no-deps . +.venv/bin/pip-audit --strict . +.venv/bin/pip-licenses --python=.venv-app/bin/python +``` + +The audit needs network access for resolution and current advisory data. The +license allow-list should only change after reviewing a new dependency. The +project currently has no runtime dependencies. + +These gates do not analyze the standard library, Git, malicious packages, +provenance, or undisclosed vulnerabilities. No broader hardening tools are in +scope. + +The network integration test runs by default. Disable the `RUN_CLONE_TEST` +build parameter for an offline unit-test-only run. If the HTTPS clone requires +authentication, configure Git credentials on the agent; interactive prompts +are disabled and clone failures fail the build. + +The pipeline checks out and tests only. It does not publish artifacts, send +checks, push commits, or deploy files.