docs: move advanced details out of readme
Chado/repo-fix/pipeline/head This commit looks good Details

This commit is contained in:
Dandaedre 2026-10-03 23:48:11 +02:00
parent 988f12ff13
commit 6883130266
2 changed files with 122 additions and 101 deletions

128
README.md
View File

@ -4,125 +4,51 @@
![tests](https://assets.dandaedre.com/coverage/Chado/repo-fix/main/test.svg)
![test coverage](https://assets.dandaedre.com/coverage/Chado/repo-fix/main/coverage.svg)
Fix Dandaedre Gitea submodule URLs after cloning, pulling, or syncing:
Normalize Dandaedre Gitea submodule URLs after cloning, pulling, or syncing:
- `.gitmodules` uses HTTPS for CI: `https://proxy.dandaedre.com/gitea/Beffroi/reportist.git`
- `.git/config` uses SSH locally: `git@git.dandaedre.com:Beffroi/reportist.git`
- `.git/modules/**/config` uses the same SSH format for submodule remotes
(`url` and `pushurl`) and nested submodule URL entries.
- `.gitmodules` uses HTTPS through `proxy.dandaedre.com` for CI.
- Local Git configuration uses SSH through `git.dandaedre.com`.
- Initialized and nested submodule remotes are updated as well.
Requires Python 3.10 or newer and Git on your PATH. No runtime pip packages are needed; the script
uses the Python standard library and Git's config commands.
The script requires Python 3.10 or newer and Git. It has no runtime Python
dependencies and does not fetch repositories or submodules.
Preview the changes, then apply them:
## Usage
Preview changes first, then apply them:
```bash
python3 fix_submodules.py /path/to/repository --dry-run
python3 fix_submodules.py /path/to/repository
```
The project follows semantic versioning and currently reports version `0.1.0`:
The repository path defaults to the current directory and may point anywhere
inside a working tree. Repeated runs are safe.
```bash
python3 fix_submodules.py --version
```
The repository defaults to the current directory, and may also be a directory
inside a working tree. Run the script again whenever a pull or `git submodule
sync` changes the URLs. Repeated runs are safe. Dry runs do not write files.
Recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on `dandaedre.com`,
`git.dandaedre.com`, and `proxy.dandaedre.com`. Old URLs such as
`git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository paths and
`.git` suffixes are preserved; the proxy's `/gitea/` prefix is translated.
Missing local URL entries are added from `.gitmodules`. Existing local URLs
retain their own repository paths. Other hosts and relative URLs are skipped.
Unrelated configuration, including `active`, is preserved.
Works with `.git` directories and pointer files, including linked worktrees
(whose local config is shared). Existing configs under Git's `modules`
directories are scanned recursively, including nested and deinitialized
submodules. Their remotes (including `origin`) are repaired; the selected
repository's own remotes are unchanged. Nothing is fetched. Nested working-tree
`.gitmodules` files are not edited; run the script inside those submodules to
normalize their declarations as well.
Commit `.gitmodules` changes for CI. Local config changes are not committed.
After fixing URLs, initialize submodules with:
After applying the changes, commit `.gitmodules` when it changed. Local Git
configuration is intentionally not committed. You can then initialize the
submodules:
```bash
git -C /path/to/repository submodule update --init --recursive
```
Run tests:
Show the current semantic version with:
```bash
python3 fix_submodules.py --version
```
## Development
Run the dependency-free unit suite with:
```bash
python3 -m unittest discover -s tests -v
```
The unit suite includes a small deterministic fuzz test and malformed-value
cases for URL parsing. They are intended to catch crashes and unsafe path
shapes, not to replace a dedicated fuzzer.
## Advanced
For the same coverage check as CI, install the pinned tools and run:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-ci.txt
.venv/bin/coverage run -m unittest discover -s tests -v
.venv/bin/coverage report
```
Branch coverage must remain at or above 80%. This deliberately modest floor
keeps the check useful without forcing low-value tests.
## Jenkins test CI
The `Jenkinsfile` uses the `rust_no_docker` agent from the Ravenous pipeline.
Configure a Jenkins Pipeline from SCM (or a multibranch pipeline) for this
repository with script path `Jenkinsfile`. The agent needs `python3` and `git`
on PATH and network access to PyPI, vulnerability data, and the Gitea HTTPS
proxy. No uv or Docker is required; CI creates isolated virtual environments
and installs the pinned tools. Use the Gitea webhook URL from the Jenkinsfile
if your Jenkins Gitea integration is configured at that address.
Every build runs the unit tests with coverage. Before testing, it runs two
small supply-chain gates: `pip-audit` 2.10.1 fails on known vulnerabilities or
dependency-collection errors, and `pip-licenses` 5.5.5 allows only MIT-licensed
installed application packages. The CI environment also pins coverage 7.16.1
and setuptools 80.9.0 in `requirements-ci.txt`. Run the gates locally with:
```bash
.venv/bin/pip-audit --strict .
.venv/bin/pip-licenses --python=.venv-app/bin/python
```
The audit needs network access for package resolution and current advisory
data. The license gate inspects the isolated application environment; update
the explicit allow-list only after reviewing a new dependency. The project has
no runtime dependencies today. These checks do not analyze the standard
library, Git, malicious packages, provenance, or undisclosed vulnerabilities;
no broader hardening tools are in scope.
By default, CI also shallow-clones
`Ravenous/ravenous_roll20.git` through the HTTPS proxy into a temporary directory
and verifies dry-run safety, both URL formats, repeatability, recovery after
`git submodule sync`, and correction of the old SSH hostname. The clone is
removed after the test. It does not fetch submodules or run code from Ravenous;
SSH connectivity and recursive submodule downloads are not tested.
Run the same network test locally:
```bash
python3 -m unittest discover -s integration -v
```
The HTTPS clone URL corresponds to
`git@dandaedre.com:Ravenous/ravenous_roll20.git` and avoids requiring an SSH key
for this CI test. If authentication is required, configure Git credentials on
the agent; prompts are disabled. Clone failures fail the build. Disable the
`RUN_CLONE_TEST` build parameter for an offline unit-test-only run.
This pipeline only checks out and tests. It does not publish artifacts, send
checks, push commits, or deploy files. Test results appear in the console log.
- [URL handling and Git layout support](docs/advanced.md#url-handling-and-git-layouts)
- [Testing, fuzz cases, and coverage](docs/advanced.md#testing-and-coverage)
- [Jenkins CI and dependency gates](docs/advanced.md#jenkins-ci)

95
docs/advanced.md Normal file
View File

@ -0,0 +1,95 @@
# Advanced usage
## URL handling and Git layouts
`repo-fix` recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on
`dandaedre.com`, `git.dandaedre.com`, and `proxy.dandaedre.com`. Legacy URLs
such as `git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository
paths and `.git` suffixes are preserved, and the proxy's `/gitea/` prefix is
translated when switching between HTTPS and SSH.
Missing local URL entries are added from `.gitmodules`. Existing local URLs
retain their own repository paths. Other hosts, relative URLs, and malformed
values are skipped. Unrelated configuration, including `active`, is preserved.
The script supports ordinary `.git` directories, `.git` pointer files, and
linked worktrees whose local configuration is shared. It recursively scans
existing configuration under Git's `modules` directories, including nested or
deinitialized submodules, and repairs their remote `url`, `pushurl`, and nested
submodule URL entries. The selected repository's own remotes are not changed.
Nested working-tree `.gitmodules` files are not edited. Run the script inside
each nested submodule when those declarations also need normalization.
## Testing and coverage
The unit suite includes malformed-value cases and a small deterministic fuzz
test for URL parsing. These checks are intended to catch crashes and unsafe
path shapes, not to replace a dedicated fuzzer.
To run the same branch-coverage check as CI:
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-ci.txt
.venv/bin/coverage run -m unittest discover -s tests -v
.venv/bin/coverage report
```
Coverage must remain at or above 80%. The threshold is intentionally modest so
it catches meaningful regressions without encouraging low-value tests.
The opt-in integration test shallow-clones
`Ravenous/ravenous_roll20.git` through the HTTPS proxy and exercises dry-run
safety, both URL formats, repeatability, recovery after `git submodule sync`,
and correction of the legacy SSH hostname:
```bash
python3 -m unittest discover -s integration -v
```
The clone is removed after the test. It does not fetch submodules, execute code
from Ravenous, test SSH connectivity, or recursively download submodules.
## Jenkins CI
Configure Jenkins as a Pipeline from SCM or a multibranch pipeline with
`Jenkinsfile` as the script path. The pipeline uses the `rust_no_docker` agent
from Ravenous and requires Python 3.10 or newer, Git, and network access to
PyPI, vulnerability data, and the Gitea HTTPS proxy. It creates isolated
virtual environments; uv and Docker are not required.
Before testing, CI runs two focused dependency gates:
- `pip-audit` 2.10.1 fails on known vulnerabilities or dependency-collection
errors.
- `pip-licenses` 5.5.5 permits only MIT-licensed packages in the isolated
application environment.
Coverage 7.16.1 and setuptools 80.9.0 are also pinned in
`requirements-ci.txt`. Create the isolated application environment and run the
gates locally with:
```bash
python3 -m venv .venv-app
.venv-app/bin/python -m pip install setuptools==80.9.0
.venv-app/bin/python -m pip install --no-build-isolation --no-deps .
.venv/bin/pip-audit --strict .
.venv/bin/pip-licenses --python=.venv-app/bin/python
```
The audit needs network access for resolution and current advisory data. The
license allow-list should only change after reviewing a new dependency. The
project currently has no runtime dependencies.
These gates do not analyze the standard library, Git, malicious packages,
provenance, or undisclosed vulnerabilities. No broader hardening tools are in
scope.
The network integration test runs by default. Disable the `RUN_CLONE_TEST`
build parameter for an offline unit-test-only run. If the HTTPS clone requires
authentication, configure Git credentials on the agent; interactive prompts
are disabled and clone failures fail the build.
The pipeline checks out and tests only. It does not publish artifacts, send
checks, push commits, or deploy files.