docs: move advanced details out of readme
Chado/repo-fix/pipeline/head This commit looks good
Details
Chado/repo-fix/pipeline/head This commit looks good
Details
This commit is contained in:
parent
988f12ff13
commit
6883130266
128
README.md
128
README.md
|
|
@ -4,125 +4,51 @@
|
||||||

|

|
||||||

|

|
||||||
|
|
||||||
Fix Dandaedre Gitea submodule URLs after cloning, pulling, or syncing:
|
Normalize Dandaedre Gitea submodule URLs after cloning, pulling, or syncing:
|
||||||
|
|
||||||
- `.gitmodules` uses HTTPS for CI: `https://proxy.dandaedre.com/gitea/Beffroi/reportist.git`
|
- `.gitmodules` uses HTTPS through `proxy.dandaedre.com` for CI.
|
||||||
- `.git/config` uses SSH locally: `git@git.dandaedre.com:Beffroi/reportist.git`
|
- Local Git configuration uses SSH through `git.dandaedre.com`.
|
||||||
- `.git/modules/**/config` uses the same SSH format for submodule remotes
|
- Initialized and nested submodule remotes are updated as well.
|
||||||
(`url` and `pushurl`) and nested submodule URL entries.
|
|
||||||
|
|
||||||
Requires Python 3.10 or newer and Git on your PATH. No runtime pip packages are needed; the script
|
The script requires Python 3.10 or newer and Git. It has no runtime Python
|
||||||
uses the Python standard library and Git's config commands.
|
dependencies and does not fetch repositories or submodules.
|
||||||
|
|
||||||
Preview the changes, then apply them:
|
## Usage
|
||||||
|
|
||||||
|
Preview changes first, then apply them:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 fix_submodules.py /path/to/repository --dry-run
|
python3 fix_submodules.py /path/to/repository --dry-run
|
||||||
python3 fix_submodules.py /path/to/repository
|
python3 fix_submodules.py /path/to/repository
|
||||||
```
|
```
|
||||||
|
|
||||||
The project follows semantic versioning and currently reports version `0.1.0`:
|
The repository path defaults to the current directory and may point anywhere
|
||||||
|
inside a working tree. Repeated runs are safe.
|
||||||
|
|
||||||
```bash
|
After applying the changes, commit `.gitmodules` when it changed. Local Git
|
||||||
python3 fix_submodules.py --version
|
configuration is intentionally not committed. You can then initialize the
|
||||||
```
|
submodules:
|
||||||
|
|
||||||
The repository defaults to the current directory, and may also be a directory
|
|
||||||
inside a working tree. Run the script again whenever a pull or `git submodule
|
|
||||||
sync` changes the URLs. Repeated runs are safe. Dry runs do not write files.
|
|
||||||
|
|
||||||
Recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on `dandaedre.com`,
|
|
||||||
`git.dandaedre.com`, and `proxy.dandaedre.com`. Old URLs such as
|
|
||||||
`git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository paths and
|
|
||||||
`.git` suffixes are preserved; the proxy's `/gitea/` prefix is translated.
|
|
||||||
|
|
||||||
Missing local URL entries are added from `.gitmodules`. Existing local URLs
|
|
||||||
retain their own repository paths. Other hosts and relative URLs are skipped.
|
|
||||||
Unrelated configuration, including `active`, is preserved.
|
|
||||||
|
|
||||||
Works with `.git` directories and pointer files, including linked worktrees
|
|
||||||
(whose local config is shared). Existing configs under Git's `modules`
|
|
||||||
directories are scanned recursively, including nested and deinitialized
|
|
||||||
submodules. Their remotes (including `origin`) are repaired; the selected
|
|
||||||
repository's own remotes are unchanged. Nothing is fetched. Nested working-tree
|
|
||||||
`.gitmodules` files are not edited; run the script inside those submodules to
|
|
||||||
normalize their declarations as well.
|
|
||||||
|
|
||||||
Commit `.gitmodules` changes for CI. Local config changes are not committed.
|
|
||||||
After fixing URLs, initialize submodules with:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git -C /path/to/repository submodule update --init --recursive
|
git -C /path/to/repository submodule update --init --recursive
|
||||||
```
|
```
|
||||||
|
|
||||||
Run tests:
|
Show the current semantic version with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 fix_submodules.py --version
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
Run the dependency-free unit suite with:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 -m unittest discover -s tests -v
|
python3 -m unittest discover -s tests -v
|
||||||
```
|
```
|
||||||
|
|
||||||
The unit suite includes a small deterministic fuzz test and malformed-value
|
## Advanced
|
||||||
cases for URL parsing. They are intended to catch crashes and unsafe path
|
|
||||||
shapes, not to replace a dedicated fuzzer.
|
|
||||||
|
|
||||||
For the same coverage check as CI, install the pinned tools and run:
|
- [URL handling and Git layout support](docs/advanced.md#url-handling-and-git-layouts)
|
||||||
|
- [Testing, fuzz cases, and coverage](docs/advanced.md#testing-and-coverage)
|
||||||
```bash
|
- [Jenkins CI and dependency gates](docs/advanced.md#jenkins-ci)
|
||||||
python3 -m venv .venv
|
|
||||||
.venv/bin/python -m pip install -r requirements-ci.txt
|
|
||||||
.venv/bin/coverage run -m unittest discover -s tests -v
|
|
||||||
.venv/bin/coverage report
|
|
||||||
```
|
|
||||||
|
|
||||||
Branch coverage must remain at or above 80%. This deliberately modest floor
|
|
||||||
keeps the check useful without forcing low-value tests.
|
|
||||||
|
|
||||||
## Jenkins test CI
|
|
||||||
|
|
||||||
The `Jenkinsfile` uses the `rust_no_docker` agent from the Ravenous pipeline.
|
|
||||||
Configure a Jenkins Pipeline from SCM (or a multibranch pipeline) for this
|
|
||||||
repository with script path `Jenkinsfile`. The agent needs `python3` and `git`
|
|
||||||
on PATH and network access to PyPI, vulnerability data, and the Gitea HTTPS
|
|
||||||
proxy. No uv or Docker is required; CI creates isolated virtual environments
|
|
||||||
and installs the pinned tools. Use the Gitea webhook URL from the Jenkinsfile
|
|
||||||
if your Jenkins Gitea integration is configured at that address.
|
|
||||||
|
|
||||||
Every build runs the unit tests with coverage. Before testing, it runs two
|
|
||||||
small supply-chain gates: `pip-audit` 2.10.1 fails on known vulnerabilities or
|
|
||||||
dependency-collection errors, and `pip-licenses` 5.5.5 allows only MIT-licensed
|
|
||||||
installed application packages. The CI environment also pins coverage 7.16.1
|
|
||||||
and setuptools 80.9.0 in `requirements-ci.txt`. Run the gates locally with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/pip-audit --strict .
|
|
||||||
.venv/bin/pip-licenses --python=.venv-app/bin/python
|
|
||||||
```
|
|
||||||
|
|
||||||
The audit needs network access for package resolution and current advisory
|
|
||||||
data. The license gate inspects the isolated application environment; update
|
|
||||||
the explicit allow-list only after reviewing a new dependency. The project has
|
|
||||||
no runtime dependencies today. These checks do not analyze the standard
|
|
||||||
library, Git, malicious packages, provenance, or undisclosed vulnerabilities;
|
|
||||||
no broader hardening tools are in scope.
|
|
||||||
|
|
||||||
By default, CI also shallow-clones
|
|
||||||
`Ravenous/ravenous_roll20.git` through the HTTPS proxy into a temporary directory
|
|
||||||
and verifies dry-run safety, both URL formats, repeatability, recovery after
|
|
||||||
`git submodule sync`, and correction of the old SSH hostname. The clone is
|
|
||||||
removed after the test. It does not fetch submodules or run code from Ravenous;
|
|
||||||
SSH connectivity and recursive submodule downloads are not tested.
|
|
||||||
|
|
||||||
Run the same network test locally:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 -m unittest discover -s integration -v
|
|
||||||
```
|
|
||||||
|
|
||||||
The HTTPS clone URL corresponds to
|
|
||||||
`git@dandaedre.com:Ravenous/ravenous_roll20.git` and avoids requiring an SSH key
|
|
||||||
for this CI test. If authentication is required, configure Git credentials on
|
|
||||||
the agent; prompts are disabled. Clone failures fail the build. Disable the
|
|
||||||
`RUN_CLONE_TEST` build parameter for an offline unit-test-only run.
|
|
||||||
|
|
||||||
This pipeline only checks out and tests. It does not publish artifacts, send
|
|
||||||
checks, push commits, or deploy files. Test results appear in the console log.
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,95 @@
|
||||||
|
# Advanced usage
|
||||||
|
|
||||||
|
## URL handling and Git layouts
|
||||||
|
|
||||||
|
`repo-fix` recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on
|
||||||
|
`dandaedre.com`, `git.dandaedre.com`, and `proxy.dandaedre.com`. Legacy URLs
|
||||||
|
such as `git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository
|
||||||
|
paths and `.git` suffixes are preserved, and the proxy's `/gitea/` prefix is
|
||||||
|
translated when switching between HTTPS and SSH.
|
||||||
|
|
||||||
|
Missing local URL entries are added from `.gitmodules`. Existing local URLs
|
||||||
|
retain their own repository paths. Other hosts, relative URLs, and malformed
|
||||||
|
values are skipped. Unrelated configuration, including `active`, is preserved.
|
||||||
|
|
||||||
|
The script supports ordinary `.git` directories, `.git` pointer files, and
|
||||||
|
linked worktrees whose local configuration is shared. It recursively scans
|
||||||
|
existing configuration under Git's `modules` directories, including nested or
|
||||||
|
deinitialized submodules, and repairs their remote `url`, `pushurl`, and nested
|
||||||
|
submodule URL entries. The selected repository's own remotes are not changed.
|
||||||
|
|
||||||
|
Nested working-tree `.gitmodules` files are not edited. Run the script inside
|
||||||
|
each nested submodule when those declarations also need normalization.
|
||||||
|
|
||||||
|
## Testing and coverage
|
||||||
|
|
||||||
|
The unit suite includes malformed-value cases and a small deterministic fuzz
|
||||||
|
test for URL parsing. These checks are intended to catch crashes and unsafe
|
||||||
|
path shapes, not to replace a dedicated fuzzer.
|
||||||
|
|
||||||
|
To run the same branch-coverage check as CI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv
|
||||||
|
.venv/bin/python -m pip install -r requirements-ci.txt
|
||||||
|
.venv/bin/coverage run -m unittest discover -s tests -v
|
||||||
|
.venv/bin/coverage report
|
||||||
|
```
|
||||||
|
|
||||||
|
Coverage must remain at or above 80%. The threshold is intentionally modest so
|
||||||
|
it catches meaningful regressions without encouraging low-value tests.
|
||||||
|
|
||||||
|
The opt-in integration test shallow-clones
|
||||||
|
`Ravenous/ravenous_roll20.git` through the HTTPS proxy and exercises dry-run
|
||||||
|
safety, both URL formats, repeatability, recovery after `git submodule sync`,
|
||||||
|
and correction of the legacy SSH hostname:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m unittest discover -s integration -v
|
||||||
|
```
|
||||||
|
|
||||||
|
The clone is removed after the test. It does not fetch submodules, execute code
|
||||||
|
from Ravenous, test SSH connectivity, or recursively download submodules.
|
||||||
|
|
||||||
|
## Jenkins CI
|
||||||
|
|
||||||
|
Configure Jenkins as a Pipeline from SCM or a multibranch pipeline with
|
||||||
|
`Jenkinsfile` as the script path. The pipeline uses the `rust_no_docker` agent
|
||||||
|
from Ravenous and requires Python 3.10 or newer, Git, and network access to
|
||||||
|
PyPI, vulnerability data, and the Gitea HTTPS proxy. It creates isolated
|
||||||
|
virtual environments; uv and Docker are not required.
|
||||||
|
|
||||||
|
Before testing, CI runs two focused dependency gates:
|
||||||
|
|
||||||
|
- `pip-audit` 2.10.1 fails on known vulnerabilities or dependency-collection
|
||||||
|
errors.
|
||||||
|
- `pip-licenses` 5.5.5 permits only MIT-licensed packages in the isolated
|
||||||
|
application environment.
|
||||||
|
|
||||||
|
Coverage 7.16.1 and setuptools 80.9.0 are also pinned in
|
||||||
|
`requirements-ci.txt`. Create the isolated application environment and run the
|
||||||
|
gates locally with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv-app
|
||||||
|
.venv-app/bin/python -m pip install setuptools==80.9.0
|
||||||
|
.venv-app/bin/python -m pip install --no-build-isolation --no-deps .
|
||||||
|
.venv/bin/pip-audit --strict .
|
||||||
|
.venv/bin/pip-licenses --python=.venv-app/bin/python
|
||||||
|
```
|
||||||
|
|
||||||
|
The audit needs network access for resolution and current advisory data. The
|
||||||
|
license allow-list should only change after reviewing a new dependency. The
|
||||||
|
project currently has no runtime dependencies.
|
||||||
|
|
||||||
|
These gates do not analyze the standard library, Git, malicious packages,
|
||||||
|
provenance, or undisclosed vulnerabilities. No broader hardening tools are in
|
||||||
|
scope.
|
||||||
|
|
||||||
|
The network integration test runs by default. Disable the `RUN_CLONE_TEST`
|
||||||
|
build parameter for an offline unit-test-only run. If the HTTPS clone requires
|
||||||
|
authentication, configure Git credentials on the agent; interactive prompts
|
||||||
|
are disabled and clone failures fail the build.
|
||||||
|
|
||||||
|
The pipeline checks out and tests only. It does not publish artifacts, send
|
||||||
|
checks, push commits, or deploy files.
|
||||||
Loading…
Reference in New Issue