docs: move advanced details out of readme
Chado/repo-fix/pipeline/head This commit looks good
Details
Chado/repo-fix/pipeline/head This commit looks good
Details
This commit is contained in:
parent
988f12ff13
commit
6883130266
128
README.md
128
README.md
|
|
@ -4,125 +4,51 @@
|
|||

|
||||

|
||||
|
||||
Fix Dandaedre Gitea submodule URLs after cloning, pulling, or syncing:
|
||||
Normalize Dandaedre Gitea submodule URLs after cloning, pulling, or syncing:
|
||||
|
||||
- `.gitmodules` uses HTTPS for CI: `https://proxy.dandaedre.com/gitea/Beffroi/reportist.git`
|
||||
- `.git/config` uses SSH locally: `git@git.dandaedre.com:Beffroi/reportist.git`
|
||||
- `.git/modules/**/config` uses the same SSH format for submodule remotes
|
||||
(`url` and `pushurl`) and nested submodule URL entries.
|
||||
- `.gitmodules` uses HTTPS through `proxy.dandaedre.com` for CI.
|
||||
- Local Git configuration uses SSH through `git.dandaedre.com`.
|
||||
- Initialized and nested submodule remotes are updated as well.
|
||||
|
||||
Requires Python 3.10 or newer and Git on your PATH. No runtime pip packages are needed; the script
|
||||
uses the Python standard library and Git's config commands.
|
||||
The script requires Python 3.10 or newer and Git. It has no runtime Python
|
||||
dependencies and does not fetch repositories or submodules.
|
||||
|
||||
Preview the changes, then apply them:
|
||||
## Usage
|
||||
|
||||
Preview changes first, then apply them:
|
||||
|
||||
```bash
|
||||
python3 fix_submodules.py /path/to/repository --dry-run
|
||||
python3 fix_submodules.py /path/to/repository
|
||||
```
|
||||
|
||||
The project follows semantic versioning and currently reports version `0.1.0`:
|
||||
The repository path defaults to the current directory and may point anywhere
|
||||
inside a working tree. Repeated runs are safe.
|
||||
|
||||
```bash
|
||||
python3 fix_submodules.py --version
|
||||
```
|
||||
|
||||
The repository defaults to the current directory, and may also be a directory
|
||||
inside a working tree. Run the script again whenever a pull or `git submodule
|
||||
sync` changes the URLs. Repeated runs are safe. Dry runs do not write files.
|
||||
|
||||
Recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on `dandaedre.com`,
|
||||
`git.dandaedre.com`, and `proxy.dandaedre.com`. Old URLs such as
|
||||
`git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository paths and
|
||||
`.git` suffixes are preserved; the proxy's `/gitea/` prefix is translated.
|
||||
|
||||
Missing local URL entries are added from `.gitmodules`. Existing local URLs
|
||||
retain their own repository paths. Other hosts and relative URLs are skipped.
|
||||
Unrelated configuration, including `active`, is preserved.
|
||||
|
||||
Works with `.git` directories and pointer files, including linked worktrees
|
||||
(whose local config is shared). Existing configs under Git's `modules`
|
||||
directories are scanned recursively, including nested and deinitialized
|
||||
submodules. Their remotes (including `origin`) are repaired; the selected
|
||||
repository's own remotes are unchanged. Nothing is fetched. Nested working-tree
|
||||
`.gitmodules` files are not edited; run the script inside those submodules to
|
||||
normalize their declarations as well.
|
||||
|
||||
Commit `.gitmodules` changes for CI. Local config changes are not committed.
|
||||
After fixing URLs, initialize submodules with:
|
||||
After applying the changes, commit `.gitmodules` when it changed. Local Git
|
||||
configuration is intentionally not committed. You can then initialize the
|
||||
submodules:
|
||||
|
||||
```bash
|
||||
git -C /path/to/repository submodule update --init --recursive
|
||||
```
|
||||
|
||||
Run tests:
|
||||
Show the current semantic version with:
|
||||
|
||||
```bash
|
||||
python3 fix_submodules.py --version
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
Run the dependency-free unit suite with:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s tests -v
|
||||
```
|
||||
|
||||
The unit suite includes a small deterministic fuzz test and malformed-value
|
||||
cases for URL parsing. They are intended to catch crashes and unsafe path
|
||||
shapes, not to replace a dedicated fuzzer.
|
||||
## Advanced
|
||||
|
||||
For the same coverage check as CI, install the pinned tools and run:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements-ci.txt
|
||||
.venv/bin/coverage run -m unittest discover -s tests -v
|
||||
.venv/bin/coverage report
|
||||
```
|
||||
|
||||
Branch coverage must remain at or above 80%. This deliberately modest floor
|
||||
keeps the check useful without forcing low-value tests.
|
||||
|
||||
## Jenkins test CI
|
||||
|
||||
The `Jenkinsfile` uses the `rust_no_docker` agent from the Ravenous pipeline.
|
||||
Configure a Jenkins Pipeline from SCM (or a multibranch pipeline) for this
|
||||
repository with script path `Jenkinsfile`. The agent needs `python3` and `git`
|
||||
on PATH and network access to PyPI, vulnerability data, and the Gitea HTTPS
|
||||
proxy. No uv or Docker is required; CI creates isolated virtual environments
|
||||
and installs the pinned tools. Use the Gitea webhook URL from the Jenkinsfile
|
||||
if your Jenkins Gitea integration is configured at that address.
|
||||
|
||||
Every build runs the unit tests with coverage. Before testing, it runs two
|
||||
small supply-chain gates: `pip-audit` 2.10.1 fails on known vulnerabilities or
|
||||
dependency-collection errors, and `pip-licenses` 5.5.5 allows only MIT-licensed
|
||||
installed application packages. The CI environment also pins coverage 7.16.1
|
||||
and setuptools 80.9.0 in `requirements-ci.txt`. Run the gates locally with:
|
||||
|
||||
```bash
|
||||
.venv/bin/pip-audit --strict .
|
||||
.venv/bin/pip-licenses --python=.venv-app/bin/python
|
||||
```
|
||||
|
||||
The audit needs network access for package resolution and current advisory
|
||||
data. The license gate inspects the isolated application environment; update
|
||||
the explicit allow-list only after reviewing a new dependency. The project has
|
||||
no runtime dependencies today. These checks do not analyze the standard
|
||||
library, Git, malicious packages, provenance, or undisclosed vulnerabilities;
|
||||
no broader hardening tools are in scope.
|
||||
|
||||
By default, CI also shallow-clones
|
||||
`Ravenous/ravenous_roll20.git` through the HTTPS proxy into a temporary directory
|
||||
and verifies dry-run safety, both URL formats, repeatability, recovery after
|
||||
`git submodule sync`, and correction of the old SSH hostname. The clone is
|
||||
removed after the test. It does not fetch submodules or run code from Ravenous;
|
||||
SSH connectivity and recursive submodule downloads are not tested.
|
||||
|
||||
Run the same network test locally:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s integration -v
|
||||
```
|
||||
|
||||
The HTTPS clone URL corresponds to
|
||||
`git@dandaedre.com:Ravenous/ravenous_roll20.git` and avoids requiring an SSH key
|
||||
for this CI test. If authentication is required, configure Git credentials on
|
||||
the agent; prompts are disabled. Clone failures fail the build. Disable the
|
||||
`RUN_CLONE_TEST` build parameter for an offline unit-test-only run.
|
||||
|
||||
This pipeline only checks out and tests. It does not publish artifacts, send
|
||||
checks, push commits, or deploy files. Test results appear in the console log.
|
||||
- [URL handling and Git layout support](docs/advanced.md#url-handling-and-git-layouts)
|
||||
- [Testing, fuzz cases, and coverage](docs/advanced.md#testing-and-coverage)
|
||||
- [Jenkins CI and dependency gates](docs/advanced.md#jenkins-ci)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,95 @@
|
|||
# Advanced usage
|
||||
|
||||
## URL handling and Git layouts
|
||||
|
||||
`repo-fix` recognizes HTTP(S), SSH, Git protocol, and SCP-style URLs on
|
||||
`dandaedre.com`, `git.dandaedre.com`, and `proxy.dandaedre.com`. Legacy URLs
|
||||
such as `git@dandaedre.com:Beffroi/reportist.git` are corrected. Repository
|
||||
paths and `.git` suffixes are preserved, and the proxy's `/gitea/` prefix is
|
||||
translated when switching between HTTPS and SSH.
|
||||
|
||||
Missing local URL entries are added from `.gitmodules`. Existing local URLs
|
||||
retain their own repository paths. Other hosts, relative URLs, and malformed
|
||||
values are skipped. Unrelated configuration, including `active`, is preserved.
|
||||
|
||||
The script supports ordinary `.git` directories, `.git` pointer files, and
|
||||
linked worktrees whose local configuration is shared. It recursively scans
|
||||
existing configuration under Git's `modules` directories, including nested or
|
||||
deinitialized submodules, and repairs their remote `url`, `pushurl`, and nested
|
||||
submodule URL entries. The selected repository's own remotes are not changed.
|
||||
|
||||
Nested working-tree `.gitmodules` files are not edited. Run the script inside
|
||||
each nested submodule when those declarations also need normalization.
|
||||
|
||||
## Testing and coverage
|
||||
|
||||
The unit suite includes malformed-value cases and a small deterministic fuzz
|
||||
test for URL parsing. These checks are intended to catch crashes and unsafe
|
||||
path shapes, not to replace a dedicated fuzzer.
|
||||
|
||||
To run the same branch-coverage check as CI:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements-ci.txt
|
||||
.venv/bin/coverage run -m unittest discover -s tests -v
|
||||
.venv/bin/coverage report
|
||||
```
|
||||
|
||||
Coverage must remain at or above 80%. The threshold is intentionally modest so
|
||||
it catches meaningful regressions without encouraging low-value tests.
|
||||
|
||||
The opt-in integration test shallow-clones
|
||||
`Ravenous/ravenous_roll20.git` through the HTTPS proxy and exercises dry-run
|
||||
safety, both URL formats, repeatability, recovery after `git submodule sync`,
|
||||
and correction of the legacy SSH hostname:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s integration -v
|
||||
```
|
||||
|
||||
The clone is removed after the test. It does not fetch submodules, execute code
|
||||
from Ravenous, test SSH connectivity, or recursively download submodules.
|
||||
|
||||
## Jenkins CI
|
||||
|
||||
Configure Jenkins as a Pipeline from SCM or a multibranch pipeline with
|
||||
`Jenkinsfile` as the script path. The pipeline uses the `rust_no_docker` agent
|
||||
from Ravenous and requires Python 3.10 or newer, Git, and network access to
|
||||
PyPI, vulnerability data, and the Gitea HTTPS proxy. It creates isolated
|
||||
virtual environments; uv and Docker are not required.
|
||||
|
||||
Before testing, CI runs two focused dependency gates:
|
||||
|
||||
- `pip-audit` 2.10.1 fails on known vulnerabilities or dependency-collection
|
||||
errors.
|
||||
- `pip-licenses` 5.5.5 permits only MIT-licensed packages in the isolated
|
||||
application environment.
|
||||
|
||||
Coverage 7.16.1 and setuptools 80.9.0 are also pinned in
|
||||
`requirements-ci.txt`. Create the isolated application environment and run the
|
||||
gates locally with:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv-app
|
||||
.venv-app/bin/python -m pip install setuptools==80.9.0
|
||||
.venv-app/bin/python -m pip install --no-build-isolation --no-deps .
|
||||
.venv/bin/pip-audit --strict .
|
||||
.venv/bin/pip-licenses --python=.venv-app/bin/python
|
||||
```
|
||||
|
||||
The audit needs network access for resolution and current advisory data. The
|
||||
license allow-list should only change after reviewing a new dependency. The
|
||||
project currently has no runtime dependencies.
|
||||
|
||||
These gates do not analyze the standard library, Git, malicious packages,
|
||||
provenance, or undisclosed vulnerabilities. No broader hardening tools are in
|
||||
scope.
|
||||
|
||||
The network integration test runs by default. Disable the `RUN_CLONE_TEST`
|
||||
build parameter for an offline unit-test-only run. If the HTTPS clone requires
|
||||
authentication, configure Git credentials on the agent; interactive prompts
|
||||
are disabled and clone failures fail the build.
|
||||
|
||||
The pipeline checks out and tests only. It does not publish artifacts, send
|
||||
checks, push commits, or deploy files.
|
||||
Loading…
Reference in New Issue