Build and publish project mdBook manuals through Jenkins
Beffroi/beffroi-tools/pipeline/head This commit looks good Details

This commit is contained in:
Dandaedre 2026-11-14 20:44:11 +01:00
parent 6ba5b06e90
commit f88450be83
26 changed files with 838 additions and 1 deletions

20
Jenkinsfile vendored
View File

@ -4,6 +4,7 @@ pipeline {
options {
skipDefaultCheckout(true)
disableConcurrentBuilds()
timeout(time: 45, unit: 'MINUTES')
}
@ -13,6 +14,7 @@ pipeline {
NO_COLOR = '1'
CARGO_TERM_COLOR = 'never'
BADGE_ROOT = '/mnt/assets/coverage'
MANUALS_ROOT = '/mnt/books/manuals'
}
stages {
@ -36,6 +38,11 @@ pipeline {
cargo install cargo-llvm-cov --version 0.9.1 --locked
fi
rustup component add llvm-tools-preview clippy rustfmt
mdbook_version=$(cat docs/manuals/mdbook-version)
if ! mdbook --version 2>/dev/null | grep -Fx "mdbook v${mdbook_version}"; then
cargo install mdbook --version "$mdbook_version" --locked
fi
command -v rsync
'''
}
}
@ -68,9 +75,22 @@ pipeline {
}
}
}
stage('Build manuals') {
steps { sh 'python3 scripts/manuals.py build' }
}
stage('Coverage') {
steps { sh 'python3 scripts/ci.py coverage' }
}
stage('Publish manuals') {
when {
allOf {
branch 'main'
not { changeRequest() }
expression { currentBuild.currentResult == 'SUCCESS' }
}
}
steps { sh 'python3 scripts/manuals.py publish --destination "$MANUALS_ROOT"' }
}
}
post {

View File

@ -12,6 +12,7 @@
<a href="https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools/src/branch/main/docs/development/releases.md">Portable releases</a> ·
<a href="#quick-start">Quick start</a> ·
<a href="https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools/src/branch/main/docs/README.md">Documentation</a> ·
<a href="https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools/src/branch/main/docs/manuals/README.md">Manuals</a> ·
<a href="https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools/src/branch/main/docs/changelogs.md">What's new</a> ·
<a href="https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools/src/branch/main/docs/development/README.md">Development</a>
</p>

View File

@ -15,6 +15,8 @@
## Shared resources
- **mdBook manuals:** [all six manuals, local preview and Jenkins publication](manuals/README.md).
- **Development:** [setup and checks](development/README.md), [CI and coverage](development/ci.md), [builds and portable releases](development/releases.md), [application artwork](development/releases.md#application-artwork).
- **File and device contracts:** [interface index](interfaces/README.md), including versioned examples and compatibility rules.
- **Starter files:** [Midra Layout catalog](templates/midra-layout/README.md), with Full HD and 4K base files in all nine exchange formats.

View File

@ -4,6 +4,8 @@
## Unreleased
- Add an mdBook manual built from the existing guides, with local assets and downloads. Jenkins publishes successful main builds under `/mnt/books/manuals/`; staging, link validation and scoped publication have regression coverage.
- Use absolute Gitea main-branch URLs for repository file links in the root README; verify linked files exist.
## v0.6.0 — 2026-11-14

View File

@ -8,7 +8,9 @@ of its successful-rename paths. Those paths must also run on a supported host;
see [FreeBSD test coverage](../ascii-names/README.md#freebsd-ci-hosts).
SCM credentials, webhooks and the publication mount are administrator setup.
CI checks formatting, Clippy, rustdoc, Python/Rust tests and
whole-workspace Rust coverage. It never creates tags or release artifacts.
whole-workspace Rust coverage. It also builds six mdBook manuals and publishes
successful non-PR main builds to `/mnt/books/manuals/`. It never creates tags or
release artifacts.
The Python reporter and cross-language parity checks are retired. CI uses the
Rust reporter's own regression tests and embedded PDF engine, plus Python
@ -85,3 +87,21 @@ Actual Jenkins archival/publication and public URL retrieval are separate
external checkpoints. Local checks do not establish that badges are live.
The asset server should mount assets read-only, disable listings, serve SVG
with restrictive CSP, and revalidate mutable URLs or use a short cache lifetime.
## Project manuals
The Tools stage installs the mdBook version pinned in
[`docs/manuals/mdbook-version`](../manuals/mdbook-version) and requires `rsync`.
The Build manuals stage stages existing documentation, builds all six books, and
checks rendered links, anchors and downloadable assets. No mdBook plugins or
additional Jenkins plugins are needed. Tests cover staging and publication into
temporary directories, including stale-file cleanup and preservation of unrelated
manuals. Publication runs after coverage, only when the whole build is successful,
on non-PR main. The helper independently enforces the branch/PR restriction.
The worker must expose the existing writable `/mnt/books` mount. The six manuals
and a shared landing page are published under `/mnt/books/manuals/`; deletion is
scoped to each owned book directory. This follows the referenced mdBook pipeline's
Cargo/build/rsync method while keeping the existing badge cleanup behavior intact.
See [manual configuration, preview and publication](../manuals/README.md) for
paths, prerequisites and update behavior.

View File

@ -4,6 +4,8 @@
## Unreleased
- Add an mdBook manual built from the existing guides, with local assets and downloads. Jenkins publishes successful main builds under `/mnt/books/manuals/`; staging, link validation and scoped publication have regression coverage.
- Use absolute Gitea main-branch URLs for repository file links in the root README; verify linked files exist.
## v0.6.0 — 2026-11-14

104
docs/manuals/README.md Normal file
View File

@ -0,0 +1,104 @@
# Project manuals
[Documentation](../README.md) · [CI and coverage](../development/ci.md)
Each project has an mdBook manual, built from the existing documentation. Edit the
original guides; the build stages them without creating a second maintained copy.
The published landing page includes all six project icons and manual links.
| Manual | Contents | Published directory |
| --- | --- | --- |
| ASCII Names | [Chapter list](ascii-names/SUMMARY.md) | `/mnt/books/manuals/ascii-names/` |
| Document Exporter | [Chapter list](exporter/SUMMARY.md) | `/mnt/books/manuals/exporter/` |
| MPC-HC Companion | [Chapter list](media-player-classic/SUMMARY.md) | `/mnt/books/manuals/media-player-classic/` |
| Midra Layout | [Chapter list](midra-layout/SUMMARY.md) | `/mnt/books/manuals/midra-layout/` |
| PPTX Reporter | [Chapter list](pptx-reporter/SUMMARY.md) | `/mnt/books/manuals/pptx-reporter/` |
| Rundown Converter | [Chapter list](rundown-converter/SUMMARY.md) | `/mnt/books/manuals/rundown-converter/` |
The landing page is `/mnt/books/manuals/index.html`. The books assume the books
server exposes that directory as `/manuals/`; each `book.toml` declares the
corresponding `site-url` for its 404 page. No public hostname is hardcoded.
## Build and preview
Requires Python 3.11+, Rust/Cargo for installation, and mdBook **0.5.4**, pinned in
[mdbook-version](mdbook-version). No mdBook plugins are required. From the repository root:
```sh
cargo install mdbook --version "$(cat docs/manuals/mdbook-version)" --locked
python3 scripts/manuals.py build
python3 scripts/manuals.py check
python3 -m http.server 8000 --directory build/manuals/site
```
Open `http://localhost:8000/manuals/`. Search, chapter navigation, themes and
printable manuals are supplied by mdBook. To use a separately installed binary,
pass `--mdbook /path/to/mdbook` to the build command.
Generated source trees are in `build/manuals/sources/<project>/`; rendered files
are in `build/manuals/site/manuals/`. Both are ignored. All books must build and
pass rendered-link validation before the output is made available for publication.
The validator checks local assets, downloads, chapter links and heading fragments;
it does not request external websites. mdBook's generated 404 heading links are
adjusted to remain on the 404 page despite its HTML base URL.
## Maintain a manual
Each project directory here contains a `book.toml` and `SUMMARY.md`:
- `SUMMARY.md` lists canonical Markdown chapters with paths relative to itself.
Add or reorder entries here. The linked files remain readable in the repository.
- `book.toml` configures the staged mdBook. Its `[manual]` table provides the
repository-relative project icon used by the landing page; the staging script
removes this table before calling mdBook.
- Run the helper rather than running `mdbook build` directly in these configuration
directories: it creates the `src` tree and rewrites chapter and asset links.
- Included chapters link to other included chapters locally. References to guides
outside the current book, source code and repository directories link to Gitea.
Referenced screenshots, icons, text quick starts, schema examples and starter
files are copied into the book with their original bytes. Midra's catalog includes
all 18 starter files, including native PPTX, FIG and AI downloads.
- Fenced code blocks retain their literal command examples. Keep documentation
links in ordinary Markdown or quoted HTML `href`/`src` attributes; the helper
currently handles inline Markdown links without optional title attributes.
- Update the tool's changelog for documentation changes. Keep generated HTML and
staging files out of Git.
## Jenkins publication
The repository [Jenkinsfile](../../Jenkinsfile) follows the supplied
[Deathwatch mdBook pipeline](https://proxy.dandaedre.com/gitea/Books/deathwatch-d10/raw/branch/main/Jenkinsfile):
use `rust_no_docker`, install mdBook with Cargo, build HTML, then publish using
`rsync -a --delete`. Here mdBook is pinned, installation errors fail the build,
and no unused preprocessors or custom Jenkins warning-parser plugins are installed.
Rendered-link validation supplies a build gate before publication.
Jenkins needs `rsync` on PATH and an existing writable `/mnt/books` mount visible
inside the worker jail. The helper creates `manuals/` and the six book directories;
it fails if the parent mount is missing. Web server routing and that mount are
administrator setup. Jenkins does not create them.
Every branch builds the manuals. Only a successful non-PR `main` build publishes,
after the test and coverage stages. Concurrent builds of the same job are disabled.
The publication helper also rejects any environment without `BRANCH_NAME=main`
or with `CHANGE_ID` set.
```sh
# Run by Jenkins, with its existing branch/PR environment:
python3 scripts/manuals.py publish --destination /mnt/books/manuals
```
Publication validates the whole rendered set before writing, refuses symlinked
source/destination directories, and runs `rsync --delete` separately inside each
of the six owned book directories. It never synchronizes or deletes the shared
`/mnt/books` or `manuals` root. Unrelated manuals and root files survive. The landing
page is replaced atomically after all book synchronizations succeed. Each book's
rsync is an in-place update, not an atomic release of the entire collection; a
failed rsync fails Jenkins and leaves the existing landing page in place. A later
successful build retries the complete set. Removing a manual configuration does
not automatically delete its previously published directory.
Existing badge archival/publication stays separate and still archives exactly
three SVGs. This workflow does not create tags, releases or release binaries.
Local builds and temporary publication tests do not mean the live books server
has been updated; that happens when this commit runs successfully on Jenkins main.

View File

@ -0,0 +1,6 @@
# Summary
- [User guide](../../ascii-names/README.md)
- [Builds and portable releases](../../development/releases.md)
- [Development](../../development/README.md)
- [Changelog](../../ascii-names/changelog.md)

View File

@ -0,0 +1,25 @@
[book]
title = "ASCII Names Manual"
description = "Clean up filenames while preserving file contents."
language = "en"
src = "src"
[build]
create-missing = false
[output.html]
site-url = "/manuals/ascii-names/"
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools"
[output.html.fold]
enable = true
level = 1
[output.html.playground]
runnable = false
# Consumed by scripts/manuals.py when staging the book and landing page.
[manual]
icon = "crates/ascii-names/assets/app-icon.png"

View File

@ -0,0 +1,5 @@
# Summary
- [User guide](../../exporter/README.md)
- [Development](../../development/README.md)
- [Changelog](../../exporter/changelog.md)

View File

@ -0,0 +1,25 @@
[book]
title = "Document Exporter Manual"
description = "Export presentations and PDF pages, and compress PDFs."
language = "en"
src = "src"
[build]
create-missing = false
[output.html]
site-url = "/manuals/exporter/"
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools"
[output.html.fold]
enable = true
level = 1
[output.html.playground]
runnable = false
# Consumed by scripts/manuals.py when staging the book and landing page.
[manual]
icon = "tools/assets/exporter/app-icon.png"

View File

@ -0,0 +1 @@
0.5.4

View File

@ -0,0 +1,6 @@
# Summary
- [Setup and workflows](../../../companion-modules/media-player-classic/README.md)
- [Connection help and presets](../../../companion-modules/media-player-classic/companion/HELP.md)
- [Upstream attribution](../../../companion-modules/media-player-classic/UPSTREAM.md)
- [Changelog](../../media-player-classic/changelog.md)

View File

@ -0,0 +1,25 @@
[book]
title = "MPC-HC Companion Manual"
description = "Control Media Player Classic Home Cinema from Companion."
language = "en"
src = "src"
[build]
create-missing = false
[output.html]
site-url = "/manuals/media-player-classic/"
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools"
[output.html.fold]
enable = true
level = 1
[output.html.playground]
runnable = false
# Consumed by scripts/manuals.py when staging the book and landing page.
[manual]
icon = "companion-modules/media-player-classic/companion/assets/app-icon.png"

View File

@ -0,0 +1,25 @@
# Summary
- [Introduction](../../midra-layout/README.md)
- [User guide contents](../../midra-layout/guide/README.md)
- [Layers and editing](../../midra-layout/guide/editing.md)
- [Canvas and reference images](../../midra-layout/guide/canvas.md)
- [Formats, crop and precision](../../midra-layout/guide/formats-and-crop.md)
- [Import, export and project files](../../midra-layout/guide/files.md)
- [Menus and preferences](../../midra-layout/guide/preferences.md)
- [Midra connection and transfers](../../midra-layout/guide/device.md)
- [Starter files for every format](../../templates/midra-layout/README.md)
- [Resolution and cropping model](../../midra-layout/resolutions.md)
- [Project file contract](../../interfaces/midra-layout.md)
- [Geometry exchange contract](../../interfaces/midra-layout-exchange.md)
- [Settings contract](../../interfaces/midra-layout-settings.md)
- [Recovery contract](../../interfaces/midra-layout-recovery.md)
- [Device import contract](../../interfaces/midra-device-import.md)
- [Device send contract](../../interfaces/midra-device-send.md)
- [Input preview contract](../../interfaces/midra-input-preview.md)
- [Hardware verification](../../midra-layout/device-check.md)
- [Viewport performance](../../midra-layout/guide/performance.md)
- [Updating screenshots](../../midra-layout/images/README.md)
- [Plan and limitations](../../midra-layout/plan.md)
- [Development](../../development/README.md)
- [Changelog](../../midra-layout/changelog.md)

View File

@ -0,0 +1,25 @@
[book]
title = "Midra Layout Manual"
description = "Prepare screen layouts, exchange geometry and review device transfers."
language = "en"
src = "src"
[build]
create-missing = false
[output.html]
site-url = "/manuals/midra-layout/"
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools"
[output.html.fold]
enable = true
level = 1
[output.html.playground]
runnable = false
# Consumed by scripts/manuals.py when staging the book and landing page.
[manual]
icon = "crates/midra-layout/assets/app-icon.png"

View File

@ -0,0 +1,8 @@
# Summary
- [Getting started](../../pptx-reporter/README.md)
- [Visual walkthrough](../../pptx-reporter/visual-guide.md)
- [Report formats and timing](../../pptx-reporter/report-formats.md)
- [Builds and portable releases](../../development/releases.md)
- [Development](../../development/README.md)
- [Changelog](../../pptx-reporter/changelog.md)

View File

@ -0,0 +1,25 @@
[book]
title = "PPTX Reporter Manual"
description = "Inspect presentation videos and export timing reports."
language = "en"
src = "src"
[build]
create-missing = false
[output.html]
site-url = "/manuals/pptx-reporter/"
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools"
[output.html.fold]
enable = true
level = 1
[output.html.playground]
runnable = false
# Consumed by scripts/manuals.py when staging the book and landing page.
[manual]
icon = "crates/pptx-reporter/assets/app-icon.png"

View File

@ -0,0 +1,8 @@
# Summary
- [User guide](../../rundown-converter/README.md)
- [Sample findings](../../rundown-converter/corpus.md)
- [Plan and limitations](../../rundown-converter/plan.md)
- [Builds and portable releases](../../development/releases.md)
- [Development](../../development/README.md)
- [Changelog](../../rundown-converter/changelog.md)

View File

@ -0,0 +1,25 @@
[book]
title = "Rundown Converter Manual"
description = "Convert spreadsheet rundowns into validated show schedules."
language = "en"
src = "src"
[build]
create-missing = false
[output.html]
site-url = "/manuals/rundown-converter/"
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools"
[output.html.fold]
enable = true
level = 1
[output.html.playground]
runnable = false
# Consumed by scripts/manuals.py when staging the book and landing page.
[manual]
icon = "crates/rundown-converter/assets/app-icon.png"

View File

@ -4,6 +4,8 @@
## Unreleased
- Add an mdBook manual built from the existing guides, with local assets and downloads. Jenkins publishes successful main builds under `/mnt/books/manuals/`; staging, link validation and scoped publication have regression coverage.
- Use absolute Gitea main-branch URLs for repository file links in the root README; verify linked files exist.
## v0.6.0 — 2026-11-14

View File

@ -4,6 +4,8 @@
## Unreleased
- Add an mdBook manual built from the existing guides, with local assets and downloads. Jenkins publishes successful main builds under `/mnt/books/manuals/`; staging, link validation and scoped publication have regression coverage.
- Replace the long overview with a visual quick start, move detailed workflows into topic guides, and centralize all nine starter-file formats in one catalog. Add real application screenshots with reproducible capture notes; retain the existing template files and validation contracts.
- Clarify shared Midra links: warn with other linked layer/folder names while allowing link edits; disable conflicting scoped sends and list all overlapping destinations. Preserve conflict-free folder/selection sends and remapping. Remove nested-folder/additional-codec backlog items and replace the extra-links proposal with this shared-destination workflow. Regression tests cover warning text, send blocking and scope isolation.

View File

@ -4,6 +4,8 @@
## Unreleased
- Add an mdBook manual built from the existing guides, with local assets and downloads. Jenkins publishes successful main builds under `/mnt/books/manuals/`; staging, link validation and scoped publication have regression coverage.
- Use absolute Gitea main-branch URLs for repository file links in the root README; verify linked files exist.
## v0.6.0 — 2026-11-14

View File

@ -4,6 +4,8 @@
## Unreleased
- Add an mdBook manual built from the existing guides, with local assets and downloads. Jenkins publishes successful main builds under `/mnt/books/manuals/`; staging, link validation and scoped publication have regression coverage.
- Use absolute Gitea main-branch URLs for repository file links in the root README; verify linked files exist.
## v0.6.0 — 2026-11-14

282
scripts/manuals.py Normal file
View File

@ -0,0 +1,282 @@
#!/usr/bin/env python3
"""Stage, build, check and publish the project's mdBook manuals (Python 3.11+)."""
import argparse
import html
from html.parser import HTMLParser
import os
from pathlib import Path
import re
import shutil
import subprocess
import tempfile
import tomllib
from urllib.parse import quote, unquote, urljoin, urlsplit
ROOT = Path(__file__).resolve().parents[1]
CONFIG = ROOT / "docs/manuals"
OUTPUT = ROOT / "build/manuals"
REPOSITORY = "https://proxy.dandaedre.com/gitea/Beffroi/beffroi-tools/src/branch/main/"
DOWNLOADS = {".png", ".jpg", ".jpeg", ".svg", ".gif", ".json", ".csv", ".dxf",
".html", ".drawio", ".pptx", ".fig", ".ai", ".txt"}
LINK = re.compile(r"(?P<start>\]\()(?P<url>[^\s)]+)(?P<end>\))"
r"|(?P<attr>(?:href|src)=[\"'])(?P<html>[^\"']+)(?P<quote>[\"'])")
def source_path(root, base, path):
"""Resolve an existing source inside the repository, never following a symlink."""
target = Path(os.path.abspath(base / unquote(path)))
relative = target.relative_to(root)
if target.resolve() != target or not target.exists():
raise ValueError(f"Missing or symlinked manual source: {relative}")
return relative
def chapter_path(path):
# Avoid mdBook's special handling of README.md when rewriting links.
return path.with_name("index.md") if path.name == "README.md" else path
def books(root=ROOT):
result = []
for config in sorted((root / "docs/manuals").glob("*/book.toml")):
slug = config.parent.name
if not re.fullmatch(r"[a-z][a-z0-9-]+", slug):
raise ValueError(f"Invalid manual directory: {slug}")
data = tomllib.loads(config.read_text())
summary = config.with_name("SUMMARY.md").read_text()
chapters = {}
def entry(match):
path = source_path(root, config.parent, match[1])
if path.suffix != ".md" or not (root / path).is_file() or path in chapters:
raise ValueError(f"Invalid or duplicate chapter: {path}")
chapters[path] = chapter_path(path) if chapters else Path("index.md")
return f"]({chapters[path].as_posix()})"
staged_summary = re.sub(r"\]\(([^)]+)\)", entry, summary)
if not chapters:
raise ValueError(f"Empty manual: {slug}")
icon = source_path(root, root, data["manual"]["icon"])
result.append(dict(slug=slug, config=config, data=data, chapters=chapters,
summary=staged_summary, icon=icon))
if not result:
raise ValueError("No manual configurations found")
return result
def stage_book(book, destination, root=ROOT):
src = destination / "src"
src.mkdir(parents=True)
# mdBook sees only its own configuration, not the staging metadata.
config = book["config"].read_text().split("[manual]", 1)[0]
(destination / "book.toml").write_text(config)
(src / "SUMMARY.md").write_text(book["summary"])
def copy_asset(path):
target = src / path
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(root / path, target)
copy_asset(book["icon"])
for original, staged in book["chapters"].items():
def rewrite(url):
parsed = urlsplit(html.unescape(url))
if parsed.scheme or parsed.netloc or not parsed.path:
return url
path = source_path(root, (root / original).parent, parsed.path)
suffix = ("?" + parsed.query if parsed.query else "")
suffix += ("#" + parsed.fragment if parsed.fragment else "")
if path in book["chapters"]:
target = book["chapters"][path].with_suffix(".html")
elif (root / path).is_file() and path.suffix.lower() in DOWNLOADS:
copy_asset(path)
target = path
else:
# Out-of-book documentation/code stays a working repository link.
return REPOSITORY + quote(path.as_posix()) + suffix
return quote(os.path.relpath(target, staged.parent).replace(os.sep, "/")) + suffix
def replace(match):
if match["url"] is not None:
return match["start"] + rewrite(match["url"]) + match["end"]
return match["attr"] + rewrite(match["html"]) + match["quote"]
# Commands and examples must retain their literal paths and markup.
chunks = re.split(r"(^```[^\n]*\n.*?^```[^\n]*(?:\n|$))",
(root / original).read_text(), flags=re.M | re.S)
text = "".join(chunk if i % 2 else LINK.sub(replace, chunk)
for i, chunk in enumerate(chunks))
portal = os.path.relpath("../index.html", staged.parent).replace(os.sep, "/")
text = f'<p><a href="{portal}">All manuals</a></p>\n\n' + text
target = src / staged
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(text)
class Page(HTMLParser):
def __init__(self, text):
super().__init__(convert_charrefs=True)
self.links = []
self.ids = set()
self.base = None
self.feed(text)
def handle_starttag(self, tag, attrs):
attrs = dict(attrs)
if tag == "base":
self.base = attrs.get("href")
return
for key in ("id", "name"):
if key in attrs:
self.ids.add(attrs[key])
for key in ("href", "src"):
if key in attrs:
self.links.append(attrs[key])
def validate_site(site, manuals):
"""Check real rendered pages, including chapter fragments and downloads."""
site = site.absolute()
required = [site / "index.html"] + [site / b["slug"] / "index.html" for b in manuals]
if any(not path.is_file() for path in required):
raise ValueError("Manual output is incomplete: missing landing page or book index")
if any(path.is_symlink() for path in [site, *site.rglob("*")]):
raise ValueError("Manual output must not contain symlinks")
pages = {p: Page(p.read_text()) for p in site.rglob("*.html")}
count = 0
for file, page in pages.items():
for url in page.links:
location = "/manuals/" + file.relative_to(site).as_posix()
parsed = urlsplit(urljoin(page.base or location, url))
if parsed.scheme or parsed.netloc:
continue
path = unquote(parsed.path)
if not path.startswith("/manuals/"):
raise ValueError(f"Link escapes manual site: {file}: {url}")
dest = Path(os.path.abspath(site / path.removeprefix("/manuals/")))
if not dest.is_relative_to(site):
raise ValueError(f"Link escapes manual site: {file}: {url}")
if dest.is_dir():
dest /= "index.html"
if not dest.exists():
raise ValueError(f"Broken manual link: {file}: {url}")
fragment = unquote(parsed.fragment)
if fragment and dest in pages and fragment not in pages[dest].ids:
raise ValueError(f"Broken manual anchor: {file}: {url}")
count += 1
print(f"Validated {len(pages)} HTML pages and {count} local links/assets.", flush=True)
def landing_page(manuals):
cards = []
for book in manuals:
info = book["data"]["book"]
cards.append(f'<a class="card" href="{book["slug"]}/index.html">'
f'<img src="{book["slug"]}/{book["icon"].as_posix()}" alt="">'
f'<h2>{html.escape(info["title"])}</h2>'
f'<p>{html.escape(info["description"])}</p></a>')
return '''<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>beffroi-tools manuals</title><style>
:root { color-scheme: light dark; font-family: system-ui, sans-serif; }
body { max-width: 1000px; margin: 3rem auto; padding: 0 1.5rem; line-height: 1.5; }
main { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: 1rem; }
.card { color: inherit; text-decoration: none; border: 1px solid #8888; border-radius: .8rem; padding: 1.5rem; }
.card:hover, .card:focus-visible { outline: 2px solid #7777dd; }
.card img { width: 64px; height: 64px; } h2 { font-size: 1.25rem; }
</style></head><body><h1>beffroi-tools manuals</h1>
<p>Guides and reference for local document, show and media workflows.</p><main>
''' + "\n".join(cards) + "\n</main></body></html>\n"
def build(mdbook, root=ROOT, output=OUTPUT):
manuals = books(root)
version = (root / "docs/manuals/mdbook-version").read_text().strip()
actual = subprocess.check_output([mdbook, "--version"], text=True).strip()
if actual != f"mdbook v{version}":
raise ValueError(f"Expected mdbook v{version}, got {actual}")
if output.is_symlink():
raise ValueError("Refusing symlinked manual build directory")
if output.exists():
shutil.rmtree(output)
output.parent.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(prefix=".manuals-", dir=output.parent) as temp:
work = Path(temp)
site = work / "site/manuals"
site.mkdir(parents=True)
for book in manuals:
source = work / "sources" / book["slug"]
stage_book(book, source, root)
subprocess.run([mdbook, "build", str(source), "--dest-dir", str(site / book["slug"])],
check=True)
# mdBook's 404 <base> otherwise sends its own heading links to index.html.
not_found = site / book["slug"] / "404.html"
not_found.write_text(not_found.read_text().replace('href="#', 'href="404.html#'))
(site / "index.html").write_text(landing_page(manuals))
validate_site(site, manuals)
work.rename(output)
print(f"Manuals ready: {output / 'site/manuals/index.html'}", flush=True)
def safe_destination(path):
if not path.is_absolute():
raise ValueError("Publication destination must be absolute")
for component in [*reversed(path.parents), path]:
if component.is_symlink() or (component.exists() and not component.is_dir()):
raise ValueError(f"Unsafe publication directory: {component}")
def publish(site, destination, manuals, environment=None):
environment = os.environ if environment is None else environment
if environment.get("BRANCH_NAME") != "main" or environment.get("CHANGE_ID"):
raise ValueError("Only non-PR main builds may publish manuals")
safe_destination(destination)
# A missing mount is an infrastructure error, not a reason to create /mnt/books.
if not destination.parent.is_dir():
raise ValueError(f"Publication parent/mount is missing: {destination.parent}")
for book in manuals:
safe_destination(destination / book["slug"])
if (destination / "index.html").is_symlink() or (destination / "index.html").is_dir():
raise ValueError("Unsafe manual landing-page destination")
validate_site(site, manuals)
destination.mkdir(exist_ok=True)
# Delete stale pages only within each explicitly owned book, never the shared root.
for book in manuals:
target = destination / book["slug"]
target.mkdir(exist_ok=True)
subprocess.run(["rsync", "-a", "--delete", str(site / book["slug"]) + "/",
str(target) + "/"], check=True)
with tempfile.NamedTemporaryFile(prefix=".manual-index-", dir=destination, delete=False) as temp:
temporary = Path(temp.name)
try:
shutil.copyfile(site / "index.html", temporary)
temporary.chmod(0o644)
temporary.replace(destination / "index.html")
finally:
temporary.unlink(missing_ok=True)
print(f"Published {len(manuals)} manuals to {destination}", flush=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest="command", required=True)
builder = commands.add_parser("build")
builder.add_argument("--mdbook", default="mdbook")
commands.add_parser("check")
publisher = commands.add_parser("publish")
publisher.add_argument("--destination", type=Path, default=Path("/mnt/books/manuals"))
args = parser.parse_args()
try:
if args.command == "build":
build(args.mdbook)
elif args.command == "check":
validate_site(OUTPUT / "site/manuals", books())
else:
publish(OUTPUT / "site/manuals", args.destination, books())
except (ValueError, OSError, subprocess.CalledProcessError) as error:
parser.exit(1, f"Manuals: {error}\n")
if __name__ == "__main__":
main()

187
tests/test_manuals.py Normal file
View File

@ -0,0 +1,187 @@
import importlib.util
from pathlib import Path
import shutil
import subprocess
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location("manuals", ROOT / "scripts/manuals.py")
manuals = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(manuals)
class ManualTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
def write(self, name, text):
path = self.root / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text)
return path
def book(self):
self.write("docs/manuals/example/book.toml", '''[book]
title = "Example"
description = "Example manual"
src = "src"
[manual]
icon = "docs/icon.png"
''')
self.write("docs/manuals/example/SUMMARY.md", "# Summary\n\n"
"- [Intro](../../tool/README.md)\n- [Details](../../tool/detail.md)\n")
self.write("docs/icon.png", "image fixture")
self.write("docs/tool/README.md", '# Intro\n\n[Detail](detail.md#values)\n'
'[Other](../other.md)\n![Icon](../icon.png)\n'
'<img src="../icon.png">\n[Data](../sample.json)\n'
'```md\n[Literal](missing.md)\n```\n')
self.write("docs/tool/detail.md", "# Details\n\n## Values\n\n[Home](README.md)\n")
self.write("docs/other.md", "# Other")
self.write("docs/sample.json", '{"value": 42}')
return manuals.books(self.root)[0]
def test_stage_rebases_chapters_assets_and_repository_links_without_copying_prose(self):
book = self.book()
before = (self.root / "docs/tool/README.md").read_bytes()
stage = self.root / "stage"
manuals.stage_book(book, stage, self.root)
intro = (stage / "src/index.md").read_text()
self.assertIn("](docs/tool/detail.html#values)", intro)
self.assertIn(manuals.REPOSITORY + "docs/other.md", intro)
self.assertIn('src="docs/icon.png"', intro)
self.assertIn("[Data](docs/sample.json)", intro)
self.assertIn("[Literal](missing.md)", intro)
self.assertIn("[Home](../../index.html)", (stage / "src/docs/tool/detail.md").read_text())
self.assertEqual((stage / "src/docs/sample.json").read_bytes(),
(self.root / "docs/sample.json").read_bytes())
self.assertFalse((stage / "src/docs/other.md").exists())
self.assertNotIn("[manual]", (stage / "book.toml").read_text())
self.assertEqual((self.root / "docs/tool/README.md").read_bytes(), before)
def test_missing_duplicate_outside_and_symlinked_sources_are_rejected(self):
self.book()
summary = self.root / "docs/manuals/example/SUMMARY.md"
for links in ("../../missing.md", "../../../../outside.md"):
summary.write_text(f"- [Bad]({links})\n")
with self.assertRaises(ValueError):
manuals.books(self.root)
summary.write_text("- [One](../../tool/README.md)\n- [Again](../../tool/README.md)\n")
with self.assertRaises(ValueError):
manuals.books(self.root)
summary.write_text("- [Link](../../linked.md)\n")
(self.root / "docs/linked.md").symlink_to(self.root / "docs/tool/README.md")
with self.assertRaises(ValueError):
manuals.books(self.root)
def site(self):
site = self.root / "site"
self.write("site/index.html", '<a href="example/index.html">Example</a>')
self.write("site/example/index.html", '<h1 id="intro">Intro</h1>'
'<a href="../index.html">Manuals</a><a href="#intro">Intro</a>'
'<a href="downloads/file.json">File</a>')
self.write("site/example/downloads/file.json", '{"value": 42}')
return site, [{"slug": "example"}]
def test_rendered_validation_checks_fragments_downloads_and_subpath_404_base(self):
site, books = self.site()
page = self.write("site/example/404.html", '<base href="/manuals/example/">'
'<h1 id="missing">Missing</h1><a href="404.html#missing">Missing</a>')
manuals.validate_site(site, books)
for url in ("index.html#missing", "absent.json", "../../outside.html"):
page.write_text(f'<a href="{url}">Invalid</a>')
with self.assertRaises(ValueError):
manuals.validate_site(site, books)
page.unlink()
(site / "example/downloads/file.json").unlink()
with self.assertRaises(ValueError):
manuals.validate_site(site, books)
@unittest.skipUnless(shutil.which("rsync"), "rsync is required for publication integration")
def test_publish_replaces_only_owned_manual_and_landing_page(self):
site, books = self.site()
destination = self.root / "public/manuals"
self.write("public/manuals/example/stale.html", "stale")
sibling = self.write("public/manuals/unrelated-book/keep.html", "keep")
root_file = self.write("public/manuals/keep.txt", "keep")
self.write("public/manuals/index.html", "old")
manuals.publish(site, destination, books, {"BRANCH_NAME": "main"})
self.assertFalse((destination / "example/stale.html").exists())
self.assertEqual((destination / "index.html").read_bytes(), (site / "index.html").read_bytes())
self.assertEqual((destination / "example/downloads/file.json").read_bytes(),
(site / "example/downloads/file.json").read_bytes())
self.assertEqual(sibling.read_text(), "keep")
self.assertEqual(root_file.read_text(), "keep")
def test_publish_rejects_non_main_pr_incomplete_output_and_missing_mount(self):
site, books = self.site()
destination = self.root / "manuals"
for env in ({}, {"BRANCH_NAME": "feature"}, {"BRANCH_NAME": "main", "CHANGE_ID": "3"}):
with self.assertRaises(ValueError):
manuals.publish(site, destination, books, env)
with self.assertRaises(ValueError):
manuals.publish(site, destination, books + [{"slug": "absent"}], {"BRANCH_NAME": "main"})
self.assertFalse(destination.exists())
with self.assertRaises(ValueError):
manuals.publish(site, self.root / "missing-mount/manuals", books, {"BRANCH_NAME": "main"})
self.assertFalse((self.root / "missing-mount").exists())
def test_publish_rejects_symlinked_output_and_destination_before_writing(self):
site, books = self.site()
destination = self.root / "manuals"
destination.mkdir()
outside = self.root / "outside"
outside.mkdir()
(destination / "example").symlink_to(outside, target_is_directory=True)
with self.assertRaises(ValueError):
manuals.publish(site, destination, books, {"BRANCH_NAME": "main"})
self.assertEqual(list(outside.iterdir()), [])
(destination / "example").unlink()
(site / "escape").symlink_to(outside, target_is_directory=True)
with self.assertRaises(ValueError):
manuals.publish(site, destination, books, {"BRANCH_NAME": "main"})
self.assertEqual(list(destination.iterdir()), [])
def test_all_six_manifests_stage_real_assets_and_every_midra_starter(self):
books = manuals.books()
self.assertEqual({book["slug"] for book in books},
{"ascii-names", "exporter", "media-player-classic", "midra-layout",
"pptx-reporter", "rundown-converter"})
for book in books:
manuals.stage_book(book, self.root / book["slug"])
assets = ROOT / "docs/templates/midra-layout"
for original in assets.iterdir():
if original.suffix == ".md":
continue
copied = self.root / "midra-layout/src/docs/templates/midra-layout" / original.name
self.assertEqual(original.read_bytes(), copied.read_bytes())
def test_failed_build_does_not_leave_a_publishable_partial_or_old_site(self):
self.book()
self.write("docs/manuals/mdbook-version", "0.5.4\n")
self.write("build/manuals/site/manuals/index.html", "old")
tool = self.write("fake-mdbook", '#!/bin/sh\n'
'if [ "$1" = "--version" ]; then echo "mdbook v0.5.4"; else exit 7; fi\n')
tool.chmod(0o755)
output = self.root / "build/manuals"
with self.assertRaises(subprocess.CalledProcessError):
manuals.build(str(tool), self.root, output)
self.assertFalse(output.exists())
self.assertEqual(list(output.parent.iterdir()), [])
def test_rsync_failure_keeps_previous_landing_page(self):
site, books = self.site()
destination = self.root / "manuals"
landing = self.write("manuals/index.html", "old landing")
with patch.object(manuals.subprocess, "run",
side_effect=subprocess.CalledProcessError(23, "rsync")):
with self.assertRaises(subprocess.CalledProcessError):
manuals.publish(site, destination, books, {"BRANCH_NAME": "main"})
self.assertEqual(landing.read_text(), "old landing")
if __name__ == "__main__":
unittest.main()