test: add reproducible synthetic legacy migration baseline
Beffroi Reporting CI Beffroi Reporting integration CI failed.
Beffroi/beffroi-reporting/pipeline/head There was a failure building this commit Details

This commit is contained in:
Dandaedre 2026-09-15 16:09:27 +02:00
parent 509a940e88
commit 6763fb99f5
7 changed files with 222 additions and 5 deletions

View File

@ -56,10 +56,11 @@ Keep cross-repository milestones here consistent with those component checklists
- [x] P0.3 Record initial ownership boundaries, coupling, and phased acceptance criteria here.
- [x] P0.4 Link this tracker and the actual root instruction file from the README; distinguish current and target paths.
- [ ] P0.5 Complete a file/function-level inventory before extraction, including rule/watchlist classification, report fields, CLI behavior, licenses, fonts, and external dependencies. Record source revisions and destination decisions without copying private evidence into this tracker.
- [ ] P0.6 Establish reproducible legacy baselines using reviewed synthetic/sanitized fixtures: expected findings, collection completeness, report structure, and representative PDFs. Record exact commands, tool versions, and failures.
- [x] P0.6 Establish reproducible legacy baselines using reviewed synthetic/sanitized fixtures: expected findings, collection completeness, report structure, and representative PDFs. Record exact commands, tool versions, and failures.
Exit: every behavior in the first extraction slice has an owner and an observable
baseline. Inventory and baseline execution have not yet been completed.
baseline. The first synthetic baseline is complete; full inventory and private
incident evidence mapping remain open under P0.5 and component tasks.
## Pass 1 — Define contracts and compatibility
@ -161,3 +162,12 @@ reports remain understandable/recreatable and individual tools need no macro rep
- Validation: checked new documentation links, unique task IDs, checkbox syntax, and whitespace. No runtime implementation changed or pipeline tests ran.
- Commit handoff: component documentation commits are pinned by the enclosing integration commit; use its gitlinks to identify the exact revisions.
- Next: complete the source inventories and sanitized baselines in each component, coordinated by P0.5–P0.6. Implementation milestones remain unchecked.
### 2026-09-15 — Inventory and executable baseline pass
- Completed P0.6 for five synthetic diagnostic captures and two synthetic report examples; added `scripts/check-legacy-baseline.py` and [reproduction instructions](docs/migration-baseline.md).
- Component inventories classify collector/normalizer functions, 53 event rules, seven watchlist entries, generic layout/fields, and private source ownership. First contract decisions are drafts, not implemented schemas.
- Validation: five expected snapshots, seven rendered PDFs with extracted titles, and three invalid-input checks passed with Typst 0.15.1. First-page visual review covered technical and Compact reports. Component CI file-presence checks passed locally; Jenkins itself was not run.
- Preserved concurrent component checkout updates that renamed instructions to `AGENT.md` and corrected reports CI license paths. References remain unchanged.
- Limits: no native Windows collection (PowerShell unavailable), migrated engine/renderer/adapter, private historical rendering, full pagination review, or advisory revalidation. P0.5 remains open for the remaining detailed historical/packaging inventory.
- Next: implement WD1/RT1 versioned schemas and synthetic validation examples; finish BR0.5 claim/evidence mapping before moving private incidents.

View File

@ -57,6 +57,9 @@ git submodule update --init --recursive
acceptance criteria, and the next-pass handoff. [AGENTS.md](AGENTS.md) defines
repository ownership and rules for agents working across the workspace.
The first executable migration check is the [synthetic legacy baseline](docs/migration-baseline.md).
It records diagnostic behavior and renders example PDFs before extraction.
## License
Root-level software and orchestration code is licensed under the **MIT License**.

@ -1 +1 @@
Subproject commit 35498b3d8e4e09ef1e4f993491b0837878d663db
Subproject commit 10c59ddaa3d22730244416a7dcd1842d1e00e6e3

View File

@ -0,0 +1,65 @@
# First migration baseline
## Run
Requirements: Python 3, Git, Typst, Poppler's `pdftotext`, DejaVu Sans and DejaVu
Sans Mono fonts. Dependencies are checked, not installed automatically.
Choose a new output directory:
```bash
python3 scripts/check-legacy-baseline.py --output /tmp/beffroi-baseline-review
```
The runner reads an explicit source allowlist from legacy renderer commit
`1d381904fff3d378a7c0e04588e75310a9b7ed60`, stages it in a temporary directory,
and executes synthetic fixtures owned by the destination components. It never
materializes legacy real captures/reports or modifies reference working trees.
Temporary sources are removed after execution.
Outputs: five diagnostic PDFs, two example report PDFs, observed diagnostic JSON,
and a manifest of source revision, fixture hashes, Typst version, available
DejaVu fonts, and negative checks. Existing output directories are rejected.
PDFs are review artifacts, not committed content.
## Verification
- Five snapshots match reviewed expected diagnostic output: collection coverage,
findings, grouping, severity/confidence, and system facts.
- Independent assertions cover schema-1 service failure, schema-2 event counts and
grouping, incomplete-collection confidence, Compact unavailable checks,
unexpected collection status, and six watchlist candidate matches.
- Seven PDFs compile, have PDF signatures, and contain expected extracted titles.
- Missing author/date and an unknown theme fail with expected errors.
This is a legacy comparison harness, **not the new integration pipeline**.
The new diagnostic schema, renderer, adapter and full pipeline remain
unimplemented. The old regression entry point mixes synthetic tests with private
captures and is deliberately not executed by this harness.
## Observations and limits
Initial run: 2026-09-15, Linux, Typst `0.15.1 (586e1bd4)`. PowerShell is absent;
no native collection ran. Font versions and metadata affect PDF bytes, so byte
identity is not asserted. The fixture dates explicitly fix analysis reference time.
Schema-1 produces one service failure. Full/partial schema-2 preserves unknown
events and caps incomplete-collection confidence at 50. Compact unavailable checks
with zero counts remain unavailable. Its legacy display still shows a high numeric
score despite partial collection: this is observed behavior, not an accepted new
health contract. Generic synthetic reports still acquire the Beffroi logo.
First-page visual review covered the technical report and Compact diagnostic:
both were readable without visible clipping and the diagnostic showed its partial
collection warning. Other pages, long-content pagination and alternate branding
remain open. Synthetic checks do not verify historical content or advisory
accuracy. Linked vendor guidance has not been revalidated.
## Next pass
- [Diagnostic inventory and draft contract](../win-diagnostics/docs/migration-inventory.md)
- [Renderer inventory and draft field mapping](../reportist/docs/migration-inventory.md)
- [Private preservation and adapter decisions](../beffroi-reports/docs/migration-inventory.md)
Implement versioned schemas with valid/invalid synthetic examples, then extract
one offline diagnostic and generic rendering slice against this baseline. Finish
private incident evidence mapping before claiming historical migration is verified.

@ -1 +1 @@
Subproject commit 2c1c085ca2940e3908c16035398d739a8773644c
Subproject commit 3b73d1a03c25c1d9ac56959768b894c64b607980

View File

@ -0,0 +1,139 @@
#!/usr/bin/env python3
"""Reproduce the migration baseline without modifying the legacy repositories."""
import argparse
import hashlib
import json
from pathlib import Path
import shutil
import subprocess
import tempfile
ROOT = Path(__file__).resolve().parents[1]
SOURCE = ROOT / "references/typist_reporter"
REVISION = "1d381904fff3d378a7c0e04588e75310a9b7ed60"
FIXTURES = ROOT / "win-diagnostics/tests/fixtures/legacy"
REPORTS = ROOT / "reportist/tests/fixtures/legacy"
EXPECTED = ROOT / "win-diagnostics/tests/legacy-baseline.json"
# Explicit allowlist: never materialize real reports or diagnostic captures.
SOURCE_FILES = (
"LICENSE", "report.typ", "diagnostics.typ", "modules/model.typ",
"modules/diagnostics-model.typ", "modules/diagnostics-normalize.typ",
"config/themes.yaml", "config/diagnostics-rules.yaml",
"config/diagnostics-watchlist.yaml", "assets/beffroi.png",
)
BRIDGE = '''#import "modules/diagnostics-normalize.typ": normalize
#let data = normalize(json(sys.inputs.at("fixture")),
rules: yaml("config/diagnostics-rules.yaml"), source-name: "synthetic fixture")
#metadata((
schema_version: data.schema_version,
mode: data.mode,
collection_incomplete: data.collection_incomplete,
analysis_confidence: data.analysis_confidence,
health_score: data.health_score,
event_counts: data.event_counts,
issue_counts: data.issue_counts,
system: data.system,
findings: data.findings,
event_groups: data.event_groups,
coverage: data.coverage,
)) <baseline>
'''
def run(*args, cwd=None):
return subprocess.run(args, cwd=cwd, check=True, capture_output=True).stdout
def verify_semantics(results):
v1 = results["windiagnostics-v1"]
assert v1["system"]["ram_total_gb"] == 31.82
assert any(f["title"] == "Service Control Manager · Event 7000"
for f in v1["findings"])
full = results["windiagnostics-v2-full"]
assert full["event_counts"] == dict(critical=0, error=1, warning=2, information=1, unknown=0)
assert full["collection_incomplete"] and full["analysis_confidence"] <= 50
assert any(e["level"] == 3 and e["count"] == 2 for e in full["event_groups"])
compact = results["windiagnostics-v2-compact"]
assert not any(f["title"] == "Devices report problems" for f in compact["findings"])
partial = results["windiagnostics-v2-partial"]
assert any(c["section"] == "services" and c["severity"] == "warning"
for c in partial["coverage"])
watch = results["windiagnostics-watchlist"]
assert [f["watch_id"] for f in watch["findings"] if "watch_id" in f] == [
"winring0", "dell-dbutil", "gigabyte-oc-guru", "obs-overlay", "obs-audio-hooks", "smb1"]
assert all(f["weight_factor"] == 0 for f in watch["findings"] if "watch_id" in f)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True, help="New directory for PDFs and observed JSON")
args = parser.parse_args()
for tool in ("git", "typst", "pdftotext"):
if not shutil.which(tool):
parser.error(f"{tool} is required; install it explicitly before running this check")
if args.output.exists():
parser.error("output directory already exists; choose a new directory")
output = args.output.resolve()
output.mkdir(parents=True)
results = {}
manifest = {"source_revision": REVISION, "typst": run("typst", "--version").decode().strip(),
"fixture_sha256": {}, "pdfs": []}
with tempfile.TemporaryDirectory(prefix="beffroi-legacy-baseline-") as directory:
work = Path(directory)
for name in SOURCE_FILES:
target = work / name
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(run("git", "-C", str(SOURCE), "show", f"{REVISION}:{name}"))
(work / "baseline.typ").write_text(BRIDGE)
for fixture in sorted(FIXTURES.glob("*.json")):
shutil.copyfile(fixture, work / fixture.name)
manifest["fixture_sha256"][fixture.name] = hashlib.sha256(fixture.read_bytes()).hexdigest()
raw = run("typst", "query", "--root", directory, "--input", f"fixture={fixture.name}",
"baseline.typ", "<baseline>", "--field", "value", "--one", cwd=work)
results[fixture.stem] = json.loads(raw)
pdf = output / f"{fixture.stem}.pdf"
run("typst", "compile", "--root", directory, "--input", f"diagnostics={fixture.name}",
"--input", "author=Synthetic baseline", "diagnostics.typ", str(pdf), cwd=work)
manifest["pdfs"].append(pdf.name)
verify_semantics(results)
for fixture in sorted(REPORTS.glob("*.yaml")):
shutil.copyfile(fixture, work / fixture.name)
manifest["fixture_sha256"][fixture.name] = hashlib.sha256(fixture.read_bytes()).hexdigest()
pdf = output / f"report-{fixture.stem}.pdf"
run("typst", "compile", "--root", directory, "--input", f"report={fixture.name}",
"report.typ", str(pdf), cwd=work)
manifest["pdfs"].append(pdf.name)
for name, content, message in (
("missing-author", 'metadata:\n Date: "2026-09-15"\n', "metadata.Author"),
("missing-date", 'metadata:\n Author: "Synthetic"\n', "metadata.Date"),
("unknown-theme", 'theme: unknown\nmetadata:\n Author: "Synthetic"\n Date: "2026-09-15"\n', "Unknown report theme"),
):
(work / "invalid.yaml").write_text(content)
result = subprocess.run(["typst", "compile", "--input", "report=invalid.yaml",
"report.typ", str(work / "invalid.pdf")], cwd=work, capture_output=True)
assert result.returncode != 0 and message in result.stderr.decode(), name
manifest["negative_checks"] = ["missing-author", "missing-date", "unknown-theme"]
for name in manifest["pdfs"]:
assert (output / name).read_bytes().startswith(b"%PDF-"), name
text = run("pdftotext", str(output / name), "-").decode()
expected_title = {"report-minimal.pdf": "Short report title",
"report-technical.pdf": "Service Interruption Investigation"}.get(
name, "Windows diagnostic assessment")
assert expected_title in " ".join(text.split()), f"Missing rendered title: {name}"
manifest["fonts"] = [font for font in run("typst", "fonts").decode().splitlines()
if "DejaVu" in font]
(output / "diagnostics.json").write_text(json.dumps(results, indent=2, ensure_ascii=False) + "\n")
(output / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
if not EXPECTED.exists():
parser.exit(1, f"Observed baseline written to {output}; reviewed expected snapshot is missing\n")
assert results == json.loads(EXPECTED.read_text()), "Diagnostic baseline differs; inspect diagnostics.json"
print(f"PASS: 5 diagnostic snapshots, 7 PDFs, 3 invalid inputs; artifacts: {output}")
if __name__ == "__main__":
try:
main()
except subprocess.CalledProcessError as error:
raise SystemExit(error.stderr.decode(errors="replace")) from error

@ -1 +1 @@
Subproject commit 861757f0d165429e789e042cf4d015ab5b72f85c
Subproject commit c0d7f42e8ca31e7f8de4f3475211a52b8db9e3d4