feat: define generic report v1 contract and YAML JSON validator
Reportist CI Reportist passed all currently implemented CI checks.
Beffroi/reportist/pipeline/head This commit looks good
Details
Reportist CI Reportist passed all currently implemented CI checks.
Beffroi/reportist/pipeline/head This commit looks good
Details
This commit is contained in:
parent
3b73d1a03c
commit
e2ada627d0
|
|
@ -102,3 +102,6 @@ core.*
|
|||
#
|
||||
# uv.lock / Cargo.lock should normally be committed when applicable.
|
||||
# =============================================================================
|
||||
|
||||
# Isolated Jenkins validation environment
|
||||
.ci-venv/
|
||||
|
|
|
|||
|
|
@ -37,10 +37,10 @@ visual baseline without importing private reports into generic fixtures.
|
|||
|
||||
Depends on Pass 0 for the selected slice.
|
||||
|
||||
- [ ] RT1.1 Define `schema/report.schema.json` with explicit schema version, required/optional fields, metadata, branding, and generic report content; preserve useful legacy compatibility deliberately.
|
||||
- [ ] RT1.2 Map existing report fields and diagnostics presentation needs to generic content structures; keep evidence interpretation outside the renderer.
|
||||
- [ ] RT1.3 Define external asset resolution, theme overrides, missing branding behavior, and neutral defaults without organisation-specific names, authors, logos, or incident terminology.
|
||||
- [ ] RT1.4 Specify YAML/JSON validation errors, unsupported schema handling, and schema evolution/migration policy.
|
||||
- [x] RT1.1 Define `schema/report.schema.json` with explicit schema version, required/optional fields, metadata, branding, and generic report content; preserve useful legacy compatibility deliberately.
|
||||
- [x] RT1.2 Map existing report fields and diagnostics presentation needs to generic content structures; keep evidence interpretation outside the renderer.
|
||||
- [x] RT1.3 Define external asset resolution, theme overrides, missing branding behavior, and neutral defaults without organisation-specific names, authors, logos, or incident terminology.
|
||||
- [x] RT1.4 Specify YAML/JSON validation errors, unsupported schema handling, and schema evolution/migration policy.
|
||||
- [ ] RT1.5 Define a simple CLI with explicit input and output paths, documented exit codes, and installation requirements; publish synthetic valid/invalid examples.
|
||||
|
||||
Exit: unrelated callers can author valid reports without knowing internal Typst
|
||||
|
|
@ -101,3 +101,11 @@ contract is sufficient for any report-producing application.
|
|||
- Validation: macro baseline command rendered both report PDFs and rejected missing author/date and unknown theme. Reviewed the technical report first page; later-page/long-content/alternate-branding coverage remains open.
|
||||
- Existing Beffroi logo coupling remains visible in legacy output. No generic renderer implementation or public schema is claimed yet.
|
||||
- Next: RT1 schema/validation fixtures, then RT2 generic rendering extraction.
|
||||
|
||||
### 2026-09-15 — Versioned contract pass
|
||||
|
||||
- Completed RT1.1–RT1.4: generic report v1 schema, YAML/JSON validator and paired synthetic examples, optional branding/path contract, generic blocks, and version/legacy mapping documentation.
|
||||
- Evidence: [report contract](docs/report-contract.md), `schema/report.schema.json`, `tools/validate.py`, `examples/`, `tests/test_contract.py`.
|
||||
- Validation: eight unittest cases with many invalid-input subcases passed on Python 3.11 and 3.14, including an isolated standalone copy. Covers YAML/JSON parity, minimal reports, invalid dates/versions, duplicate keys, generic branding shape, nested blocks and parser errors.
|
||||
- Jenkins now runs the contract tests. Rendering, actual asset loading and visual branding are not implemented or validated by these schema tests.
|
||||
- Next: finish RT1.5 exact rendering command/output behavior, then RT2 extraction. No render command is claimed yet.
|
||||
|
|
|
|||
|
|
@ -46,15 +46,18 @@ pipeline {
|
|||
* - print `typst --version`.
|
||||
*/
|
||||
|
||||
/*
|
||||
* TODO: Validate Report Schema
|
||||
*
|
||||
* Once schema/report.schema.json exists:
|
||||
*
|
||||
* - validate the schema itself;
|
||||
* - validate examples/*.yaml;
|
||||
* - reject malformed report definitions.
|
||||
*/
|
||||
stage('Validate Contract') {
|
||||
steps {
|
||||
sh '''
|
||||
set -eu
|
||||
command -v uv >/dev/null 2>&1 || { echo "uv is required for the isolated validation environment" >&2; exit 1; }
|
||||
uv venv --python 3.11 --allow-existing .ci-venv
|
||||
uv pip sync --python .ci-venv/bin/python requirements.txt
|
||||
.ci-venv/bin/python --version
|
||||
.ci-venv/bin/python -m unittest discover -s tests -p 'test_*.py'
|
||||
'''
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* TODO: Render Example
|
||||
|
|
@ -87,7 +90,7 @@ pipeline {
|
|||
conclusion: 'SUCCESS',
|
||||
name: 'Reportist CI',
|
||||
summary: 'Reportist passed all currently implemented CI checks.',
|
||||
text: 'Repository validation completed successfully. Renderer validation will be enabled as functionality is migrated.',
|
||||
text: 'Repository and report contract tests passed. PDF renderer implementation and rendering validation remain pending.',
|
||||
title: 'Reportist CI passed'
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -26,6 +26,12 @@ See [AGENT.md](AGENT.md) for repository ownership and development rules.
|
|||
[AGENTS-TASKS.md](AGENTS-TASKS.md) tracks migration passes, completed work,
|
||||
acceptance criteria, and the next-pass handoff.
|
||||
|
||||
## Contract validation
|
||||
|
||||
The version-1 schema, standalone validator, and synthetic examples are available.
|
||||
See [contract documentation](docs/report-contract.md) for setup and validation commands.
|
||||
Collection/analysis and PDF rendering are still separate migration tasks.
|
||||
|
||||
## License
|
||||
|
||||
Software source code is licensed under the **MIT License**.
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ Defaults use A4, DejaVu Sans and DejaVu Sans Mono. No external Typst packages ar
|
|||
imported. The new CLI must accept explicit input/output paths independently of
|
||||
legacy incident directory conventions.
|
||||
|
||||
## Field mapping for version 1 (draft, not implemented)
|
||||
## Field mapping for version 1
|
||||
|
||||
| Existing fields | Proposed treatment |
|
||||
| --- | --- |
|
||||
|
|
@ -58,8 +58,9 @@ legacy incident directory conventions.
|
|||
| `actions_title`, `actions`: title, status, text, commands | Generic action display; commands are inert text. |
|
||||
| `diagnostics.hypotheses` and lists | Map to generic sections/cards or a documented generic assessment block. |
|
||||
|
||||
Schema validation, YAML/JSON parity, optional-field behavior, branding, and the
|
||||
standalone CLI remain RT1/RT2 tasks; this is not a released interface.
|
||||
Version-1 schema validation, YAML/JSON parity, optional-field and branding shape
|
||||
checks are now implemented; see [report contract](report-contract.md). Rendering,
|
||||
asset loading, and actual branding/layout behavior remain unimplemented.
|
||||
|
||||
## Baseline
|
||||
|
||||
|
|
@ -77,4 +78,5 @@ this is a known coupling to remove. Other pages, long-content pagination,
|
|||
alternate branding, all optional constructs, and font-version reproducibility
|
||||
still require tests.
|
||||
|
||||
Next: RT1 schema and validation examples, then the generic single-report slice.
|
||||
Next: define the rendering command precisely and implement the generic
|
||||
single-report slice against the validated contract.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,108 @@
|
|||
# Report contract version 1
|
||||
|
||||
[report.schema.json](../schema/report.schema.json) is the implemented JSON Schema
|
||||
Draft 2020-12 interface for generic YAML/JSON report definitions. Validation is
|
||||
available now; PDF rendering is the next implementation slice.
|
||||
|
||||
## Standalone validation
|
||||
|
||||
Python 3.11 or newer is required. From this component checkout:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt
|
||||
.venv/bin/python tools/validate.py examples/example-report.yaml
|
||||
.venv/bin/python tools/validate.py examples/example-report.json
|
||||
.venv/bin/python -m unittest discover -s tests -p 'test_*.py'
|
||||
```
|
||||
|
||||
On Windows substitute `.venv\Scripts\python.exe`. With uv, use `uv venv` and
|
||||
`uv pip sync --python .venv/bin/python requirements.txt`. The dependency lock pins
|
||||
transitive versions. Validation needs no Typst or sibling repositories and works
|
||||
from any working directory using explicit paths. It writes no files. Exit codes:
|
||||
0 valid, 1 invalid/unreadable input or missing dependencies, 2 CLI usage error.
|
||||
Errors identify the offending JSON path; unsupported file extensions are rejected.
|
||||
|
||||
## Required and optional content
|
||||
|
||||
Only `schema_version: 1`, nonblank `title`, and `metadata` containing nonblank
|
||||
`Author` and a calendar-valid `Date` string (`YYYY-MM-DD`) are required.
|
||||
`examples/minimal-report.json` demonstrates omitted optional fields. The larger
|
||||
YAML/JSON examples have identical parsed content and contain no organisation assets
|
||||
or actual incident data.
|
||||
|
||||
Optional fields retain useful legacy structure:
|
||||
|
||||
- `id`, `subtitle`, `report_type` (`technical` or `briefing`), `status`, `severity`.
|
||||
- `summary`, `closing`: supplied prose, never generated from technical observations.
|
||||
- `timeline`: rows with `time`, `event`, optional `notes`.
|
||||
- `findings`: title, optional ID/severity/status, separate `evidence` and `assessment`,
|
||||
and optional `evidence_refs` containing artifact, JSON pointer, and optional label.
|
||||
- `sections`, `appendices`: titled sections containing generic content blocks.
|
||||
- `actions_title`, `actions`: action titles, optional status/text/command strings.
|
||||
- `provenance`: generic source records with artifact and optional SHA-256/schema
|
||||
version/producer, and optional `rendered_by` tool name/version. Do not fabricate
|
||||
missing historical provenance or claim a render before it happens.
|
||||
|
||||
Finding IDs, when supplied, must be unique. Evidence pointers are syntax-checked;
|
||||
this validator does not open referenced evidence or prove report assertions.
|
||||
Additional metadata labels may contain JSON scalar values. Unknown fields elsewhere
|
||||
are rejected, including legacy `diagnostics`, top-level `organization` and named
|
||||
`theme` fields. Status is a supplied nonblank display label; severity is one of
|
||||
`info`, `low`, `medium`, `high`, `critical`, `warning`, `ok`, `success`, `unanalysed`.
|
||||
|
||||
## Blocks
|
||||
|
||||
| `type` | Required content | Optional content |
|
||||
| --- | --- | --- |
|
||||
| `text` | `text` | — |
|
||||
| `bullets` | nonempty `items` of strings | — |
|
||||
| `steps` | nonempty `items` of strings | — |
|
||||
| `callout` | `text` | `title`, `kind` (severity token) |
|
||||
| `code` | `text` | — |
|
||||
| `subsection` | `title`, nonempty nested `content` | — |
|
||||
|
||||
Unknown or misspelled blocks fail validation. Text and commands are data, not code
|
||||
for the validator to execute. The renderer must preserve that boundary.
|
||||
|
||||
## Branding and path resolution
|
||||
|
||||
`branding` is optional and contains optional `organization`, `logo`, and `theme`.
|
||||
A theme is an object with optional `font`, `mono_font`, and six-digit hex colors:
|
||||
`ink`, `muted`, `line`, `paper_soft`, `accent`, `info`, `warning`, `critical`,
|
||||
`success`. No organisation, author, logo, or incident name is hard-coded.
|
||||
|
||||
Logo and artifact paths use relative forward-slash paths without parent/dot
|
||||
segments, backslashes, drive letters, or control characters. For the future
|
||||
renderer, resolve assets relative to the input report directory; missing branding
|
||||
means neutral defaults with no logo, and a missing specified asset is an error.
|
||||
This pass validates the contract shape, not asset availability or actual layout.
|
||||
The intended rendering CLI will accept one report and an explicit output path;
|
||||
there is no implemented render command yet.
|
||||
|
||||
## YAML/JSON behavior
|
||||
|
||||
Input must fit the JSON data model: string mapping keys, strings, finite numbers,
|
||||
booleans, null, arrays, and objects. Quote dates/timestamps and identifier-like
|
||||
values so YAML cannot reinterpret them. Duplicate keys, non-finite numbers, unsafe
|
||||
tags, recursive aliases, and YAML merge keys are rejected. Acyclic aliases are
|
||||
allowed when their expanded values satisfy the same schema. Multiple YAML documents
|
||||
are not accepted. YAML and JSON validation use the same schema and semantic checks.
|
||||
|
||||
## Legacy migration and compatibility
|
||||
|
||||
Add an explicit version and title to legacy reports, retain `metadata.Date` and
|
||||
`metadata.Author`, and move `organization` into `branding.organization`. Resolve
|
||||
named legacy palettes into explicit `branding.theme` values. Supply a logo only
|
||||
when the report's owner provides it. Map diagnostic hypotheses and fixed domain
|
||||
sections into generic sections/cards; they are not a required renderer feature.
|
||||
Do not relabel Windows raw captures as report inputs.
|
||||
|
||||
Unsupported versions fail validation. Breaking structural or semantic changes
|
||||
require a new schema version and migration notes. The pinned legacy examples remain
|
||||
unchanged for visual comparison; the new examples show contract-v1 authoring.
|
||||
|
||||
See [JSON Schema validation](https://json-schema.org/draft/2020-12/json-schema-validation)
|
||||
and [jsonschema format checking](https://python-jsonschema.readthedocs.io/en/stable/validate/)
|
||||
for the underlying validation rules. Local semantic checks additionally reject
|
||||
repeated finding IDs.
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
{
|
||||
"id": "INC-2026-001",
|
||||
"title": "Service Interruption Investigation",
|
||||
"subtitle": "Incident analysis and shift handoff",
|
||||
"report_type": "technical",
|
||||
"status": "open",
|
||||
"severity": "high",
|
||||
"metadata": {
|
||||
"Date": "2026-09-05",
|
||||
"Author": "Your Name",
|
||||
"Service": "Example API",
|
||||
"Environment": "Production",
|
||||
"Owner": "Operations"
|
||||
},
|
||||
"summary": "The API is available but intermittently slow. Database connection exhaustion is the strongest current lead. No permanent data loss has been observed.\n",
|
||||
"timeline": [
|
||||
{
|
||||
"time": "09:15",
|
||||
"event": "Alert received",
|
||||
"notes": "Latency exceeded the five-second threshold."
|
||||
},
|
||||
{
|
||||
"time": "09:22",
|
||||
"event": "Initial mitigation",
|
||||
"notes": "One unhealthy application instance was removed from rotation."
|
||||
}
|
||||
],
|
||||
"findings": [
|
||||
{
|
||||
"title": "Connection pool saturation",
|
||||
"severity": "high",
|
||||
"status": "confirmed",
|
||||
"evidence": "Pool utilization remained at 100 percent during each latency spike.\n",
|
||||
"assessment": "Requests are waiting for database connections. Query duration and pool sizing both need review.\n",
|
||||
"id": "pool-saturation",
|
||||
"evidence_refs": [
|
||||
{
|
||||
"artifact": "observations.json",
|
||||
"pointer": "/metrics/pool_utilization",
|
||||
"label": "Synthetic metric"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "No evidence of data loss",
|
||||
"severity": "info",
|
||||
"status": "confirmed",
|
||||
"evidence": "Record counts and replication checks are consistent across nodes.\n",
|
||||
"assessment": "Continue monitoring, but recovery work can prioritize availability.\n"
|
||||
}
|
||||
],
|
||||
"sections": [
|
||||
{
|
||||
"title": "Operational precautions",
|
||||
"content": [
|
||||
{
|
||||
"type": "bullets",
|
||||
"items": [
|
||||
"Avoid schema changes during diagnosis.",
|
||||
"Record exact timestamps for new latency spikes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "callout",
|
||||
"title": "Escalation condition",
|
||||
"kind": "critical",
|
||||
"text": "Escalate immediately if replication lag exceeds five minutes."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Diagnostic procedure",
|
||||
"content": [
|
||||
{
|
||||
"type": "steps",
|
||||
"items": [
|
||||
"Capture application and database metrics.",
|
||||
"Correlate slow requests with query duration.",
|
||||
"Test a pool-size adjustment in staging."
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "subsection",
|
||||
"title": "Useful command",
|
||||
"content": [
|
||||
{
|
||||
"type": "code",
|
||||
"text": "systemctl status example-api"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"actions_title": "Next-shift plan",
|
||||
"actions": [
|
||||
{
|
||||
"title": "Capture a diagnostic bundle",
|
||||
"status": "pending",
|
||||
"text": "Collect logs and metrics covering the next occurrence.",
|
||||
"commands": [
|
||||
"journalctl -u example-api --since '1 hour ago'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Validate mitigation in staging",
|
||||
"status": "pending",
|
||||
"text": "Load-test the proposed connection-pool configuration."
|
||||
}
|
||||
],
|
||||
"closing": "Service is operational with intermittent degradation. The next priority is to capture a complete occurrence and validate the pool adjustment in staging.\n",
|
||||
"schema_version": 1,
|
||||
"branding": {
|
||||
"organization": "Example Organization",
|
||||
"theme": {
|
||||
"accent": "#086f83"
|
||||
}
|
||||
},
|
||||
"appendices": [
|
||||
{
|
||||
"title": "Example provenance",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "All content is synthetic."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,95 @@
|
|||
id: INC-2026-001
|
||||
title: Service Interruption Investigation
|
||||
subtitle: Incident analysis and shift handoff
|
||||
report_type: technical
|
||||
status: open
|
||||
severity: high
|
||||
metadata:
|
||||
Date: '2026-09-05'
|
||||
Author: Your Name
|
||||
Service: Example API
|
||||
Environment: Production
|
||||
Owner: Operations
|
||||
summary: 'The API is available but intermittently slow. Database connection exhaustion
|
||||
is the strongest current lead. No permanent data loss has been observed.
|
||||
|
||||
'
|
||||
timeline:
|
||||
- time: 09:15
|
||||
event: Alert received
|
||||
notes: Latency exceeded the five-second threshold.
|
||||
- time: 09:22
|
||||
event: Initial mitigation
|
||||
notes: One unhealthy application instance was removed from rotation.
|
||||
findings:
|
||||
- title: Connection pool saturation
|
||||
severity: high
|
||||
status: confirmed
|
||||
evidence: 'Pool utilization remained at 100 percent during each latency spike.
|
||||
|
||||
'
|
||||
assessment: 'Requests are waiting for database connections. Query duration and pool
|
||||
sizing both need review.
|
||||
|
||||
'
|
||||
id: pool-saturation
|
||||
evidence_refs:
|
||||
- artifact: observations.json
|
||||
pointer: /metrics/pool_utilization
|
||||
label: Synthetic metric
|
||||
- title: No evidence of data loss
|
||||
severity: info
|
||||
status: confirmed
|
||||
evidence: 'Record counts and replication checks are consistent across nodes.
|
||||
|
||||
'
|
||||
assessment: 'Continue monitoring, but recovery work can prioritize availability.
|
||||
|
||||
'
|
||||
sections:
|
||||
- title: Operational precautions
|
||||
content:
|
||||
- type: bullets
|
||||
items:
|
||||
- Avoid schema changes during diagnosis.
|
||||
- Record exact timestamps for new latency spikes.
|
||||
- type: callout
|
||||
title: Escalation condition
|
||||
kind: critical
|
||||
text: Escalate immediately if replication lag exceeds five minutes.
|
||||
- title: Diagnostic procedure
|
||||
content:
|
||||
- type: steps
|
||||
items:
|
||||
- Capture application and database metrics.
|
||||
- Correlate slow requests with query duration.
|
||||
- Test a pool-size adjustment in staging.
|
||||
- type: subsection
|
||||
title: Useful command
|
||||
content:
|
||||
- type: code
|
||||
text: systemctl status example-api
|
||||
actions_title: Next-shift plan
|
||||
actions:
|
||||
- title: Capture a diagnostic bundle
|
||||
status: pending
|
||||
text: Collect logs and metrics covering the next occurrence.
|
||||
commands:
|
||||
- journalctl -u example-api --since '1 hour ago'
|
||||
- title: Validate mitigation in staging
|
||||
status: pending
|
||||
text: Load-test the proposed connection-pool configuration.
|
||||
closing: 'Service is operational with intermittent degradation. The next priority
|
||||
is to capture a complete occurrence and validate the pool adjustment in staging.
|
||||
|
||||
'
|
||||
schema_version: 1
|
||||
branding:
|
||||
organization: Example Organization
|
||||
theme:
|
||||
accent: '#086f83'
|
||||
appendices:
|
||||
- title: Example provenance
|
||||
content:
|
||||
- type: text
|
||||
text: All content is synthetic.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
{
|
||||
"schema_version": 1,
|
||||
"title": "Synthetic minimal report",
|
||||
"metadata": {
|
||||
"Date": "2026-09-15",
|
||||
"Author": "Example Author"
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
jsonschema==4.26.0
|
||||
rfc3339-validator==0.1.4
|
||||
PyYAML==6.0.3
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# This file was autogenerated by uv via the following command:
|
||||
# uv pip compile --python-version 3.11 --universal requirements.in -o requirements.txt
|
||||
attrs==26.1.0
|
||||
# via
|
||||
# jsonschema
|
||||
# referencing
|
||||
jsonschema==4.26.0
|
||||
# via -r requirements.in
|
||||
jsonschema-specifications==2025.9.1
|
||||
# via jsonschema
|
||||
pyyaml==6.0.3
|
||||
# via -r requirements.in
|
||||
referencing==0.37.0
|
||||
# via
|
||||
# jsonschema
|
||||
# jsonschema-specifications
|
||||
rfc3339-validator==0.1.4
|
||||
# via -r requirements.in
|
||||
rpds-py==2026.6.3
|
||||
# via
|
||||
# jsonschema
|
||||
# referencing
|
||||
six==1.17.0
|
||||
# via rfc3339-validator
|
||||
typing-extensions==4.16.0 ; python_full_version < '3.13'
|
||||
# via referencing
|
||||
|
|
@ -0,0 +1,523 @@
|
|||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Reportist structured report version 1",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"schema_version": {
|
||||
"type": "integer",
|
||||
"const": 1
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"subtitle": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"id": {
|
||||
"type": "string",
|
||||
"pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]*$(?![\\s\\S])"
|
||||
},
|
||||
"report_type": {
|
||||
"enum": [
|
||||
"technical",
|
||||
"briefing"
|
||||
]
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"severity": {
|
||||
"enum": [
|
||||
"info",
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"critical",
|
||||
"warning",
|
||||
"ok",
|
||||
"success",
|
||||
"unanalysed"
|
||||
]
|
||||
},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"Date": {
|
||||
"type": "string",
|
||||
"format": "date"
|
||||
},
|
||||
"Author": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"Date",
|
||||
"Author"
|
||||
],
|
||||
"additionalProperties": {
|
||||
"type": [
|
||||
"string",
|
||||
"number",
|
||||
"boolean",
|
||||
"null"
|
||||
]
|
||||
}
|
||||
},
|
||||
"branding": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"organization": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"logo": {
|
||||
"type": "string",
|
||||
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.?(?:/|$))[^\\\\:\\x00-\\x1f]+$(?![\\s\\S])"
|
||||
},
|
||||
"theme": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"font": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"mono_font": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"ink": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"muted": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"line": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"paper_soft": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"accent": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"info": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"warning": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"critical": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
},
|
||||
"success": {
|
||||
"type": "string",
|
||||
"pattern": "^#[0-9a-fA-F]{6}$(?![\\s\\S])"
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"closing": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"timeline": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"time": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"event": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"notes": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"time",
|
||||
"event"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"findings": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]*$(?![\\s\\S])"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"severity": {
|
||||
"enum": [
|
||||
"info",
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"critical",
|
||||
"warning",
|
||||
"ok",
|
||||
"success",
|
||||
"unanalysed"
|
||||
]
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"evidence": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"assessment": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"evidence_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"artifact": {
|
||||
"type": "string",
|
||||
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.?(?:/|$))[^\\\\:\\x00-\\x1f]+$(?![\\s\\S])"
|
||||
},
|
||||
"pointer": {
|
||||
"type": "string",
|
||||
"pattern": "^(?:/(?:[^~/]|~[01])*)*$(?![\\s\\S])"
|
||||
},
|
||||
"label": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"artifact",
|
||||
"pointer"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"uniqueItems": true
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"title"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"sections": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/section"
|
||||
}
|
||||
},
|
||||
"appendices": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/section"
|
||||
}
|
||||
},
|
||||
"actions_title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"actions": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"text": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"commands": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"title"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"provenance": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"sources": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"artifact": {
|
||||
"type": "string",
|
||||
"pattern": "^(?!/)(?!.*(?:^|/)\\.\\.?(?:/|$))[^\\\\:\\x00-\\x1f]+$(?![\\s\\S])"
|
||||
},
|
||||
"sha256": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-f0-9]{64}$(?![\\s\\S])"
|
||||
},
|
||||
"schema_version": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"producer": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name",
|
||||
"version"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"artifact"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"rendered_by": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"name",
|
||||
"version"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"schema_version",
|
||||
"title",
|
||||
"metadata"
|
||||
],
|
||||
"additionalProperties": false,
|
||||
"$defs": {
|
||||
"section": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"content": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/block"
|
||||
},
|
||||
"minItems": 1
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"title",
|
||||
"content"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
"block": {
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"const": "text"
|
||||
},
|
||||
"text": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"text"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"const": "bullets"
|
||||
},
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"minItems": 1
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"items"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"const": "steps"
|
||||
},
|
||||
"items": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"minItems": 1
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"items"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"const": "callout"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"kind": {
|
||||
"enum": [
|
||||
"info",
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"critical",
|
||||
"warning",
|
||||
"ok",
|
||||
"success",
|
||||
"unanalysed"
|
||||
]
|
||||
},
|
||||
"text": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"text"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"const": "code"
|
||||
},
|
||||
"text": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"text"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"const": "subsection"
|
||||
},
|
||||
"title": {
|
||||
"type": "string",
|
||||
"pattern": "\\S"
|
||||
},
|
||||
"content": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/$defs/block"
|
||||
},
|
||||
"minItems": 1
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"type",
|
||||
"title",
|
||||
"content"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
import copy
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
spec = importlib.util.spec_from_file_location("report_validation", ROOT / "tools/validate.py")
|
||||
validation = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(validation)
|
||||
|
||||
|
||||
class ReportContractTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.document = validation.load_report(ROOT / "examples/example-report.json")
|
||||
|
||||
def test_yaml_json_parity_and_minimal_report(self):
|
||||
self.assertEqual(self.document, validation.load_report(ROOT / "examples/example-report.yaml"))
|
||||
self.assertEqual(validation.validate_document(self.document), [])
|
||||
self.assertEqual(validation.validate_document(validation.load_report(ROOT / "examples/minimal-report.json")), [])
|
||||
|
||||
def test_optional_branding_replacement(self):
|
||||
self.document["branding"] = {"organization": "Independent Example", "logo": "assets/example.svg",
|
||||
"theme": {"accent": "#112233", "font": "Example Font"}}
|
||||
self.assertEqual(validation.validate_document(self.document), [])
|
||||
del self.document["branding"]
|
||||
self.assertEqual(validation.validate_document(self.document), [])
|
||||
|
||||
def test_required_fields_and_versions(self):
|
||||
for path, value in [(("schema_version",), 2), (("schema_version",), True),
|
||||
(("title",), " "), (("metadata", "Author"), ""),
|
||||
(("metadata", "Date"), "2026-02-30"),
|
||||
(("metadata", "Date"), "2026-09-15T00:00:00Z")]:
|
||||
with self.subTest(path=path):
|
||||
doc = copy.deepcopy(self.document)
|
||||
target = doc
|
||||
for key in path[:-1]:
|
||||
target = target[key]
|
||||
target[path[-1]] = value
|
||||
self.assertTrue(validation.validate_document(doc))
|
||||
for key in ("Date", "Author"):
|
||||
doc = copy.deepcopy(self.document)
|
||||
del doc["metadata"][key]
|
||||
self.assertTrue(validation.validate_document(doc))
|
||||
|
||||
def test_legacy_domain_fields_rejected(self):
|
||||
for key in ("diagnostics", "organization", "theme", "collect"):
|
||||
with self.subTest(key=key):
|
||||
doc = copy.deepcopy(self.document)
|
||||
doc[key] = {}
|
||||
self.assertTrue(validation.validate_document(doc))
|
||||
|
||||
def test_all_blocks_and_unknown_nested_blocks(self):
|
||||
blocks = [
|
||||
{"type": "text", "text": "Text"}, {"type": "bullets", "items": ["Item"]},
|
||||
{"type": "steps", "items": ["Step"]}, {"type": "callout", "text": "Note", "kind": "warning"},
|
||||
{"type": "code", "text": "echo example"},
|
||||
]
|
||||
self.document["sections"] = [{"title": "Blocks", "content": [
|
||||
{"type": "subsection", "title": "Nested", "content": blocks}]}]
|
||||
self.assertEqual(validation.validate_document(self.document), [])
|
||||
blocks.append({"type": "execute", "command": "anything"})
|
||||
self.assertTrue(validation.validate_document(self.document))
|
||||
blocks.pop()
|
||||
blocks[0]["extra"] = "typo"
|
||||
self.assertTrue(validation.validate_document(self.document))
|
||||
|
||||
def test_bad_branding_and_references(self):
|
||||
for path in ("../secret.svg", "/logo.svg", "C:\\logo.svg", "assets/../logo.svg", "logo.svg\n"):
|
||||
with self.subTest(path=path):
|
||||
self.document["branding"]["logo"] = path
|
||||
self.assertTrue(validation.validate_document(self.document))
|
||||
self.setUp()
|
||||
self.document["branding"]["theme"]["accent"] = "not-a-color"
|
||||
self.assertTrue(validation.validate_document(self.document))
|
||||
self.setUp()
|
||||
self.document["findings"][0]["evidence_refs"][0]["pointer"] = "/bad~3escape"
|
||||
self.assertTrue(validation.validate_document(self.document))
|
||||
self.setUp()
|
||||
self.document["findings"].append(copy.deepcopy(self.document["findings"][0]))
|
||||
self.assertIn("duplicate finding", " ".join(validation.validate_document(self.document)))
|
||||
|
||||
def test_cli_parsing_errors_and_non_json_yaml(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
for suffix, content in [
|
||||
("yaml", "title: first\ntitle: second\n"),
|
||||
("yaml", "metadata: {Date: 2026-09-15}\n"),
|
||||
("yaml", "value: .nan\n"), ("yaml", "value: &cycle [*cycle]\n"),
|
||||
("yaml", "1: bad key\n"), ("yaml", "value: !!python/object:invalid {}\n"),
|
||||
("yaml", "base: &base {a: 1}\nvalue: {<<: *base}\n"),
|
||||
("json", '{"title":"first","title":"second"}'),
|
||||
("json", '{"value":1e999}'), ("json", '{"value":NaN}'),
|
||||
("txt", "{}"), ("yaml", "[unclosed"),
|
||||
]:
|
||||
with self.subTest(content=content):
|
||||
path = Path(temp) / ("invalid." + suffix)
|
||||
path.write_text(content)
|
||||
result = subprocess.run([sys.executable, str(ROOT / "tools/validate.py"), str(path)],
|
||||
cwd=temp, text=True, capture_output=True)
|
||||
self.assertEqual(result.returncode, 1, result.stdout)
|
||||
self.assertNotIn("Traceback", result.stderr)
|
||||
|
||||
def test_cli_from_another_directory(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
result = subprocess.run([sys.executable, str(ROOT / "tools/validate.py"),
|
||||
str(ROOT / "examples/example-report.yaml")], cwd=temp, text=True, capture_output=True)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -0,0 +1,117 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Validate Reportist contract v1 from a YAML or JSON report."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
try:
|
||||
from jsonschema import Draft202012Validator, FormatChecker
|
||||
import yaml
|
||||
except ImportError:
|
||||
raise SystemExit("Install validation dependencies with: python -m pip install -r requirements.txt")
|
||||
|
||||
SCHEMA = json.loads((Path(__file__).resolve().parents[1] / "schema/report.schema.json").read_text())
|
||||
Draft202012Validator.check_schema(SCHEMA)
|
||||
VALIDATOR = Draft202012Validator(SCHEMA, format_checker=FormatChecker())
|
||||
|
||||
|
||||
def unique_object(pairs):
|
||||
result = {}
|
||||
for key, value in pairs:
|
||||
if not isinstance(key, str):
|
||||
raise ValueError("mapping keys must be strings")
|
||||
if key in result:
|
||||
raise ValueError(f"duplicate mapping key: {key}")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
class ReportLoader(yaml.SafeLoader):
|
||||
"""Safe YAML restricted to the JSON data model, without duplicate keys."""
|
||||
|
||||
|
||||
def construct_mapping(loader, node):
|
||||
return unique_object((loader.construct_object(k), loader.construct_object(v))
|
||||
for k, v in node.value)
|
||||
|
||||
|
||||
ReportLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, construct_mapping)
|
||||
|
||||
|
||||
def check_json_value(value, ancestors=None):
|
||||
ancestors = set() if ancestors is None else ancestors
|
||||
if value is None or isinstance(value, (str, bool, int)):
|
||||
return
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
raise ValueError("non-finite numbers are not supported")
|
||||
return
|
||||
if isinstance(value, (dict, list)):
|
||||
if id(value) in ancestors:
|
||||
raise ValueError("cyclic YAML aliases are not supported")
|
||||
ancestors.add(id(value))
|
||||
if isinstance(value, dict):
|
||||
if any(not isinstance(key, str) for key in value):
|
||||
raise ValueError("mapping keys must be strings")
|
||||
children = value.values()
|
||||
else:
|
||||
children = value
|
||||
for child in children:
|
||||
check_json_value(child, ancestors)
|
||||
ancestors.remove(id(value))
|
||||
return
|
||||
raise ValueError("report values must use JSON types; quote YAML dates and timestamps")
|
||||
|
||||
|
||||
def load_report(path):
|
||||
path = Path(path)
|
||||
text = path.read_text(encoding="utf-8")
|
||||
if path.suffix.lower() == ".json":
|
||||
document = json.loads(text, object_pairs_hook=unique_object)
|
||||
elif path.suffix.lower() in (".yaml", ".yml"):
|
||||
document = yaml.load(text, Loader=ReportLoader)
|
||||
else:
|
||||
raise ValueError("expected a .yaml, .yml, or .json report")
|
||||
check_json_value(document)
|
||||
return document
|
||||
|
||||
|
||||
def validate_document(document):
|
||||
check_json_value(document)
|
||||
errors = []
|
||||
for error in VALIDATOR.iter_errors(document):
|
||||
path = "/" + "/".join(str(p).replace("~", "~0").replace("/", "~1") for p in error.absolute_path)
|
||||
# oneOf's headline is intentionally concise; its path locates the offending block.
|
||||
errors.append(f"{path}: {error.message}")
|
||||
if errors:
|
||||
return sorted(errors)
|
||||
ids = set()
|
||||
for index, finding in enumerate(document.get("findings", [])):
|
||||
if "id" in finding:
|
||||
if finding["id"] in ids:
|
||||
errors.append(f"/findings/{index}/id: duplicate finding ID")
|
||||
ids.add(finding["id"])
|
||||
return errors
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("input", type=Path)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
errors = validate_document(load_report(args.input))
|
||||
except (OSError, ValueError, yaml.YAMLError, RecursionError) as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
if errors:
|
||||
print("\n".join(errors), file=sys.stderr)
|
||||
return 1
|
||||
print(f"Valid report v1: {args.input}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Reference in New Issue