feat: correlate reliability history
Win Diagnostics CI Win Diagnostics passed all currently implemented CI checks.
Beffroi/win-diagnostics/pipeline/head This commit looks good Details

This commit is contained in:
Dandaedre 2026-09-16 00:05:14 +02:00
parent 757866e637
commit 343f505506
Signed by: Dandaedre
GPG Key ID: 440C6592E8A26D99
7 changed files with 167 additions and 13 deletions

View File

@ -64,9 +64,7 @@ the public interface, with collection limitations documented.
Depends on normalized evidence and the diagnostic contract.
- [x] WD3.1 Port reusable rules and watchlist matching with traceable evidence; keep investigation-specific conclusions and organisational recommendations outside this engine.
- [ ] WD3.2 Preserve and test event grouping, deduplication, correlation, unknown-event behavior, collection completeness, severity, and confidence.
- Primary event grouping, unknown events, watchlist gates, completeness, severity,
and confidence are covered; reliability-history deduplication remains pending.
- [x] WD3.2 Preserve and test event grouping, deduplication, correlation, unknown-event behavior, collection completeness, severity, and confidence.
- [x] WD3.3 Make time-dependent analysis repeatable using a defined reference time; document intentional differences from legacy behavior.
- [x] WD3.4 Emit schema-valid `diagnostic.json`; verify every finding's evidence reference resolves and raw observations remain available.
- [x] WD3.5 Run sanitized regression fixtures for known diagnostic situations without the original machine; record exact commands and results.
@ -80,15 +78,26 @@ Depends on the working diagnostic slice.
- [x] WD4.1 Correct Jenkins' instruction-file check; enable schema, fixture, and analysis checks as they become available, with accurate coverage reporting.
- [ ] WD4.2 Validate native collection on supported Windows/PowerShell environments; clearly document anything not tested by the non-Windows Jenkins agent.
- [ ] WD4.3 Document installation, collection, offline analysis, output schema, privacy, dependencies, and troubleshooting from a fresh standalone clone.
- [ ] WD4.4 Verify the standalone clone needs neither Reportist, beffroi-reports, the macro repository, nor legacy sources; audit fixtures for identifying data.
- [ ] WD4.5 Record a verified component revision and public commands for the integration repository's sanitized pipeline test; keep integration orchestration there.
- [x] WD4.3 Document installation, collection, offline analysis, output schema, privacy, dependencies, and troubleshooting from a fresh standalone clone.
- [x] WD4.4 Verify the standalone clone needs neither Reportist, beffroi-reports, the macro repository, nor legacy sources; audit fixtures for identifying data.
- [x] WD4.5 Record a verified component revision and public commands for the integration repository's sanitized pipeline test; keep integration orchestration there.
Exit: a fresh clone collects/analyses supported Windows evidence and produces a
valid diagnostic result independently of reporting tools.
## Handoff log
### 2026-09-16 — Reliability deduplication and standalone handoff
- Completed WD3.2 and WD4.3–WD4.5. Reviewed warning/critical reliability records
supplement bounded event logs while same-second/provider/ID/message duplicates
are omitted. Every retained record keeps its raw JSON pointer.
- Documented standalone installation, analysis, validation, privacy,
troubleshooting, and native-collection limitations. Runtime requires no sibling,
macro, renderer, reports, or legacy checkout.
- Native collection remains WD2.1–WD2.2/WD4.2 and requires a supported Windows
PowerShell worker; it is not claimed from Linux fixture validation.
### 2026-09-15 — Complete event and watchlist catalog
- Completed WD3.1 and advanced WD3.2 for the selected event-analysis scope: all 53 event rules

View File

@ -50,6 +50,9 @@ required for reproducibility. Existing output requires `--force`. Exit codes are
CLI usage. See the [rule migration status](docs/rules.md). Native Windows
collection and broader non-event evidence correlation are still migration work.
See [standalone use](docs/standalone.md) for installation, exact commands,
privacy guidance, troubleshooting, and native-collection limitations.
## License
Software source code is licensed under the **MIT License**.

View File

@ -99,8 +99,11 @@ python tools/analyse.py capture.json \
Input must be inside the explicit evidence root so output retains a portable
source path. Existing output requires `--force`. Exit codes are 0 for a validated
artifact, 1 for input, validation, or I/O failure, and 2 for invalid invocation.
The current rule catalog contains the first 15 reviewed event classifications;
other captured events remain traceable `unanalysed` findings. See [rules.md](rules.md).
The current catalog contains all 53 reviewed event classifications and seven
advisory watchlist entries. Actionable reliability-history records supplement
bounded event logs and are deduplicated by provider, event ID, normalized message,
and timestamp second. Other events remain traceable `unanalysed` findings. See
[rules.md](rules.md).
## Mapping legacy captures
@ -126,8 +129,8 @@ remain migration comparisons, not new-schema examples.
Consumers select by explicit artifact version and reject unsupported versions.
Breaking field or semantic changes require a new version and documented migration.
Do not relabel raw captures or silently reshape historical artifacts. Contract v1
is the initial implementation. The offline analyser is a bounded migration slice,
not yet the complete legacy rule/watchlist engine or collector.
is the initial implementation. Event/watchlist analysis is migrated; native
collection and broader non-event health analysis remain pending.
Schema and format behavior follow [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12/json-schema-validation)
and the [jsonschema validator documentation](https://python-jsonschema.readthedocs.io/en/stable/validate/).

View File

@ -41,3 +41,7 @@ OBS audio hooks, and SMBv1. Matches are exact and case-insensitive; driver paths
match by basename, versions and enabled states are enforced where specified, and
OBS/Intel prerequisite gates must pass. Results remain warning-level candidates
whose descriptions explicitly avoid claims of exploitation or causation.
Actionable warning/critical reliability-history records supplement bounded event
logs. Records duplicating an event by provider, ID, whitespace-normalized message,
and timestamp second are omitted; distinct records retain their raw pointer.

47
docs/standalone.md Normal file
View File

@ -0,0 +1,47 @@
# Standalone use
## Install and verify
Python 3.11 or newer is required for offline validation and analysis. Reportist,
beffroi-reports, the integration workspace, and legacy sources are not runtime
dependencies.
```bash
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python -m unittest discover -s tests -p 'test_*.py'
```
```bash
.venv/bin/python tools/analyse.py capture.json \
--evidence-root . --output diagnostic.json \
--reference-time 2026-09-15T12:00:00Z \
--producer-version "$(git rev-parse HEAD)"
.venv/bin/python tools/validate.py diagnostic.json --evidence-root .
```
Both commands return 0 on success, 1 for input/validation/I/O failures, and 2 for
invalid CLI usage. Existing analysis output requires `--force`.
## Collection status
Native collection has not yet been migrated into this standalone repository or
validated on a supported Windows worker. Offline analysis is supported and tested
on sanitized schema-1/2 captures. Missing, skipped, unavailable, or truncated
sections are limitations and never evidence of system health.
## Privacy and troubleshooting
Captures can contain host names, user paths, software inventory, device IDs,
events, and security configuration. Store real captures in the ignored
`collection/`, `captures/`, or `local-data/` directories and review every fixture
before committing it.
- “input must be inside evidence root”: choose the capture's containing directory.
- Unsupported schema: retain the source unchanged and use schema 1 or 2.
- Invalid timestamp: use timezone-qualified ISO 8601 times.
- Existing output: choose a new path or pass `--force` deliberately.
- Evidence mismatch: validate against the unchanged capture used for analysis.
The fixtures use synthetic values. The integration repository pins the verified
revision and owns cross-component orchestration.

View File

@ -3,6 +3,7 @@ import hashlib
import importlib.util
import json
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
@ -104,6 +105,36 @@ class AnalyseTests(unittest.TestCase):
self.assertEqual(result["findings"][0]["confidence"], 0)
self.assertEqual(len(result["findings"][0]["evidence"]), 3)
def test_reliability_history_adds_actionable_events_without_duplicates(self):
capture = analyser.validation.load_json(FIXTURES / "windiagnostics-v2-full.json")
capture = copy.deepcopy(capture)
event = {"TimeCreated": "2026-09-14T12:00:00.100Z",
"ProviderName": "Service Control Manager", "Id": 7000,
"Message": "Synthetic service failure"}
capture["sections"]["recent_system_events"]["data"] = [event]
capture["sections"]["recent_system_events"]["count"] = 1
capture["sections"]["recent_application_events"]["data"] = []
capture["sections"]["recent_application_events"]["count"] = 0
capture["sections"]["reliability_records"]["data"] = [
{"TimeGenerated": "2026-09-14T12:00:00.900Z",
"SourceName": "Service Control Manager", "EventIdentifier": "7000",
"Message": "Synthetic service failure"},
{"TimeGenerated": "2026-09-13T12:00:00Z",
"SourceName": "Service Control Manager", "EventIdentifier": 7000,
"Message": "Separate reliability failure"},
{"TimeGenerated": "2026-09-13T12:00:00Z", "SourceName": "EventLog",
"EventIdentifier": 1, "Message": "Unreviewed reliability noise"},
]
capture["sections"]["reliability_records"]["count"] = 3
result = analyser.analyse(capture, "capture.json", "0" * 64,
self.reference_time, "test-revision")
finding = next(item for item in result["findings"]
if item["id"] == "event-service-control-manager-7000")
self.assertEqual(finding["evidence"], ["event-recent_system_events-1",
"event-reliability-records-2"])
self.assertNotIn("event-reliability-records-1",
{item["id"] for item in result["observations"]})
def test_watchlist_fixture_emits_six_traceable_candidates(self):
result = self.run_fixture("windiagnostics-watchlist.json")
findings = {finding["id"]: finding for finding in result["findings"]
@ -220,6 +251,21 @@ class AnalyseTests(unittest.TestCase):
"--reference-time", self.reference_time, "--producer-version", "test-revision",
], cwd=self.work, text=True, capture_output=True)
self.assertEqual(result.returncode, 0, result.stderr)
def test_standalone_component_copy_analyses_fixture(self):
standalone = self.work / "standalone"
shutil.copytree(ROOT, standalone, ignore=shutil.ignore_patterns(
".git", ".ci-venv", "__pycache__", "*.pyc"))
fixture = standalone / "tests" / "fixtures" / "legacy" / "windiagnostics-v1.json"
output = self.work / "standalone-diagnostic.json"
result = subprocess.run([
sys.executable, str(standalone / "tools" / "analyse.py"), str(fixture),
"--evidence-root", str(fixture.parent), "--output", str(output),
"--reference-time", self.reference_time,
"--producer-version", "standalone-test",
], cwd=self.work, text=True, capture_output=True)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(json.loads(output.read_text())["schema_version"], 1)
self.assertTrue(output.is_file())

View File

@ -363,6 +363,20 @@ def selected_event_sections(capture):
return [(name, sections[name]) for name in names if name in sections]
def event_identity(event):
provider = str(event.get("ProviderName", "Unknown provider"))
raw_id = event.get("Id", "unknown")
event_id = raw_id if type(raw_id) in (int, str) else "unknown"
return provider, event_id
def event_signature(provider, event_id, timestamp, message):
normalized_message = " ".join(str(message or "").split())
parsed = valid_timestamp(timestamp)
second = datetime.fromisoformat(parsed.replace("Z", "+00:00")).replace(microsecond=0) if parsed else timestamp
return provider, str(event_id), second, normalized_message
def analyse(capture, artifact, artifact_hash, reference_time, producer_version):
version = capture.get("schema_version")
if type(version) is not int or version not in (1, 2):
@ -383,6 +397,7 @@ def analyse(capture, artifact, artifact_hash, reference_time, producer_version):
collection_status = "complete" if complete else "partial"
observations = []
groups = OrderedDict()
event_signatures = set()
for section_name, section in selected_event_sections(capture):
data = section.get("data", []) if isinstance(section, dict) else []
if not isinstance(data, list):
@ -398,9 +413,36 @@ def analyse(capture, artifact, artifact_hash, reference_time, producer_version):
if observed_at:
observation["observed_at"] = observed_at
observations.append(observation)
provider = str(event.get("ProviderName", "Unknown provider"))
raw_event_id = event.get("Id", "unknown")
event_id = raw_event_id if type(raw_event_id) in (int, str) else "unknown"
provider, event_id = event_identity(event)
groups.setdefault((provider, event_id), []).append(observation_id)
event_signatures.add(event_signature(provider, event_id, event.get("TimeCreated"),
event.get("Message")))
reliability = capture["sections"].get("reliability_records")
reliability_data = reliability.get("data", []) if isinstance(reliability, dict) else []
if isinstance(reliability_data, list):
for index, record in enumerate(reliability_data):
if not isinstance(record, dict):
continue
provider = str(record.get("SourceName", "Unknown provider"))
raw_id = record.get("EventIdentifier", "unknown")
event_id = int(raw_id) if isinstance(raw_id, str) and raw_id.isdigit() else raw_id
rule = EVENT_RULES.get((provider, event_id))
if not rule or rule["severity"] not in ("warning", "critical"):
continue
signature = event_signature(provider, event_id, record.get("TimeGenerated"),
record.get("Message"))
if signature in event_signatures:
continue
event_signatures.add(signature)
observation_id = f"event-reliability-records-{index + 1}"
observation = {"id": observation_id, "section": "reliability_records",
"source_pointer": pointer("sections", "reliability_records", "data", index),
"value": record}
observed_at = valid_timestamp(record.get("TimeGenerated"))
if observed_at:
observation["observed_at"] = observed_at
observations.append(observation)
groups.setdefault((provider, event_id), []).append(observation_id)
findings = []