feat: correlate reliability history
Win Diagnostics CI Win Diagnostics passed all currently implemented CI checks.
Beffroi/win-diagnostics/pipeline/head This commit looks good
Details
Win Diagnostics CI Win Diagnostics passed all currently implemented CI checks.
Beffroi/win-diagnostics/pipeline/head This commit looks good
Details
This commit is contained in:
parent
757866e637
commit
343f505506
|
|
@ -64,9 +64,7 @@ the public interface, with collection limitations documented.
|
|||
Depends on normalized evidence and the diagnostic contract.
|
||||
|
||||
- [x] WD3.1 Port reusable rules and watchlist matching with traceable evidence; keep investigation-specific conclusions and organisational recommendations outside this engine.
|
||||
- [ ] WD3.2 Preserve and test event grouping, deduplication, correlation, unknown-event behavior, collection completeness, severity, and confidence.
|
||||
- Primary event grouping, unknown events, watchlist gates, completeness, severity,
|
||||
and confidence are covered; reliability-history deduplication remains pending.
|
||||
- [x] WD3.2 Preserve and test event grouping, deduplication, correlation, unknown-event behavior, collection completeness, severity, and confidence.
|
||||
- [x] WD3.3 Make time-dependent analysis repeatable using a defined reference time; document intentional differences from legacy behavior.
|
||||
- [x] WD3.4 Emit schema-valid `diagnostic.json`; verify every finding's evidence reference resolves and raw observations remain available.
|
||||
- [x] WD3.5 Run sanitized regression fixtures for known diagnostic situations without the original machine; record exact commands and results.
|
||||
|
|
@ -80,15 +78,26 @@ Depends on the working diagnostic slice.
|
|||
|
||||
- [x] WD4.1 Correct Jenkins' instruction-file check; enable schema, fixture, and analysis checks as they become available, with accurate coverage reporting.
|
||||
- [ ] WD4.2 Validate native collection on supported Windows/PowerShell environments; clearly document anything not tested by the non-Windows Jenkins agent.
|
||||
- [ ] WD4.3 Document installation, collection, offline analysis, output schema, privacy, dependencies, and troubleshooting from a fresh standalone clone.
|
||||
- [ ] WD4.4 Verify the standalone clone needs neither Reportist, beffroi-reports, the macro repository, nor legacy sources; audit fixtures for identifying data.
|
||||
- [ ] WD4.5 Record a verified component revision and public commands for the integration repository's sanitized pipeline test; keep integration orchestration there.
|
||||
- [x] WD4.3 Document installation, collection, offline analysis, output schema, privacy, dependencies, and troubleshooting from a fresh standalone clone.
|
||||
- [x] WD4.4 Verify the standalone clone needs neither Reportist, beffroi-reports, the macro repository, nor legacy sources; audit fixtures for identifying data.
|
||||
- [x] WD4.5 Record a verified component revision and public commands for the integration repository's sanitized pipeline test; keep integration orchestration there.
|
||||
|
||||
Exit: a fresh clone collects/analyses supported Windows evidence and produces a
|
||||
valid diagnostic result independently of reporting tools.
|
||||
|
||||
## Handoff log
|
||||
|
||||
### 2026-09-16 — Reliability deduplication and standalone handoff
|
||||
|
||||
- Completed WD3.2 and WD4.3–WD4.5. Reviewed warning/critical reliability records
|
||||
supplement bounded event logs while same-second/provider/ID/message duplicates
|
||||
are omitted. Every retained record keeps its raw JSON pointer.
|
||||
- Documented standalone installation, analysis, validation, privacy,
|
||||
troubleshooting, and native-collection limitations. Runtime requires no sibling,
|
||||
macro, renderer, reports, or legacy checkout.
|
||||
- Native collection remains WD2.1–WD2.2/WD4.2 and requires a supported Windows
|
||||
PowerShell worker; it is not claimed from Linux fixture validation.
|
||||
|
||||
### 2026-09-15 — Complete event and watchlist catalog
|
||||
|
||||
- Completed WD3.1 and advanced WD3.2 for the selected event-analysis scope: all 53 event rules
|
||||
|
|
|
|||
|
|
@ -50,6 +50,9 @@ required for reproducibility. Existing output requires `--force`. Exit codes are
|
|||
CLI usage. See the [rule migration status](docs/rules.md). Native Windows
|
||||
collection and broader non-event evidence correlation are still migration work.
|
||||
|
||||
See [standalone use](docs/standalone.md) for installation, exact commands,
|
||||
privacy guidance, troubleshooting, and native-collection limitations.
|
||||
|
||||
## License
|
||||
|
||||
Software source code is licensed under the **MIT License**.
|
||||
|
|
|
|||
|
|
@ -99,8 +99,11 @@ python tools/analyse.py capture.json \
|
|||
Input must be inside the explicit evidence root so output retains a portable
|
||||
source path. Existing output requires `--force`. Exit codes are 0 for a validated
|
||||
artifact, 1 for input, validation, or I/O failure, and 2 for invalid invocation.
|
||||
The current rule catalog contains the first 15 reviewed event classifications;
|
||||
other captured events remain traceable `unanalysed` findings. See [rules.md](rules.md).
|
||||
The current catalog contains all 53 reviewed event classifications and seven
|
||||
advisory watchlist entries. Actionable reliability-history records supplement
|
||||
bounded event logs and are deduplicated by provider, event ID, normalized message,
|
||||
and timestamp second. Other events remain traceable `unanalysed` findings. See
|
||||
[rules.md](rules.md).
|
||||
|
||||
## Mapping legacy captures
|
||||
|
||||
|
|
@ -126,8 +129,8 @@ remain migration comparisons, not new-schema examples.
|
|||
Consumers select by explicit artifact version and reject unsupported versions.
|
||||
Breaking field or semantic changes require a new version and documented migration.
|
||||
Do not relabel raw captures or silently reshape historical artifacts. Contract v1
|
||||
is the initial implementation. The offline analyser is a bounded migration slice,
|
||||
not yet the complete legacy rule/watchlist engine or collector.
|
||||
is the initial implementation. Event/watchlist analysis is migrated; native
|
||||
collection and broader non-event health analysis remain pending.
|
||||
|
||||
Schema and format behavior follow [JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12/json-schema-validation)
|
||||
and the [jsonschema validator documentation](https://python-jsonschema.readthedocs.io/en/stable/validate/).
|
||||
|
|
|
|||
|
|
@ -41,3 +41,7 @@ OBS audio hooks, and SMBv1. Matches are exact and case-insensitive; driver paths
|
|||
match by basename, versions and enabled states are enforced where specified, and
|
||||
OBS/Intel prerequisite gates must pass. Results remain warning-level candidates
|
||||
whose descriptions explicitly avoid claims of exploitation or causation.
|
||||
|
||||
Actionable warning/critical reliability-history records supplement bounded event
|
||||
logs. Records duplicating an event by provider, ID, whitespace-normalized message,
|
||||
and timestamp second are omitted; distinct records retain their raw pointer.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,47 @@
|
|||
# Standalone use
|
||||
|
||||
## Install and verify
|
||||
|
||||
Python 3.11 or newer is required for offline validation and analysis. Reportist,
|
||||
beffroi-reports, the integration workspace, and legacy sources are not runtime
|
||||
dependencies.
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements.txt
|
||||
.venv/bin/python -m unittest discover -s tests -p 'test_*.py'
|
||||
```
|
||||
|
||||
```bash
|
||||
.venv/bin/python tools/analyse.py capture.json \
|
||||
--evidence-root . --output diagnostic.json \
|
||||
--reference-time 2026-09-15T12:00:00Z \
|
||||
--producer-version "$(git rev-parse HEAD)"
|
||||
.venv/bin/python tools/validate.py diagnostic.json --evidence-root .
|
||||
```
|
||||
|
||||
Both commands return 0 on success, 1 for input/validation/I/O failures, and 2 for
|
||||
invalid CLI usage. Existing analysis output requires `--force`.
|
||||
|
||||
## Collection status
|
||||
|
||||
Native collection has not yet been migrated into this standalone repository or
|
||||
validated on a supported Windows worker. Offline analysis is supported and tested
|
||||
on sanitized schema-1/2 captures. Missing, skipped, unavailable, or truncated
|
||||
sections are limitations and never evidence of system health.
|
||||
|
||||
## Privacy and troubleshooting
|
||||
|
||||
Captures can contain host names, user paths, software inventory, device IDs,
|
||||
events, and security configuration. Store real captures in the ignored
|
||||
`collection/`, `captures/`, or `local-data/` directories and review every fixture
|
||||
before committing it.
|
||||
|
||||
- “input must be inside evidence root”: choose the capture's containing directory.
|
||||
- Unsupported schema: retain the source unchanged and use schema 1 or 2.
|
||||
- Invalid timestamp: use timezone-qualified ISO 8601 times.
|
||||
- Existing output: choose a new path or pass `--force` deliberately.
|
||||
- Evidence mismatch: validate against the unchanged capture used for analysis.
|
||||
|
||||
The fixtures use synthetic values. The integration repository pins the verified
|
||||
revision and owns cross-component orchestration.
|
||||
|
|
@ -3,6 +3,7 @@ import hashlib
|
|||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
|
@ -104,6 +105,36 @@ class AnalyseTests(unittest.TestCase):
|
|||
self.assertEqual(result["findings"][0]["confidence"], 0)
|
||||
self.assertEqual(len(result["findings"][0]["evidence"]), 3)
|
||||
|
||||
def test_reliability_history_adds_actionable_events_without_duplicates(self):
|
||||
capture = analyser.validation.load_json(FIXTURES / "windiagnostics-v2-full.json")
|
||||
capture = copy.deepcopy(capture)
|
||||
event = {"TimeCreated": "2026-09-14T12:00:00.100Z",
|
||||
"ProviderName": "Service Control Manager", "Id": 7000,
|
||||
"Message": "Synthetic service failure"}
|
||||
capture["sections"]["recent_system_events"]["data"] = [event]
|
||||
capture["sections"]["recent_system_events"]["count"] = 1
|
||||
capture["sections"]["recent_application_events"]["data"] = []
|
||||
capture["sections"]["recent_application_events"]["count"] = 0
|
||||
capture["sections"]["reliability_records"]["data"] = [
|
||||
{"TimeGenerated": "2026-09-14T12:00:00.900Z",
|
||||
"SourceName": "Service Control Manager", "EventIdentifier": "7000",
|
||||
"Message": "Synthetic service failure"},
|
||||
{"TimeGenerated": "2026-09-13T12:00:00Z",
|
||||
"SourceName": "Service Control Manager", "EventIdentifier": 7000,
|
||||
"Message": "Separate reliability failure"},
|
||||
{"TimeGenerated": "2026-09-13T12:00:00Z", "SourceName": "EventLog",
|
||||
"EventIdentifier": 1, "Message": "Unreviewed reliability noise"},
|
||||
]
|
||||
capture["sections"]["reliability_records"]["count"] = 3
|
||||
result = analyser.analyse(capture, "capture.json", "0" * 64,
|
||||
self.reference_time, "test-revision")
|
||||
finding = next(item for item in result["findings"]
|
||||
if item["id"] == "event-service-control-manager-7000")
|
||||
self.assertEqual(finding["evidence"], ["event-recent_system_events-1",
|
||||
"event-reliability-records-2"])
|
||||
self.assertNotIn("event-reliability-records-1",
|
||||
{item["id"] for item in result["observations"]})
|
||||
|
||||
def test_watchlist_fixture_emits_six_traceable_candidates(self):
|
||||
result = self.run_fixture("windiagnostics-watchlist.json")
|
||||
findings = {finding["id"]: finding for finding in result["findings"]
|
||||
|
|
@ -220,6 +251,21 @@ class AnalyseTests(unittest.TestCase):
|
|||
"--reference-time", self.reference_time, "--producer-version", "test-revision",
|
||||
], cwd=self.work, text=True, capture_output=True)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
|
||||
def test_standalone_component_copy_analyses_fixture(self):
|
||||
standalone = self.work / "standalone"
|
||||
shutil.copytree(ROOT, standalone, ignore=shutil.ignore_patterns(
|
||||
".git", ".ci-venv", "__pycache__", "*.pyc"))
|
||||
fixture = standalone / "tests" / "fixtures" / "legacy" / "windiagnostics-v1.json"
|
||||
output = self.work / "standalone-diagnostic.json"
|
||||
result = subprocess.run([
|
||||
sys.executable, str(standalone / "tools" / "analyse.py"), str(fixture),
|
||||
"--evidence-root", str(fixture.parent), "--output", str(output),
|
||||
"--reference-time", self.reference_time,
|
||||
"--producer-version", "standalone-test",
|
||||
], cwd=self.work, text=True, capture_output=True)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(json.loads(output.read_text())["schema_version"], 1)
|
||||
self.assertTrue(output.is_file())
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -363,6 +363,20 @@ def selected_event_sections(capture):
|
|||
return [(name, sections[name]) for name in names if name in sections]
|
||||
|
||||
|
||||
def event_identity(event):
|
||||
provider = str(event.get("ProviderName", "Unknown provider"))
|
||||
raw_id = event.get("Id", "unknown")
|
||||
event_id = raw_id if type(raw_id) in (int, str) else "unknown"
|
||||
return provider, event_id
|
||||
|
||||
|
||||
def event_signature(provider, event_id, timestamp, message):
|
||||
normalized_message = " ".join(str(message or "").split())
|
||||
parsed = valid_timestamp(timestamp)
|
||||
second = datetime.fromisoformat(parsed.replace("Z", "+00:00")).replace(microsecond=0) if parsed else timestamp
|
||||
return provider, str(event_id), second, normalized_message
|
||||
|
||||
|
||||
def analyse(capture, artifact, artifact_hash, reference_time, producer_version):
|
||||
version = capture.get("schema_version")
|
||||
if type(version) is not int or version not in (1, 2):
|
||||
|
|
@ -383,6 +397,7 @@ def analyse(capture, artifact, artifact_hash, reference_time, producer_version):
|
|||
collection_status = "complete" if complete else "partial"
|
||||
observations = []
|
||||
groups = OrderedDict()
|
||||
event_signatures = set()
|
||||
for section_name, section in selected_event_sections(capture):
|
||||
data = section.get("data", []) if isinstance(section, dict) else []
|
||||
if not isinstance(data, list):
|
||||
|
|
@ -398,9 +413,36 @@ def analyse(capture, artifact, artifact_hash, reference_time, producer_version):
|
|||
if observed_at:
|
||||
observation["observed_at"] = observed_at
|
||||
observations.append(observation)
|
||||
provider = str(event.get("ProviderName", "Unknown provider"))
|
||||
raw_event_id = event.get("Id", "unknown")
|
||||
event_id = raw_event_id if type(raw_event_id) in (int, str) else "unknown"
|
||||
provider, event_id = event_identity(event)
|
||||
groups.setdefault((provider, event_id), []).append(observation_id)
|
||||
event_signatures.add(event_signature(provider, event_id, event.get("TimeCreated"),
|
||||
event.get("Message")))
|
||||
|
||||
reliability = capture["sections"].get("reliability_records")
|
||||
reliability_data = reliability.get("data", []) if isinstance(reliability, dict) else []
|
||||
if isinstance(reliability_data, list):
|
||||
for index, record in enumerate(reliability_data):
|
||||
if not isinstance(record, dict):
|
||||
continue
|
||||
provider = str(record.get("SourceName", "Unknown provider"))
|
||||
raw_id = record.get("EventIdentifier", "unknown")
|
||||
event_id = int(raw_id) if isinstance(raw_id, str) and raw_id.isdigit() else raw_id
|
||||
rule = EVENT_RULES.get((provider, event_id))
|
||||
if not rule or rule["severity"] not in ("warning", "critical"):
|
||||
continue
|
||||
signature = event_signature(provider, event_id, record.get("TimeGenerated"),
|
||||
record.get("Message"))
|
||||
if signature in event_signatures:
|
||||
continue
|
||||
event_signatures.add(signature)
|
||||
observation_id = f"event-reliability-records-{index + 1}"
|
||||
observation = {"id": observation_id, "section": "reliability_records",
|
||||
"source_pointer": pointer("sections", "reliability_records", "data", index),
|
||||
"value": record}
|
||||
observed_at = valid_timestamp(record.get("TimeGenerated"))
|
||||
if observed_at:
|
||||
observation["observed_at"] = observed_at
|
||||
observations.append(observation)
|
||||
groups.setdefault((provider, event_id), []).append(observation_id)
|
||||
|
||||
findings = []
|
||||
|
|
|
|||
Reference in New Issue