From 5a28720fea7311bb2ea8c56abad2d3796e0acb05 Mon Sep 17 00:00:00 2001 From: Dandaedre Date: Tue, 3 Nov 2026 12:10:52 +0100 Subject: [PATCH] Separate rundown processing stages and plan buffer corrections --- docs/rundown-converter-plan.md | 132 +++++++++++++++++++++++++++++---- docs/rundown-converter.md | 47 +++++++++--- 2 files changed, 153 insertions(+), 26 deletions(-) diff --git a/docs/rundown-converter-plan.md b/docs/rundown-converter-plan.md index a3e7f40..3e0ac01 100644 --- a/docs/rundown-converter-plan.md +++ b/docs/rundown-converter-plan.md @@ -12,13 +12,24 @@ Use the reporter's Rust/clap/eframe approach and optional desktop feature. A Python prototype is optional for investigating a difficult format, not a second maintained implementation. -The processing pipeline is: +Source extraction prepares tables before the five distinct processing stages: ```text -source readers -> source tables -> header/column detection -> neutral rundown - -> timing resolution and validation -> Ontime adapter -> file +source readers -> source tables + 1. infer and verify column meanings across rows + 2. identify untimed gaps and record them in the draft IR + 3. construct the complete IR from source times and gap constraints + 4. validate schedule consistency + 5. export the validated IR to Ontime JSON ``` +Each stage has a separate result and diagnostics. A later stage must not silently +reinterpret an earlier decision. Unresolved column meanings block stage 2; +unresolved gap boundaries remain explicit constraints for stage 3. If a complete +schedule cannot be established, return an actionable error rather than exporting +partial guesses. Stage 4 is an independent validation pass over the complete IR, +not a side effect of export. All exporters consume the same validated IR. + Readers retain original cell text, typed values, number formats, merged-cell ranges and source coordinates when available. A table model separates extraction from interpretation so format-specific quirks do not leak into the Ontime adapter. @@ -46,7 +57,7 @@ only when required decisions are resolved. | Rundown/session | Ordered entries, label, date/day context, stage/track where established | | Entry | Stable identity, cue, title, speaker, notes, additional original columns, event/break/marker kind | | Timing | Optional explicit start/end, duration, day offset, fixed/relative intent, inferred/resolved values | -| Gap | Known previous end and next start, interval, provenance, explicit or inferred classification | +| Gap | Untimed break, boundary references/constraints, resolved interval when known, provenance, no timer display | | Decision | Selected mapping/rule, evidence, alternatives, user overrides | | Diagnostic | Stable code, severity, source reference, explanation, corrective action | @@ -57,12 +68,25 @@ reportable rather than overwritten. Convert to Ontime milliseconds only in the adapter. Stable identities should be reproducible for unchanged input; do not promise identity stability across arbitrary source edits. -## Detection and clarification +## 1. Infer and verify column meanings Combine normalized French/English header aliases, cell types, number formats, value patterns and consistency across rows. Normalize accents, casing, spacing and line breaks without losing original text. Evaluate candidate header bands and mappings jointly instead of assigning columns from individual keywords. +Infer meaning primarily from data across the entire column: one ambiguous cell +can be resolved by evidence in other rows. A column must retain one semantic role +and a compatible unit interpretation throughout the selected table. Different +textual formats may express the same role; roles must not switch row by row. + +Require the chosen interpretation to fit every relevant nonempty data cell, not +just a majority or a sample. If no single meaning fits, return +`INCONSISTENT_COLUMN_MEANING` with conflicting cell references and ask the user to +clarify headers, correct values or explicitly separate distinct tables. An +explicit mapping must also pass this check. Empty cells provide no evidence; +recognized headings/totals are structural rows with recorded classifications, +not convenient exceptions used to hide contradictory event data. Check semantic +consistency here; chronological contradictions belong to stage 4. Account for title blocks above tables, merged/multiline headers, repeated headers, blank separators, continuation rows, section labels, footnotes and totals. @@ -91,12 +115,38 @@ GUI mapping panel should select table, header rows, column roles and units. Reusable mapping profiles should verify expected headers before applying; a changed document must not silently inherit an unsuitable profile. -## Timing resolution and export contract +## 2. Identify gaps and incorporate them into the IR + +Identify breaks with no timer to display before deriving event durations. Use +source labels, structural evidence and independently established event ends and +following starts. Record each gap in the draft IR with its boundary references, +known times and unresolved constraints. A named untimed lunch row is a gap, not +an event with a missing duration. A break explicitly intended as a timed cue +remains a different entry kind. + +Gaps consume schedule time but never create a timer or show a duration as an +event. Their intervals may appear in the diagnostic preview for explanation. +Resolve unknown boundaries in stage 3 when sufficient evidence exists; otherwise +ask for clarification. A long start-to-start interval alone is insufficient to +identify a break. No optional synthetic break timer is generated for these gaps. + +## 3. Construct the complete IR from times Apply the policies in the [tool overview](rundown-converter.md#timing-and-breaks). Solve constraints within a session while preserving explicit anchors. An explicit start/end pair yields duration; a start/duration pair yields end. If all three -are present, validate agreement using a documented precision tolerance. +are present, retain all three plus calculated values for stage 4 to compare. +Complete the ordered entries and gap intervals, preserving provenance and +explicit versus inferred facts. Missing facts that cannot be derived must produce +a clarification error; do not call an unresolved schedule complete. + +## 4. Check schedule consistency + +Validate the complete IR independently of the reader and export target. Check +start/end/duration agreement using a documented precision tolerance, time going +backward within each session/track, negative durations, overlaps and gap boundaries. +Keep source order; never sort away a backward time or repair it by assuming a new +day. Validate that gaps do not overlap events or accidentally consume cue time. Do not infer an end across an unknown break, another session, another day or a parallel track. Midnight rollover must be supported by evidence or an explicit @@ -104,14 +154,15 @@ setting. Treat unresolved overlap, negative duration, invalid time and unavailab final duration as actionable errors. Define zero-duration markers separately from missing-duration events. Never silently clamp or wrap unsupported target values. +## 5. Generate importable output + The Ontime adapter will map absolute starts to independent schedule entries and link only entries explicitly intended to follow their predecessor. Duration-lock, end-lock and count-to-end playback are distinct choices: preserving a scheduled start must not silently enable count-to-end or automated playback. Keep default end actions manual. Validate target limits, ordering, grouping and project schema. -An established gap remains empty by default; do not use a delay entry to represent -an ordinary lunch break. If visible inferred break cues are implemented, label -and track them separately from source events. +An established untimed gap remains empty; do not use a delay entry or synthetic +timer to represent it. Preserve the independent start after it. Export via a temporary file and publish only after complete validation, with explicit overwrite behavior and cleanup on errors/cancellation. Produce neutral @@ -119,6 +170,46 @@ JSON separately from native Ontime JSON with unmistakable filenames. Diagnostic preview may remain available on failure, but an invalid project must not be published. Keep local source paths out of exported project notes by default. +## Stretch goals + +### Additional exports + +Generate a Typst report (editable source and rendered PDF) from the validated IR. +Re-export XLS/ODS spreadsheets with explicit, normalized columns suitable for +Ontime import. Verify accepted file types and exact header/unit requirements +against the target Ontime release before claiming direct import support; if a +requested format needs conversion, document that limitation explicitly. Include +round-trip import tests for timing and gaps. Keep untimed gaps free of timer rows +in every output. These exports do not become intermediate steps for JSON export. + +### Optional buffer adjustment + +Offer an explicit correction mode that removes time from event rows to reserve +buffer in a cramped schedule. Keep the original IR and create a separate proposed +revision with a before/after preview and a per-row adjustment record. Never apply +this automatically during inference, validation or export. + +Let the user choose a target buffer (duration or percentage), scope/session, +distribution (equal cuts or proportional cuts), minimum event durations and +protected rows. By default distribute cuts across all eligible timed rows in the +selected scope; untimed gaps, markers and protected rows contribute nothing. +Reject infeasible requests with the maximum achievable buffer. Allocate integer +rounding remainders deterministically so cuts sum exactly to the requested buffer. + +Define where the saved time goes: either slack after each shortened event while +keeping its start, or a consolidated buffer before a selected anchor. Consolidation +requires explicit permission to move affected intermediate starts; preserve fixed +session/restart anchors and protected timings. Never lengthen an existing break +by accident or count existing slack as newly recovered time. If a source already +overlaps, distinguish cuts needed to eliminate the overlap from additional cuts +needed to create the requested buffer. Column ambiguities must be resolved first; +correction must not conceal malformed times or silently choose their meaning. + +Run the complete stage-4 validation again on any proposed revision before stage-5 +export. Report remaining conflicts and export only a valid selected revision. +Test insufficient capacity, minimum durations, protected rows, anchor preservation, +rounding, existing gaps, overlap repair and GUI/CLI parity. + ## CLI and desktop experience Proposed CLI operations are inspect, validate and convert; finalize syntax after @@ -152,14 +243,19 @@ errors must remain visible with direct corrective actions. defaults. Include the new member in workspace CI, documentation and coverage. - [ ] Implement the required source readers and extraction diagnostics. Test unsupported/encrypted/corrupt inputs and formula/merged-cell behavior as relevant. -- [ ] Implement detection, explicit mappings and source-located errors; verify - competing headers and misleading time formats fail clearly. -- [ ] Implement timing constraints and gaps. Test known/unknown lunch gaps, - start-only schedules, contradictory triples, missing final duration, overlaps, - midnight/day transitions, zero-duration markers and multiple sessions/tracks. +- [ ] Stage 1: implement whole-column inference, explicit mappings and located + errors. Test ambiguous cells clarified by other rows, incompatible meanings in + one column, structural rows and misleading headers/units. +- [ ] Stage 2: identify untimed gaps and populate draft IR constraints. Test named + and unnamed breaks, unknown boundaries and distinction from timed break cues. +- [ ] Stage 3: construct the complete IR from times and gap constraints. Test + start-only schedules, missing final duration and provenance of inferred values. +- [ ] Stage 4: independently validate backward times, negative durations, + contradictory triples, overlaps, gap boundaries, midnight/day transitions, + zero-duration markers and multiple sessions/tracks. - [ ] Implement inspect/validate/convert CLI and atomic exports. Test CLI status, output collisions, cancellation, source preservation and deterministic results. -- [ ] Implement the Ontime adapter and neutral JSON export. Check native imports +- [ ] Stage 5: implement the Ontime adapter and neutral JSON export. Check native imports against the pinned parser and manually inspect real Ontime behavior; don't rely solely on JSON snapshots. Verify that later anchors survive earlier edits and gaps remain visible in the schedule; document playback implications. @@ -175,6 +271,10 @@ errors must remain visible with direct corrective actions. production GUI coverage and reporter behavior. Record unavailable manual checks. - [ ] Extend manual packaging/release preparation for both binaries and supported platforms, including only dependencies this tool needs. CI must not publish releases. +- [ ] Stretch: add Typst source/PDF reports and XLS/ODS re-export with verified + Ontime-compatible columns and documented format support. +- [ ] Stretch: add explicit buffer adjustment, before/after review, feasibility + checks and revalidation, with the cases listed above. - [ ] Commit each completed implementation task with its validation summary. ## Documentation conventions diff --git a/docs/rundown-converter.md b/docs/rundown-converter.md index b5ebeb0..fb7d33a 100644 --- a/docs/rundown-converter.md +++ b/docs/rundown-converter.md @@ -19,12 +19,22 @@ and durations, discard questionable rows, or fill unknown durations with zero. ## Planned workflow -1. Open or drop a document, then select a sheet or table if several are plausible. -2. Inspect detected headers, sample rows and a normalized schedule preview. -3. Resolve highlighted ambiguities through column mappings or clearer source - headers. Ordinary unambiguous files should require no manual mapping. -4. Export native Ontime JSON and optionally a neutral JSON representation for - inspection or reuse. Open the result or its output folder. +Open or drop a document and select its sheet/table, then follow five stages: + +1. Infer column meanings from data across rows. An ambiguous cell can be clarified + by other rows, but one meaning must fit the whole column; otherwise return an + error asking for clearer headers, corrected values or an explicit mapping. +2. Identify untimed gaps and incorporate them into the draft intermediate + representation (IR). These breaks occupy schedule time without a shown timer. +3. Construct the complete IR from times and gap constraints, preserving original + and inferred values. Ask for clarification when required facts remain unknown. +4. Check the complete schedule for inconsistencies, including backward times, + negative durations, overlaps and contradictory start/end/duration values. +5. Generate native Ontime JSON from the validated IR, with optional neutral JSON + for inspection or reuse. Open the result or its output folder. + +Review and correction controls expose each stage's decisions and errors. CLI and +GUI follow this same sequence; export cannot bypass unresolved errors. The CLI and GUI will use the same detection, validation and export code. CLI failures will identify the source location and return a nonzero exit status. @@ -59,8 +69,8 @@ strategies in [Smart time entry](https://docs.getontime.no/quick-tips/smart-time Its [delay documentation](https://docs.getontime.no/quick-tips/managing-delays/) also describes schedule gaps absorbing delays. The planned default export will retain a known unnamed break as a gap, with the following start unlinked. -An optional visible “Break (inferred)” cue can be evaluated during implementation; -it must be visibly synthetic and must not change the following scheduled start. +Untimed gaps remain in the IR without generating a synthetic timer or displaying +a timer duration. An explicitly timed break cue remains distinct from such gaps. Schedule preservation does not imply automatic playback or guarantee that an operator will start an event on time. @@ -70,14 +80,31 @@ Native Ontime JSON is the primary target: it lets the exporter control time relationships and project structure without a second column-mapping step. Ontime supports project files and spreadsheet interchange; see its [import/export documentation](https://docs.getontime.no/quick-tips/import-export/). -An XLSX export is a possible later convenience for human editing, not a required -intermediate format. Legacy XLS output is not planned. +Stretch goals are a Typst report (editable source and PDF) and XLS/ODS re-export +with normalized columns directly suitable for Ontime import. Actual file-format +and column compatibility must be verified against the target release; any required +format conversion will be documented. These outputs share the validated IR and +are not intermediate steps for native JSON export. The existing reporter targets Ontime 4. The new adapter will initially target that family, with the exact tested release/schema recorded before implementation is accepted. Import and timing behavior must be verified in that release; consulting documentation alone does not establish compatibility. +## Stretch goal: reserve buffer time + +An optional correction mode will shorten timed rows to create a requested buffer +in an overcrowded schedule. It will offer equal or proportional cuts, minimum +durations and protected rows, with a before/after preview. Original values remain +available. Untimed breaks do not contribute time cuts. + +Saved time can stay as slack after each event, preserving starts, or be collected +before a chosen anchor if the user explicitly allows intermediate starts to move. +Fixed restart times stay protected. Impossible requests must report the available +capacity; existing overlaps must be accounted for before claiming new buffer. +Every adjusted schedule must pass validation again before export. See the +[buffer design](rundown-converter-plan.md#optional-buffer-adjustment). + ## Examples needed next Provide representative originals, including difficult or ambiguous layouts and