Separate rundown processing stages and plan buffer corrections

This commit is contained in:
Dandaedre 2026-11-03 12:10:52 +01:00
parent 48414bf279
commit 5a28720fea
2 changed files with 153 additions and 26 deletions

View File

@ -12,13 +12,24 @@ Use the reporter's Rust/clap/eframe approach and optional desktop feature.
A Python prototype is optional for investigating a difficult format, not a
second maintained implementation.
The processing pipeline is:
Source extraction prepares tables before the five distinct processing stages:
```text
source readers -> source tables -> header/column detection -> neutral rundown
-> timing resolution and validation -> Ontime adapter -> file
source readers -> source tables
1. infer and verify column meanings across rows
2. identify untimed gaps and record them in the draft IR
3. construct the complete IR from source times and gap constraints
4. validate schedule consistency
5. export the validated IR to Ontime JSON
```
Each stage has a separate result and diagnostics. A later stage must not silently
reinterpret an earlier decision. Unresolved column meanings block stage 2;
unresolved gap boundaries remain explicit constraints for stage 3. If a complete
schedule cannot be established, return an actionable error rather than exporting
partial guesses. Stage 4 is an independent validation pass over the complete IR,
not a side effect of export. All exporters consume the same validated IR.
Readers retain original cell text, typed values, number formats, merged-cell
ranges and source coordinates when available. A table model separates extraction
from interpretation so format-specific quirks do not leak into the Ontime adapter.
@ -46,7 +57,7 @@ only when required decisions are resolved.
| Rundown/session | Ordered entries, label, date/day context, stage/track where established |
| Entry | Stable identity, cue, title, speaker, notes, additional original columns, event/break/marker kind |
| Timing | Optional explicit start/end, duration, day offset, fixed/relative intent, inferred/resolved values |
| Gap | Known previous end and next start, interval, provenance, explicit or inferred classification |
| Gap | Untimed break, boundary references/constraints, resolved interval when known, provenance, no timer display |
| Decision | Selected mapping/rule, evidence, alternatives, user overrides |
| Diagnostic | Stable code, severity, source reference, explanation, corrective action |
@ -57,12 +68,25 @@ reportable rather than overwritten. Convert to Ontime milliseconds only in the
adapter. Stable identities should be reproducible for unchanged input; do not
promise identity stability across arbitrary source edits.
## Detection and clarification
## 1. Infer and verify column meanings
Combine normalized French/English header aliases, cell types, number formats,
value patterns and consistency across rows. Normalize accents, casing, spacing
and line breaks without losing original text. Evaluate candidate header bands
and mappings jointly instead of assigning columns from individual keywords.
Infer meaning primarily from data across the entire column: one ambiguous cell
can be resolved by evidence in other rows. A column must retain one semantic role
and a compatible unit interpretation throughout the selected table. Different
textual formats may express the same role; roles must not switch row by row.
Require the chosen interpretation to fit every relevant nonempty data cell, not
just a majority or a sample. If no single meaning fits, return
`INCONSISTENT_COLUMN_MEANING` with conflicting cell references and ask the user to
clarify headers, correct values or explicitly separate distinct tables. An
explicit mapping must also pass this check. Empty cells provide no evidence;
recognized headings/totals are structural rows with recorded classifications,
not convenient exceptions used to hide contradictory event data. Check semantic
consistency here; chronological contradictions belong to stage 4.
Account for title blocks above tables, merged/multiline headers, repeated headers,
blank separators, continuation rows, section labels, footnotes and totals.
@ -91,12 +115,38 @@ GUI mapping panel should select table, header rows, column roles and units.
Reusable mapping profiles should verify expected headers before applying; a
changed document must not silently inherit an unsuitable profile.
## Timing resolution and export contract
## 2. Identify gaps and incorporate them into the IR
Identify breaks with no timer to display before deriving event durations. Use
source labels, structural evidence and independently established event ends and
following starts. Record each gap in the draft IR with its boundary references,
known times and unresolved constraints. A named untimed lunch row is a gap, not
an event with a missing duration. A break explicitly intended as a timed cue
remains a different entry kind.
Gaps consume schedule time but never create a timer or show a duration as an
event. Their intervals may appear in the diagnostic preview for explanation.
Resolve unknown boundaries in stage 3 when sufficient evidence exists; otherwise
ask for clarification. A long start-to-start interval alone is insufficient to
identify a break. No optional synthetic break timer is generated for these gaps.
## 3. Construct the complete IR from times
Apply the policies in the [tool overview](rundown-converter.md#timing-and-breaks).
Solve constraints within a session while preserving explicit anchors. An explicit
start/end pair yields duration; a start/duration pair yields end. If all three
are present, validate agreement using a documented precision tolerance.
are present, retain all three plus calculated values for stage 4 to compare.
Complete the ordered entries and gap intervals, preserving provenance and
explicit versus inferred facts. Missing facts that cannot be derived must produce
a clarification error; do not call an unresolved schedule complete.
## 4. Check schedule consistency
Validate the complete IR independently of the reader and export target. Check
start/end/duration agreement using a documented precision tolerance, time going
backward within each session/track, negative durations, overlaps and gap boundaries.
Keep source order; never sort away a backward time or repair it by assuming a new
day. Validate that gaps do not overlap events or accidentally consume cue time.
Do not infer an end across an unknown break, another session, another day or a
parallel track. Midnight rollover must be supported by evidence or an explicit
@ -104,14 +154,15 @@ setting. Treat unresolved overlap, negative duration, invalid time and unavailab
final duration as actionable errors. Define zero-duration markers separately from
missing-duration events. Never silently clamp or wrap unsupported target values.
## 5. Generate importable output
The Ontime adapter will map absolute starts to independent schedule entries and
link only entries explicitly intended to follow their predecessor. Duration-lock,
end-lock and count-to-end playback are distinct choices: preserving a scheduled
start must not silently enable count-to-end or automated playback. Keep default
end actions manual. Validate target limits, ordering, grouping and project schema.
An established gap remains empty by default; do not use a delay entry to represent
an ordinary lunch break. If visible inferred break cues are implemented, label
and track them separately from source events.
An established untimed gap remains empty; do not use a delay entry or synthetic
timer to represent it. Preserve the independent start after it.
Export via a temporary file and publish only after complete validation, with
explicit overwrite behavior and cleanup on errors/cancellation. Produce neutral
@ -119,6 +170,46 @@ JSON separately from native Ontime JSON with unmistakable filenames. Diagnostic
preview may remain available on failure, but an invalid project must not be
published. Keep local source paths out of exported project notes by default.
## Stretch goals
### Additional exports
Generate a Typst report (editable source and rendered PDF) from the validated IR.
Re-export XLS/ODS spreadsheets with explicit, normalized columns suitable for
Ontime import. Verify accepted file types and exact header/unit requirements
against the target Ontime release before claiming direct import support; if a
requested format needs conversion, document that limitation explicitly. Include
round-trip import tests for timing and gaps. Keep untimed gaps free of timer rows
in every output. These exports do not become intermediate steps for JSON export.
### Optional buffer adjustment
Offer an explicit correction mode that removes time from event rows to reserve
buffer in a cramped schedule. Keep the original IR and create a separate proposed
revision with a before/after preview and a per-row adjustment record. Never apply
this automatically during inference, validation or export.
Let the user choose a target buffer (duration or percentage), scope/session,
distribution (equal cuts or proportional cuts), minimum event durations and
protected rows. By default distribute cuts across all eligible timed rows in the
selected scope; untimed gaps, markers and protected rows contribute nothing.
Reject infeasible requests with the maximum achievable buffer. Allocate integer
rounding remainders deterministically so cuts sum exactly to the requested buffer.
Define where the saved time goes: either slack after each shortened event while
keeping its start, or a consolidated buffer before a selected anchor. Consolidation
requires explicit permission to move affected intermediate starts; preserve fixed
session/restart anchors and protected timings. Never lengthen an existing break
by accident or count existing slack as newly recovered time. If a source already
overlaps, distinguish cuts needed to eliminate the overlap from additional cuts
needed to create the requested buffer. Column ambiguities must be resolved first;
correction must not conceal malformed times or silently choose their meaning.
Run the complete stage-4 validation again on any proposed revision before stage-5
export. Report remaining conflicts and export only a valid selected revision.
Test insufficient capacity, minimum durations, protected rows, anchor preservation,
rounding, existing gaps, overlap repair and GUI/CLI parity.
## CLI and desktop experience
Proposed CLI operations are inspect, validate and convert; finalize syntax after
@ -152,14 +243,19 @@ errors must remain visible with direct corrective actions.
defaults. Include the new member in workspace CI, documentation and coverage.
- [ ] Implement the required source readers and extraction diagnostics. Test
unsupported/encrypted/corrupt inputs and formula/merged-cell behavior as relevant.
- [ ] Implement detection, explicit mappings and source-located errors; verify
competing headers and misleading time formats fail clearly.
- [ ] Implement timing constraints and gaps. Test known/unknown lunch gaps,
start-only schedules, contradictory triples, missing final duration, overlaps,
midnight/day transitions, zero-duration markers and multiple sessions/tracks.
- [ ] Stage 1: implement whole-column inference, explicit mappings and located
errors. Test ambiguous cells clarified by other rows, incompatible meanings in
one column, structural rows and misleading headers/units.
- [ ] Stage 2: identify untimed gaps and populate draft IR constraints. Test named
and unnamed breaks, unknown boundaries and distinction from timed break cues.
- [ ] Stage 3: construct the complete IR from times and gap constraints. Test
start-only schedules, missing final duration and provenance of inferred values.
- [ ] Stage 4: independently validate backward times, negative durations,
contradictory triples, overlaps, gap boundaries, midnight/day transitions,
zero-duration markers and multiple sessions/tracks.
- [ ] Implement inspect/validate/convert CLI and atomic exports. Test CLI status,
output collisions, cancellation, source preservation and deterministic results.
- [ ] Implement the Ontime adapter and neutral JSON export. Check native imports
- [ ] Stage 5: implement the Ontime adapter and neutral JSON export. Check native imports
against the pinned parser and manually inspect real Ontime behavior; don't rely
solely on JSON snapshots. Verify that later anchors survive earlier edits and
gaps remain visible in the schedule; document playback implications.
@ -175,6 +271,10 @@ errors must remain visible with direct corrective actions.
production GUI coverage and reporter behavior. Record unavailable manual checks.
- [ ] Extend manual packaging/release preparation for both binaries and supported
platforms, including only dependencies this tool needs. CI must not publish releases.
- [ ] Stretch: add Typst source/PDF reports and XLS/ODS re-export with verified
Ontime-compatible columns and documented format support.
- [ ] Stretch: add explicit buffer adjustment, before/after review, feasibility
checks and revalidation, with the cases listed above.
- [ ] Commit each completed implementation task with its validation summary.
## Documentation conventions

View File

@ -19,12 +19,22 @@ and durations, discard questionable rows, or fill unknown durations with zero.
## Planned workflow
1. Open or drop a document, then select a sheet or table if several are plausible.
2. Inspect detected headers, sample rows and a normalized schedule preview.
3. Resolve highlighted ambiguities through column mappings or clearer source
headers. Ordinary unambiguous files should require no manual mapping.
4. Export native Ontime JSON and optionally a neutral JSON representation for
inspection or reuse. Open the result or its output folder.
Open or drop a document and select its sheet/table, then follow five stages:
1. Infer column meanings from data across rows. An ambiguous cell can be clarified
by other rows, but one meaning must fit the whole column; otherwise return an
error asking for clearer headers, corrected values or an explicit mapping.
2. Identify untimed gaps and incorporate them into the draft intermediate
representation (IR). These breaks occupy schedule time without a shown timer.
3. Construct the complete IR from times and gap constraints, preserving original
and inferred values. Ask for clarification when required facts remain unknown.
4. Check the complete schedule for inconsistencies, including backward times,
negative durations, overlaps and contradictory start/end/duration values.
5. Generate native Ontime JSON from the validated IR, with optional neutral JSON
for inspection or reuse. Open the result or its output folder.
Review and correction controls expose each stage's decisions and errors. CLI and
GUI follow this same sequence; export cannot bypass unresolved errors.
The CLI and GUI will use the same detection, validation and export code. CLI
failures will identify the source location and return a nonzero exit status.
@ -59,8 +69,8 @@ strategies in [Smart time entry](https://docs.getontime.no/quick-tips/smart-time
Its [delay documentation](https://docs.getontime.no/quick-tips/managing-delays/)
also describes schedule gaps absorbing delays. The planned default export will
retain a known unnamed break as a gap, with the following start unlinked.
An optional visible “Break (inferred)” cue can be evaluated during implementation;
it must be visibly synthetic and must not change the following scheduled start.
Untimed gaps remain in the IR without generating a synthetic timer or displaying
a timer duration. An explicitly timed break cue remains distinct from such gaps.
Schedule preservation does not imply automatic playback or guarantee that an
operator will start an event on time.
@ -70,14 +80,31 @@ Native Ontime JSON is the primary target: it lets the exporter control time
relationships and project structure without a second column-mapping step.
Ontime supports project files and spreadsheet interchange; see its
[import/export documentation](https://docs.getontime.no/quick-tips/import-export/).
An XLSX export is a possible later convenience for human editing, not a required
intermediate format. Legacy XLS output is not planned.
Stretch goals are a Typst report (editable source and PDF) and XLS/ODS re-export
with normalized columns directly suitable for Ontime import. Actual file-format
and column compatibility must be verified against the target release; any required
format conversion will be documented. These outputs share the validated IR and
are not intermediate steps for native JSON export.
The existing reporter targets Ontime 4. The new adapter will initially target
that family, with the exact tested release/schema recorded before implementation
is accepted. Import and timing behavior must be verified in that release;
consulting documentation alone does not establish compatibility.
## Stretch goal: reserve buffer time
An optional correction mode will shorten timed rows to create a requested buffer
in an overcrowded schedule. It will offer equal or proportional cuts, minimum
durations and protected rows, with a before/after preview. Original values remain
available. Untimed breaks do not contribute time cuts.
Saved time can stay as slack after each event, preserving starts, or be collected
before a chosen anchor if the user explicitly allows intermediate starts to move.
Fixed restart times stay protected. Impossible requests must report the available
capacity; existing overlaps must be accounted for before claiming new buffer.
Every adjusted schedule must pass validation again before export. See the
[buffer design](rundown-converter-plan.md#optional-buffer-adjustment).
## Examples needed next
Provide representative originals, including difficult or ambiguous layouts and