beffroi-tools/docs/rundown-converter-plan.md

317 lines
19 KiB
Markdown

# Rundown converter development plan
Status: first XLSX/CSV implementation is available with a shared core, CLI and
egiu interface. User-facing scope and current limitations are in
[Rundown converter](rundown-converter.md); [corpus findings](rundown-corpus.md)
record real inputs and unresolved questions. Unchecked tasks below remain future
work; broader design requirements are not claims of implemented support.
## Architecture
Workspace member: `crates/rundown-converter/`, with binaries
`rundown-converter` and `rundown-converter-gui`. It uses the reporter's
Rust/clap/eframe approach and optional desktop feature.
A Python prototype is optional for investigating a difficult format, not a
second maintained implementation.
Source extraction prepares tables before the five distinct processing stages:
```text
source readers -> source tables
1. infer and verify column meanings across rows
2. identify untimed gaps and record them in the draft IR
3. construct the complete IR from source times and gap constraints
4. validate schedule consistency
5. export the validated IR to Ontime JSON
```
Each stage has a separate result and diagnostics. A later stage must not silently
reinterpret an earlier decision. Unresolved column meanings block stage 2;
unresolved gap boundaries remain explicit constraints for stage 3. If a complete
schedule cannot be established, return an actionable error rather than exporting
partial guesses. Stage 4 is an independent validation pass over the complete IR,
not a side effect of export. All exporters consume the same validated IR.
Readers retain original cell text, typed values, number formats, merged-cell
ranges and source coordinates when available. A table model separates extraction
from interpretation so format-specific quirks do not leak into the Ontime adapter.
PDF extraction must retain page/position information and expose unreliable reading
order; image-only documents must produce an actionable unsupported/extraction
error until a tested reader exists. Never execute document macros or formulas.
Missing or stale formula results require explicit handling.
Planned core modules cover readers, detection, neutral model, timing, diagnostics,
export and orchestration. GUI and CLI are thin clients of the same core; neither
contains independent parsing rules. Extract shared reporter UI utilities only
when useful code reuse is demonstrated; do not couple the new model to videos,
FFprobe, Typst or the reporter's sequential timer assumptions.
## Neutral representation
Use a versioned serializable model independent of Ontime fields and units.
Represent unresolved facts explicitly; validation produces an exportable model
only when required decisions are resolved.
| Entity | Information to retain |
| --- | --- |
| Document | Source identity, reader, sheet/table choices, model version, interpretation settings |
| Source reference | File, sheet/page, row/cell or bounding region, original text |
| Rundown/session | Ordered entries, label, date/day context, stage/track where established |
| Entry | Stable identity, cue, title, speaker, notes, additional original columns, event/break/marker kind |
| Timing | Optional explicit start/end, duration, day offset, fixed/relative intent, inferred/resolved values |
| Gap | Untimed break, boundary references/constraints, resolved interval when known, provenance, no timer display |
| Decision | Selected mapping/rule, evidence, alternatives, user overrides |
| Diagnostic | Stable code, severity, source reference, explanation, corrective action |
Use integer time units, not floating-point arithmetic, for normalized durations
and schedule calculations. Keep wall-clock times plus date/day offsets separate
from elapsed durations. Preserve given and derived values so conflicts are
reportable rather than overwritten. Convert to Ontime milliseconds only in the
adapter. Stable identities should be reproducible for unchanged input; do not
promise identity stability across arbitrary source edits.
## 1. Infer and verify column meanings
Combine normalized French/English header aliases, cell types, number formats,
value patterns and consistency across rows. Normalize accents, casing, spacing
and line breaks without losing original text. Evaluate candidate header bands
and mappings jointly instead of assigning columns from individual keywords.
Infer meaning primarily from data across the entire column: one ambiguous cell
can be resolved by evidence in other rows. A column must retain one semantic role
and a compatible unit interpretation throughout the selected table. Different
textual formats may express the same role; roles must not switch row by row.
Require the chosen interpretation to fit every relevant nonempty data cell, not
just a majority or a sample. If no single meaning fits, return
`INCONSISTENT_COLUMN_MEANING` with conflicting cell references and ask the user to
clarify headers, correct values or explicitly separate distinct tables. An
explicit mapping must also pass this check. Empty cells provide no evidence;
recognized headings/totals are structural rows with recorded classifications,
not convenient exceptions used to hide contradictory event data. Check semantic
consistency here; chronological contradictions belong to stage 4.
Account for title blocks above tables, merged/multiline headers, repeated headers,
blank separators, continuation rows, section labels, footnotes and totals.
Account for clock forms such as `09:30`/`9h30`, explicit units such as `15 min`,
spreadsheet time serials, formula results, decimal commas and CSV delimiters.
An unlabelled `10:30` or bare `15` is not sufficient evidence for a time role/unit.
Formatting and keywords are evidence, not proof.
Calibrate acceptance thresholds and the separation between competing mappings
against examples. Do not present heuristic scores as statistical probabilities.
Report the winning evidence and meaningful alternatives. If required roles or
row classifications remain ambiguous, block export; nonessential columns can
remain preserved metadata with a warning. Never silently drop a possible cue.
Example diagnostic, with proposed wording:
```text
AMBIGUOUS_TIME_COLUMN — Sheet “Jour 1”, C4:C28, header “Temps”
Values such as “10:30” could be start times or durations.
Clarify the header to “Début” or “Durée (mm:ss)”, or map column C and its units.
No Ontime file was written.
```
Mappings must be possible without modifying originals. CLI options/config and a
GUI mapping panel should select table, header rows, column roles and units.
Reusable mapping profiles should verify expected headers before applying; a
changed document must not silently inherit an unsuitable profile.
## 2. Identify gaps and incorporate them into the IR
Identify breaks with no timer to display before deriving event durations. Use
source labels, structural evidence and independently established event ends and
following starts. Record each gap in the draft IR with its boundary references,
known times and unresolved constraints. A named untimed lunch row is a gap, not
an event with a missing duration. A break explicitly intended as a timed cue
remains a different entry kind.
Gaps consume schedule time but never create a timer or show a duration as an
event. Their intervals may appear in the diagnostic preview for explanation.
Resolve unknown boundaries in stage 3 when sufficient evidence exists; otherwise
ask for clarification. A long start-to-start interval alone is insufficient to
identify a break. No optional synthetic break timer is generated for these gaps.
## 3. Construct the complete IR from times
Apply the policies in the [tool overview](rundown-converter.md#timing-and-breaks).
Solve constraints within a session while preserving explicit anchors. An explicit
start/end pair yields duration; a start/duration pair yields end. If all three
are present, retain all three plus calculated values for stage 4 to compare.
Complete the ordered entries and gap intervals, preserving provenance and
explicit versus inferred facts. Missing facts that cannot be derived must produce
a clarification error; do not call an unresolved schedule complete.
## 4. Check schedule consistency
Validate the complete IR independently of the reader and export target. Check
start/end/duration agreement using a documented precision tolerance, time going
backward within each session/track, negative durations, overlaps and gap boundaries.
Keep source order; never sort away a backward time or repair it by assuming a new
day. Validate that gaps do not overlap events or accidentally consume cue time.
Do not infer an end across an unknown break, another session, another day or a
parallel track. Midnight rollover must be supported by evidence or an explicit
setting. Treat unresolved overlap, negative duration, invalid time and unavailable
final duration as actionable errors. Define zero-duration markers separately from
missing-duration events. Never silently clamp or wrap unsupported target values.
## 5. Generate importable output
The Ontime adapter will map absolute starts to independent schedule entries and
link only entries explicitly intended to follow their predecessor. Duration-lock,
end-lock and count-to-end playback are distinct choices: preserving a scheduled
start must not silently enable count-to-end or automated playback. Keep default
end actions manual. Validate target limits, ordering, grouping and project schema.
An established untimed gap remains empty; do not use a delay entry or synthetic
timer to represent it. Preserve the independent start after it.
Export via a temporary file and publish only after complete validation, with
explicit overwrite behavior and cleanup on errors/cancellation. Produce neutral
JSON separately from native Ontime JSON with unmistakable filenames. Diagnostic
preview may remain available on failure, but an invalid project must not be
published. Keep local source paths out of exported project notes by default.
## Stretch goals
### Additional exports
Generate a Typst report (editable source and rendered PDF) from the validated IR.
Re-export XLS/ODS spreadsheets with explicit, normalized columns suitable for
Ontime import. Verify accepted file types and exact header/unit requirements
against the target Ontime release before claiming direct import support; if a
requested format needs conversion, document that limitation explicitly. Include
round-trip import tests for timing and gaps. Keep untimed gaps free of timer rows
in every output. These exports do not become intermediate steps for JSON export.
### Optional buffer adjustment
Offer an explicit correction mode that removes time from event rows to reserve
buffer in a cramped schedule. Keep the original IR and create a separate proposed
revision with a before/after preview and a per-row adjustment record. Never apply
this automatically during inference, validation or export.
Let the user choose a target buffer (duration or percentage), scope/session,
distribution (equal cuts or proportional cuts), minimum event durations and
protected rows. By default distribute cuts across all eligible timed rows in the
selected scope; untimed gaps, markers and protected rows contribute nothing.
Reject infeasible requests with the maximum achievable buffer. Allocate integer
rounding remainders deterministically so cuts sum exactly to the requested buffer.
Define where the saved time goes: either slack after each shortened event while
keeping its start, or a consolidated buffer before a selected anchor. Consolidation
requires explicit permission to move affected intermediate starts; preserve fixed
session/restart anchors and protected timings. Never lengthen an existing break
by accident or count existing slack as newly recovered time. If a source already
overlaps, distinguish cuts needed to eliminate the overlap from additional cuts
needed to create the requested buffer. Column ambiguities must be resolved first;
correction must not conceal malformed times or silently choose their meaning.
Run the complete stage-4 validation again on any proposed revision before stage-5
export. Report remaining conflicts and export only a valid selected revision.
Test insufficient capacity, minimum durations, protected rows, anchor preservation,
rounding, existing gaps, overlap repair and GUI/CLI parity.
## CLI and desktop experience
Proposed CLI operations are inspect, validate and convert; finalize syntax after
sample review. Support explicit output, sheet/table selection, mappings, units,
neutral JSON and overwrite. Provide useful progress, plain redirected logs,
`NO_COLOR`, documented exit codes and structured diagnostics for automation.
CLI operation must not depend on GUI availability or interactive prompts.
Match the reporter's compact egui layout, purple primary action, native file
picker, drag/drop, recent inputs, output controls, Settings, fixed bottom progress
and Cancel, scrollable content, and Results with Open/Open folder/Copy path.
Add a Review pane for source samples, mappings and the resolved schedule; show
anchored starts, inferred durations, breaks and blockers clearly. Keep controls
usable in small windows and at larger text sizes. Conversion runs on a worker
thread; controls reflect running/cancelled/completed states. Parsing and export
errors must remain visible with direct corrective actions.
## Ordered task list and acceptance gates
- [x] Inspect existing CLI, GUI, Ontime export and repository documentation.
- [x] Check official Ontime documentation for schedule gaps and import options.
- [x] Specify neutral model, ambiguity policy, start-time priority and break rules.
- [x] Add separate tool documentation and link it from the repository overview.
- [x] Inspect both supplied XLSX workbooks and both PDFs; download five public
workbook layouts with a provenance manifest. Record findings separately.
- [x] Run a bounded PDF guessing experiment over all 17 supplied pages using
text coordinates and vector table borders. Record recovery and validator
results in the [PDF feasibility trial](rundown-corpus.md#pdf-feasibility-trial).
Pause importer work: column recovery succeeds, but missing durations and
simultaneous tracks prevent dependable whole-document timer conversion.
- [x] Add bounded XLSX and UTF-8 CSV/TSV readers, synthetic regression fixtures,
merged-block handling and source-located formula errors.
- [x] Verify exported event times and a 90-minute gap through the Ontime 4.15.1
native rundown parser and metadata calculation at the recorded source commit.
- [x] Add initial Setup/Review/Results GUI, source preview, column/unit selectors,
shared profile controls and worker/cancellation tests.
- [x] Keep inference evidence and the source table on Review after failures;
add column-role dropdowns, row classifications and title/start/end/duration corrections.
- [x] Save/overwrite/save-as portable resolution profiles and explicitly remember,
reuse or forget validated resolutions matched to the complete sheet.
- [x] Redesign Review into Resolve rows / Schedule preview / Columns, with one
issue at a time, row navigation/filtering, a full-height correction table and
separate saved-resolution controls. Preserve pending edits and previous results
until recheck; verify compact layout and dropdown interaction.
- [x] Merge schedule preview and row corrections with All rows / Included rows /
Needs attention filters, preserving ignored rows for restoration and displaying
attached notes individually without duplicating their timers.
- [ ] Add recent-file persistence, direct start/end/title editing and
comprehensive native GUI interaction checks.
- [ ] Inventory supplied examples by format/layout; record intended interpretations
and the user's Ontime version. Decide first-release input coverage from these.
- [ ] Build sanitized fixtures and an expected-result matrix before tuning rules.
Include difficult examples and some held-out layouts to detect overfitting.
- [ ] Confirm a pinned Ontime schema/release and create a minimal import experiment
with independent starts and a lunch gap; verify actual schedule and playback.
- [x] Scaffold the Rust member, core model, optional GUI and CLI; retain reporter
defaults. Include the new member in workspace CI, documentation and coverage.
- [ ] Implement the required source readers and extraction diagnostics. Test
unsupported/encrypted/corrupt inputs and formula/merged-cell behavior as relevant.
- [x] Stage 1: implement whole-column inference, explicit mappings and located
errors. Test ambiguous cells clarified by other rows, incompatible meanings in
one column, structural rows and misleading headers/units.
- [x] Stage 2: identify untimed gaps and populate draft IR constraints. Test named
and unnamed breaks, unknown boundaries and distinction from timed break cues.
- [x] Stage 3: construct the complete IR from times and gap constraints. Test
start-only schedules, missing final duration and provenance of inferred values.
- [x] Stage 4: independently validate backward times, negative durations,
contradictory triples, overlaps, gap boundaries and explicit markers.
- [ ] Add explicit midnight/day and multiple-track interpretations; current
behavior rejects these rather than guessing.
- [x] Implement inspect/validate/convert CLI and atomic exports. Test CLI status,
output collisions, cancellation, source preservation and deterministic results.
- [ ] Stage 5: implement the Ontime adapter and neutral JSON export. Check native imports
against the pinned parser and manually inspect real Ontime behavior; don't rely
solely on JSON snapshots. Verify that later anchors survive earlier edits and
gaps remain visible in the schedule; document playback implications.
- [ ] Implement desktop Setup/Review/Results, mapping corrections and diagnostics.
Verify GUI/CLI parity; manually inspect representative and ambiguous inputs,
resized windows, larger text, long paths, progress and cancellation.
- [ ] Run the supported corpus end to end: each source cue must be represented or
have an explicit reviewed exclusion; every unresolved ambiguity must block export.
- [ ] Complete usage examples, supported-format matrix, limitations and illustrated
GUI instructions. Add runnable quick-start commands only once they exist.
- [x] Run appropriate Python CI/packaging tests and all Rust gates listed in
[Development](development.md#checks), plus whole-workspace coverage. Preserve
production GUI coverage and reporter behavior. Record unavailable manual checks.
- [ ] Extend manual packaging/release preparation for both binaries and supported
platforms, including only dependencies this tool needs. CI must not publish releases.
- [ ] Stretch: add Typst source/PDF reports and XLS/ODS re-export with verified
Ontime-compatible columns and documented format support.
- [ ] Stretch: add explicit buffer adjustment, before/after review, feasibility
checks and revalidation, with the cases listed above.
- [ ] Commit each completed implementation task with its validation summary.
## Documentation conventions
Keep README as a short repository entry point with quick-start examples and links.
Each tool has its own usage page; detailed design/task tracking stays separate.
Use one H1 per page, sentence-case headings, fenced code with a language, ordinary
relative Markdown links and consistent tables/lists. Label planned features and
commands clearly. The existing illustrated guide remains explicitly scoped to the
PPTX reporter; add a separate converter guide when a working GUI can be captured.