344 lines
22 KiB
Markdown
344 lines
22 KiB
Markdown
# Rundown converter development plan
|
|
|
|
Status: the XLSX/CSV release candidate is available with a shared core, CLI and
|
|
egui interface. User-facing scope and current limitations are in
|
|
[Rundown converter](rundown-converter.md); [corpus findings](rundown-corpus.md)
|
|
record real inputs and unresolved questions. Unchecked tasks below remain future
|
|
work; broader design requirements are not claims of implemented support.
|
|
|
|
## Completion audit
|
|
|
|
Audited against the `0.3.0-rc.1` source and documentation on 2026-11-03. The
|
|
five required workflow stages and the requested shared CLI/GUI review experience
|
|
are implemented. Remaining work is acceptance hardening, explicitly unsupported
|
|
schedule models, packaging and stretch goals.
|
|
|
|
| Area | Status | Evidence or remaining work |
|
|
| --- | --- | --- |
|
|
| 1. Whole-column inference | Complete | Header/data evidence, manual mapping and whole-column contradictions are implemented and tested. Explicit roles can select the start component of a mixed range and apply one documented extraction policy to a mixed duration column. |
|
|
| 2. Untimed gaps | Complete | Named and inferred gaps enter the IR; keep/extend previous/add to next resolutions are available. |
|
|
| 3. Complete IR | Complete for supported single-day schedules | Explicit and inferred timing provenance, notes, metadata, corrections and exclusions are retained. |
|
|
| 4. Consistency checks | Complete for supported single-track schedules | Conflicting triples, backward/negative timing, overlaps and gap boundaries block export. |
|
|
| 5. Ontime output | Complete | Native Ontime JSON, neutral JSON and normalized XLSX share the validated IR; native parsing was checked against Ontime 4.15.1. |
|
|
| CLI and desktop review | Complete for the release-candidate workflow | Inspect/validate/convert, Setup/Review/Results, mapping, row editing, profiles, remembered resolutions and pending/recheck behavior are present. Recent-file persistence and broader native/platform interaction checks remain. |
|
|
| Public evaluation corpus | Expanded, still not a benchmark | Ten external files now cover XLSX and CSV successes, deliberate failures and manual-resolution cases. A fully reviewed expected-result matrix and held-out accuracy exercise remain. |
|
|
| PDF input | Trial complete; importer paused | The bounded 17-page experiment recovered tables but could not establish dependable sequential timers. |
|
|
| Parallel tracks and midnight/day rollover | Unsupported by design | These require explicit models; current behavior stops instead of guessing. |
|
|
| Portable release bundles | Open | Workspace builds include the converter, but release packaging is still reporter-oriented. |
|
|
| XLSX re-export stretch goal | Complete | Writes Ontime-importable normalized columns. |
|
|
| Typst/PDF report stretch goal | Open | No report exporter is implemented. |
|
|
| Buffer-adjustment stretch goal | Open | No proposal, feasibility or redistribution workflow is implemented. |
|
|
| ODS | Intentionally excluded | Ontime's file importer does not expose ODS upload; XLSX is the supported spreadsheet export. |
|
|
|
|
## Architecture
|
|
|
|
Workspace member: `crates/rundown-converter/`, with binaries
|
|
`rundown-converter` and `rundown-converter-gui`. It uses the reporter's
|
|
Rust/clap/eframe approach and optional desktop feature.
|
|
A Python prototype is optional for investigating a difficult format, not a
|
|
second maintained implementation.
|
|
|
|
Source extraction prepares tables before the five distinct processing stages:
|
|
|
|
```text
|
|
source readers -> source tables
|
|
1. infer and verify column meanings across rows
|
|
2. identify untimed gaps and record them in the draft IR
|
|
3. construct the complete IR from source times and gap constraints
|
|
4. validate schedule consistency
|
|
5. export the validated IR to Ontime JSON
|
|
```
|
|
|
|
Each stage has a separate result and diagnostics. A later stage must not silently
|
|
reinterpret an earlier decision. Unresolved column meanings block stage 2;
|
|
unresolved gap boundaries remain explicit constraints for stage 3. If a complete
|
|
schedule cannot be established, return an actionable error rather than exporting
|
|
partial guesses. Stage 4 is an independent validation pass over the complete IR,
|
|
not a side effect of export. All exporters consume the same validated IR.
|
|
|
|
Readers retain original cell text, typed values, number formats, merged-cell
|
|
ranges and source coordinates when available. A table model separates extraction
|
|
from interpretation so format-specific quirks do not leak into the Ontime adapter.
|
|
PDF extraction must retain page/position information and expose unreliable reading
|
|
order; image-only documents must produce an actionable unsupported/extraction
|
|
error until a tested reader exists. Never execute document macros or spreadsheet
|
|
functions. A bounded arithmetic parser may calculate same-sheet references when
|
|
cached results are missing; unsupported or stale results require explicit handling.
|
|
|
|
Planned core modules cover readers, detection, neutral model, timing, diagnostics,
|
|
export and orchestration. GUI and CLI are thin clients of the same core; neither
|
|
contains independent parsing rules. Extract shared reporter UI utilities only
|
|
when useful code reuse is demonstrated; do not couple the new model to videos,
|
|
FFprobe, Typst or the reporter's sequential timer assumptions.
|
|
|
|
## Neutral representation
|
|
|
|
Use a versioned serializable model independent of Ontime fields and units.
|
|
Represent unresolved facts explicitly; validation produces an exportable model
|
|
only when required decisions are resolved.
|
|
|
|
| Entity | Information to retain |
|
|
| --- | --- |
|
|
| Document | Source identity, reader, sheet/table choices, model version, interpretation settings |
|
|
| Source reference | File, sheet/page, row/cell or bounding region, original text |
|
|
| Rundown/session | Ordered entries, label, date/day context, stage/track where established |
|
|
| Entry | Stable identity, cue, title, notes, custom-field metadata, additional original columns, event/break/marker kind |
|
|
| Timing | Optional explicit start/end, duration, day offset, fixed/relative intent, inferred/resolved values |
|
|
| Gap | Untimed break, boundary references/constraints, resolved interval when known, provenance, no timer display |
|
|
| Decision | Selected mapping/rule, evidence, alternatives, user overrides |
|
|
| Diagnostic | Stable code, severity, source reference, explanation, corrective action |
|
|
|
|
Use integer time units, not floating-point arithmetic, for normalized durations
|
|
and schedule calculations. Keep wall-clock times plus date/day offsets separate
|
|
from elapsed durations. Preserve given and derived values so conflicts are
|
|
reportable rather than overwritten. Convert to Ontime milliseconds only in the
|
|
adapter. Stable identities should be reproducible for unchanged input; do not
|
|
promise identity stability across arbitrary source edits.
|
|
|
|
## 1. Infer and verify column meanings
|
|
|
|
Combine normalized French/English header aliases, cell types, number formats,
|
|
value patterns and consistency across rows. Normalize accents, casing, spacing
|
|
and line breaks without losing original text. Evaluate candidate header bands
|
|
and mappings jointly instead of assigning columns from individual keywords.
|
|
Infer meaning primarily from data across the entire column: one ambiguous cell
|
|
can be resolved by evidence in other rows. A column must retain one semantic role
|
|
and a compatible unit interpretation throughout the selected table. Different
|
|
textual formats may express the same role; roles must not switch row by row.
|
|
|
|
Require the chosen interpretation to fit every relevant nonempty data cell, not
|
|
just a majority or a sample. If no single meaning fits, return
|
|
`INCONSISTENT_COLUMN_MEANING` with conflicting cell references and ask the user to
|
|
clarify headers, correct values or explicitly separate distinct tables. An
|
|
explicit mapping must also pass this check. Empty cells provide no evidence;
|
|
recognized headings/totals are structural rows with recorded classifications,
|
|
not convenient exceptions used to hide contradictory event data. Check semantic
|
|
consistency here; chronological contradictions belong to stage 4.
|
|
|
|
Account for title blocks above tables, merged/multiline headers, repeated headers,
|
|
blank separators, continuation rows, section labels, footnotes and totals.
|
|
Account for clock forms such as `09:30`/`9h30`, explicit units such as `15 min`,
|
|
spreadsheet time serials, formula results, decimal commas and CSV delimiters.
|
|
An unlabelled `10:30` or bare `15` is not sufficient evidence for a time role/unit.
|
|
Formatting and keywords are evidence, not proof.
|
|
|
|
Calibrate acceptance thresholds and the separation between competing mappings
|
|
against examples. Do not present heuristic scores as statistical probabilities.
|
|
Report the winning evidence and meaningful alternatives. If required roles or
|
|
row classifications remain ambiguous, block export; nonessential columns can
|
|
remain preserved metadata with a warning. Never silently drop a possible cue.
|
|
|
|
Example diagnostic, with proposed wording:
|
|
|
|
```text
|
|
AMBIGUOUS_TIME_COLUMN — Sheet “Jour 1”, C4:C28, header “Temps”
|
|
Values such as “10:30” could be start times or durations.
|
|
Clarify the header to “Début” or “Durée (mm:ss)”, or map column C and its units.
|
|
No Ontime file was written.
|
|
```
|
|
|
|
Mappings must be possible without modifying originals. CLI options/config and a
|
|
GUI mapping panel should select table, header rows, column roles and units.
|
|
Reusable mapping profiles should verify expected headers before applying; a
|
|
changed document must not silently inherit an unsuitable profile.
|
|
|
|
## 2. Identify gaps and incorporate them into the IR
|
|
|
|
Identify breaks with no timer to display before deriving event durations. Use
|
|
source labels, structural evidence and independently established event ends and
|
|
following starts. Record each gap in the draft IR with its boundary references,
|
|
known times and unresolved constraints. A named untimed lunch row is a gap, not
|
|
an event with a missing duration. A break explicitly intended as a timed cue
|
|
remains a different entry kind.
|
|
|
|
Gaps consume schedule time but never create a timer or show a duration as an
|
|
event. Their intervals may appear in the diagnostic preview for explanation.
|
|
Resolve unknown boundaries in stage 3 when sufficient evidence exists; otherwise
|
|
ask for clarification. A long start-to-start interval alone is insufficient to
|
|
identify a break. No optional synthetic break timer is generated for these gaps.
|
|
|
|
## 3. Construct the complete IR from times
|
|
|
|
Apply the policies in the [tool overview](rundown-converter.md#timing-and-breaks).
|
|
Solve constraints within a session while preserving explicit anchors. An explicit
|
|
start/end pair yields duration; a start/duration pair yields end. If all three
|
|
are present, retain all three plus calculated values for stage 4 to compare.
|
|
Complete the ordered entries and gap intervals, preserving provenance and
|
|
explicit versus inferred facts. Missing facts that cannot be derived must produce
|
|
a clarification error; do not call an unresolved schedule complete.
|
|
|
|
## 4. Check schedule consistency
|
|
|
|
Validate the complete IR independently of the reader and export target. Check
|
|
start/end/duration agreement using a documented precision tolerance, time going
|
|
backward within each session/track, negative durations, overlaps and gap boundaries.
|
|
Keep source order; never sort away a backward time or repair it by assuming a new
|
|
day. Validate that gaps do not overlap events or accidentally consume cue time.
|
|
|
|
Do not infer an end across an unknown break, another session, another day or a
|
|
parallel track. Midnight rollover must be supported by evidence or an explicit
|
|
setting. Treat unresolved overlap, negative duration, invalid time and unavailable
|
|
final duration as actionable errors. Define zero-duration markers separately from
|
|
missing-duration events. Never silently clamp or wrap unsupported target values.
|
|
|
|
## 5. Generate importable output
|
|
|
|
The Ontime adapter will map absolute starts to independent schedule entries and
|
|
link only entries explicitly intended to follow their predecessor. Duration-lock,
|
|
end-lock and count-to-end playback are distinct choices: preserving a scheduled
|
|
start must not silently enable count-to-end or automated playback. Keep default
|
|
end actions manual. Validate target limits, ordering, grouping and project schema.
|
|
An established untimed gap remains empty; do not use a delay entry or synthetic
|
|
timer to represent it. Preserve the independent start after it.
|
|
|
|
Export via a temporary file and publish only after complete validation, with
|
|
explicit overwrite behavior and cleanup on errors/cancellation. Produce neutral
|
|
JSON separately from native Ontime JSON with unmistakable filenames. Diagnostic
|
|
preview may remain available on failure, but an invalid project must not be
|
|
published. Keep local source paths out of exported project notes by default.
|
|
|
|
## Stretch goals
|
|
|
|
### Additional exports
|
|
|
|
Generate a Typst report (editable source and rendered PDF) from the validated IR.
|
|
XLSX re-export provides explicit, normalized columns suitable for Ontime import;
|
|
ODS remains unavailable because Ontime's current file importer does not expose
|
|
ODS upload. Keep untimed gaps free of timer rows in every output. These exports
|
|
do not become intermediate steps for JSON export.
|
|
|
|
### Optional buffer adjustment
|
|
|
|
Offer an explicit correction mode that removes time from event rows to reserve
|
|
buffer in a cramped schedule. Keep the original IR and create a separate proposed
|
|
revision with a before/after preview and a per-row adjustment record. Never apply
|
|
this automatically during inference, validation or export.
|
|
|
|
Let the user choose a target buffer (duration or percentage), scope/session,
|
|
distribution (equal cuts or proportional cuts), minimum event durations and
|
|
protected rows. By default distribute cuts across all eligible timed rows in the
|
|
selected scope; untimed gaps, markers and protected rows contribute nothing.
|
|
Reject infeasible requests with the maximum achievable buffer. Allocate integer
|
|
rounding remainders deterministically so cuts sum exactly to the requested buffer.
|
|
|
|
Define where the saved time goes: either slack after each shortened event while
|
|
keeping its start, or a consolidated buffer before a selected anchor. Consolidation
|
|
requires explicit permission to move affected intermediate starts; preserve fixed
|
|
session/restart anchors and protected timings. Never lengthen an existing break
|
|
by accident or count existing slack as newly recovered time. If a source already
|
|
overlaps, distinguish cuts needed to eliminate the overlap from additional cuts
|
|
needed to create the requested buffer. Column ambiguities must be resolved first;
|
|
correction must not conceal malformed times or silently choose their meaning.
|
|
|
|
Run the complete stage-4 validation again on any proposed revision before stage-5
|
|
export. Report remaining conflicts and export only a valid selected revision.
|
|
Test insufficient capacity, minimum durations, protected rows, anchor preservation,
|
|
rounding, existing gaps, overlap repair and GUI/CLI parity.
|
|
|
|
## CLI and desktop experience
|
|
|
|
Proposed CLI operations are inspect, validate and convert; finalize syntax after
|
|
sample review. Support explicit output, sheet/table selection, mappings, units,
|
|
neutral JSON and overwrite. Provide useful progress, plain redirected logs,
|
|
`NO_COLOR`, documented exit codes and structured diagnostics for automation.
|
|
CLI operation must not depend on GUI availability or interactive prompts.
|
|
|
|
Match the reporter's compact egui layout, purple primary action, native file
|
|
picker, drag/drop, recent inputs, output controls, Settings, fixed bottom progress
|
|
and Cancel, scrollable content, and Results with Open/Open folder/Copy path.
|
|
Add a Review pane for source samples, mappings and the resolved schedule; show
|
|
anchored starts, inferred durations, breaks and blockers clearly. Keep controls
|
|
usable in small windows and at larger text sizes. Conversion runs on a worker
|
|
thread; controls reflect running/cancelled/completed states. Parsing and export
|
|
errors must remain visible with direct corrective actions.
|
|
|
|
## Ordered task list and acceptance gates
|
|
|
|
- [x] Inspect existing CLI, GUI, Ontime export and repository documentation.
|
|
- [x] Check official Ontime documentation for schedule gaps and import options.
|
|
- [x] Specify neutral model, ambiguity policy, start-time priority and break rules.
|
|
- [x] Add separate tool documentation and link it from the repository overview.
|
|
- [x] Inspect both supplied XLSX workbooks and both PDFs; download five public
|
|
workbook layouts with a provenance manifest. Record findings separately.
|
|
- [x] Run a bounded PDF guessing experiment over all 17 supplied pages using
|
|
text coordinates and vector table borders. Record recovery and validator
|
|
results in the [PDF feasibility trial](rundown-corpus.md#pdf-feasibility-trial).
|
|
Pause importer work: column recovery succeeds, but missing durations and
|
|
simultaneous tracks prevent dependable whole-document timer conversion.
|
|
- [x] Add bounded XLSX and UTF-8 CSV/TSV readers, synthetic regression fixtures,
|
|
merged-block handling and source-located formula errors.
|
|
- [x] Verify exported event times and a 90-minute gap through the Ontime 4.15.1
|
|
native rundown parser and metadata calculation at the recorded source commit.
|
|
- [x] Add initial Setup/Review/Results GUI, source preview, column/unit selectors,
|
|
shared profile controls and worker/cancellation tests.
|
|
- [x] Keep inference evidence and the source table on Review after failures;
|
|
add column-role dropdowns, row classifications and title/start/end/duration corrections.
|
|
- [x] Save/overwrite/save-as portable resolution profiles and explicitly remember,
|
|
reuse or forget validated resolutions matched to the complete sheet.
|
|
- [x] Redesign Review into Resolve rows / Schedule preview / Columns, with one
|
|
issue at a time, row navigation/filtering, a full-height correction table and
|
|
separate saved-resolution controls. Preserve pending edits and previous results
|
|
until recheck; verify compact layout and dropdown interaction.
|
|
- [x] Merge schedule preview and row corrections with All rows / Included rows /
|
|
Needs attention filters, preserving ignored rows for restoration and displaying
|
|
attached notes individually without duplicating their timers.
|
|
- [ ] Add recent-file persistence and comprehensive native GUI interaction checks.
|
|
- [ ] Inventory supplied examples by format/layout; record intended interpretations
|
|
and the user's Ontime version. Decide first-release input coverage from these.
|
|
- [ ] Build sanitized fixtures and an expected-result matrix before tuning rules.
|
|
Include difficult examples and some held-out layouts to detect overfitting.
|
|
- [ ] Confirm a pinned Ontime schema/release and create a minimal import experiment
|
|
with independent starts and a lunch gap; verify actual schedule and playback.
|
|
- [x] Scaffold the Rust member, core model, optional GUI and CLI; retain reporter
|
|
defaults. Include the new member in workspace CI, documentation and coverage.
|
|
- [ ] Implement the required source readers and extraction diagnostics. Test
|
|
unsupported/encrypted/corrupt inputs and formula/merged-cell behavior as relevant.
|
|
- [x] Stage 1: implement whole-column inference, explicit mappings and located
|
|
errors. Test ambiguous cells clarified by other rows, incompatible meanings in
|
|
one column, structural rows and misleading headers/units.
|
|
- [x] Keep one Duration column role while offering nested whole-cell, minimum,
|
|
maximum, sum, margin and split extraction policies. Let a Start mapping select
|
|
only the first clock in a mixed range; retain margin gaps and split entries in
|
|
the IR.
|
|
- [x] Stage 2: identify untimed gaps and populate draft IR constraints. Test named
|
|
and unnamed breaks, unknown boundaries and distinction from timed break cues.
|
|
- [x] Stage 3: construct the complete IR from times and gap constraints. Test
|
|
start-only schedules, missing final duration and provenance of inferred values.
|
|
- [x] Stage 4: independently validate backward times, negative durations,
|
|
contradictory triples, overlaps, gap boundaries and explicit markers.
|
|
- [ ] Add explicit midnight/day and multiple-track interpretations; current
|
|
behavior rejects these rather than guessing.
|
|
- [x] Implement inspect/validate/convert CLI and atomic exports. Test CLI status,
|
|
output collisions, cancellation, source preservation and deterministic results.
|
|
- [ ] Stage 5: implement the Ontime adapter and neutral JSON export. Check native imports
|
|
against the pinned parser and manually inspect real Ontime behavior; don't rely
|
|
solely on JSON snapshots. Verify that later anchors survive earlier edits and
|
|
gaps remain visible in the schedule; document playback implications.
|
|
- [ ] Implement desktop Setup/Review/Results, mapping corrections and diagnostics.
|
|
Verify GUI/CLI parity; manually inspect representative and ambiguous inputs,
|
|
resized windows, larger text, long paths, progress and cancellation.
|
|
- [ ] Run the supported corpus end to end: each source cue must be represented or
|
|
have an explicit reviewed exclusion; every unresolved ambiguity must block export.
|
|
- [ ] Complete usage examples, supported-format matrix, limitations and illustrated
|
|
GUI instructions. Add runnable quick-start commands only once they exist.
|
|
- [x] Run appropriate Python CI/packaging tests and all Rust gates listed in
|
|
[Development](development.md#checks), plus whole-workspace coverage. Preserve
|
|
production GUI coverage and reporter behavior. Record unavailable manual checks.
|
|
- [ ] Extend manual packaging/release preparation for both binaries and supported
|
|
platforms, including only dependencies this tool needs. CI must not publish releases.
|
|
- [x] Stretch: add XLSX re-export with verified Ontime-compatible columns and
|
|
documented format support.
|
|
- [ ] Stretch: add Typst source/PDF reports.
|
|
- [ ] Stretch: add explicit buffer adjustment, before/after review, feasibility
|
|
checks and revalidation, with the cases listed above.
|
|
- [x] Commit each completed implementation task with its validation summary.
|
|
|
|
## Documentation conventions
|
|
|
|
Keep README as a short repository entry point with quick-start examples and links.
|
|
Each tool has its own usage page; detailed design/task tracking stays separate.
|
|
Use one H1 per page, sentence-case headings, fenced code with a language, ordinary
|
|
relative Markdown links and consistent tables/lists. Label planned features and
|
|
commands clearly. The existing illustrated guide remains explicitly scoped to the
|
|
PPTX reporter; add a separate converter guide when a working GUI can be captured.
|