147 lines
9.6 KiB
Markdown
147 lines
9.6 KiB
Markdown
# Rundown evaluation corpus
|
|
|
|
The first implementation was evaluated on two supplied XLSX workbooks (three
|
|
sheets), two supplied PDFs, and five downloaded public XLSX workbooks. Source
|
|
originals remain in ignored `rundowns/` and `downloads/rundown-corpus/`; they are
|
|
not redistributed in Git. Automated regression tests use synthetic data.
|
|
|
|
The [public source manifest](../crates/rundown-converter/tests/public-corpus.json)
|
|
records download URLs, source pages, SHA-256 hashes and retrieval date. Downloads
|
|
are manual research inputs, never a network dependency of CI. Templates are not
|
|
a representative accuracy benchmark; these observations establish specific
|
|
behaviors and limitations, not a general success percentage.
|
|
|
|
## Supplied examples
|
|
|
|
| Example | Evidence and result |
|
|
| --- | --- |
|
|
| ESTIVALES, OUVERTURE | Correctly identifies Heure, Durée and Contenu / Détail. Shared merged durations remain one timer; merged windows with individual durations become consecutive cues with leftover slack. Untimed production directions need a note classification. Final 11:45 cue lacks duration and blocks export pending clarification. |
|
|
| ESTIVALES, CLOTURE | Five timing blocks. With the explicit `untimed_as_notes` policy, converts to five timers; all original continuation/direction rows remain in the IR and exported notes. This policy is a test scenario, not a confirmed interpretation from the owner. |
|
|
| Plénière 2, Feuil1 | Both C and D are named TIMER and contain plausible durations. Returns `AMBIGUOUS_COLUMN`; parent-segment versus child-cue timing needs clarification. No automatic conversion is claimed. |
|
|
| Montage PDF | Text is extractable, but includes overlapping logistics tasks, missing durations and different locations. Not a single established timer sequence. |
|
|
| Exploitation/démontage PDF | Text is extractable; page-spanning directions, simultaneous activities and untimed arrivals require track and row classification. PDF import remains a stretch goal. |
|
|
|
|
Private originals contain production details and contacts. Tests and documentation
|
|
do not reproduce those details. An unchanged local file can be inspected with
|
|
`rundown-converter inspect INPUT --json`; this output contains the original cells,
|
|
so treat it like the source document.
|
|
|
|
## PDF feasibility trial
|
|
|
|
A bounded follow-up trial attempted automatic table recovery from all 17 pages
|
|
of the two supplied PDFs. Result: useful text/table recovery, but no dependable
|
|
whole-document timer conversion. PDF importer work is paused; the application
|
|
continues to return `PDF_REQUIRES_REVIEW`. This is a finding about these samples,
|
|
not evidence that all PDF rundowns are unsuitable.
|
|
|
|
### Extraction attempts
|
|
|
|
First compare `pdftotext -layout` with word coordinates from
|
|
`pdftotext -bbox-layout`. Plain reading order sometimes joins a clock and its
|
|
activity, while multiline cells and vertically centred text prevent reliable
|
|
row detection from baselines alone. The rendered first page of each document
|
|
was inspected to compare the recovered coordinates with the visible grid.
|
|
|
|
Then use `pdftocairo -svg` to recover the table borders. In these files the borders
|
|
are thin filled rectangles. A temporary Python standard-library prototype
|
|
clusters their coordinates into column boundaries and row bands, assigns words
|
|
by their bounding-box centres, and records missing horizontal borders as possible
|
|
vertical cell spans. This is specific to ruled, text-based PDFs; it does not
|
|
establish support for scans, borderless tables or other PDF drawing conventions.
|
|
|
|
The first geometry pass lost a column on two Montage continuation pages where
|
|
the left border was absent. It also missed text above the first visible border
|
|
on some pages. A second pass aligns column boundaries across the document and
|
|
extends the outer row bounds to include body text. It recovers four columns for
|
|
Montage and five for Exploitation on every page.
|
|
|
|
| Measurement | Montage | Exploitation/démontage |
|
|
| --- | --- | --- |
|
|
| Pages | 4 | 13 |
|
|
| Nonempty geometric row bands | 33 | 123 |
|
|
| Body words assigned to cells | 714 | 2,645 |
|
|
| Repeated page-heading words excluded | 16 | 39 |
|
|
| Populated time cells | 13 | 46 |
|
|
| Explicit start/end ranges | 1 | 15 |
|
|
| Start-only cells | 9 | 31 |
|
|
| Other time-cell values | `TBC`, `01H/2H`, `2H MAX` | None |
|
|
|
|
All non-heading words were assigned in this second pass. That is extraction
|
|
coverage, **not** proof that every cell boundary, inherited start or event grouping
|
|
is correct. Row bands are geometric subdivisions, not a count of semantic events.
|
|
Missing grid borders and cross-page continuations still need interpretation.
|
|
|
|
### Timing experiment and stop decision
|
|
|
|
For a deliberately generous trial, write the recovered rows to temporary CSVs
|
|
with guessed Start/Title/Owner/Location/Notes headers and enable the existing
|
|
`untimed_as_notes` policy. No start or duration is filled from the next row.
|
|
Keep page/bounding-box and possible-merge evidence in separate temporary JSON;
|
|
this CSV trial does not resolve or automatically apply those ambiguous spans.
|
|
|
|
Montage stops at column validation on its three unresolved time-cell values.
|
|
Exploitation reaches IR construction but reports 31 missing durations. Its 12
|
|
orphan-note diagnostics are downstream consequences of unresolved preceding
|
|
events, not 12 additional extraction failures. A shared time cell can cover
|
|
several simultaneous tasks; treating them as consecutive cues is unjustified.
|
|
|
|
As a diagnostic only, test the explicit-range subset separately. Montage's single
|
|
range validates, but that excludes almost the entire document. Exploitation's
|
|
15 ranges yield 14 candidate timers and one untimed break, with six overlapping
|
|
predecessor pairs and three gap-overlap diagnostics. This subset also omits the
|
|
31 start-only entries, so it is not an acceptable converted rundown even if
|
|
those conflicts were removed. **No partial Ontime project was published.**
|
|
|
|
The remaining obstacles are semantic: selecting simultaneous tracks, deciding
|
|
whether untimed rows are instructions or events, distinguishing arrivals and
|
|
cutoff markers from timers, and interpreting uncertain overnight deadlines.
|
|
Neither a following start nor a missing table border supplies that information.
|
|
Per the requested stop condition, no experimental reader or new runtime dependency
|
|
was added. Revisit with a sequential PDF containing explicit durations/end times,
|
|
or after adding explicit track and row-role selection to the neutral model.
|
|
Temporary extractions and the prototype stay outside Git; no private PDF content
|
|
or contact details are redistributed.
|
|
|
|
## Public examples
|
|
|
|
| Source | Result and useful variation |
|
|
| --- | --- |
|
|
| [Stagetimer](https://stagetimer.io/blog/what-is-a-run-of-show/) | Header below title blocks, Excel serial times, formula placeholders. The populated row has inconsistent start/end/duration values; validation returns `CONFLICTING_TIMES`. |
|
|
| [Attenditai](https://attenditai.com/templates/run-of-show-template) | Instruction sheet plus blank formula-driven schedule. Formula cells without usable cached values require recalculation/population; no invented events. |
|
|
| [Joinways](https://joinways.app/solutions/run-of-show-example) | AM/PM times, section labels embedded in the clock column, parallel setup tasks. Unmerged labels require explicit row classifications before column inference can succeed; overlaps then require separate tracks. |
|
|
| [LiveFocus](https://livefocus.net/guides/event-run-of-show-template) | Three sheets, cached formula durations in minutes and fractional-day clocks. Minute units are inferred from start/end pairs. Blank formula rows and full-width merged footer are structural. The completed example reports overlapping activities. |
|
|
| [Novus](https://examples.novusstreamsolutions.com/templates/planning/event-run-of-show-csv) | String clock times and two-part durations. Whole-column comparison with start/end pairs establishes `hh:mm`; the populated XLSX sheet converts without a profile. |
|
|
|
|
Two additional institutional sources (Moira Shire and Monash University) rejected
|
|
requests with HTTP 403. A speaker toolkit page exposed no direct workbook link.
|
|
No authentication or access restrictions were bypassed.
|
|
|
|
## Ontime verification
|
|
|
|
Generated native JSON was run through Ontime's actual rundown parser and metadata
|
|
calculation from version 4.15.1, source commit
|
|
[`0fe52483eb756228589321b83c3b0f810dfb36b1`](https://github.com/cpvalente/ontime/commit/0fe52483eb756228589321b83c3b0f810dfb36b1).
|
|
The synthetic lunch example preserves all four events, their start/end/duration
|
|
values and unlinked starts; Ontime calculates a 5,400,000 ms gap before resumption.
|
|
The five-event closing-sheet scenario and eleven-event public Novus example
|
|
also preserve every timer through that parser. This verifies parser and schedule
|
|
semantics, not an operated live show or every
|
|
Ontime project section. The user's installed version still needs confirmation.
|
|
|
|
## Regression coverage
|
|
|
|
Synthetic tests cover whole-column contradictions, duplicate TIMER headers,
|
|
ambiguous duration units resolved by other rows, Excel time values and merges,
|
|
notes/markers, gaps, conflicting triples, backward times, negative durations,
|
|
output collisions, source preservation, cancellation, CLI exit codes and desktop
|
|
worker state. Neither private originals nor public binaries are CI fixtures.
|
|
|
|
## Validation of the first implementation
|
|
|
|
All workspace Rust tests and 43 Python tests passed, along with formatting,
|
|
Clippy and warning-free Rustdoc. Whole-workspace line coverage is 86.20% against
|
|
the 75% gate, including production GUI/PDF code. The native converter Setup window
|
|
was visually inspected; headless egui tests exercise Review/Results, compact
|
|
layouts, worker completion, errors and export. Full interactive operation on
|
|
Windows/macOS and a live Ontime playback session have not been verified.
|