167 lines
13 KiB
Markdown
167 lines
13 KiB
Markdown
# Rundown evaluation corpus
|
||
|
||
The supplied corpus now includes five XLSX workbooks (seven sheets) and two PDFs.
|
||
The five workbooks are tracked at the user's request, with normalized filenames
|
||
and unchanged contents; see the [fixture inventory](../rundowns/README.md) and
|
||
[original-name/checksum manifest](../rundowns/sources.json). Four offline integration
|
||
tests exercise all seven sheets through the real reader and pipeline. PDFs and
|
||
generated exports remain ignored. Seven public XLSX downloads and three public
|
||
CSV files remain local evaluation inputs under `downloads/rundown-corpus/`.
|
||
|
||
The [public source manifest](../crates/rundown-converter/tests/public-corpus.json)
|
||
records download URLs, source pages, SHA-256 hashes and retrieval date. Downloads
|
||
are manual research inputs, never a network dependency of CI. Templates are not
|
||
a representative accuracy benchmark; these observations establish specific
|
||
behaviors and limitations, not a general success percentage.
|
||
|
||
## Supplied examples
|
||
|
||
| Example | Evidence and result |
|
||
| --- | --- |
|
||
| ESTIVALES, OUVERTURE | Correctly identifies Heure, Durée and Contenu / Détail. Shared merged durations remain one timer; merged windows with individual durations become consecutive cues with leftover slack. Untimed production directions need a note classification. Final 11:45 cue lacks duration and blocks export pending clarification. |
|
||
| ESTIVALES, CLOTURE | Five timing blocks. With the explicit `untimed_as_notes` policy, converts to five timers; all original continuation/direction rows remain in the IR; exported notes depend on the selected Notes columns. This policy is a test scenario, not a confirmed interpretation from the owner. |
|
||
| Plénière 2, Feuil1 | Both C and D are named TIMER and contain plausible durations. Returns `AMBIGUOUS_COLUMN`; parent-segment versus child-cue timing needs clarification. No automatic conversion is claimed. |
|
||
| GIFAV 2026 Auditorium | `Horaires` contains consistent parent-session start/end ranges and is detected as a combined Start–end column. `Durée` mixes child ranges with speaking/QA text, so it cannot be flattened into one duration role. Mapping Start–end to A and Title to D, retaining presentations as notes, validates Thursday with 11 entries and three gaps. Friday reaches schedule validation with 10 entries and three gaps, then correctly reports the source overlap between the 13:45–15:45 poster session and the 14:45–15:45 following session. |
|
||
| Montage PDF | Text is extractable, but includes overlapping logistics tasks, missing durations and different locations. Not a single established timer sequence. |
|
||
| Exploitation/démontage PDF | Text is extractable; page-spanning directions, simultaneous activities and untimed arrivals require track and row classification. PDF import remains a stretch goal. |
|
||
|
||
The supplied originals contain production details and contacts. The five tracked
|
||
workbooks were explicitly approved for Git inclusion by the user; this does not
|
||
assert a blanket redistribution license for third-party material. Test assertions
|
||
and documentation avoid reproducing their production text. An unchanged local file can be inspected with
|
||
`rundown-converter inspect INPUT --json`; this output contains the original cells,
|
||
so treat it like the source document.
|
||
|
||
## PDF feasibility trial
|
||
|
||
A bounded follow-up trial attempted automatic table recovery from all 17 pages
|
||
of the two supplied PDFs. Result: useful text/table recovery, but no dependable
|
||
whole-document timer conversion. PDF importer work is paused; the application
|
||
continues to return `PDF_REQUIRES_REVIEW`. This is a finding about these samples,
|
||
not evidence that all PDF rundowns are unsuitable.
|
||
|
||
### Extraction attempts
|
||
|
||
First compare `pdftotext -layout` with word coordinates from
|
||
`pdftotext -bbox-layout`. Plain reading order sometimes joins a clock and its
|
||
activity, while multiline cells and vertically centred text prevent reliable
|
||
row detection from baselines alone. The rendered first page of each document
|
||
was inspected to compare the recovered coordinates with the visible grid.
|
||
|
||
Then use `pdftocairo -svg` to recover the table borders. In these files the borders
|
||
are thin filled rectangles. A temporary Python standard-library prototype
|
||
clusters their coordinates into column boundaries and row bands, assigns words
|
||
by their bounding-box centres, and records missing horizontal borders as possible
|
||
vertical cell spans. This is specific to ruled, text-based PDFs; it does not
|
||
establish support for scans, borderless tables or other PDF drawing conventions.
|
||
|
||
The first geometry pass lost a column on two Montage continuation pages where
|
||
the left border was absent. It also missed text above the first visible border
|
||
on some pages. A second pass aligns column boundaries across the document and
|
||
extends the outer row bounds to include body text. It recovers four columns for
|
||
Montage and five for Exploitation on every page.
|
||
|
||
| Measurement | Montage | Exploitation/démontage |
|
||
| --- | --- | --- |
|
||
| Pages | 4 | 13 |
|
||
| Nonempty geometric row bands | 33 | 123 |
|
||
| Body words assigned to cells | 714 | 2,645 |
|
||
| Repeated page-heading words excluded | 16 | 39 |
|
||
| Populated time cells | 13 | 46 |
|
||
| Explicit start/end ranges | 1 | 15 |
|
||
| Start-only cells | 9 | 31 |
|
||
| Other time-cell values | `TBC`, `01H/2H`, `2H MAX` | None |
|
||
|
||
All non-heading words were assigned in this second pass. That is extraction
|
||
coverage, **not** proof that every cell boundary, inherited start or event grouping
|
||
is correct. Row bands are geometric subdivisions, not a count of semantic events.
|
||
Missing grid borders and cross-page continuations still need interpretation.
|
||
|
||
### Timing experiment and stop decision
|
||
|
||
For a deliberately generous trial, write the recovered rows to temporary CSVs
|
||
with guessed Start/Title/Owner/Location/Notes headers and enable the existing
|
||
`untimed_as_notes` policy. No start or duration is filled from the next row.
|
||
Keep page/bounding-box and possible-merge evidence in separate temporary JSON;
|
||
this CSV trial does not resolve or automatically apply those ambiguous spans.
|
||
|
||
Montage stops at column validation on its three unresolved time-cell values.
|
||
Exploitation reaches IR construction but reports 31 missing durations. Its 12
|
||
orphan-note diagnostics are downstream consequences of unresolved preceding
|
||
events, not 12 additional extraction failures. A shared time cell can cover
|
||
several simultaneous tasks; treating them as consecutive cues is unjustified.
|
||
|
||
As a diagnostic only, test the explicit-range subset separately. Montage's single
|
||
range validates, but that excludes almost the entire document. Exploitation's
|
||
15 ranges yield 14 candidate timers and one untimed break, with six overlapping
|
||
predecessor pairs and three gap-overlap diagnostics. This subset also omits the
|
||
31 start-only entries, so it is not an acceptable converted rundown even if
|
||
those conflicts were removed. **No partial Ontime project was published.**
|
||
|
||
The remaining obstacles are semantic: selecting simultaneous tracks, deciding
|
||
whether rows without a timing are instructions or events, distinguishing arrivals and
|
||
cutoff markers from timers, and interpreting uncertain overnight deadlines.
|
||
Neither a following start nor a missing table border supplies that information.
|
||
Per the requested stop condition, no experimental reader or new runtime dependency
|
||
was added. Revisit with a sequential PDF containing explicit durations/end times,
|
||
or after adding explicit track and row-role selection to the neutral model.
|
||
Temporary extractions and the prototype stay outside Git; no private PDF content
|
||
or contact details are redistributed.
|
||
|
||
## Public examples
|
||
|
||
| Source | Result and useful variation |
|
||
| --- | --- |
|
||
| [Stagetimer](https://stagetimer.io/blog/what-is-a-run-of-show/) | Header below title blocks, Excel serial times, formula placeholders. The populated row has inconsistent start/end/duration values; validation returns `CONFLICTING_TIMES`. |
|
||
| [Attenditai](https://attenditai.com/templates/run-of-show-template) | Instruction sheet plus blank formula-driven schedule. Formula cells without usable cached values require recalculation/population; no invented events. |
|
||
| [Joinways](https://joinways.app/solutions/run-of-show-example) | AM/PM times, section labels embedded in the clock column, parallel setup tasks. Unmerged labels require explicit row classifications before column inference can succeed; overlaps then require separate tracks. |
|
||
| [LiveFocus](https://livefocus.net/guides/event-run-of-show-template) | Three sheets, cached formula durations in minutes and fractional-day clocks. Minute units are inferred from start/end pairs. Blank formula rows and full-width merged footer are structural. The completed example reports overlapping activities. |
|
||
| [Novus](https://examples.novusstreamsolutions.com/templates/planning/event-run-of-show-csv) | String clock times and two-part durations. Whole-column comparison with start/end pairs establishes `hh:mm`; the populated XLSX sheet converts without a profile. |
|
||
| [Events Authority](https://eventsauthority.com/corporate-event-planning/run-show-template/) | Two sheets, a title block, Excel clock values and chained formulas. The main schedule infers start/end/title and validates nine events plus one gap after its total row is explicitly ignored. The preparation sheet is correctly kept separate. |
|
||
| [Arena Takeover](https://arenatakeover.com/resources.html) | A sports cue sheet mixes countdowns (`T-30:00`), elapsed time, quarter clock, halftime and final-game tokens in one clock column. It fails whole-column validation instead of pretending these are one wall-clock schedule; durations also require an explicit unit. |
|
||
| [LASSO broadcast](https://www.lasso.io/rundown-templates/) | A compact broadcast CSV uses `Item` as its title header and `mm:ss` durations. Manual mapping reaches IR construction, where a cue without a duration remains an actionable blocker. |
|
||
| [LASSO corporate](https://www.lasso.io/rundown-templates/) | A conference CSV interleaves timed cues with session headings and end markers. Manual mapping exposes those rows for classification and leaves the final missing duration unresolved rather than silently dropping it. |
|
||
| [The AV Company](https://theavcompany.com/blog/run-of-show) | Instructions, blank template and populated example sheets, with a two-level visual header, Excel fractional-day durations, continuation notes and a final `--` marker. Explicit header/column choices and three row classifications produce a valid 20-event schedule. The workbook also contains formatting far beyond the visible data columns, which remains harmless but verbose in inference evidence. |
|
||
|
||
Two additional institutional sources (Moira Shire and Monash University) rejected
|
||
requests with HTTP 403. A speaker toolkit page exposed no direct workbook link.
|
||
No authentication or access restrictions were bypassed.
|
||
|
||
The second search deliberately retained cases that fail or need review. They add
|
||
relative sports clocks, broadcast `mm:ss`, structural session rows, two-level
|
||
headers, preparation sheets and wide production metadata; a corpus of only
|
||
successful single-track schedules would not exercise the converter's stop rules.
|
||
|
||
## Ontime verification
|
||
|
||
Generated native JSON was run through Ontime's actual rundown parser and metadata
|
||
calculation from version 4.15.1, source commit
|
||
[`0fe52483eb756228589321b83c3b0f810dfb36b1`](https://github.com/cpvalente/ontime/commit/0fe52483eb756228589321b83c3b0f810dfb36b1).
|
||
The synthetic lunch example preserves all four events, their start/end/duration
|
||
values and unlinked starts; Ontime calculates a 5,400,000 ms gap before resumption.
|
||
The five-event closing-sheet scenario and eleven-event public Novus example
|
||
also preserve every timer through that parser. This verifies parser and schedule
|
||
semantics, not an operated live show or every
|
||
Ontime project section. The user's installed version still needs confirmation.
|
||
|
||
## Regression coverage
|
||
|
||
Synthetic tests cover whole-column contradictions, duplicate TIMER headers,
|
||
ambiguous duration units resolved by other rows, Excel time values and merges,
|
||
notes/markers, gaps, conflicting triples, backward times, negative durations,
|
||
output collisions, source preservation, cancellation, CLI exit codes and desktop
|
||
worker state. `tests/supplied_corpus.rs` additionally tests French header detection
|
||
and typed Excel durations in both Moebius files, duplicate timers in Plénière 2,
|
||
merged blocks and source-row retention in ESTIVALES, and parent ranges, gaps and
|
||
the real Friday overlap in GIFAV. These tests are offline and never write exports
|
||
beside the fixtures. Public downloads remain outside CI.
|
||
|
||
## Validation of the first implementation
|
||
|
||
All workspace Rust tests and 43 Python tests passed, along with formatting,
|
||
Clippy and warning-free Rustdoc. Whole-workspace line coverage is 86.20% against
|
||
the 75% gate, including production GUI/PDF code. The native converter Setup window
|
||
was visually inspected; headless egui tests exercise Review/Results, compact
|
||
layouts, worker completion, errors and export. Full interactive operation on
|
||
Windows/macOS and a live Ontime playback session have not been verified.
|