diff --git a/docs/rundown-converter-plan.md b/docs/rundown-converter-plan.md index 6e42dd3..74a4ff8 100644 --- a/docs/rundown-converter-plan.md +++ b/docs/rundown-converter-plan.md @@ -237,6 +237,11 @@ errors must remain visible with direct corrective actions. - [x] Add separate tool documentation and link it from the repository overview. - [x] Inspect both supplied XLSX workbooks and both PDFs; download five public workbook layouts with a provenance manifest. Record findings separately. +- [x] Run a bounded PDF guessing experiment over all 17 supplied pages using + text coordinates and vector table borders. Record recovery and validator + results in the [PDF feasibility trial](rundown-corpus.md#pdf-feasibility-trial). + Pause importer work: column recovery succeeds, but missing durations and + simultaneous tracks prevent dependable whole-document timer conversion. - [x] Add bounded XLSX and UTF-8 CSV/TSV readers, synthetic regression fixtures, merged-block handling and source-located formula errors. - [x] Verify exported event times and a 90-minute gap through the Ontime 4.15.1 diff --git a/docs/rundown-converter.md b/docs/rundown-converter.md index dc5387b..cfd7092 100644 --- a/docs/rundown-converter.md +++ b/docs/rundown-converter.md @@ -77,7 +77,10 @@ Processing is local. XLSX preserves merged ranges and cached formula values; formulas and macros are never executed. Recalculate and save formula-based workbooks before importing. The converter cannot determine whether cached values are stale. CSV/TSV uses UTF-8, quoted fields and comma/semicolon/tab detection. -Legacy XLS, ODS, Word, PDF and scans are not yet supported. Input files are limited +Legacy XLS, ODS, Word, PDF and scans are not yet supported. A +[PDF feasibility trial](rundown-corpus.md#pdf-feasibility-trial) recovered table +text from the supplied PDFs but could not establish a complete timer schedule; +PDF importer work is paused for now. Input files are limited to 32 MiB; XLSX expanded content is limited to 128 MiB. ## Timing and breaks diff --git a/docs/rundown-corpus.md b/docs/rundown-corpus.md index 47d17b9..7fdd9e0 100644 --- a/docs/rundown-corpus.md +++ b/docs/rundown-corpus.md @@ -26,6 +26,82 @@ do not reproduce those details. An unchanged local file can be inspected with `rundown-converter inspect INPUT --json`; this output contains the original cells, so treat it like the source document. +## PDF feasibility trial + +A bounded follow-up trial attempted automatic table recovery from all 17 pages +of the two supplied PDFs. Result: useful text/table recovery, but no dependable +whole-document timer conversion. PDF importer work is paused; the application +continues to return `PDF_REQUIRES_REVIEW`. This is a finding about these samples, +not evidence that all PDF rundowns are unsuitable. + +### Extraction attempts + +First compare `pdftotext -layout` with word coordinates from +`pdftotext -bbox-layout`. Plain reading order sometimes joins a clock and its +activity, while multiline cells and vertically centred text prevent reliable +row detection from baselines alone. The rendered first page of each document +was inspected to compare the recovered coordinates with the visible grid. + +Then use `pdftocairo -svg` to recover the table borders. In these files the borders +are thin filled rectangles. A temporary Python standard-library prototype +clusters their coordinates into column boundaries and row bands, assigns words +by their bounding-box centres, and records missing horizontal borders as possible +vertical cell spans. This is specific to ruled, text-based PDFs; it does not +establish support for scans, borderless tables or other PDF drawing conventions. + +The first geometry pass lost a column on two Montage continuation pages where +the left border was absent. It also missed text above the first visible border +on some pages. A second pass aligns column boundaries across the document and +extends the outer row bounds to include body text. It recovers four columns for +Montage and five for Exploitation on every page. + +| Measurement | Montage | Exploitation/démontage | +| --- | --- | --- | +| Pages | 4 | 13 | +| Nonempty geometric row bands | 33 | 123 | +| Body words assigned to cells | 714 | 2,645 | +| Repeated page-heading words excluded | 16 | 39 | +| Populated time cells | 13 | 46 | +| Explicit start/end ranges | 1 | 15 | +| Start-only cells | 9 | 31 | +| Other time-cell values | `TBC`, `01H/2H`, `2H MAX` | None | + +All non-heading words were assigned in this second pass. That is extraction +coverage, **not** proof that every cell boundary, inherited start or event grouping +is correct. Row bands are geometric subdivisions, not a count of semantic events. +Missing grid borders and cross-page continuations still need interpretation. + +### Timing experiment and stop decision + +For a deliberately generous trial, write the recovered rows to temporary CSVs +with guessed Start/Title/Owner/Location/Notes headers and enable the existing +`untimed_as_notes` policy. No start or duration is filled from the next row. +Keep page/bounding-box and possible-merge evidence in separate temporary JSON; +this CSV trial does not resolve or automatically apply those ambiguous spans. + +Montage stops at column validation on its three unresolved time-cell values. +Exploitation reaches IR construction but reports 31 missing durations. Its 12 +orphan-note diagnostics are downstream consequences of unresolved preceding +events, not 12 additional extraction failures. A shared time cell can cover +several simultaneous tasks; treating them as consecutive cues is unjustified. + +As a diagnostic only, test the explicit-range subset separately. Montage's single +range validates, but that excludes almost the entire document. Exploitation's +15 ranges yield 14 candidate timers and one untimed break, with six overlapping +predecessor pairs and three gap-overlap diagnostics. This subset also omits the +31 start-only entries, so it is not an acceptable converted rundown even if +those conflicts were removed. **No partial Ontime project was published.** + +The remaining obstacles are semantic: selecting simultaneous tracks, deciding +whether untimed rows are instructions or events, distinguishing arrivals and +cutoff markers from timers, and interpreting uncertain overnight deadlines. +Neither a following start nor a missing table border supplies that information. +Per the requested stop condition, no experimental reader or new runtime dependency +was added. Revisit with a sequential PDF containing explicit durations/end times, +or after adding explicit track and row-role selection to the neutral model. +Temporary extractions and the prototype stay outside Git; no private PDF content +or contact details are redistributed. + ## Public examples | Source | Result and useful variation |