129 lines
5.4 KiB
Markdown
129 lines
5.4 KiB
Markdown
# Chunked Rendering v2 — Design Spec
|
|
|
|
Status: **planned** (v1 shipped; v1 boundary limitation is detected + warned, not yet fixed)
|
|
|
|
## Why chunking exists
|
|
|
|
The render is a single `ffmpeg -filter_complex` pass that opens every `-i` input up
|
|
front. On memory-constrained machines (an 8 GB VM, or a WSL2 rig whose VM RAM is a
|
|
slice of the host) the aggregate decoder + filter buffers OOM-kill the process — or,
|
|
on WSL2, the whole VM. `render_chunk_slides` (in `~/.gnommo.conf` `[performance]`, or
|
|
`--chunk-slides N`) splits the timeline into groups of N slides, renders each as a
|
|
partial render, and concatenates the chunks with `-c copy`.
|
|
|
|
Each chunk is a partial render built via `build_render_plan(slide_range=(start,end))`,
|
|
which passes `time_range=(time_offset, render_end_time)` to the event extractors. See
|
|
`partial-rendering-spec.md` for the partial-render mechanics chunking reuses.
|
|
|
|
## The v1 defect this fixes
|
|
|
|
`_extract_video_events` / `_extract_audio_events` keep an event **only if its start
|
|
falls inside the window**:
|
|
|
|
```python
|
|
if start_time < range_start or start_time >= range_end:
|
|
continue
|
|
```
|
|
|
|
So a clip that **began in an earlier chunk and is still playing across the boundary**
|
|
is dropped from every later chunk it overlaps. The full (non-chunked) render is
|
|
correct; the chunked render silently diverges.
|
|
|
|
Who is affected:
|
|
|
|
- **Overlay videos that span slides:** `end_on: next_video`, `loop`, long `take`/`end`
|
|
(multi-slide backgrounds / persistent picture-in-picture).
|
|
- **Looping background audio/music:** started once early, meant to underlie the whole
|
|
video — dropped from every chunk after the first. Highest blast radius.
|
|
- **Not affected:** `end_on: next_slide` clips (they end exactly at a slide marker =
|
|
a chunk seam), per-slide content, and the full-screen `plan.background` (a separate
|
|
always-included input).
|
|
|
|
v1 status: `cli._chunk_boundary_span_warnings(plan, groups)` detects spanning
|
|
video/outro/audio events and prints a loud warning before rendering, so the divergence
|
|
is never silent. It does not yet correct the output.
|
|
|
|
## v2 algorithm
|
|
|
|
### 1. Overlap inclusion (not start-inside)
|
|
|
|
Replace the start-inside test with an overlap test in both extractors:
|
|
|
|
```python
|
|
# keep the event if it overlaps [range_start, range_end)
|
|
if end_time <= range_start or start_time >= range_end:
|
|
continue
|
|
```
|
|
|
|
### 2. Per-event seek for clips that began earlier
|
|
|
|
A clip included by overlap whose `start_time < range_start` is already mid-playback at
|
|
the chunk boundary. It must resume at the correct frame, not restart. Compute how far
|
|
into the clip the window begins and carry it as a **per-event seek**:
|
|
|
|
```
|
|
into = range_start - start_time # seconds of the clip already elapsed
|
|
base = video_source.skip or 0.0
|
|
# non-looping clip:
|
|
skip_override = base + into
|
|
# looping clip (end_on: loop / next_video that wraps):
|
|
period = (video_source.duration or 0) - base # one loop's playable length
|
|
skip_override = base + (into % period) if period > 0 else base
|
|
```
|
|
|
|
Then clamp the event to the window and let the offset pass zero it:
|
|
|
|
```
|
|
start_time = max(start_time, range_start) # -> 0 after time_offset subtraction
|
|
end_time = min(end_time, range_end)
|
|
```
|
|
|
|
The seam for the *seek* is `VideoEvent.skip_override` (already added, inert until v2):
|
|
the renderer prefers it over `video_source.skip` when set. `-ss {skip}` is applied as
|
|
an input option, so ffmpeg decodes to that point — frame-accurate for the codecs in
|
|
use.
|
|
|
|
### 3. Audio equivalent
|
|
|
|
`AudioEvent` has no `skip_override` / `end_time` yet. v2 adds both (or derives end from
|
|
`audio_def.duration` / `loop` → `total_duration`) and applies the same overlap + seek.
|
|
For looping music the seek is the loop-phase modulo above; `ignore_pauses` and
|
|
`overlap` (crossfade) interactions must be re-checked at the seam.
|
|
|
|
### 4. Outro events
|
|
|
|
`OutroEvent` also spans (start_time/end_time). Extend the same treatment; outros
|
|
normally live in the final chunk so this is lower priority but should be covered for
|
|
completeness.
|
|
|
|
## Concat-seam correctness (the risk to validate)
|
|
|
|
Chunks are joined with `-c copy`, so the two sides of a seam must be frame-aligned:
|
|
chunk *k* ends showing the clip at position `P`, chunk *k+1* must resume at exactly `P`
|
|
via `skip_override`. Validation plan:
|
|
|
|
1. Pick a project with a known multi-slide overlay video **and** looping music
|
|
(or synthesize one).
|
|
2. Render it full (reference) and chunked (small `chunk_slides`, so a seam falls mid
|
|
clip).
|
|
3. Compare: identical duration; frame diff at ±3 frames around each seam below a
|
|
threshold; audio cross-correlation shows no gap/jump; the overlay is present in
|
|
every chunk it overlaps (the v1 bug is gone).
|
|
4. Assert `_chunk_boundary_span_warnings` returns empty for the fixed path.
|
|
|
|
## Work items
|
|
|
|
- [ ] `_extract_video_events`: overlap test + `skip_override` (loop-aware).
|
|
- [ ] `_extract_audio_events`: add `end_time`/`skip_override`, overlap + seek.
|
|
- [ ] `AudioEvent`: `skip_override` field; renderer audio path honors it.
|
|
- [ ] `OutroEvent`: same treatment.
|
|
- [ ] Frame/audio seam-diff test (chunked vs full) in the test suite.
|
|
- [ ] Flip `_chunk_boundary_span_warnings` from "will be dropped" to a debug-only
|
|
assertion once v2 is the default.
|
|
|
|
## Seams already in place (v2 preparations, shipped)
|
|
|
|
- `VideoEvent.skip_override` (models.py) — inert; renderer honors it when set.
|
|
- `cli._chunk_boundary_span_warnings` — detection + warning.
|
|
- v1-limitation comments at both filter sites in `transformer.py`.
|