diff --git a/docs/chunking_v2.md b/docs/chunking_v2.md new file mode 100644 index 0000000..235240d --- /dev/null +++ b/docs/chunking_v2.md @@ -0,0 +1,128 @@ +# Chunked Rendering v2 — Design Spec + +Status: **planned** (v1 shipped; v1 boundary limitation is detected + warned, not yet fixed) + +## Why chunking exists + +The render is a single `ffmpeg -filter_complex` pass that opens every `-i` input up +front. On memory-constrained machines (an 8 GB VM, or a WSL2 rig whose VM RAM is a +slice of the host) the aggregate decoder + filter buffers OOM-kill the process — or, +on WSL2, the whole VM. `render_chunk_slides` (in `~/.gnommo.conf` `[performance]`, or +`--chunk-slides N`) splits the timeline into groups of N slides, renders each as a +partial render, and concatenates the chunks with `-c copy`. + +Each chunk is a partial render built via `build_render_plan(slide_range=(start,end))`, +which passes `time_range=(time_offset, render_end_time)` to the event extractors. See +`partial-rendering-spec.md` for the partial-render mechanics chunking reuses. + +## The v1 defect this fixes + +`_extract_video_events` / `_extract_audio_events` keep an event **only if its start +falls inside the window**: + +```python +if start_time < range_start or start_time >= range_end: + continue +``` + +So a clip that **began in an earlier chunk and is still playing across the boundary** +is dropped from every later chunk it overlaps. The full (non-chunked) render is +correct; the chunked render silently diverges. + +Who is affected: + +- **Overlay videos that span slides:** `end_on: next_video`, `loop`, long `take`/`end` + (multi-slide backgrounds / persistent picture-in-picture). +- **Looping background audio/music:** started once early, meant to underlie the whole + video — dropped from every chunk after the first. Highest blast radius. +- **Not affected:** `end_on: next_slide` clips (they end exactly at a slide marker = + a chunk seam), per-slide content, and the full-screen `plan.background` (a separate + always-included input). + +v1 status: `cli._chunk_boundary_span_warnings(plan, groups)` detects spanning +video/outro/audio events and prints a loud warning before rendering, so the divergence +is never silent. It does not yet correct the output. + +## v2 algorithm + +### 1. Overlap inclusion (not start-inside) + +Replace the start-inside test with an overlap test in both extractors: + +```python +# keep the event if it overlaps [range_start, range_end) +if end_time <= range_start or start_time >= range_end: + continue +``` + +### 2. Per-event seek for clips that began earlier + +A clip included by overlap whose `start_time < range_start` is already mid-playback at +the chunk boundary. It must resume at the correct frame, not restart. Compute how far +into the clip the window begins and carry it as a **per-event seek**: + +``` +into = range_start - start_time # seconds of the clip already elapsed +base = video_source.skip or 0.0 +# non-looping clip: +skip_override = base + into +# looping clip (end_on: loop / next_video that wraps): +period = (video_source.duration or 0) - base # one loop's playable length +skip_override = base + (into % period) if period > 0 else base +``` + +Then clamp the event to the window and let the offset pass zero it: + +``` +start_time = max(start_time, range_start) # -> 0 after time_offset subtraction +end_time = min(end_time, range_end) +``` + +The seam for the *seek* is `VideoEvent.skip_override` (already added, inert until v2): +the renderer prefers it over `video_source.skip` when set. `-ss {skip}` is applied as +an input option, so ffmpeg decodes to that point — frame-accurate for the codecs in +use. + +### 3. Audio equivalent + +`AudioEvent` has no `skip_override` / `end_time` yet. v2 adds both (or derives end from +`audio_def.duration` / `loop` → `total_duration`) and applies the same overlap + seek. +For looping music the seek is the loop-phase modulo above; `ignore_pauses` and +`overlap` (crossfade) interactions must be re-checked at the seam. + +### 4. Outro events + +`OutroEvent` also spans (start_time/end_time). Extend the same treatment; outros +normally live in the final chunk so this is lower priority but should be covered for +completeness. + +## Concat-seam correctness (the risk to validate) + +Chunks are joined with `-c copy`, so the two sides of a seam must be frame-aligned: +chunk *k* ends showing the clip at position `P`, chunk *k+1* must resume at exactly `P` +via `skip_override`. Validation plan: + +1. Pick a project with a known multi-slide overlay video **and** looping music + (or synthesize one). +2. Render it full (reference) and chunked (small `chunk_slides`, so a seam falls mid + clip). +3. Compare: identical duration; frame diff at ±3 frames around each seam below a + threshold; audio cross-correlation shows no gap/jump; the overlay is present in + every chunk it overlaps (the v1 bug is gone). +4. Assert `_chunk_boundary_span_warnings` returns empty for the fixed path. + +## Work items + +- [ ] `_extract_video_events`: overlap test + `skip_override` (loop-aware). +- [ ] `_extract_audio_events`: add `end_time`/`skip_override`, overlap + seek. +- [ ] `AudioEvent`: `skip_override` field; renderer audio path honors it. +- [ ] `OutroEvent`: same treatment. +- [ ] Frame/audio seam-diff test (chunked vs full) in the test suite. +- [ ] Flip `_chunk_boundary_span_warnings` from "will be dropped" to a debug-only + assertion once v2 is the default. + +## Seams already in place (v2 preparations, shipped) + +- `VideoEvent.skip_override` (models.py) — inert; renderer honors it when set. +- `cli._chunk_boundary_span_warnings` — detection + warning. +- v1-limitation comments at both filter sites in `transformer.py`. diff --git a/gnommo/cli.py b/gnommo/cli.py index c3c8ed8..19a874e 100644 --- a/gnommo/cli.py +++ b/gnommo/cli.py @@ -4136,6 +4136,58 @@ def _writeback_video_metadata(plan, project_path, config) -> None: print(f" Updated videos.json: {written}") +def _chunk_boundary_span_warnings(plan, groups) -> list[str]: + """Detect events that span a chunk boundary. + + KNOWN LIMITATION (chunking v1): the render's time-range filter keeps only events + whose *start* falls inside a chunk's window (transformer.py `_extract_video_events` + / `_extract_audio_events`: `if start_time < range_start: continue`). An overlay + video or audio clip that began in an earlier chunk and is still playing across the + boundary is therefore DROPPED from every later chunk it overlaps — so the chunked + output silently diverges from a full render. This detector makes that non-silent. + The real fix (overlap-inclusion + per-event seek) is chunking v2 — see + docs/chunking_v2.md. + """ + slide_start = {e.slide_id: e.start_time for e in plan.slide_events} + boundaries = [] # (slide_id, output_time) at each chunk seam after the first + for g in groups[1:]: + t = slide_start.get(g[0]) + if t is not None: + boundaries.append((g[0], t)) + if not boundaries: + return [] + + warnings: list[str] = [] + + def _check(start, end, label, kind): + crossed = [(sid, bt) for sid, bt in boundaries if start < bt < end] + if crossed: + sid, bt = crossed[0] + extra = f" (and {len(crossed) - 1} more)" if len(crossed) > 1 else "" + warnings.append( + f"{kind} '{label}' plays {_format_time(start)}–{_format_time(end)} and " + f"crosses the chunk boundary at {sid} ({_format_time(bt)}){extra}; it will " + f"be DROPPED from the chunk(s) after the boundary." + ) + + for e in plan.video_events: + _check(e.start_time, e.end_time, + getattr(e.video_source, "source_file", e.video_id), "video") + for e in plan.outro_events: + _check(e.start_time, e.end_time, + getattr(e.video_source, "source_file", e.video_id), "outro video") + for e in plan.audio_events: + ad = e.audio_def + if getattr(ad, "loop", False): + end = plan.total_duration + elif getattr(ad, "duration", None) is not None: + end = e.start_time + ad.duration + else: + continue # unknown length — can't judge span + _check(e.start_time, end, ad.file, "audio") + return warnings + + def _chunked_render( project_path: Path, verbose: bool, @@ -4146,6 +4198,7 @@ def _chunked_render( slide_ids: list[str], out_dir: Path, final_output: Path, + plan=None, ) -> int: """Render in slide-based chunks then concatenate — avoids filter graph OOM.""" import math @@ -4158,6 +4211,30 @@ def _chunked_render( f"\n Auto-chunking: {len(slide_ids)} slides → {len(groups)} chunks of ≤{chunk_size}" ) + # Warn about events that span a chunk boundary — chunking v1 drops these from + # the later chunk, so the output would silently differ from a full render. + if plan is not None: + _span_warnings = _chunk_boundary_span_warnings(plan, groups) + if _span_warnings: + banner = " " + "!" * 64 + print(f"\n{banner}", file=sys.stderr) + print(" CHUNK-BOUNDARY WARNING — chunked output will differ from a full render:", + file=sys.stderr) + for w in _span_warnings: + print(f" - {w}", file=sys.stderr) + print(" These overlay video/audio clips span a chunk seam. Chunking v1 keeps", + file=sys.stderr) + print(" only clips that START inside a chunk, so the ones above vanish after", + file=sys.stderr) + print(" the boundary. Options: render this project WITHOUT chunking (lower", + file=sys.stderr) + print(" cpu_limit instead), or move the chunk size so no seam splits them.", + file=sys.stderr) + print(f"{banner}\n", file=sys.stderr) + _render_log("CHUNK-BOUNDARY WARNING (v1 drops cross-boundary clips):") + for w in _span_warnings: + _render_log(f" - {w}") + chunks_dir = out_dir / "chunks" chunks_dir.mkdir(parents=True, exist_ok=True) @@ -5112,6 +5189,7 @@ def _cmd_render_impl( _slide_ids, out_dir, output_path, + plan=plan, ) plan.output_path = output_path @@ -5197,7 +5275,7 @@ def _cmd_render_impl( _render_log("FFmpeg command:\n" + generate_ffmpeg_command_string(plan, output_path)) except Exception as _e: _render_log(f"(could not serialize ffmpeg command: {_e})") - render(plan, output_path, verbose=verbose) + render(plan, output_path, verbose=verbose, log=_render_log) print(f" Output: {output_path}") if _render_gateable: diff --git a/gnommo/models.py b/gnommo/models.py index ca50416..ef389e2 100644 --- a/gnommo/models.py +++ b/gnommo/models.py @@ -429,6 +429,11 @@ class VideoEvent: cutout: "CutoutDefinition" cutout_name: str = "" # resolved cutout name (e.g. "fullscreen"), for display layer: str = "above" # "above" = on top of slides; "below" = behind slides + # Chunking v2 seam (see docs/chunking_v2.md): when a clip began before this + # chunk's window, the render must seek into it so it resumes mid-clip instead of + # restarting at the boundary. None = play from video_source.skip (the v1/default). + # Set only by the v2 overlap-inclusion path; the renderer prefers it when present. + skip_override: Optional[float] = None @dataclass diff --git a/gnommo/preprocessor.py b/gnommo/preprocessor.py index 83259fe..ce78f8f 100644 --- a/gnommo/preprocessor.py +++ b/gnommo/preprocessor.py @@ -428,7 +428,12 @@ def _oom_postmortem(returncode: int, log_text: str) -> None: sys.stdout.flush() -def run_ffmpeg_with_progress(cmd, duration, description="Processing", loglevel=None): +def run_ffmpeg_with_progress(cmd, duration, description="Processing", loglevel=None, + progress_hook=None): + """progress_hook(seconds): optional callback invoked ~every 5s with the current + output timestamp. The render passes one that logs the active slide/asset + RAM, + so the last line flushed before a hard kill (even a whole-VM WSL crash) names + where in the timeline memory blew up.""" from collections import deque cmd = cmd.copy() @@ -492,6 +497,7 @@ def run_ffmpeg_with_progress(cmd, duration, description="Processing", loglevel=N last_percent = 0 seen_any_progress = False last_log_line = "" + last_hook = 0.0 logs = deque(maxlen=_LOG_TAIL) def draw(percent, suffix=""): @@ -553,6 +559,12 @@ def run_ffmpeg_with_progress(cmd, duration, description="Processing", loglevel=N last_update = time.time() seen_any_progress = True draw(last_percent, "") + if progress_hook is not None and (time.time() - last_hook) >= 5.0: + last_hook = time.time() + try: + progress_hook(t_s) + except Exception: + pass except ValueError: pass diff --git a/gnommo/renderer.py b/gnommo/renderer.py index 56f4a8c..641eee4 100644 --- a/gnommo/renderer.py +++ b/gnommo/renderer.py @@ -159,7 +159,64 @@ def _build_crossfade_loop_filter( return filters -def render(plan: RenderPlan, output_path: Path, verbose: bool = False) -> None: +def _fmt_t(s: float) -> str: + s = max(0, int(s)) + return f"{s // 60:02d}:{s % 60:02d}" + + +def _active_slide_at(plan: RenderPlan, t: float): + """slide_id enabled at output time t (last slide whose window has opened).""" + cur = None + for e in plan.slide_events: + if e.start_time <= t < e.end_time: + return e.slide_id + if e.start_time <= t: + cur = e.slide_id + return cur + + +def _active_videos_at(plan: RenderPlan, t: float) -> list: + """source_file of every video/outro event playing at output time t.""" + names = [] + for e in list(plan.video_events) + list(plan.outro_events): + if e.start_time <= t < e.end_time: + src = getattr(getattr(e, "video_source", None), "source_file", None) + names.append(src or "?") + return names + + +def _log_render_timeline(plan: RenderPlan, log) -> None: + """Static output-time -> asset map (log only). Lets any render position — incl. + the last one before a hard kill — be mapped to a slide/asset after the fact.""" + log("[timeline] video/outro assets by output time (RAM-relevant inputs):") + rows = [ + (e.start_time, e.end_time, getattr(getattr(e, "video_source", None), "source_file", "?")) + for e in list(plan.video_events) + list(plan.outro_events) + ] + for start, end, src in sorted(rows): + log(f" {_fmt_t(start)}-{_fmt_t(end)} {src}") + + +def _make_render_progress_hook(plan: RenderPlan, log): + from .preprocessor import _mem_snapshot + + def hook(t: float) -> None: + mem = _mem_snapshot() + memstr = "" + if mem and mem[1]: + avail, total = mem + memstr = f"{100 * (total - avail) / total:.0f}% mem, {avail / 1e9:.1f}GB free | " + vids = _active_videos_at(plan, t) + vidstr = ", ".join(vids) if vids else "(none)" + log( + f"[render {_fmt_t(t)}/{_fmt_t(plan.total_duration)}] {memstr}" + f"slide={_active_slide_at(plan, t)} | videos: {vidstr}" + ) + + return hook + + +def render(plan: RenderPlan, output_path: Path, verbose: bool = False, log=None) -> None: """ Render the final video using FFmpeg. @@ -167,6 +224,11 @@ def render(plan: RenderPlan, output_path: Path, verbose: bool = False) -> None: 1. Scales background video (if present) or creates solid color 2. Overlays talking head at configured position 3. Overlays slides at their configured positions with time-based enable + + `log`: optional callback (line -> None) writing to the render log only. When + given, a static timeline map is logged and the render position + active + slide/asset + RAM are logged ~every 5s, so a hard kill's last flushed line + pinpoints which asset the memory blew up on. """ # Ensure output directory exists output_path.parent.mkdir(parents=True, exist_ok=True) @@ -179,9 +241,17 @@ def render(plan: RenderPlan, output_path: Path, verbose: bool = False) -> None: print(" ".join(cmd)) print() + hook = None + if log is not None: + try: + _log_render_timeline(plan, log) + except Exception: + pass + hook = _make_render_progress_hook(plan, log) + # Run with progress bar and ETA result = run_ffmpeg_with_progress( - cmd, duration=plan.total_duration, description="Rendering" + cmd, duration=plan.total_duration, description="Rendering", progress_hook=hook ) if result.returncode != 0: @@ -519,7 +589,10 @@ def build_ffmpeg_command(plan: RenderPlan, output_path: Path) -> list[str]: video_path = _resolve_video_path( videos_dir, event.video_source, shared_assets_dir, project_path ) - skip = event.video_source.skip or 0.0 + # Chunking v2 (docs/chunking_v2.md): a clip that began before this chunk + # resumes mid-clip via skip_override. None today (v1) → the source's own skip. + skip = event.skip_override if getattr(event, "skip_override", None) is not None \ + else (event.video_source.skip or 0.0) # How long this clip needs to play in the output clip_duration = event.end_time - event.start_time diff --git a/gnommo/transformer.py b/gnommo/transformer.py index 15550bd..a7fd37c 100644 --- a/gnommo/transformer.py +++ b/gnommo/transformer.py @@ -1321,7 +1321,13 @@ def _extract_video_events( # end_on is None and marker_type == "narration": runs to end end_time = total_duration - # Filter by time range + # Filter by time range. + # CHUNKING v1 LIMITATION: this keeps only clips whose START is inside the + # window, so a clip that began in an earlier chunk and is still playing + # across the boundary is DROPPED here — the chunked output then differs from + # a full render. cli._chunk_boundary_span_warnings surfaces this. The v2 fix + # is overlap-inclusion + a per-event seek (skip_override); see + # docs/chunking_v2.md. if start_time < range_start or start_time >= range_end: continue end_time = min(end_time, range_end) @@ -1361,6 +1367,9 @@ def _extract_audio_events( elif marker_id.startswith("audio:"): audio_id = marker_id[6:] if audio_id is not None and audio_id in audio: + # CHUNKING v1 LIMITATION (same as video, worse for looping background + # music): a clip started before the window is dropped from later chunks. + # v2 = overlap-inclusion + audio seek; see docs/chunking_v2.md. if timing.timestamp < range_start or timing.timestamp >= range_end: continue start_time = max(0, timing.timestamp - AUDIO_OFFSET_SECONDS)