Adding changes version 1

2026-02-06 17:56:05 +01:00
parent 93fa820275
commit fdd275ac0e
30 changed files with 7068 additions and 888 deletions
@@ -0,0 +1,317 @@
+# Partial Rendering Specification
+
+## Overview
+
+Enable rendering of specific sections of a video (e.g., slides 1-10, then 10-20) instead of the full video. This is useful for:
+- Faster iteration during development
+- Re-rendering specific sections after fixes
+- Parallel rendering of segments that can be concatenated later
+
+## Scope (v1)
+
+**In scope:**
+- Camera state tracking (cumulative state must be computed from t=0)
+- Time offset adjustment for all events
+- Slide range filtering
+- Input video seeking
+
+**Out of scope (v1):**
+- Audio events crossing range boundaries
+- Triggered video duration edge cases
+- Events are assumed to begin at their marker timestamp and never "carry over"
+
+## Current Architecture Analysis
+
+### 1. Camera State Management
+
+**Current behavior** (`transformer.py:250-332`):
+- Camera state is **cumulative** across the transcript
+- `_extract_camera_events()` walks through ALL markers sequentially
+- Each marker type (Zoom/Tilt/Pan) only modifies its property while preserving others
+- Example: `[Zoom2]` then `[TiltLeft]` = both zoom AND tilt active
+
+**Problem for partial rendering**:
+If we start rendering at slide 10, we need the camera state AS IT WOULD BE after processing slides 1-9.
+
+**Solution**:
+Separate "state computation" from "event generation":
+1. Always walk through ALL transcript markers to compute cumulative state
+2. Track the "initial state" at the start of the render range
+3. Only emit CameraEvents for markers WITHIN the render range
+4. First event in partial render must transition FROM the computed initial state
+
+### 2. Time Signature Adjustment
+
+**Current behavior**:
+All timing uses absolute timestamps from `transcript.csv`:
+- `SlideEvent.start_time/end_time`
+- `VideoEvent.start_time/end_time`
+- `AudioEvent.start_time`
+- `CameraEvent.time`
+- FFmpeg expressions: `enable=between(t, start, end)`
+- Camera animation: `if(between(t, 1.000, 1.200), ...)`
+
+**Problem for partial rendering**:
+If slide 10 starts at t=10.0s and we render from there, FFmpeg expects t=0 at the start of output.
+
+**Solution**:
+Apply a `time_offset` to all events after extraction:
+```
+new_time = original_time - time_offset
+```
+Where `time_offset` = start time of first slide/event in range.
+
+### 3. Input Video Seeking
+
+**Current behavior**:
+- Always-visible videos (talking head) start from the beginning
+- FFmpeg processes entire input duration
+
+**Problem for partial rendering**:
+Need to seek into source videos to the correct position.
+
+**Solution**:
+Add `-ss <seek_time>` before input files for always-visible videos:
+```
+ffmpeg -ss 10.0 -i talking_head.mov ...
+```
+
+---
+
+## Proposed API
+
+### Command Line Interface
+
+```bash
+# Render full video (current behavior)
+gnommo render example/project.json output.mp4
+
+# Render specific slide range
+gnommo render example/project.json output.mp4 --slides S1:S10
+gnommo render example/project.json output.mp4 --slides S10:S20
+gnommo render example/project.json output.mp4 --slides S5:  # S5 to end
+
+# Render specific time range (alternative)
+gnommo render example/project.json output.mp4 --time 0:60
+gnommo render example/project.json output.mp4 --time 60:120
+```
+
+### Internal API
+
+New parameters for `build_render_plan()`:
+```python
+def build_render_plan(
+    ...
+    slide_range: Optional[tuple[str, Optional[str]]] = None,  # (start_slide, end_slide)
+    # OR
+    time_range: Optional[tuple[float, Optional[float]]] = None,  # (start_time, end_time)
+) -> RenderPlan:
+```
+
+New field on `RenderPlan`:
+```python
+@dataclass
+class RenderPlan:
+    ...
+    time_offset: float = 0.0  # Offset to subtract from all timestamps
+    initial_camera_state: CameraState = field(default_factory=CameraState)  # State at render start
+    input_seek_time: float = 0.0  # Seek position for input videos
+```
+
+---
+
+## Implementation Details
+
+### Phase 1: Compute Full State, Filter Events
+
+Modify `_extract_camera_events()` to accept a time range:
+
+```python
+def _extract_camera_events(
+    transcript: list[TimedWord],
+    time_range: Optional[tuple[float, float]] = None,  # (start, end)
+) -> tuple[list[CameraEvent], CameraState]:
+    """
+    Returns:
+        - List of CameraEvents within time_range
+        - Initial CameraState at start of time_range
+    """
+    events: list[CameraEvent] = []
+    current_state = CameraState()
+    initial_state = CameraState()
+    start_time, end_time = time_range or (0.0, float('inf'))
+
+    found_start = False
+
+    for timed_word in transcript:
+        if not timed_word.is_marker:
+            continue
+
+        marker_id = timed_word.marker_id
+        if not marker_id or marker_id not in CAMERA_PRESETS:
+            continue
+
+        # Always update current_state (full walk)
+        preset = CAMERA_PRESETS[marker_id]
+        new_state = _apply_preset(current_state, marker_id, preset)
+
+        # Capture state just before we enter the render range
+        if not found_start and timed_word.time >= start_time:
+            initial_state = current_state  # State BEFORE this marker
+            found_start = True
+
+        # Only emit events within range
+        if start_time <= timed_word.time < end_time:
+            events.append(CameraEvent(
+                time=timed_word.time,
+                target_state=new_state,
+                duration=0.2,
+                easing="ease-out",
+            ))
+
+        current_state = new_state
+
+    return events, initial_state
+```
+
+### Phase 2: Apply Time Offset
+
+After extracting events, apply offset to all timestamps:
+
+```python
+def _apply_time_offset(plan: RenderPlan, offset: float) -> RenderPlan:
+    """Shift all timestamps by offset (subtract offset from all times)."""
+
+    # Adjust slide events
+    for event in plan.slide_events:
+        event.start_time -= offset
+        event.end_time -= offset
+
+    # Adjust video events
+    for event in plan.video_events:
+        event.start_time -= offset
+        event.end_time -= offset
+
+    # Adjust audio events
+    for event in plan.audio_events:
+        event.start_time = max(0, event.start_time - offset)
+
+    # Adjust camera events
+    for event in plan.camera_events:
+        event.time -= offset
+
+    # Adjust total duration
+    plan.total_duration -= offset
+    plan.time_offset = offset
+    plan.input_seek_time = offset
+
+    return plan
+```
+
+### Phase 3: FFmpeg Seeking
+
+Modify `build_ffmpeg_command()` to add seeking:
+
+```python
+def build_ffmpeg_command(plan: RenderPlan, output_path: Path) -> list[str]:
+    cmd = ["ffmpeg", "-y"]
+
+    # Add seek for always-visible videos
+    for video_id, video_source, cutout in plan.narration_videos:
+        video_path = _resolve_video_path(videos_dir, video_source)
+        if plan.input_seek_time > 0:
+            cmd.extend(["-ss", str(plan.input_seek_time)])  # Seek BEFORE -i
+        cmd.extend(["-i", str(video_path)])
+        ...
+```
+
+### Phase 4: Initial Camera State Handling
+
+If `initial_camera_state` is not default, inject a "virtual" camera event at t=0:
+
+```python
+def build_camera_transform(
+    camera_events: list[CameraEvent],
+    initial_state: CameraState,  # NEW PARAMETER
+    ...
+) -> str:
+    # If initial state differs from default, prepend a virtual event
+    if not initial_state.is_default():
+        initial_event = CameraEvent(
+            time=0.0,
+            target_state=initial_state,
+            duration=0.0,  # Instant - no transition
+            easing="linear",
+        )
+        camera_events = [initial_event] + camera_events
+    ...
+```
+
+---
+
+## FFmpeg Optimization
+
+**Only emit filters for events within range.**
+
+When rendering a partial range, the `RenderPlan` should only contain events within that range. This means:
+- Fewer inputs added to the FFmpeg command (only slides/videos/audio actually used)
+- Fewer overlay filters in filter_complex
+- Fewer `between(t, start, end)` enable expressions to evaluate per frame
+
+Example: Full video has 50 slides, rendering S40:S50 only:
+- **Before**: 50 slide inputs, 50 overlay filters
+- **After**: 10 slide inputs, 10 overlay filters
+
+This is achieved naturally by filtering events in `build_render_plan()` before constructing the plan - the renderer already only processes events present in the plan.
+
+---
+
+## Edge Cases (v1 Simplified)
+
+### 1. Camera state from before range
+If rendering S5:S10 but there's a camera event at the S4 marker:
+- Camera state from S4 must be captured as `initial_camera_state`
+- Rendered output starts with that state already applied at t=0
+
+### 2. Events filter by marker position
+All events (slides, videos, audio) are filtered by whether their START marker falls within the range.
+- Events beginning outside range are excluded
+- No "carry over" or boundary-crossing logic needed
+
+---
+
+## Testing Strategy
+
+### Unit Tests
+1. Camera state computation maintains state across full transcript
+2. Time offset correctly shifts all event types
+3. Initial camera state correctly captured at boundary
+
+### Integration Tests
+1. Render slides 1-5, then 5-10, concatenate, compare to full render
+2. Camera state continuity across segment boundaries
+3. Audio alignment after seeking
+
+### Manual Verification
+1. Visual inspection of camera state at segment boundaries
+2. Audio sync verification
+
+---
+
+## Future Enhancements
+
+### Parallel Rendering Pipeline
+```bash
+# Render in parallel, then concatenate
+gnommo render proj.json seg1.mp4 --slides S1:S10 &
+gnommo render proj.json seg2.mp4 --slides S10:S20 &
+gnommo render proj.json seg3.mp4 --slides S20: &
+wait
+ffmpeg -f concat -i segments.txt -c copy final.mp4
+```
+
+### Smart Re-rendering
+Track which slides changed and only re-render affected segments.
+
+### Preview Mode
+Quick low-quality render of specific section for review.