Skip to content

perf(export): skip unused audio decoding in still previews - #3322

Open
vedanth-jadhav wants to merge 1 commit into
CapSoftware:mainfrom
vedanth-jadhav:perf/preview-skip-audio
Open

vedanth-jadhav wants to merge 1 commit into
CapSoftware:mainfrom
vedanth-jadhav:perf/preview-skip-audio

Conversation

@vedanth-jadhav

@vedanth-jadhav vedanth-jadhav commented Oct 26, 2026 •

Copy link
Copy Markdown

Still-image export previews currently build an audio-capable exporter and wait for full recorded PCM, although PreviewSource only borrows video decoders and cursor events. A long recording therefore incurs a large audio decode/allocation before one JPEG can be returned.

Use the existing build_video_only() in render_preview and render_preview_with_config. These are the CLI export-preview path and the GPUI fallback when no editor instance is available. The open-editor preview path already reuses its editor and is unchanged. The production change is two builder calls; no new cache, API, dependency or unsafe code.

Complete preview measurements

Source API Before, s After, s Before peak RSS, MiB After peak RSS, MiB
1h AAC microphone disk config 2.010 0.160 1495.8 170.8
1h AAC microphone supplied config 2.009 0.174 1495.5 170.7
1h independent mic + system AAC disk config 2.419 0.194 2819.3 171.0
1h independent mic + system AAC supplied config 2.212 0.167 2819.0 170.9
10s source control disk config 0.168 0.162 175.7 170.7
10s source control supplied config 0.167 0.174 176.1 170.2
No recorded audio control disk config 0.157 0.155 170.8 170.5
No recorded audio control supplied config 0.155 0.167 170.7 170.9

Affected long-source calls take 91.3–92.4% less time, removing about 1.8–2.2 seconds per request, with 88.6–93.9% lower whole-process peak RSS. This is cold standalone/fallback preview preparation, not an open editor's normal seek latency or total desktop-app memory.

Five independent processes per revision/API/workload, alternating revision order: 80 previews total, after excluded baseline pilots. Every workload has identical JPEG bytes and result metadata across revisions and both APIs; only measured render time is excluded from the identity comparison. Timers include builder/device/decoder preparation, frame rendering, JPEG/base64 encoding and return from the public preview API; output-file serialization and process startup are excluded. Supplied-config runs also time the initial ProjectConfiguration::load used to supply that config.

The short/audio-free controls vary rather than universally improve: worst median is supplied-config/no-audio +11.45ms (7.4%); its process ranges overlap (145.8–169.7ms before, 146.5–173.1ms after). Supplied-config/10s is +7.46ms, with overlapping ranges. No causal control gain or universal speedup is claimed.

Linux x86_64, AMD EPYC 9V74, two-CPU quota and affinity 0,1; Rust 1.88, FFmpeg 7.1.5, Mesa 25.0.7 Lavapipe with two threads. Shipping root release: opt-level=s, thin LTO, one codegen unit. Harness: two Tokio workers, 16MiB worker stacks; warm filesystem/shader caches; no overlapping compilation. Synthetic 320×180/1fps H.264 display, 48kHz stereo AAC microphone and independently encoded 660Hz system track, four-second unit-speed selection, frame time 1s, 640×360/15fps preview. No camera/cursor drawing, music or captions in these measurements. RSS includes native/render allocations. Physical GPUs and macOS are unmeasured.

Measured baseline 586522a. Rebased onto f6046f1; intervening caption-word changes do not change these preview builders/recorded-audio loaders and the fixtures have no captions. Final checks are rerun after rebase. Raw samples, saved binaries, outputs/hashes and drivers stay outside the production diff.

Correctness and deliberate behavior

  • Native differential tests compare the previous eager builder with both public preview APIs, normal and cursor-only settings. JPEG bytes, dimensions, total frames and estimated-size bits must match.
  • A separate test corrupts an AAC packet while retaining a valid container. The eager audio builder rejects it; both video-only previews still return the exact original JPEG and metadata. Audio payload decoding is deliberately no longer a condition for producing a silent still image. Metadata/header validation remains in ProjectRecordingsMeta; this does not promise success with missing or unprobeable sources.
  • Video/cursor loading, source-duration/timeline calculation, supplied-config handling, subtitle behavior and cursor-only overrides remain in the existing builder/render pipeline. MP4 audio validation and streaming are outside this change.

Validation: 47 library tests pass; both new native tests pass when explicitly run with FFmpeg/ffprobe and a wgpu adapter. Rebased format/check, strict library/example Clippy and diff checks pass. The built CLI returns exit0 on valid and damaged-audio projects; its valid JPEG/metadata exactly match the measured candidate. Strict all-target Clippy encounters the unchanged Linux irrefutable let … else in mp4.rs:2120; it is also present on the baseline. No suppression or unrelated fix is included. New native tests are ignored by default because they need FFmpeg/ffprobe and a wgpu adapter, matching existing exporter integration tests.

Reproduction

Copy the new example onto the baseline before building it, save the binaries separately, then alternate them on the same disposable Studio project. Use a four-second unit-speed cut from a long recording with one/two AAC tracks for the affected cases; use a short recording and video-only project as controls.

cargo build -p cap-export --release --locked --example preview-audio-benchmark
# Save each revision's executable as /tmp/preview-before and /tmp/preview-after.
project=/path/to/disposable-studio.cap
for i in 1 2 3 4 5; do
  versions="before after"
  if [ $((i % 2)) -eq 0 ]; then versions="after before"; fi
  for version in $versions; do
    taskset -c 0,1 /tmp/preview-$version "$project" "/tmp/preview-$i-$version.json"
    taskset -c 0,1 /tmp/preview-$version "$project" "/tmp/preview-config-$i-$version.json" config
  done
done
cargo test -p cap-export --lib --locked
cargo test -p cap-export --lib --locked video_only_preview_tests -- --ignored --test-threads=1

Use native FFmpeg setup and a valid Vulkan ICD/XDG runtime on headless Linux; set CAP_RENDER_FORCE_SOFTWARE_ADAPTER=1 LP_NUM_THREADS=2 for the software case. Example output paths must not already exist. Compare result JSON after excluding frame_render_time_ms; JPEG bytes and all remaining metadata must match.

Current main/open PRs rechecked. This is separate from #3318, which streams and strictly validates MP4 audio; previews do not consume audio. #2703 concerns screenshot-editor startup, and #2807/#2854 concern GPU/editor frame transport. No overlapping still-preview audio-preparation fix found.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Both standalone preview APIs now use build_video_only() to skip recorded-audio decoding.

  • Still previews skip recorded-audio decoding before returning a frame.

Intentional behavior acknowledged by vedanth-jadhav: damaged audio payloads no longer prevent a silent still preview; metadata checks remain.

Reviews (1) · Last reviewed commit: "perf(export): skip unused audio decoding..." · Reviewed by Greptile

@vercel

vercel Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Someone is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant