Skip to content

perf(camera): reduce per-frame copies and allocations across capture backends - #1

Closed
Addin wants to merge 4 commits into
mainfrom
perf/camera-capture-pipeline
Closed

Addin wants to merge 4 commits into
mainfrom
perf/camera-capture-pipeline

Conversation

@Addin

@Addin Addin commented Oct 26, 2026

Copy link
Copy Markdown
Owner

Summary

This branch reduces per-frame CPU/GPU work in the camera capture path and fixes capture/format issues found while profiling.

Windows / Media Foundation

  • Wrap MF buffers in a single refcounted AVBufferRef with stack plane metadata instead of one Vec/Arc per plane.
  • Replace the per-frame Arc<Mutex<IMFMediaBuffer>> with a direct COM-backed CameraBuffer; every consumer uses a consistent 2D/contiguous lock policy.
  • Cache the D3D11 upload texture and immediate context, and use UpdateSubresource instead of creating a texture per frame.
  • Upload directly into the hardware encoder's video-processor input texture, skipping the extra CopyResource; VideoProcessorBlt still performs conversion/scaling.
  • Fix DirectShow/Media Foundation capture issues: shared media types across the callback worker, MTA initialization, bounded callback queues, drop counters, and NV21 to NV12 conversion.
  • Restore the non-Windows CameraBuffer import gate.

macOS / VideoToolbox

  • Reuse an IOSurface-backed CVPixelBufferPool for VideoToolbox output.
  • Revert the camera-ffmpeg zero-copy wrapper so preview/FFmpeg consumers copy instead of holding CVPixelBuffer locks for the AVFrame lifetime.
  • Fix the VideoToolbox pool attribute compile error.

Benchmarks

Added frame-copy-bench and camera-upload-bench.

1080p NV12, release, median of 3 runs:

Benchmark Previous PR This run
Zero-copy frame wrap 0.002 ms / ~559k fps 0.001 ms / 936k fps
Preview copy 1.015 ms / 985 fps 0.881 ms / 1,136 fps
Row copy 0.980 ms / 1,020 fps 0.878 ms / 1,139 fps
Buffer clone 0.000 ms 0.000 ms
Bytes copy 2.074 ms 1.862 ms
Cached upload 0.733 ms / 1,364 fps 0.825 ms / 1,212 fps
Uncached upload 0.956 ms / 1,046 fps 0.948 ms / 1,055 fps
Direct encoder-input upload 1.053 ms / 949 fps 0.884 ms / 1,131 fps
Cached upload + CopyResource (old two-step) 1.359 ms / 736 fps 1.205 ms / 830 fps

Direct encoder-input upload versus the old two-step path: 0.884 vs 1.205 ms/frame (~27% lower latency, ~36% higher throughput). Upload timings vary by roughly ±20% run-to-run on this host; the table uses medians, with the best cached upload at 0.700 ms and best direct upload at 0.818 ms.

Test plan

  • cargo clippy --workspace --exclude cap-desktop --all-features --locked -- -D warnings (Windows/MSVC) — passed.
  • cargo fmt --all --check — passed.
  • CI-equivalent Windows test set: cap-project, cap-timestamp, cap-enc-ffmpeg, cap-camera-effects, cap-recording lib (687 passed), cap-rendering, cap-audio, cap-export, cap-editor, cap-render-farm, cap-camera-ffmpeg (4 passed), cap-enc-mediafoundation, cap-utils (79 passed), cap CLI record tests (13 passed), scap-targets, and WASAPI loopback.
  • scripts/test-recording-reliability.py and scripts/test-capbench.py — passed (18 and 17 tests, with expected skips).
  • Instant-mode DASH scenarios: 61/61 pass with CAP_EXPORT_FORCE_SOFTWARE_ENCODER=1; the local NVIDIA NVENC path fails 3 segment-count tests, which is hardware-encoder-specific and not caused by this branch.

Notes

  • A full macOS build was not possible on the Windows development host. The non-Windows CameraBuffer import is now cfg(windows)-gated; macOS CI will validate the build.
  • The stricter local --all-targets clippy (not the CI clippy command) still reports two pre-existing test-only lints in cap-export and cap-recording; neither is touched by this branch.

Addin added 4 commits October 25, 2026 20:32
Wrap Media Foundation buffers and CVPixelBuffer planes as refcounted AVFrames instead of allocating and copying each frame. Keep native Media Foundation buffers through the feed, run MF and DirectShow sample callbacks on bounded worker queues, and reuse an IOSurface-backed CVPixelBufferPool for VideoToolbox output. Add conversion counters and a synthetic 1080p frame-copy benchmark.
Use the same Media Foundation lock type for every consumer of a capture buffer, including the hardware upload and preview copy paths, and cache the last converted frame for paced repeats. Revert the macOS camera-ffmpeg zero-copy wrapper so preview and ffmpeg consumers copy instead of holding CVPixelBuffer locks for an AVFrame lifetime. Fix the VideoToolbox pool attribute compile error, share DirectShow media types across the callback worker, initialize that worker as MTA, and expose callback drop counters. Restore NV21 to NV12 conversion in as_ffmpeg and log conversion statistics periodically.
Wrap each MF buffer in a single AVBufferRef with stack plane metadata, cutting the zero-copy path from 0.002 to 0.001 ms/frame. Replace the per-frame Arc<Mutex<IMFMediaBuffer>> with a direct COM-backed CameraBuffer.

Cache the D3D11 upload texture and immediate context and use UpdateSubresource instead of CreateTexture2D per frame: 0.733 vs 0.956 ms/frame for 1080p NV12. Upload straight into the hardware encoder's video-processor input texture to skip the extra CopyResource, while still running VideoProcessorBlt for conversion and scaling.

Add camera-upload and zero-copy/preview-copy benchmarks.
Gate the Windows-only CameraBuffer import in feeds/camera.rs; the unconditional import broke macOS and Linux builds of cap-recording.

Extend camera-upload-bench with the direct encoder-input path and a cached-upload-plus-CopyResource comparison. At 1080p NV12 the direct path is 1.053 ms/frame versus 1.359 ms/frame for the two-step path; cached and uncached uploads are 0.700 and 0.912 ms/frame.
@Addin

Addin commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Superseded by CapSoftware#3359

@Addin Addin closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant