Repository navigation
Conversation
Wrap Media Foundation buffers and CVPixelBuffer planes as refcounted AVFrames instead of allocating and copying each frame. Keep native Media Foundation buffers through the feed, run MF and DirectShow sample callbacks on bounded worker queues, and reuse an IOSurface-backed CVPixelBufferPool for VideoToolbox output. Add conversion counters and a synthetic 1080p frame-copy benchmark.
Use the same Media Foundation lock type for every consumer of a capture buffer, including the hardware upload and preview copy paths, and cache the last converted frame for paced repeats. Revert the macOS camera-ffmpeg zero-copy wrapper so preview and ffmpeg consumers copy instead of holding CVPixelBuffer locks for an AVFrame lifetime. Fix the VideoToolbox pool attribute compile error, share DirectShow media types across the callback worker, initialize that worker as MTA, and expose callback drop counters. Restore NV21 to NV12 conversion in as_ffmpeg and log conversion statistics periodically.
Wrap each MF buffer in a single AVBufferRef with stack plane metadata, cutting the zero-copy path from 0.002 to 0.001 ms/frame. Replace the per-frame Arc<Mutex<IMFMediaBuffer>> with a direct COM-backed CameraBuffer. Cache the D3D11 upload texture and immediate context and use UpdateSubresource instead of CreateTexture2D per frame: 0.733 vs 0.956 ms/frame for 1080p NV12. Upload straight into the hardware encoder's video-processor input texture to skip the extra CopyResource, while still running VideoProcessorBlt for conversion and scaling. Add camera-upload and zero-copy/preview-copy benchmarks.
Gate the Windows-only CameraBuffer import in feeds/camera.rs; the unconditional import broke macOS and Linux builds of cap-recording. Extend camera-upload-bench with the direct encoder-input path and a cached-upload-plus-CopyResource comparison. At 1080p NV12 the direct path is 1.053 ms/frame versus 1.359 ms/frame for the two-step path; cached and uncached uploads are 0.700 and 0.912 ms/frame.
Owner
Author
|
Superseded by CapSoftware#3359 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch reduces per-frame CPU/GPU work in the camera capture path and fixes capture/format issues found while profiling.
Windows / Media Foundation
AVBufferRefwith stack plane metadata instead of oneVec/Arcper plane.Arc<Mutex<IMFMediaBuffer>>with a direct COM-backedCameraBuffer; every consumer uses a consistent 2D/contiguous lock policy.UpdateSubresourceinstead of creating a texture per frame.CopyResource;VideoProcessorBltstill performs conversion/scaling.CameraBufferimport gate.macOS / VideoToolbox
CVPixelBufferPoolfor VideoToolbox output.Benchmarks
Added
frame-copy-benchandcamera-upload-bench.1080p NV12, release, median of 3 runs:
Direct encoder-input upload versus the old two-step path: 0.884 vs 1.205 ms/frame (~27% lower latency, ~36% higher throughput). Upload timings vary by roughly ±20% run-to-run on this host; the table uses medians, with the best cached upload at 0.700 ms and best direct upload at 0.818 ms.
Test plan
cargo clippy --workspace --exclude cap-desktop --all-features --locked -- -D warnings(Windows/MSVC) — passed.cargo fmt --all --check— passed.scripts/test-recording-reliability.pyandscripts/test-capbench.py— passed (18 and 17 tests, with expected skips).CAP_EXPORT_FORCE_SOFTWARE_ENCODER=1; the local NVIDIA NVENC path fails 3 segment-count tests, which is hardware-encoder-specific and not caused by this branch.Notes
CameraBufferimport is nowcfg(windows)-gated; macOS CI will validate the build.--all-targetsclippy (not the CI clippy command) still reports two pre-existing test-only lints incap-exportandcap-recording; neither is touched by this branch.