diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 3e33a58c..415bb2d5 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -39,7 +39,7 @@ edit yourself keeps the mode you give it until the next `config set`; run | `capabilities.agent-device.enabled` | **Both sides.** On a worker: run one agent-device daemon on this worker, started and watched by Simlock. It sees Simlock's devices and no others, refuses `boot` and `shutdown`, and keeps its files in `~/.simlock/agent-device`. If agent-device is missing, older than 0.21.28, speaks another RPC protocol or needs a newer Node than Simlock runs on, it is not started and Simlock runs on without it. A daemon a killed Simlock left behind is stopped at the next start with `capabilities.agent-device.enabled` `true`; with it off, Simlock does not look for one. With `http.enabled` also `true`, the agent-device CLI reaches it at `/v1/agent-device` (see [HTTP-API.md](HTTP-API.md#v1agent-device)). On a gateway: serve `/v1/agent-device` and send each agent to a worker whose agent-device is available; the gateway runs no daemon of its own. | `false` | | `capabilities.agent-device.leaseTtlCapMs` | **Both sides.** Caps the TTL of the leases the agent-device CLI makes and renews through Simlock: a longer `ttlMs` is lowered to it, and a lease that sends none gets it. A whole number of milliseconds from `5000` to `lease.maxTtlMs`; a value outside that range stops the daemon at start, on a gateway too. Used only when `capabilities.agent-device.enabled` is `true`. | unset (no cap) | | `capabilities.agent-device.command` | **Worker side.** The agent-device executable: a name looked up on `PATH`, or a path. A gateway ignores it and logs a warning. Used only when `capabilities.agent-device.enabled` is `true`. | `agent-device` | -| `capabilities.stream.enabled` | **Both sides.** Offer a live view of a leased simulator's or emulator's screen. On a worker: answer `GET /v1/stream/leases/{leaseId}` (see [HTTP-API.md](HTTP-API.md#get-v1streamleasesleaseid)) by capturing the screen of the device that lease holds. On a gateway: serve the same URL and relay each request to the worker that holds the lease. Off by default because a stream is a capture pinned to a leased device — with it off, nothing is running until someone asks to look, and nothing is running when they stop. Read at start; changing it needs a restart. | `false` | +| `capabilities.stream.enabled` | **Both sides.** Offer a live view of a leased device's screen: a simulator, an emulator or a physical iPhone. On a worker: answer `GET /v1/stream/leases/{leaseId}` (see [HTTP-API.md](HTTP-API.md#get-v1streamleasesleaseid)) by capturing the screen of the device that lease holds. On a gateway: serve the same URL and relay each request to the worker that holds the lease. Off by default because a stream is a capture pinned to a leased device — with it off, nothing is running until someone asks to look, and nothing is running when they stop. Read at start; changing it needs a restart. | `false` | | `capabilities.stream.fps` | **Worker side.** The frame rate the encoder is asked for. A capture sends full frames only when the device's screen changes, and on a still screen only small keep-alive frames, so this is a ceiling and not a promise: an untouched home screen stays far below it, which is what keeps an idle stream cheap. A whole number from `1` to `60`. | `30` | | `capabilities.stream.maxDimension` | **Worker side.** Cap on the encoded picture's longest side, in pixels; the other side follows the device's aspect ratio. This is the lever that costs the most bandwidth — a device that encodes at about 6 Mbit/s native is nearer 1.4 Mbit/s at `720`. A whole number from `96` to `4096`. | `720` | | `capabilities.stream.bitrateKbps` | **Worker side.** Target bitrate in kilobits per second, handed to the encoder as an average rather than a ceiling; a frame that needs more takes it. A whole number from `100` to `40000`; the iOS encoder takes nothing lower. | `1500` | @@ -78,6 +78,7 @@ edit yourself keeps the mode you give it until the next `config set`; run | `stalledTransition.minimumThresholdMs` | Floor under the multiplied estimate, for a driver whose estimate is near zero. | `1 minute` | | `drivers.ios.deviceRoot` | The CoreSimulator device set Simlock owns and scopes every `simctl` call to. See [Device roots](#device-roots). | `${SIMLOCK_HOME}/devices/ios` | | `drivers.ios.screenCaptureCommand` | The program that captures a simulator's screen for `capabilities.stream.enabled`. Unset, the one `pnpm build` put beside the daemon. | unset | +| `drivers.ios.deviceCaptureCommand` | The program that captures a physical iPhone's screen for `capabilities.stream.enabled`. Unset, the one `pnpm build` put beside the daemon. | unset | | `drivers.android.deviceRoot` | The AVD home Simlock owns; exported as `ANDROID_AVD_HOME` to every `avdmanager`/`emulator` call. See [Device roots](#device-roots). | `${SIMLOCK_HOME}/devices/android` | | `drivers.android.adbServerPort` | TCP port for Simlock's own adb server. Must not be the shared server's `5037`. Startup fails closed if it is occupied. | `5038` | | `ios.defaultMode` | The device mode an iOS lease request gets when it names none: `slim` or `full`. Every worker makes both kinds whatever this says. See [Device mode: slim and full](#device-mode-slim-and-full). | `full` | @@ -607,16 +608,19 @@ sends almost nothing. `capabilities.stream.fps` is a ceiling rather than a rate reason. On a Mac an iOS capture costs about 50-80 MB of memory per device being watched. -An iOS simulator's screen and an Android emulator's screen can be streamed. A -lease on a physical iPhone or Android phone is refused with -`STREAM_NOT_SUPPORTED`, naming that as the reason rather than failing for one -the reader has to guess. - -- **iOS** needs the capture program `pnpm build` builds on a Mac with Xcode - selected. Without it an iOS stream is refused with - `STREAM_UNAVAILABLE`, saying why; `simlock status` says why too when no - platform on the worker can stream. `drivers.ios.screenCaptureCommand` names a - program at another path. +An iOS simulator's screen, a physical iPhone's screen and an Android +emulator's screen can be streamed. A lease on a physical Android phone is +refused with `STREAM_NOT_SUPPORTED`, naming that as the reason rather than +failing for one the reader has to guess. + +- **iOS** needs the capture programs `pnpm build` builds on a Mac with Xcode + selected: one for a simulator and one for a physical iPhone. Without the one + a device needs, that stream is refused with `STREAM_UNAVAILABLE`, saying why; + `simlock status` says why too when no platform on the worker can stream. A + physical iPhone additionally needs a worker on Xcode 27 — the device program + refuses a Mac whose CoreDevice major is another. + `drivers.ios.screenCaptureCommand` names another simulator program and + `drivers.ios.deviceCaptureCommand` another iPhone one. - **Android** needs nothing more than the SDK the emulators already use. Each stream copies a small screen server into the emulator, and removes it when the stream ends. An emulator encodes in software unless its GPU mode gives it diff --git a/docs/CONSOLE.md b/docs/CONSOLE.md index e1e1518c..ef397020 100644 --- a/docs/CONSOLE.md +++ b/docs/CONSOLE.md @@ -258,10 +258,10 @@ live on the page. What you are looking at is either moving or gone. The panel is on every lease's page, and streams wherever the worker the lease runs on has screen streaming turned on (see [`capabilities.stream.enabled`](CONFIGURATION.md)). -Where it is off, the panel says so before you press anything, and the button -cannot be pressed; where the lease is on a device a screen cannot be -captured from — a physical iPhone or Android phone — it says that a live screen -comes only from a simulator or an emulator. Either way it starts nothing. +Simulators, emulators and physical iPhones all stream. Where it is off, the +panel says so before you press anything, and the button cannot be pressed; it +starts nothing. An iPhone streams on a worker running Xcode 27: on another +worker, the panel reports that the screen cannot be captured there and why. The first picture can take a while on a slow emulator. The panel waits up to 90 seconds for it, longer than a worker takes to give up, so a capture that cannot diff --git a/docs/HTTP-API.md b/docs/HTTP-API.md index f25ff98e..856458f4 100644 --- a/docs/HTTP-API.md +++ b/docs/HTTP-API.md @@ -942,7 +942,9 @@ worth asking for twice rather than holding open in advance. Authorization is the same as every other lease route: a token sees its own leases, an operator sees any, and anything else is a `404`. `STREAM_NOT_SUPPORTED` (415) means the device behind the lease can never produce a picture here — a -physical phone. `STREAM_UNAVAILABLE` (503) means it could but cannot now: a +physical Android phone. A physical iPhone is not refused that way: it streams +like a simulator, on a worker that can capture it. `STREAM_UNAVAILABLE` +(503) means it could but cannot now: a capture program that is not installed, or a device that painted nothing within 30 seconds; asking again is the right move. A stream that stops receiving frames for 30 seconds ends with an error rather than freezing on its last picture. `WORKER_UNREACHABLE` on diff --git a/docs/internal/VENDORED-CODE.md b/docs/internal/VENDORED-CODE.md index 1b05eeb5..6a75bd40 100644 --- a/docs/internal/VENDORED-CODE.md +++ b/docs/internal/VENDORED-CODE.md @@ -15,7 +15,8 @@ license text(s) they require under `LICENSES/`, and a `NOTICE.md`. | What | Upstream | Version / commit | License | Where it lives | Consumed by | Re-verify with | |---|---|---|---|---|---|---| | `@agent-device/proxy` — the transport proxy in front of an agent-device daemon, packed from a branch of agent-device that is not on npm | `callstack/agent-device`, `simlock-integration` branch | commit `253dca023edf13410f7928c76e9e604e5e5871a0` | MIT (Callstack) | `vendor/agent-device-proxy-0.21.28-dev-253dca0.tgz`; license inside the archive at `package/LICENSE` | `package.json`'s `"@agent-device/proxy": "file:vendor/…tgz"`, which `src/agent-device/host/` imports | `tar -tzf vendor/agent-device-proxy-*.tgz \| grep LICENSE`; the commit is what `scripts/slow-e2e.sh` checks a local checkout against, so keep it the first 40-char hash in `vendor/README.md` | -| serve-sim's capture and encode pipeline — CoreSimulator framebuffer capture and the H.264 encode path, patched by us, with four files rewritten by us that stay serve-sim's modified files | `EvanBacon/serve-sim` | commit `6d009686ed08f7d8bd19bca9f9f67a9237931f44` | Apache-2.0 for serve-sim's files and our rewrites of four of them; MIT for the three files that are ours entirely | `vendor/screen-capture/Sources/simlock-screen-capture/`, license at `vendor/screen-capture/LICENSES/serve-sim.APACHE-2.0.txt` | the `simlock-screen-capture` program `pnpm build` puts in dist/bin, run by `src/drivers/ios/screen-capture.ts` | the `curl … \| diff -u -` command in [`vendor/screen-capture/NOTICE.md`](../../vendor/screen-capture/NOTICE.md), per file | +| serve-sim's capture and encode pipeline — CoreSimulator framebuffer capture and the H.264 encode path, patched by us, with four files rewritten by us that stay serve-sim's modified files | `EvanBacon/serve-sim` | commit `6d009686ed08f7d8bd19bca9f9f67a9237931f44` | Apache-2.0 for serve-sim's files and our rewrites of four of them; MIT for the files that are ours entirely | `vendor/screen-capture/Sources/`, licenses at `vendor/screen-capture/LICENSES/serve-sim.APACHE-2.0.txt` and `…/idevice.MIT.txt` | the `simlock-screen-capture` and `simlock-device-capture` programs `pnpm build` puts in dist/bin, run by `src/drivers/ios/screen-capture.ts` | the `curl … \| diff -u -` command in [`vendor/screen-capture/NOTICE.md`](../../vendor/screen-capture/NOTICE.md), per file | +| the HEVC capability string the iPhone's media-stream offer advertises for payload 100 — one string inside one of our own files | `jkcoxson/idevice` | `master` at the time of taking; the string is what its receiver documents | MIT | `vendor/screen-capture/LICENSES/idevice.MIT.txt`, the string in `vendor/screen-capture/Sources/SimlockMediaBridge/SimlockMediaBridge.m` | `simlock-device-capture`, whose offer a physical iPhone accepts without painting torn pictures | the string in `Sources/SimlockMediaBridge/SimlockMediaBridge.m` against idevice's receiver, and its `LICENSE.txt` at `https://raw.githubusercontent.com/jkcoxson/idevice/master/LICENSE.txt` | | `scrcpy-server` — the Android screen-streaming server artifact, byte-identical to upstream's release asset | `Genymobile/scrcpy` | release `v5.0.1`; revision built `dc8a55a24c27ae78ae202b5a3288da681d0f141a` | Apache-2.0; the Kotlin standard library inside it adds BSD-3-Clause (ThreeTen-BP) and BSL-1.0 (Boost) code | `vendor/scrcpy/scrcpy-server-v5.0.1`, licenses at `vendor/scrcpy/LICENSES/scrcpy.APACHE-2.0.txt` and `vendor/scrcpy/LICENSES/threetenbp.BSD-3-Clause.txt`, digest in `vendor/scrcpy/SHA256SUMS.txt` | the Android driver's screen capture, which pushes it unmodified to a leased emulator; the client that drives it is ours, in `src/drivers/android/`, and is not vendored | `cd vendor/scrcpy && shasum -a 256 -c SHA256SUMS.txt`, and the same against upstream's own release `SHA256SUMS.txt`, whose GPG signature `vendor/scrcpy/NOTICE.md` says how to check | `vendor/scrcpy/NOTICE.md` names what is *inside* that artifact besides scrcpy's diff --git a/docs/internal/agent-rules/toolchain.md b/docs/internal/agent-rules/toolchain.md index 8a4bde3c..4df2f642 100644 --- a/docs/internal/agent-rules/toolchain.md +++ b/docs/internal/agent-rules/toolchain.md @@ -66,8 +66,11 @@ that goes through the reviews its diff's kind gets and the merge gate: `pnpm build` runs `scripts/build-capture.mjs`: it copies the vendored scrcpy server into `dist/vendor/scrcpy/` after checking its pinned SHA-256, and on a Mac with Xcode selected it builds `vendor/screen-capture/` (Swift) into -`dist/bin/simlock-screen-capture`. Elsewhere the Swift build is skipped and iOS -streams report unavailable. +`dist/bin/simlock-screen-capture` and `dist/bin/simlock-device-capture`, one +product per `swift build`, each whether or not the other built. Elsewhere the +Swift build is skipped and iOS streams report unavailable. Neither build checks +a CoreDevice version: the device program is the one place that enforces that +pin. Neither runs in the unit, e2e or console lanes: both read a booted device's screen, which means real devices and the slow lane. The lanes cover what this @@ -108,7 +111,7 @@ The machine can run a line only if: | ---------- | ---------------------------------------------------------- | | iOS | `uname -s` is `Darwin`; `xcrun simctl list runtimes` lists one | | Android | `command -v emulator adb` finds both | -| Physical iOS | `uname -s` is `Darwin`; `SIMLOCK_PHYSICAL_IOS` is the UDID of an iPhone on USB (trusted, Developer Mode on, unlocked, Auto-Lock Never); `SIMLOCK_PHYSICAL_IOS_APP` is a `.app` signed for it. Run `scripts/slow-e2e.sh --physical e2e/slow-ios-physical.test.ts` | +| Physical iOS | `uname -s` is `Darwin`; `SIMLOCK_PHYSICAL_IOS` is the UDID of an iPhone on USB (trusted, Developer Mode on, unlocked, Auto-Lock Never); `SIMLOCK_PHYSICAL_IOS_APP` is a `.app` signed for it. `/usr/libexec/PlistBuddy -c "Print :CFBundleShortVersionString" /Library/Developer/PrivateFrameworks/CoreDevice.framework/Resources/Info.plist` prints `642.` followed by a number (streaming a phone needs Xcode 27's CoreDevice; the framework is machine-wide and does not follow `xcode-select`). DeviceHub and Xcode are not running during the lane. Run `scripts/slow-e2e.sh --physical e2e/slow-ios-physical.test.ts` | | agent-device through simlock | `node -v` is 22.12.0 or later (agent-device's own minimum); the iOS check and `uname -m` is `arm64` (the fixture app is arm64 only), or the Android check; `SIMLOCK_E2E_AGENT_DEVICE` (in `.slow-lane.env`) is the absolute path of `bin/agent-device.mjs` in an install of the agent-device version `package.json` pins for `@agent-device/proxy` (`npm install -g agent-device@`, or a checkout at tag `v` built with `pnpm install && pnpm build && pnpm build:android`, the last building the Android snapshot helper); room for two devices: `node -p "os.totalmem() / 2**30"` prints at least 7 for iOS or 12 for Android (simlock's 4 GiB OS reserve plus two of its default 1.5 GiB iOS or 4 GiB Android per-device budgets). Run `scripts/slow-e2e.sh e2e/slow-agent-device.test.ts` or `scripts/slow-e2e.sh e2e/slow-agent-device-gateway.test.ts` | | Physical Android | `command -v adb` finds it; `adb devices -l` lists `$SIMLOCK_PHYSICAL_ANDROID` on a USB transport (a `usb:` field) in state `device`; `SIMLOCK_PHYSICAL_ANDROID_APK` names an APK. Run `scripts/slow-e2e.sh --physical e2e/slow-android-physical.test.ts` | | Physical Android, a person at the phone | the Physical Android checks above, and `SIMLOCK_PHYSICAL_PERSON` is `1`. Run `scripts/slow-e2e.sh --physical-manual e2e/slow-android-physical-manual.test.ts` | diff --git a/e2e/helpers/index.ts b/e2e/helpers/index.ts index 589b32aa..9fb0cbb1 100644 --- a/e2e/helpers/index.ts +++ b/e2e/helpers/index.ts @@ -20,6 +20,8 @@ export { DEVICE_READY_MS, waitFor } from "./wait.js"; export { events } from "./events.js"; export type { RecordedEvent } from "./events.js"; export { waitForDeviceState, waitForLeaseCount } from "./status.js"; +export { SCREEN_TAGS, readScreenFrames, type ScreenFrame } from "./screen-frames.js"; +export { REPO_ROOT } from "./repo-root.js"; export type { CliResult } from "./cli.js"; export { ANDROID, diff --git a/e2e/helpers/screen-frames.ts b/e2e/helpers/screen-frames.ts new file mode 100644 index 00000000..ac6abb73 --- /dev/null +++ b/e2e/helpers/screen-frames.ts @@ -0,0 +1,35 @@ +// why: the stream's envelope is one contract both iOS stream lanes read -- a big-endian length, a +// one-byte tag, then the payload, written by the capture programs and handed to a viewer byte for +// byte by the stream module. A lane that claims a stream carries a key frame reads it through here, +// so the two lanes cannot drift into checking two different formats. +export const SCREEN_TAGS = { description: 0x01, keyframe: 0x02, meta: 0x05 } as const; + +export interface ScreenFrame { + readonly tag: number; + readonly payload: Uint8Array; +} + +// why: the body is an endless stream, so this reads frames until it has what it came for and then +// lets go, rather than reading to an end that never comes. +export async function readScreenFrames( + reader: ReadableStreamDefaultReader, + enough: (frames: readonly ScreenFrame[]) => boolean, +): Promise { + const frames: ScreenFrame[] = []; + let buffer = new Uint8Array(0); + while (!enough(frames)) { + const step = await reader.read(); + if (step.done) break; + const joined = new Uint8Array(buffer.length + step.value.length); + joined.set(buffer); + joined.set(step.value, buffer.length); + buffer = joined; + while (buffer.length >= 5) { + const length = new DataView(buffer.buffer, buffer.byteOffset).getUint32(0); + if (buffer.length < 4 + length) break; + frames.push({ payload: buffer.slice(5, 4 + length), tag: buffer[4]! }); + buffer = buffer.slice(4 + length); + } + } + return frames; +} diff --git a/e2e/slow-ios-physical-manual.test.ts b/e2e/slow-ios-physical-manual.test.ts index 490df27f..e2aca282 100644 --- a/e2e/slow-ios-physical-manual.test.ts +++ b/e2e/slow-ios-physical-manual.test.ts @@ -1,7 +1,11 @@ +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; import { describe, expect, it } from "vitest"; import { freeLoopbackPort, waitFor, withDaemon, type TestEnv } from "./helpers/index.js"; +const execFileAsync = promisify(execFile); + const UDID = process.env["SIMLOCK_PHYSICAL_IOS"]; const APP = process.env["SIMLOCK_PHYSICAL_IOS_APP"]; const PERSON = process.env["SIMLOCK_PHYSICAL_PERSON"]; @@ -24,15 +28,25 @@ interface StatusDevice { readonly absentReason?: string; } -async function startDaemon(): Promise { +async function startDaemon( + options: { stream?: boolean } = {}, +): Promise { const port = await freeLoopbackPort(); - return withDaemon({ + const env = await withDaemon({ configOverrides: { + ...(options.stream === true + ? { + capabilities: { + stream: { bitrateKbps: 1500, enabled: true, fps: 30, maxDimension: 720 }, + }, + } + : {}), health: { probeIntervalMs: 2_000, stableObservations: 2 }, http: { enabled: true, host: "127.0.0.1", port }, }, driver: "real", }); + return { ...env, port }; } async function deviceOf(env: TestEnv, udid: string): Promise { @@ -74,6 +88,14 @@ async function enroll(env: TestEnv, id: string): Promise { expect(added.code, `device add failed: ${added.stderr}`).toBe(0); } +/** why: `pgrep -x` matches the program's own name only; a pattern match would also find the shell running this check. */ +async function deviceCapturesRunning(): Promise { + const found = await execFileAsync("pgrep", ["-x", "simlock-device-capture"]).catch(() => ({ + stdout: "", + })); + return found.stdout.split("\n").filter((line) => line.trim() !== "").length; +} + describe.skipIf(process.platform !== "darwin" || missing !== undefined)( missing === undefined ? "iOS physical phone with a person (real devicectl)" @@ -131,6 +153,50 @@ describe.skipIf(process.platform !== "darwin" || missing !== undefined)( }, ); + it( + "while the iPhone streams, a person locking it ends the lease as device-lost and leaves no device capture running within 10 s", + { timeout: 15 * MINUTE }, + async () => { + const env = await startDaemon({ stream: true }); + await enroll(env, udid); + await expectAfterPrompt(env, udid, "keep the phone unlocked and plugged in", "ready"); + const leased = await env.cli( + ["lease", "--platform", "ios", "--physical", "--agent-id", "phone-lane-lock", "--detach"], + { timeout: 2 * MINUTE }, + ); + expect(leased.code, `lease failed: ${leased.stderr}`).toBe(0); + const leaseId = (leased.json as { lease: { id: string } }).lease.id; + + const created = await env.cli(["token", "create", "--role", "operator"]); + expect(created.code, created.stderr).toBe(0); + const viewer = new AbortController(); + const response = await fetch( + `http://127.0.0.1:${String(env.port)}/v1/stream/leases/${encodeURIComponent(leaseId)}`, + { + headers: { authorization: `Bearer ${(created.json as { secret: string }).secret}` }, + signal: viewer.signal, + }, + ).catch(() => undefined); + expect(response?.status, "the leased iPhone did not start streaming").toBe(200); + + await expectAfterPrompt(env, udid, "LOCK the phone now", "absent", "locked"); + + const released = (await env.events("1h")) + .map((entry) => ({ + event: entry.event, + payload: entry.payload as { leaseId?: string; reason?: string }, + })) + .filter((entry) => entry.event === "lease.released" && entry.payload.leaseId === leaseId); + expect(released.map((entry) => entry.payload.reason)).toContain("device-lost"); + await waitFor(async () => (await deviceCapturesRunning()) === 0, { + interval: 500, + label: "the device capture stopped once the locked iPhone's lease ended", + timeout: 10_000, + }); + viewer.abort(); + }, + ); + it("ends with device remove", { timeout: 5 * MINUTE }, async () => { const env = await startDaemon(); await enroll(env, udid); diff --git a/e2e/slow-ios-physical.test.ts b/e2e/slow-ios-physical.test.ts index a46d569e..6f5041c4 100644 --- a/e2e/slow-ios-physical.test.ts +++ b/e2e/slow-ios-physical.test.ts @@ -1,17 +1,38 @@ -import { execFile } from "node:child_process"; -import { access, mkdtemp, readFile } from "node:fs/promises"; +import { execFile, spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { access, mkdtemp, readFile, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { promisify } from "node:util"; import { describe, expect, it } from "vitest"; -import { type TestEnv, waitFor, waitForDeviceState, withHttpDaemon } from "./helpers/index.js"; +import { + REPO_ROOT, + SCREEN_TAGS as TAG, + type TestEnv, + readScreenFrames as readFrames, + waitFor, + waitForDeviceState, + withHttpDaemon, +} from "./helpers/index.js"; const execFileAsync = promisify(execFile); const UDID = process.env["SIMLOCK_PHYSICAL_IOS"]; const APP = process.env["SIMLOCK_PHYSICAL_IOS_APP"]; +const DEVICE_PROGRAM = join(REPO_ROOT, "dist", "bin", "simlock-device-capture"); +const SETTINGS_ARGS = [ + "--codec", + "avcc", + "--fps", + "30", + "--bitrate", + "1500000", + "--max-dimension", + "720", +] as const; + const PROBE_INTERVAL_MS = 2_000; const MINUTE = 60_000; @@ -104,6 +125,9 @@ async function rig(): Promise { configOverrides: { health: { probeIntervalMs: PROBE_INTERVAL_MS, stableObservations: 2 }, http: { enabled: true, host: "127.0.0.1", port }, + capabilities: { + stream: { bitrateKbps: 1500, enabled: true, fps: 30, maxDimension: 720 }, + }, }, driver: "real", })); @@ -373,5 +397,269 @@ describe.skipIf(process.platform !== "darwin" || missing !== undefined)( await uninstallDirect(udid, bundle); expect(await installedBundles(udid)).not.toContain(bundle); }); + + // why: the Physical iOS row of the toolchain requires CoreDevice 642 (Xcode 27), so a lane that + // got this far has the toolchain the device program needs; a missing program is a broken build, + // not a machine that cannot run the lane, and skipping it would hide that. + it("the device capture program is built", { timeout: MINUTE }, () => { + expect( + existsSync(DEVICE_PROGRAM), + `no ${DEVICE_PROGRAM} -- run pnpm build on a Mac with Xcode 27`, + ).toBe(true); + }); + + it( + "the device program refuses a Mac whose CoreDevice major is not 642, and names the version, the missing framework or the unreadable version as its reason", + { timeout: 5 * MINUTE }, + async () => { + const rows = [ + ["650.3", "this Mac has CoreDevice 650.3"], + ["641.9", "this Mac has CoreDevice 641.9"], + ["6420.1", "this Mac has CoreDevice 6420.1"], + ["64.2", "this Mac has CoreDevice 64.2"], + ["", "this Mac's CoreDevice version cannot be read"], + ] as const; + for (const [version, reason] of rows) { + const plist = await aCoreDevicePlist(version); + const run = await runDeviceProgram(plist, udid); + expect(run.code, `CoreDevice ${version} is not 642: ${run.stderr}`).toBe(69); + expect(lastLineOf(run.stderr)).toBe( + `Physical iPhone streams need Xcode 27 (CoreDevice 642.x); ${reason}`, + ); + } + + const absent = await runDeviceProgram( + join(await mkdtemp(join(tmpdir(), "simlock-coredevice-")), "absent.plist"), + udid, + ); + expect(absent.code, "a Mac without CoreDevice cannot stream an iPhone").toBe(69); + expect(lastLineOf(absent.stderr)).toBe( + "Physical iPhone streams need Xcode 27 (CoreDevice 642.x); this Mac has no CoreDevice", + ); + }, + ); + + it( + "the device program goes on for every CoreDevice 642.x spelling and, for a UDID it cannot find, exits 1 within 26 s naming it as not connected", + { timeout: 10 * MINUTE }, + async () => { + const unknown = "FFFFFFFF-0000-4000-8000-000000000000"; + for (const version of ["642.16", "642.0", "642", " 642.3 "]) { + const plist = await aCoreDevicePlist(version); + const started = Date.now(); + const run = await runDeviceProgram(plist, unknown); + expect( + run.code, + `CoreDevice ${version} is 642.x, so the version is not what refused`, + ).toBe(1); + const elapsed = Date.now() - started; + expect( + elapsed, + `CoreDevice ${version} found no device within the 25 s start bound`, + ).toBeLessThan(26_000); + expect(lastLineOf(run.stderr)).toContain(unknown); + expect(lastLineOf(run.stderr)).toMatch(/not connected/iu); + } + }, + ); + + it( + "the device program asked to stop while it is still starting goes away inside the caller's grace, on its own and not by force", + { timeout: 5 * MINUTE }, + async () => { + // why: 642.x and a UDID that is not there leave the program in its 25 s start, which is when a + // stop is hardest to honour — the caller SIGKILLs 2 s after it asks, and a program that cannot + // leave a start it cannot cancel is SIGKILLed with the iPhone's media stream left up. Closing + // stdin is the stop the driver uses; exit 0 within 2 s is that stop having been taken. + const plist = await aCoreDevicePlist("642.0"); + const child = spawn( + DEVICE_PROGRAM, + ["FFFFFFFF-0000-4000-8000-000000000000", ...SETTINGS_ARGS], + { + env: { ...process.env, SIMLOCK_COREDEVICE_INFO_PLIST: plist }, + stdio: ["pipe", "pipe", "pipe"], + }, + ); + let stderr = ""; + child.stderr.on("data", (chunk: Buffer) => { + stderr += chunk.toString("utf8"); + }); + await new Promise((resolve) => setTimeout(resolve, 5_000)); + expect(child.exitCode, "the program ended before it was asked to").toBeNull(); + const askedAt = Date.now(); + child.stdin.end(); + const ended = await new Promise<{ + code: number | null; + signal: NodeJS.Signals | null; + }>((resolve) => { + child.once("exit", (exitCode, exitSignal) => + resolve({ code: exitCode, signal: exitSignal }), + ); + }); + expect( + Date.now() - askedAt, + `the stop was not taken within 2 s, so the caller killed it: ${stderr}`, + ).toBeLessThan(2_000); + expect( + ended.signal, + `a stop the program was asked for is not a signal: ${stderr}`, + ).toBeNull(); + expect(ended.code, "a stop the program was asked for is a clean end").toBe(0); + }, + ); + + it( + "a lease on the iPhone streams its screen as meta with codec h264, an avcC description and a key frame; a still screen keeps the stream open past 35 s; a second viewer on the same lease either streams or refuses with a reason while the first keeps its frames; after the lease ends no device capture remains and a new lease streams a key frame", + { timeout: 15 * MINUTE }, + async () => { + expect(await deviceCapturesRunning(), "a device capture was already running").toBe(0); + const daemon = await rig(); + await enroll(daemon.env, udid); + const grant = await leasePhysical(daemon.env, "phone-lane-stream"); + const open = async (leaseId: string, signal: AbortSignal) => { + const response = await fetch( + `${daemon.base}/v1/stream/leases/${encodeURIComponent(leaseId)}`, + { headers: daemon.auth, signal }, + ); + if (response.body === null) throw new Error("the stream answered without a body"); + return { reader: response.body.getReader(), response }; + }; + + const viewer = new AbortController(); + const first = await open(grant.lease.id, viewer.signal); + expect( + first.response.status, + await first.response + .clone() + .text() + .catch(() => ""), + ).toBe(200); + expect(first.response.headers.get("simlock-stream-codec")).toBe("h264"); + const frames = await readFrames(first.reader, (read) => + read.some((frame) => frame.tag === TAG.keyframe), + ); + expect(frames.map((frame) => frame.tag).slice(0, 3)).toEqual([ + TAG.meta, + TAG.description, + TAG.keyframe, + ]); + const meta = JSON.parse(new TextDecoder().decode(frames[0]!.payload)) as { + codec: string; + width: number; + height: number; + }; + expect(meta.codec).toBe("h264"); + expect(Math.max(meta.width, meta.height)).toBeLessThanOrEqual(720); + expect(frames[1]!.payload[0]).toBe(1); + expect(await deviceCapturesRunning()).toBe(1); + + // why: nothing touches the iPhone here, so its screen is still; the stream must stay open + // well past the 30 s silence bound, on the source's own idle repaints. + const stillUntil = Date.now() + 35_000; + const held = await readFrames(first.reader, () => Date.now() > stillUntil); + expect(held.length, "a still iPhone's stream sent nothing for 35 s").toBeGreaterThan(0); + expect((await first.reader.read()).done, "a still iPhone's stream ended inside 35 s").toBe( + false, + ); + + const second = await open(grant.lease.id, new AbortController().signal); + if (second.response.status === 200) { + await readFrames(second.reader, (read) => + read.some((frame) => frame.tag === TAG.keyframe), + ); + } else { + expect(second.response.status).toBe(503); + const refusal = (await second.response.json()) as { error?: { message?: string } }; + expect(refusal.error?.message, "a 503 says why").toBeTruthy(); + } + // why: whatever the second viewer gets, the first must keep its own stream: read it for a + // further 10 s and count what arrives. + const alongsideUntil = Date.now() + 10_000; + const alongside = await readFrames(first.reader, () => Date.now() > alongsideUntil); + expect(alongside.length, "the first viewer received no frames for 10 s").toBeGreaterThan(0); + + const released = await daemon.env.cli(["release", grant.lease.id]); + expect(released.code, released.stderr).toBe(0); + await waitFor(async () => (await deviceCapturesRunning()) === 0, { + interval: 250, + label: "the device capture stopped once its lease ended", + timeout: 5_000, + }); + + // why: the stream's own first-frame bound is 30 s, so a body that produced no key frame by + // then has already failed or will not fail visibly; the race only names the claim. + const again = await leasePhysical(daemon.env, "phone-lane-stream-2"); + const reopened = await open(again.lease.id, new AbortController().signal); + expect(reopened.response.status).toBe(200); + const keyFrameArrived = await Promise.race([ + readFrames(reopened.reader, (read) => + read.some((frame) => frame.tag === TAG.keyframe), + ).then((read) => read.some((frame) => frame.tag === TAG.keyframe)), + new Promise((resolve) => setTimeout(() => resolve(false), 30_000)), + ]); + expect( + keyFrameArrived, + "a new lease on the same iPhone streamed no key frame in 30 s", + ).toBe(true); + await daemon.env.cli(["release", again.lease.id]); + }, + ); }, ); + +/** why: `pgrep -x` matches the program's own name only; a pattern match would also find the shell running this check (F44). */ +async function deviceCapturesRunning(): Promise { + const found = await execFileAsync("pgrep", ["-x", "simlock-device-capture"]).catch(() => ({ + stdout: "", + })); + return found.stdout.split("\n").filter((line) => line.trim() !== "").length; +} + +// why: the program reads its version from SIMLOCK_COREDEVICE_INFO_PLIST only in tests (the driver +// never passes it), and `version` of "" plants a plist without the key -- the row the program is +// required to read as "this Mac's CoreDevice version cannot be read" rather than as an empty 642. +async function aCoreDevicePlist(version: string): Promise { + const dir = await mkdtemp(join(tmpdir(), "simlock-coredevice-")); + const path = join(dir, "Info.plist"); + const body = [ + '', + '', + "", + ...(version === "" + ? [] + : [` CFBundleShortVersionString${version}`]), + "", + "", + "", + ].join("\n"); + await writeFile(path, body, "utf8"); + return path; +} + +// why: runs the device program on its own, with a version file the driver never passes. `device` is +// the UDID it is asked to stream; the caller's 26 s bound is the program's own 25 s start deadline. +// `stdio[0]` is `pipe` and not `ignore` because stdin is the capture's liveness handle: the +// driver hands it a live pipe it closes only to stop, and a pipe hung up at birth is a stop asked for +// before the program began, which the program answers by exiting 0 without ever looking for a device. +async function runDeviceProgram( + infoPlist: string, + device: string, +): Promise<{ code: number; stderr: string }> { + const child = spawn(DEVICE_PROGRAM, [device, ...SETTINGS_ARGS], { + env: { ...process.env, SIMLOCK_COREDEVICE_INFO_PLIST: infoPlist }, + stdio: ["pipe", "pipe", "pipe"], + }); + let stderr = ""; + child.stderr.on("data", (chunk: Buffer) => { + stderr += chunk.toString("utf8"); + }); + const code = await new Promise((resolve, reject) => { + child.once("error", reject); + child.once("close", (exitCode) => resolve(exitCode ?? -1)); + }); + return { code, stderr }; +} + +function lastLineOf(text: string): string { + return text.trimEnd().split("\n").at(-1) ?? ""; +} diff --git a/e2e/slow-ios-stream.test.ts b/e2e/slow-ios-stream.test.ts index 27f46f4e..76c1c2bd 100644 --- a/e2e/slow-ios-stream.test.ts +++ b/e2e/slow-ios-stream.test.ts @@ -1,42 +1,21 @@ -import { execFile } from "node:child_process"; +import { execFile, spawn } from "node:child_process"; +import { existsSync } from "node:fs"; +import { join } from "node:path"; import { promisify } from "node:util"; import { describe, expect, it } from "vitest"; -import { DEVICE_READY_MS, waitFor, withHttpDaemon } from "./helpers/index.js"; +import { + DEVICE_READY_MS, + REPO_ROOT, + SCREEN_TAGS as TAG, + readScreenFrames as readFrames, + waitFor, + withHttpDaemon, +} from "./helpers/index.js"; const execFileAsync = promisify(execFile); -const TAG = { description: 0x01, keyframe: 0x02, meta: 0x05 } as const; - -interface Frame { - readonly tag: number; - readonly payload: Uint8Array; -} - -// why: the body is an endless stream, so this reads frames until it has what it came for and then -// lets go, rather than reading to an end that never comes. -async function readFrames( - reader: ReadableStreamDefaultReader, - enough: (frames: readonly Frame[]) => boolean, -): Promise { - const frames: Frame[] = []; - let buffer = new Uint8Array(0); - while (!enough(frames)) { - const step = await reader.read(); - if (step.done) break; - const joined = new Uint8Array(buffer.length + step.value.length); - joined.set(buffer); - joined.set(step.value, buffer.length); - buffer = joined; - while (buffer.length >= 5) { - const length = new DataView(buffer.buffer, buffer.byteOffset).getUint32(0); - if (buffer.length < 4 + length) break; - frames.push({ payload: buffer.slice(5, 4 + length), tag: buffer[4]! }); - buffer = buffer.slice(4 + length); - } - } - return frames; -} +const SIMULATOR_PROGRAM = join(REPO_ROOT, "dist", "bin", "simlock-screen-capture"); // why: `pgrep -x` matches the program's own name only; a pattern match would also find the shell // running this check, which is a capture that looks alive and is not. @@ -156,5 +135,48 @@ describe.skipIf(process.platform !== "darwin")( } }, ); + + it( + "the simulator capture program exits 64 on missing usage and 1 when its device is not there", + { timeout: 120_000 }, + async () => { + expect(existsSync(SIMULATOR_PROGRAM), "run pnpm build first").toBe(true); + const noArgs = await run(SIMULATOR_PROGRAM, []); + expect(noArgs.code, "no arguments is a usage error").toBe(64); + const unknownUdid = await run(SIMULATOR_PROGRAM, [ + "00000000-0000-0000-0000-000000000000", + "--codec", + "avcc", + "--fps", + "30", + "--bitrate", + "1500000", + "--max-dimension", + "720", + ]); + expect(unknownUdid.code, "a device that is not there fails to start").toBe(1); + }, + ); }, ); + +// why: `execFile` rejects on a non-zero exit, which is the whole claim here; the exit code is what +// the test reads, so the rejection is unwrapped rather than caught per call site. `stdio[0]` is +// `pipe` and not `ignore` because stdin is the capture's liveness handle: the driver hands it a live +// pipe it closes only to stop, so a pipe hung up at birth is a stop asked for before the program +// began -- which the iPhone program answers by going at once with exit 0, reporting no reason. +async function run( + program: string, + args: readonly string[], +): Promise<{ code: number; stderr: string }> { + const child = spawn(program, args, { stdio: ["pipe", "pipe", "pipe"] }); + let stderr = ""; + child.stderr.on("data", (chunk: Buffer) => { + stderr += chunk.toString("utf8"); + }); + const code = await new Promise((resolve, reject) => { + child.once("error", reject); + child.once("close", (exitCode) => resolve(exitCode ?? -1)); + }); + return { code, stderr }; +} diff --git a/scripts/build-capture.mjs b/scripts/build-capture.mjs index 5b6c7e51..85c52b22 100644 --- a/scripts/build-capture.mjs +++ b/scripts/build-capture.mjs @@ -1,9 +1,11 @@ #!/usr/bin/env node // `pnpm build:capture`: puts the screen-capture artifacts beside the compiled daemon, which looks for // them there. The scrcpy server is copied into dist/vendor/scrcpy/ once its SHA-256 matches the pin in -// vendor/scrcpy/SHA256SUMS.txt; a mismatch fails the build. On macOS with a Swift toolchain the iOS -// capture program in vendor/screen-capture/ is built and copied to dist/bin/simlock-screen-capture. -// Anywhere else it is skipped with a note, and the daemon reports iOS streams as unavailable. +// vendor/scrcpy/SHA256SUMS.txt; a mismatch fails the build. On macOS with a Swift toolchain each +// program of vendor/screen-capture/ is built with its own `swift build --product` and copied to +// dist/bin/ — one failing does not skip the other, and the daemon reports the streams it lost. +// Anywhere else the Swift build is skipped with a note, and the daemon reports iOS streams as +// unavailable. // Each artifact ships with its upstream's NOTICE.md and LICENSES/ in dist/vendor//, because the // package publishes dist/ and Apache-2.0 §4(a) asks that whoever gets a binary gets its license too. import { spawnSync } from "node:child_process"; @@ -39,30 +41,56 @@ function copyScrcpyServer() { copyNotices("scrcpy"); } -function buildIosCapture() { - if (process.platform !== "darwin") { - console.log("build-capture: not macOS, skipping the iOS screen-capture program"); - return; - } - const packagePath = join(root, "vendor/screen-capture"); - const scratch = join(packagePath, ".build/out"); - const built = spawnSync( +// why: one product per call, so a build that fails on one program is not what skips the other; +// the caller decides what the pair of outcomes means. +function buildAndCopySwiftProduct(product, what, packagePath, scratch) { + const result = spawnSync( "swift", - ["build", "-c", "release", "--package-path", packagePath, "--scratch-path", scratch], + [ + "build", + "-c", + "release", + "--product", + product, + "--package-path", + packagePath, + "--scratch-path", + scratch, + ], { stdio: "inherit" }, ); - if (built.error !== undefined || built.status !== 0) { + // why: a failed spawn (no `swift` on the PATH) leaves `status` null, which is not 0 either, so + // this one test covers both the tool that is missing and the build that failed. + if (result.status !== 0) { console.warn( - "build-capture: could not build the iOS screen-capture program (is Xcode installed and selected?); " + - "iOS screen streams will be unavailable", + `build-capture: could not build ${product} (is Xcode installed and selected?); ${what} will be unavailable`, ); - return; + return false; } - const target = join(dist, "bin/simlock-screen-capture"); + const target = join(dist, "bin", product); mkdirSync(dirname(target), { recursive: true }); - copyFileSync(join(scratch, "release/simlock-screen-capture"), target); + copyFileSync(join(scratch, "release", product), target); chmodSync(target, 0o755); - copyNotices("screen-capture"); + return true; +} + +function buildIosCapture() { + if (process.platform !== "darwin") { + console.log("build-capture: not macOS, skipping the iOS screen-capture programs"); + return; + } + const packagePath = join(root, "vendor/screen-capture"); + const scratch = join(packagePath, ".build/out"); + // why: each program builds on its own -- the device program links a private framework the + // simulator program never needs, so one failing must not take the other's streams with it. + const products = [ + ["simlock-screen-capture", "iOS screen streams"], + ["simlock-device-capture", "physical iPhone streams"], + ]; + const copied = products.filter(([product, what]) => + buildAndCopySwiftProduct(product, what, packagePath, scratch), + ); + if (copied.length > 0) copyNotices("screen-capture"); } copyScrcpyServer(); diff --git a/src/build-capture.test.ts b/src/build-capture.test.ts index ec28188f..a700da24 100644 --- a/src/build-capture.test.ts +++ b/src/build-capture.test.ts @@ -47,17 +47,26 @@ function scratchRepo(server: string, digest: string): string { } // why: a `swift` that writes the program where `swift build` would, so the claim about what ships -// beside the iOS program is checked without a Swift toolchain. +// beside the iOS programs is checked without a Swift toolchain. It honours `--product` like the +// real build does, and fails the products named in SIMLOCK_FAKE_SWIFT_FAIL so a test can have one +// program build and the other not. function fakeSwift(repo: string): string { const bin = join(repo, "fake-bin"); - mkdirSync(bin); + mkdirSync(bin, { recursive: true }); writeFileSync( join(bin, "swift"), [ "#!/bin/sh", - 'while [ "$1" != "--scratch-path" ]; do shift; done', - '/bin/mkdir -p "$2/release"', - 'echo program > "$2/release/simlock-screen-capture"', + "product=''", + 'scratch=""', + 'while [ "$#" -gt 0 ]; do', + ' if [ "$1" = "--product" ]; then shift; product="$1"; fi', + ' if [ "$1" = "--scratch-path" ]; then shift; scratch="$1"; fi', + " shift", + "done", + 'case ",$SIMLOCK_FAKE_SWIFT_FAIL," in *",$product,"*) exit 1 ;; esac', + '/bin/mkdir -p "$scratch/release"', + 'echo program > "$scratch/release/$product"', ].join("\n"), ); chmodSync(join(bin, "swift"), 0o755); @@ -105,26 +114,43 @@ describe("scripts/build-capture.mjs", () => { }); it.skipIf(process.platform !== "darwin")( - "ships the iOS capture program with serve-sim's NOTICE.md and license, and leaves them out when it is not built", + "builds each iOS capture program on its own, ships each one that built with serve-sim's NOTICE.md and license, and leaves out what did not build", () => { const repo = scratchRepo("the real server", sha256("the real server")); - const run = (path: string) => - execFileSync(process.execPath, [join(repo, "scripts", "build-capture.mjs")], { - env: { ...process.env, PATH: path }, - stdio: "pipe", + const run = (path: string, fail = "") => + spawnSync(process.execPath, [join(repo, "scripts", "build-capture.mjs")], { + encoding: "utf8", + env: { ...process.env, PATH: path, SIMLOCK_FAKE_SWIFT_FAIL: fail }, }); const shipped = join(repo, "dist", "vendor", "screen-capture"); + const simulatorProgram = join(repo, "dist", "bin", "simlock-screen-capture"); + const deviceProgram = join(repo, "dist", "bin", "simlock-device-capture"); run(""); - expect(existsSync(join(repo, "dist", "bin", "simlock-screen-capture"))).toBe(false); + expect(existsSync(simulatorProgram)).toBe(false); + expect(existsSync(deviceProgram)).toBe(false); expect(existsSync(shipped)).toBe(false); run(fakeSwift(repo)); - expect(existsSync(join(repo, "dist", "bin", "simlock-screen-capture"))).toBe(true); + expect(existsSync(simulatorProgram)).toBe(true); + expect(existsSync(deviceProgram)).toBe(true); expect(readFileSync(join(shipped, "NOTICE.md"), "utf8")).toBe("notice of screen-capture"); expect(readFileSync(join(shipped, "LICENSES", "serve-sim.APACHE-2.0.txt"), "utf8")).toBe( "license of screen-capture", ); + + rmSync(join(repo, "dist"), { force: true, recursive: true }); + const deviceFailed = run(fakeSwift(repo), "simlock-device-capture"); + expect(existsSync(simulatorProgram)).toBe(true); + expect(existsSync(deviceProgram)).toBe(false); + expect(existsSync(join(shipped, "NOTICE.md"))).toBe(true); + expect(deviceFailed.stdout + deviceFailed.stderr).toMatch(/physical iPhone streams/iu); + + rmSync(join(repo, "dist"), { force: true, recursive: true }); + run(fakeSwift(repo), "simlock-screen-capture"); + expect(existsSync(simulatorProgram)).toBe(false); + expect(existsSync(deviceProgram)).toBe(true); + expect(existsSync(join(shipped, "NOTICE.md"))).toBe(true); }, ); }); diff --git a/src/daemon/main.ts b/src/daemon/main.ts index 58c74c15..5962b113 100644 --- a/src/daemon/main.ts +++ b/src/daemon/main.ts @@ -971,10 +971,13 @@ async function startGatewayDaemon(options: GatewayDaemonOptions): Promise logger.debug("Screen capture", { line }), }, diff --git a/src/drivers/ios/AGENTS.md b/src/drivers/ios/AGENTS.md index 9cc08841..f58883a6 100644 --- a/src/drivers/ios/AGENTS.md +++ b/src/drivers/ios/AGENTS.md @@ -7,14 +7,14 @@ Parent: [drivers](../AGENTS.md). Flows: [device selection](../../../docs/interna ## Files -| File | Role | -| ------------------- | ----------------------------------------------------------------------------- | -| `index.ts` | `IosSimctlDriver`: every `Driver` verb, slim pass, `simctl` passthrough rules | -| `physical.ts` | `IosPhysicalHandler`: enrol, presence, reclaim through `devicectl` | -| `devicectl-exec.ts` | `devicectlExecCommand`: the reviewed `devicectl` list | -| `slim-labels.ts` | `SLIM_CATEGORIES`, `resolveSlimCategories`, `labelsFor`, `slimSignature` | -| `prerequisites.ts` | `iosPrerequisites`: Xcode selected, license accepted, first launch done | -| `screen-capture.ts` | `createIosScreenCapture`: runs `vendor/screen-capture/` on a simulator | +| File | Role | +| ------------------- | ------------------------------------------------------------------------------------------- | +| `index.ts` | `IosSimctlDriver`: every `Driver` verb, slim pass, `simctl` passthrough rules | +| `physical.ts` | `IosPhysicalHandler`: enrol, presence, reclaim through `devicectl` | +| `devicectl-exec.ts` | `devicectlExecCommand`: the reviewed `devicectl` list | +| `slim-labels.ts` | `SLIM_CATEGORIES`, `resolveSlimCategories`, `labelsFor`, `slimSignature` | +| `prerequisites.ts` | `iosPrerequisites`: Xcode selected, license accepted, first launch done | +| `screen-capture.ts` | `createIosScreenCapture`: runs `vendor/screen-capture/` on a simulator or a physical iPhone | ## How it works @@ -81,11 +81,16 @@ Passthrough — Lease environment: virtual `SIMLOCK_IOS_DEVICE_SET=`; physical `SIMLOCK_DEVICE_UDID=`. -Screen — `screenCapture` (only when `create` is given a launcher): `simlock-screen-capture ---codec avcc --fps --bitrate --max-dimension` with `SIMLOCK_IOS_DEVICE_SET=`; stdout is -the frames, stdin its liveness. Close: stdin, 2 s, SIGKILL. Physical → `ScreenNotSupportedError`; -program missing → `ScreenUnavailableError`. Default program: dist/bin, built by `pnpm build`; -`drivers.ios.screenCaptureCommand` names another. +Screen — `screenCapture` (only when `create` is given a launcher): `simlock-screen-capture + --codec avcc --fps --bitrate --max-dimension` with `SIMLOCK_IOS_DEVICE_SET=` for a +simulator, and the same arguments without that variable for a physical iPhone run by +`simlock-device-capture`. stdout is the frames, stdin its liveness. Close: stdin, 2 s, SIGKILL. +Either program missing → `ScreenUnavailableError` (a missing iPhone program refuses an iPhone +open only). Default programs: dist/bin, built by `pnpm build`; +`drivers.ios.screenCaptureCommand` and `drivers.ios.deviceCaptureCommand` name others. Neither +program's environment carries `SIMLOCK_COREDEVICE_INFO_PLIST`; the iPhone program enforces its own +pin — a CoreDevice major other than 642 exits 69 with the reason, a 503 for the viewer. Asked to +stop while still starting, a program stops and goes inside its 2 s. Advisories — `runtime-cache-unreclaimable` (asset bundles no installed runtime mounts); `slim-runtime-unsupported` (installed runtimes below 18.5, only when `ios.defaultMode` is `slim`). @@ -128,4 +133,18 @@ record of a device's mode; `driverData` holds the slim signature. simulator is booted only when it starts, then repaints its last surface every 200 ms, so after the health monitor reboots a crashed simulator the viewer can see the old picture until they press Stop and Show screen again. A lease ending stops it as usual. +- **An iPhone rotated mid-stream keeps the first picture's scale**, as a simulator does today: the + scaler is sized once, from the first picture. An iPhone that stays connected and unlocked but + stops sending video, or a decoder that stops producing pictures, shows the last picture until + the lease ends. +- **A capture killed by SIGKILL cannot stop its CoreDevice stream**; CoreDevice is left to notice + its client is gone. The same for a stop while `startMediaStream` is still running past 1.5 s. +- **A new Xcode major turns iPhone streams off** until `vendor/screen-capture/Stubs/` is checked + against that release and the pin moves. +- **Untested on an iPhone:** DeviceHub or Xcode showing the same device during a stream; the lease + holder's `devicectl` commands during one; the source ending after the first picture; `stop()` + really stopping the stream. Nothing can observe CoreDevice's state, and a device that serves two + streams at once lets a new lease stream even after a leaked one, so the slow lane proves only + that the process is gone and that the device streams again. Two viewers of one lease ask for two + streams; the two-viewer case finds out whether it serves both (#602 would remove the second). - **A cold slim lease takes ~160 s.** MCP clients need `resetTimeoutOnProgress` or a longer timeout. diff --git a/src/drivers/ios/index.test.ts b/src/drivers/ios/index.test.ts index 0f174ed2..c715c7bc 100644 --- a/src/drivers/ios/index.test.ts +++ b/src/drivers/ios/index.test.ts @@ -17,6 +17,7 @@ import { UnknownModelError, } from "../../core/index.js"; import { + FakeBytePipeLauncher, FakeClock, MemoryFilesystem, NodeFilesystem, @@ -4499,6 +4500,53 @@ async function settledValue(promise: Promise): Promise { return state.value; } +describe("IosSimctlDriver screen capture commands", () => { + const physicalDevice = { + address: "UDID-P", + deviceId: "UDID-P", + driverData: {}, + enrolledApps: [], + physical: true, + } as const; + const settings = { bitrateKbps: 1_500, fps: 30, maxDimension: 720 }; + + // why: the daemon hands the driver its packaged programs; `drivers.ios.deviceCaptureCommand` + // replaces only the physical iPhone's program, and an empty string counts as unset exactly as + // `screenCaptureCommand` does, so one mistaken `""` does not turn iPhone streams off. + it.each([ + ["the configured program", "/worker/custom-device-capture", "/worker/custom-device-capture"], + ["the daemon's program when unset", undefined, "/dist/bin/simlock-device-capture"], + ["the daemon's program when set empty", "", "/dist/bin/simlock-device-capture"], + ] as const)("a physical iPhone's stream starts %s", async (_claim, configured, expected) => { + const filesystem = new MemoryFilesystem(); + await filesystem.mkdirp("/worker"); + await filesystem.mkdirp("/dist/bin"); + await filesystem.writeFileAtomic("/worker/custom-device-capture", ""); + await filesystem.writeFileAtomic("/dist/bin/simlock-device-capture", ""); + const launcher = new FakeBytePipeLauncher(); + const driver = await IosSimctlDriver.create({ + clock: new FakeClock(), + driverConfig: configured === undefined ? {} : { deviceCaptureCommand: configured }, + filesystem, + idGenerator: { generate: () => "device-1" }, + instanceId, + processRunner: new ScriptedProcessRunner([]), + screenCapture: { + baseEnvironment: { PATH: "/usr/bin" }, + command: "/dist/bin/simlock-screen-capture", + deviceCommand: "/dist/bin/simlock-device-capture", + launcher, + }, + simlockHome: "/home/.simlock", + }); + + const capture = driver.screenCapture; + if (capture === undefined) throw new Error("the driver offers no screen capture"); + await capture.open(physicalDevice, settings, new AbortController().signal); + expect(launcher.starts.map((start) => start.command)).toEqual([expected]); + }); +}); + describe("IosSimctlDriver defaultModels", () => { it("lists the built-in iOS preference list per class, newest first, with phone starting at iPhone 17", async () => { const driver = await createDriver(new ScriptedProcessRunner([])); diff --git a/src/drivers/ios/index.ts b/src/drivers/ios/index.ts index 290fd9aa..2dbbbf28 100644 --- a/src/drivers/ios/index.ts +++ b/src/drivers/ios/index.ts @@ -214,6 +214,7 @@ export interface IosSimctlDriverOptions { // screen at all rather than one that fails on every open. readonly screenCapture?: { readonly command: string; + readonly deviceCommand?: string; readonly launcher: BytePipeLauncher; readonly baseEnvironment: NodeJS.ProcessEnv; readonly onLog?: (line: string) => void; @@ -325,6 +326,13 @@ export class IosSimctlDriver implements Driver { baseEnvironment: options.screenCapture.baseEnvironment, clock: options.clock, command: configuredScreenCaptureCommand(options) ?? options.screenCapture.command, + ...(options.screenCapture.deviceCommand === undefined && + configuredDeviceCaptureCommand(options) === undefined + ? {} + : { + deviceCommand: + configuredDeviceCaptureCommand(options) ?? options.screenCapture.deviceCommand, + }), deviceSet: deviceRoot, filesystem: options.filesystem, launcher: options.screenCapture.launcher, @@ -1687,6 +1695,11 @@ function configuredScreenCaptureCommand(options: IosSimctlDriverOptions): string return typeof configured === "string" && configured !== "" ? configured : undefined; } +function configuredDeviceCaptureCommand(options: IosSimctlDriverOptions): string | undefined { + const configured = options.driverConfig["deviceCaptureCommand"]; + return typeof configured === "string" && configured !== "" ? configured : undefined; +} + function configuredDeviceRoot(options: IosSimctlDriverOptions): string { const configured = options.driverConfig["deviceRoot"]; diff --git a/src/drivers/ios/screen-capture.test.ts b/src/drivers/ios/screen-capture.test.ts index 540d4e64..fbe7b9ea 100644 --- a/src/drivers/ios/screen-capture.test.ts +++ b/src/drivers/ios/screen-capture.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; -import { ScreenNotSupportedError, ScreenUnavailableError } from "../../core/index.js"; +import { ScreenUnavailableError } from "../../core/index.js"; import { FakeBytePipeLauncher, FakeClock, MemoryFilesystem } from "../../ports/index.js"; import { createIosScreenCapture } from "./screen-capture.js"; @@ -11,18 +11,28 @@ const simulator = { driverData: {}, physical: false, } as const; +const physical = { + address: "UDID-1", + deviceId: "UDID-1", + driverData: {}, + enrolledApps: [], + physical: true, +} as const; const COMMAND = "/opt/simlock/simlock-screen-capture"; +const DEVICE_COMMAND = "/opt/simlock/simlock-device-capture"; -async function setup() { +async function setup(options: { deviceCommand?: string } = {}) { const filesystem = new MemoryFilesystem(); await filesystem.mkdirp("/opt/simlock"); await filesystem.writeFileAtomic(COMMAND, ""); + await filesystem.writeFileAtomic(DEVICE_COMMAND, ""); const launcher = new FakeBytePipeLauncher(); const clock = new FakeClock(); const capture = createIosScreenCapture({ baseEnvironment: { PATH: "/usr/bin" }, clock, command: COMMAND, + ...(options.deviceCommand === undefined ? {} : { deviceCommand: options.deviceCommand }), deviceSet: "/home/.simlock/devices/ios", filesystem, launcher, @@ -47,8 +57,8 @@ describe("iOS screen capture", () => { expect(launcher.starts).toEqual([]); }); - it("runs the capture program on the device's UDID, inside Simlock's device set, with the stream's settings", async () => { - const { capture, launcher } = await setup(); + it("runs the capture program on the device's UDID, inside Simlock's device set, with the stream's settings, and never the device program", async () => { + const { capture, launcher } = await setup({ deviceCommand: DEVICE_COMMAND }); await capture.open(simulator, settings, new AbortController().signal); expect(launcher.starts).toEqual([ { @@ -69,16 +79,96 @@ describe("iOS screen capture", () => { ]); }); - it("refuses a physical iPhone as not supported, without starting anything", async () => { + it("a physical iPhone starts the device program on its UDID with the stream's settings and without Simlock's device set", async () => { + const { capture, launcher } = await setup({ deviceCommand: DEVICE_COMMAND }); + await capture.open(physical, settings, new AbortController().signal); + expect(launcher.starts).toEqual([ + { + args: [ + "UDID-1", + "--codec", + "avcc", + "--fps", + "30", + "--bitrate", + "1500000", + "--max-dimension", + "720", + ], + command: DEVICE_COMMAND, + env: { PATH: "/usr/bin" }, + }, + ]); + }); + + it("refuses a physical open with the build's own reason when the device program is missing, starts nothing, and a simulator open still works", async () => { + const { capture, filesystem, launcher } = await setup({ deviceCommand: DEVICE_COMMAND }); + await filesystem.rm(DEVICE_COMMAND); + await expect( + capture.open(physical, settings, new AbortController().signal), + ).rejects.toBeInstanceOf(ScreenUnavailableError); + await expect(capture.open(physical, settings, new AbortController().signal)).rejects.toThrow( + `Physical iPhone streams need simlock-device-capture, which did not build on this Mac (it needs Xcode 27, CoreDevice 642.x); there is none at ${DEVICE_COMMAND}`, + ); + expect(launcher.starts).toEqual([]); + await expect( + capture.open(simulator, settings, new AbortController().signal), + ).resolves.toBeDefined(); + }); + + it("starts either program without SIMLOCK_COREDEVICE_INFO_PLIST, even when the daemon's environment has it", async () => { + const filesystem = new MemoryFilesystem(); + await filesystem.mkdirp("/opt/simlock"); + await filesystem.writeFileAtomic(COMMAND, ""); + await filesystem.writeFileAtomic(DEVICE_COMMAND, ""); + const launcher = new FakeBytePipeLauncher(); + const capture = createIosScreenCapture({ + baseEnvironment: { + PATH: "/usr/bin", + SIMLOCK_COREDEVICE_INFO_PLIST: "/tmp/another-coredevice.plist", + }, + clock: new FakeClock(), + command: COMMAND, + deviceCommand: DEVICE_COMMAND, + deviceSet: "/home/.simlock/devices/ios", + filesystem, + launcher, + }); + await capture.open(simulator, settings, new AbortController().signal); + await capture.open(physical, settings, new AbortController().signal); + expect(launcher.starts.map((start) => start.env)).toEqual([ + { PATH: "/usr/bin", SIMLOCK_IOS_DEVICE_SET: "/home/.simlock/devices/ios" }, + { PATH: "/usr/bin" }, + ]); + }); + + // why: `deviceCommand` is optional on the options a host builds, so the refusal has a wording of its + // own for a worker that was never given a device program at all — which is not the same claim as a + // program that did not build, and is the only thing standing between a physical open and a spawn of + // the empty string. + it("refuses a physical open when the worker was given no device program at all, and a simulator open still works", async () => { const { capture, launcher } = await setup(); await expect( - capture.open( - { ...simulator, enrolledApps: [], physical: true }, - settings, - new AbortController().signal, - ), - ).rejects.toBeInstanceOf(ScreenNotSupportedError); + capture.open(physical, settings, new AbortController().signal), + ).rejects.toBeInstanceOf(ScreenUnavailableError); + await expect(capture.open(physical, settings, new AbortController().signal)).rejects.toThrow( + "Physical iPhone streams need simlock-device-capture, which this worker was not given", + ); expect(launcher.starts).toEqual([]); + await expect( + capture.open(simulator, settings, new AbortController().signal), + ).resolves.toBeDefined(); + expect(await capture.unavailable()).toBeUndefined(); + }); + + it("reports streaming available when only the device program is missing, and the simulator program's reason when that one is", async () => { + const { capture, filesystem } = await setup({ deviceCommand: DEVICE_COMMAND }); + await filesystem.rm(DEVICE_COMMAND); + expect(await capture.unavailable()).toBeUndefined(); + + await filesystem.writeFileAtomic(DEVICE_COMMAND, ""); + await filesystem.rm(COMMAND); + expect(await capture.unavailable()).toBe(`No screen capture program at ${COMMAND}`); }); it("refuses as unavailable, and says why, when the capture program is not installed", async () => { diff --git a/src/drivers/ios/screen-capture.ts b/src/drivers/ios/screen-capture.ts index 3681f6e2..956b7d85 100644 --- a/src/drivers/ios/screen-capture.ts +++ b/src/drivers/ios/screen-capture.ts @@ -2,7 +2,6 @@ import { isPhysicalDriverDevice, type ScreenCapture, type ScreenFeed, - ScreenNotSupportedError, type ScreenSettings, ScreenUnavailableError, } from "../../core/index.js"; @@ -10,6 +9,9 @@ import type { BytePipeLauncher, BytePipeProcess, Clock, Filesystem } from "../.. export interface IosScreenCaptureOptions { readonly command: string; + // why: a physical iPhone streams through a second program (`simlock-device-capture`); when it did + // not build on this Mac the option is still set, and the missing file is what refuses the open. + readonly deviceCommand?: string; readonly launcher: BytePipeLauncher; readonly clock: Clock; readonly filesystem: Filesystem; @@ -30,23 +32,31 @@ const LOG_TAIL = 20; export function createIosScreenCapture(options: IosScreenCaptureOptions): ScreenCapture { // why: only a command that spells a path can be checked; a bare program name is resolved by the // OS at spawn, and a failure there is reported with the spawn's own words. + // why: `unavailable()` speaks only of the simulator program, so a Mac without the device program + // still reports streaming available; the device program's own absence refuses a physical open. const missing = async (): Promise => options.command.includes("/") && !(await options.filesystem.exists(options.command)) ? `No screen capture program at ${options.command}` : undefined; + const missingDeviceProgram = async (): Promise => { + const command = options.deviceCommand; + if (command === undefined) + return "Physical iPhone streams need simlock-device-capture, which this worker was not given"; + // why: only a command that spells a path can be checked before spawn; a bare program name is + // resolved by the OS and a failure there is reported with the spawn's own words. + if (!command.includes("/") || (await options.filesystem.exists(command))) return undefined; + return `Physical iPhone streams need simlock-device-capture, which did not build on this Mac (it needs Xcode 27, CoreDevice 642.x); there is none at ${command}`; + }; + return { open: async (device, settings, signal) => { - if (isPhysicalDriverDevice(device)) { - throw new ScreenNotSupportedError( - "A physical iPhone's screen cannot be streamed; only a simulator's can", - ); - } - const reason = await missing(); + const physical = isPhysicalDriverDevice(device); + const reason = physical ? await missingDeviceProgram() : await missing(); if (reason !== undefined) throw new ScreenUnavailableError(reason); if (signal.aborted) throw new ScreenUnavailableError("The capture was stopped before it started"); - return startCapture(options, device.deviceId, settings); + return startCapture(options, device.deviceId, settings, physical); }, unavailable: missing, }; @@ -56,10 +66,11 @@ function startCapture( options: IosScreenCaptureOptions, udid: string, settings: ScreenSettings, + physical: boolean, ): ScreenFeed { const tail: string[] = []; const child = options.launcher.start( - options.command, + physical ? (options.deviceCommand ?? "") : options.command, [ udid, "--codec", @@ -72,7 +83,11 @@ function startCapture( String(settings.maxDimension), ], { - env: { ...options.baseEnvironment, SIMLOCK_IOS_DEVICE_SET: options.deviceSet }, + // why: the environment the daemon carries must not reach either program: a device set named for + // a simulator would send the device program looking in the wrong place, and a + // SIMLOCK_COREDEVICE_INFO_PLIST from outside would let the daemon's environment pick which + // CoreDevice the iPhone stream believes it is running against. + env: envFor(options, physical), onStderrLine: (line) => { tail.push(line); if (tail.length > LOG_TAIL) tail.shift(); @@ -84,6 +99,12 @@ function startCapture( return { body: bodyOf(child, tail, close), close }; } +function envFor(options: IosScreenCaptureOptions, physical: boolean): NodeJS.ProcessEnv { + const { SIMLOCK_COREDEVICE_INFO_PLIST: _coreDevicePlist, ...environment } = + options.baseEnvironment; + return physical ? environment : { ...environment, SIMLOCK_IOS_DEVICE_SET: options.deviceSet }; +} + // why: a program that dies mid-picture ends its stdout like one that finished, so the body asks how // it exited and fails with its last log line rather than ending cleanly -- a viewer must be able to // tell a simulator that shut down from a stream that was asked to stop. diff --git a/src/source-hygiene.test.ts b/src/source-hygiene.test.ts index 72ff7219..2c36da11 100644 --- a/src/source-hygiene.test.ts +++ b/src/source-hygiene.test.ts @@ -149,12 +149,14 @@ const licensesClaimedOf = (name: string): string[] => { /** * why: some licenses' verbatim text never names the license -- BSD-3-Clause is a copyright line - * and three conditions -- so a name-word check would reject the verbatim text rule 3 demands. For - * those, the name is checked against the clause that makes the text that license instead: the - * third condition is what separates BSD-3-Clause from BSD-2-Clause. + * and three conditions, MIT's is a permission grant with no title -- so a name-word check would + * reject the verbatim text rule 3 demands. For those, the name is checked against the clause that + * makes the text that license instead: the third condition is what separates BSD-3-Clause from + * BSD-2-Clause, and the grant-plus-inclusion pair is what an MIT/Expat text is. */ const clauseNamingLicense: Readonly> = { "BSD-3-Clause": /Neither the name of[\s\S]+?may\s+be\s+used\s+to\s+endorse\s+or\s+promote/, + MIT: /Permission is hereby granted, free of charge[\s\S]{0,2000}?shall be included in all copies/, }; /** why: what in `text` contradicts the license `name` claims to hold, one line per mismatch. */ diff --git a/ui/src/state/screen.test.ts b/ui/src/state/screen.test.ts index 79ef4e2a..f04e86b4 100644 --- a/ui/src/state/screen.test.ts +++ b/ui/src/state/screen.test.ts @@ -4,16 +4,14 @@ import { ApiError } from "../data/api"; import { refusalOf } from "./screen"; describe("what a refused stream says", () => { - it("says which devices can be streamed when the daemon answers that this one cannot", () => { + it("says only that this device cannot be streamed, naming no platform, when the daemon answers 415", () => { const refused = new ApiError( 415, undefined, "A physical iPhone's screen cannot be streamed; only a simulator's can", ); - expect(refusalOf(refused)).toBe( - "This device cannot be streamed. A live screen is captured from a simulator or an emulator, not from a physical phone.", - ); + expect(refusalOf(refused)).toBe("This device cannot be streamed."); }); it("keeps the daemon's own reason when it named one, and says what else to check", () => { diff --git a/ui/src/state/screen.ts b/ui/src/state/screen.ts index e6aecc5f..214cbf61 100644 --- a/ui/src/state/screen.ts +++ b/ui/src/state/screen.ts @@ -85,10 +85,11 @@ export function useScreen(leaseId: string): Screen { } /** - * why: the daemon names its reason in its own words and those words are kept, because they say what its - * operator has to fix; the status only decides what to add, since a refusal the console can explain - * better than the daemon did — a device that cannot be streamed at all — would otherwise leave the - * reader with a sentence that does not say what to do. + * why: the daemon names its reason in its own words and those words reach the reader, because they + * say what its operator has to fix; the status only decides what to add or replace. A 415 is the one + * refusal whose words are not kept: every such message a driver writes names the kind of device it + * cannot stream, and which kinds can be streamed is the drivers' knowledge and not this side's, so + * the console says only that this device cannot be streamed and names no kind. */ export function refusalOf(error: unknown): string { if (!(error instanceof ApiError)) { @@ -100,7 +101,7 @@ export function refusalOf(error: unknown): string { return `${error.message} A stream is only offered where capabilities.stream.enabled is on, and only for a lease that is still held.`; } if (error.status === 415) { - return "This device cannot be streamed. A live screen is captured from a simulator or an emulator, not from a physical phone."; + return "This device cannot be streamed."; } return error.message; } diff --git a/vendor/README.md b/vendor/README.md index ab7d44a8..dbf07e9b 100644 --- a/vendor/README.md +++ b/vendor/README.md @@ -2,7 +2,7 @@ | Directory | Upstream | License | Notice | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------ | -| `screen-capture/` | `EvanBacon/serve-sim` — a capture and encode pipeline we patched, four files rewritten | Apache-2.0 (our own three files MIT) | [`screen-capture/NOTICE.md`](screen-capture/NOTICE.md) | +| `screen-capture/` | `EvanBacon/serve-sim` — a capture and encode pipeline we patched, four files rewritten; plus one HEVC capability string from `jkcoxson/idevice` (MIT) | Apache-2.0 (our own files MIT, one MIT string from idevice) | [`screen-capture/NOTICE.md`](screen-capture/NOTICE.md) | | `scrcpy/` | `Genymobile/scrcpy` — the server artifact, byte-identical, pushed unmodified to a leased emulator by the Android driver, whose client is ours in `src/drivers/android/` | Apache-2.0, plus BSD-3-Clause and BSL-1.0 code in the bundled Kotlin stdlib | [`scrcpy/NOTICE.md`](scrcpy/NOTICE.md) | A first-level directory here is third-party or derived-from-third-party code, and diff --git a/vendor/screen-capture/LICENSES/idevice.MIT.txt b/vendor/screen-capture/LICENSES/idevice.MIT.txt new file mode 100644 index 00000000..bae7492b --- /dev/null +++ b/vendor/screen-capture/LICENSES/idevice.MIT.txt @@ -0,0 +1,7 @@ +Copyright 2026 Jackson Coxson + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/screen-capture/NOTICE.md b/vendor/screen-capture/NOTICE.md index 2368bf3c..611cc9c7 100644 --- a/vendor/screen-capture/NOTICE.md +++ b/vendor/screen-capture/NOTICE.md @@ -1,9 +1,13 @@ # Vendored: serve-sim's capture and encoder -A Swift package that streams the framebuffer of one booted iOS Simulator. The -capture pipeline and the encoder came from **serve-sim** — the encoder rewritten -here, but still serve-sim's file modified; the process around them — the CLI, the -envelope, the build — is ours. Where exactly the line sits is per file, below. +A Swift package holding two programs that stream one device's screen: a +simulator's framebuffer, and a physical iPhone's media stream. The capture +pipeline and the encoders behind the simulator program came from **serve-sim** — +the encoders rewritten here, but still serve-sim's files modified; the process +around them — the CLI, the envelope, the lifecycle, the build — is ours. The +iPhone program is ours entirely, but for one HEVC capability string from +**idevice** (MIT), recorded under "What is ours". Where exactly the line sits is +per file, below. ## Upstream @@ -33,23 +37,34 @@ carries no count of our _own_ lines: a stale count still reads as a fact. ## Where every file came from -Our target is one flat directory, `Sources/simlock-screen-capture/`; upstream -keeps the same files in two targets, `Sources/SimNative/` and -`Sources/SimNativeSupport/`, and their paths are given per row. - -| File in `Sources/simlock-screen-capture/` | Upstream path | Origin | -| ----------------------------------------- | --------------------------- | --------------------------------------------------------------------------------- | -| `SimFrameworks.swift` | `Sources/SimNative/` | theirs, unchanged | -| `FramebufferSurfaceSelector.swift` | `Sources/SimNativeSupport/` | theirs, unchanged | -| `DeviceDisplaySizes.swift` | `Sources/SimNativeSupport/` | theirs, unchanged | -| `FrameCapture.swift` | `Sources/SimNative/` | theirs, **one patch** (deviation 1) | -| `H264Encoder.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | -| `VideoEncoder.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | -| `PixelBufferUtils.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | -| `Xcode.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | -| `main.swift` | — | ours, entirely: the CLI, the frame envelope, the lifecycle | -| `scripts/unframe.mjs` | — | ours, entirely: a hand-run diagnostic | -| `Package.swift` | — | ours, entirely | +The four files serve-sim wrote as they are — unchanged or with one patch — and +the four it wrote that we rewrote all live in two directories: the simulator +source files under `Sources/simlock-screen-capture/`, and the four encoders under +`Sources/SimlockCapture/` (the shared library both programs build on, which is +otherwise all ours). Upstream keeps the same files in two targets, +`Sources/SimNative/` and `Sources/SimNativeSupport/`, and their paths are given +per row. + +| File | Upstream path | Origin | +| ----------------------------------------------------------------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `simlock-screen-capture/SimFrameworks.swift` | `Sources/SimNative/` | theirs, unchanged | +| `simlock-screen-capture/FramebufferSurfaceSelector.swift` | `Sources/SimNativeSupport/` | theirs, unchanged | +| `simlock-screen-capture/DeviceDisplaySizes.swift` | `Sources/SimNativeSupport/` | theirs, unchanged | +| `simlock-screen-capture/FrameCapture.swift` | `Sources/SimNative/` | theirs, **one patch** (deviation 1) | +| `SimlockCapture/H264Encoder.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | +| `SimlockCapture/VideoEncoder.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | +| `SimlockCapture/PixelBufferUtils.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | +| `SimlockCapture/Xcode.swift` | `Sources/SimNative/` | theirs, **rewritten by us** — a modified file, Apache-2.0, `MODIFIED FILE` header | +| `SimlockCapture/Lifecycle.swift`, `SimlockCapture/Options.swift` | — | ours, entirely: the option parser, the envelope, the lifecycle, the scaler, the encode loop | +| `simlock-screen-capture/main.swift`, `simlock-screen-capture/SimulatorSource.swift` | — | ours, entirely: the simulator program and its framebuffer as a `ScreenSource` | +| `simlock-device-capture/*` (five files) | — | ours, entirely: the iPhone program — the version gate, the CoreDevice source, the RTP receiver, the decoder — rewritten from this repo's own PoC (ADR 0031) | +| `SimlockMediaBridge/*` | — | ours, entirely: the negotiator bridge; it carries the `idevice` HEVC feature string below | +| `Stubs/*` (three interfaces) | — | ours, entirely: hand-written declarations for the parts of CoreDevice we call | +| `scripts/unframe.mjs` | — | ours, entirely: a hand-run diagnostic | +| `Package.swift` | — | ours, entirely | + +To diff any of the eight serve-sim files, use the command at the end with the +upstream path from its row and the file's path under `Sources/`. Four files were rewritten rather than copied, and they are still serve-sim's files, modified: each was written from upstream's file of the same name, so each carries the @@ -186,18 +201,20 @@ curl -sL https://raw.githubusercontent.com/EvanBacon/serve-sim/6d009686ed08f7d8b | diff -u - Sources/simlock-screen-capture/FrameCapture.swift ``` -That prints three things and nothing else — the `MODIFIED FILE` header Apache §4(b) -asks for, the device-set patch, and the deleted `import SimNativeSupport` — which is -the whole claim of this notice in one screenful. If that diff ever grows past those -three, either someone edited upstream's capture logic or Apple moved something +That prints four things and nothing else — the `MODIFIED FILE` header Apache §4(b) +asks for, the device-set patch, the deleted `import SimNativeSupport`, and the +`import SimlockCapture` that replaced it — which is the whole claim of this notice +in one screenful. If that diff ever grows past those +four, either someone edited upstream's capture logic or Apple moved something upstream, and both deserve to be noticed rather than committed. For any other file in the table, substitute its upstream path in both places — for the four rewritten ones it prints a whole-file rewrite, which is what their rows say: ```sh UPSTREAM=packages/serve-sim/Sources/SimNative/SimFrameworks.swift # or SimNativeSupport/… +LOCAL=Sources/simlock-screen-capture/SimFrameworks.swift # the encoders live in Sources/SimlockCapture/ curl -sL "https://raw.githubusercontent.com/EvanBacon/serve-sim/6d009686ed08f7d8bd19bca9f9f67a9237931f44/$UPSTREAM" \ - | diff -u - "Sources/simlock-screen-capture/$(basename "$UPSTREAM")" + | diff -u - "$LOCAL" ``` Run it before and after an Xcode upgrade, and again before taking anything new @@ -224,11 +241,41 @@ URL either returns that commit's bytes or fails. ## What is ours -`main.swift`, `scripts/unframe.mjs` and `Package.swift`, plus `NOTICE.md` and the -`LICENSES/` entry's file name. Everything else in `Sources/simlock-screen-capture/` -is serve-sim's: three files unchanged, `FrameCapture.swift` with the one patch above, -and the four rewritten files, which are serve-sim's files modified by us and are -Apache-2.0 like the rest of serve-sim's code here. +`Package.swift`, `NOTICE.md` and the `LICENSES/` entries' file names, +`scripts/unframe.mjs`, the whole of `Sources/SimlockCapture/` but for the four +encoder files (`Lifecycle.swift` and `Options.swift`: the option parser, the +envelope, the lifecycle, the scaler, the encode loop), both programs' +`main.swift` and `Sources/simlock-screen-capture/SimulatorSource.swift`, the +whole of `Sources/simlock-device-capture/` and `Sources/SimlockMediaBridge/`, +and the three interfaces under `Stubs/`. Everything else is serve-sim's: three +files unchanged, `FrameCapture.swift` with the one patch above, and the four +rewritten files, which are serve-sim's files modified by us and are Apache-2.0 +like the rest of serve-sim's code here. + +### idevice (MIT): the HEVC capability string + +`SimlockMediaBridge/SimlockMediaBridge.m` replaces the feature string the offer +advertises for payload 100 (HEVC) with: + +```text +FLS;MS:-1;LF:-1;LTR;CABAC;POS:0;EOD:1;HTS:2;RR:3;AR:16/9,5/8;XR:16/9,5/8; +``` + +That string is from the **idevice** project — +(`master`, `3854a5df4a5a6dee71ffce4d8befc2ea356a8065` when taken), MIT, +`Copyright 2026 Jackson Coxson`, verbatim: +[`LICENSES/idevice.MIT.txt`](LICENSES/idevice.MIT.txt), re-fetched with: + +```sh +curl -sL https://raw.githubusercontent.com/jkcoxson/idevice/master/LICENSE.txt \ + | diff -u - LICENSES/idevice.MIT.txt +``` + +It is the receiver's documented standalone-decoder preset; every other spelling +the iPhone has been offered paints torn pictures. The file it lives in is +otherwise ours, and carrying one MIT string inside an MIT file needs no +relicensing — the notice above and the license file travel with it, which is all +the license asks. Ours is under the project's MIT license, like the rest of this repository. That is permitted rather than a relicensing: Apache-2.0 lets a licensee add its own terms to diff --git a/vendor/screen-capture/Package.swift b/vendor/screen-capture/Package.swift index b0d64de3..270cd64a 100644 --- a/vendor/screen-capture/Package.swift +++ b/vendor/screen-capture/Package.swift @@ -1,11 +1,15 @@ // swift-tools-version:6.0 // -// simlock-screen-capture — a standalone Swift CLI that streams the framebuffer of a booted -// iOS Simulator to stdout. +// Two programs that stream one device's screen on stdout, over one library: // -// Origin: see NOTICE.md for the per-file record, and LICENSES/ for the license text of the parts -// that are not ours. The private-API capture path is serve-sim's (Apache-2.0), copied from -// Sources/SimNative and Sources/SimNativeSupport with two changes: +// simlock-screen-capture the framebuffer of one booted iOS Simulator +// simlock-device-capture the HEVC media stream of one physical iPhone, through CoreDevice +// +// Both write the same envelope, take the same arguments and exit the same codes; only the +// `ScreenSource` behind them differs. See NOTICE.md for the per-file record, and LICENSES/ for the +// license text of the parts that are not ours. The simulator capture pipeline and the encoders came +// from serve-sim (Apache-2.0), copied from Sources/SimNative and Sources/SimNativeSupport with two +// changes: // // 1. `FrameCapture.findSimDevice` also searches the device set named by // SIMLOCK_IOS_DEVICE_SET. serve-sim searches only CoreSimulator's default set, and simlock @@ -19,18 +23,60 @@ // The encoders are serve-sim's files rewritten here: `H264Encoder.swift`, `VideoEncoder.swift`, // `PixelBufferUtils.swift` and `Xcode.swift` now use public VideoToolbox, CoreVideo, CoreMedia and // ImageIO, and as modified files of serve-sim they stay under Apache-2.0 (see NOTICE.md). -// -// One target, so the copied types keep their internal visibility and need no `public` edits. import PackageDescription let package = Package( name: "simlock-screen-capture", platforms: [.macOS(.v14)], targets: [ + // The shared half of both programs: the options, the envelope, the lifecycle, the scaler + // and the encoders. `public` is only what the two executables reach. + .target( + name: "SimlockCapture", + path: "Sources/SimlockCapture", + swiftSettings: [.swiftLanguageMode(.v5)] + ), .executableTarget( name: "simlock-screen-capture", + dependencies: ["SimlockCapture"], path: "Sources/simlock-screen-capture", swiftSettings: [.swiftLanguageMode(.v5)] ), + // The one piece of Objective-C in the package, and the only reason it has two languages: + // SwiftPM refuses a target that mixes Swift and Objective-C sources. + .target( + name: "SimlockMediaBridge", + path: "Sources/SimlockMediaBridge", + publicHeadersPath: "include" + ), + // A physical iPhone's stream. Only this target knows CoreDevice exists, and it links it + // `weak` (F27): on a Mac without the framework the program must still start long enough to + // print why it cannot stream, rather than dying in the loader. + // + // `-Onone` is load-bearing: the CoreDevice stubs under `Stubs/` leave enum cases out, and an + // optimizing client may treat an enum with no declared case as uninhabited. The `-I` path is + // absolute because a relative one does not resolve through the build's own working + // directory; `Context.packageDirectory` is the package's own path, so it stays put when the + // checkout moves. + .executableTarget( + name: "simlock-device-capture", + dependencies: ["SimlockCapture", "SimlockMediaBridge"], + path: "Sources/simlock-device-capture", + swiftSettings: [ + .swiftLanguageMode(.v5), + .unsafeFlags([ + "-Onone", + "-I", Context.packageDirectory + "/Stubs", + ]), + ], + linkerSettings: [ + .unsafeFlags([ + "-F", "/Library/Developer/PrivateFrameworks", + "-Xlinker", "-weak_framework", "-Xlinker", "CoreDevice", + "-Xlinker", "-weak_library", "-Xlinker", + "/Library/Developer/PrivateFrameworks/CoreDeviceUtilities.framework/Versions/A/CoreDeviceUtilities", + ]), + ] + ), ] ) diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/H264Encoder.swift b/vendor/screen-capture/Sources/SimlockCapture/H264Encoder.swift similarity index 100% rename from vendor/screen-capture/Sources/simlock-screen-capture/H264Encoder.swift rename to vendor/screen-capture/Sources/SimlockCapture/H264Encoder.swift diff --git a/vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift b/vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift new file mode 100644 index 00000000..5db6bab6 --- /dev/null +++ b/vendor/screen-capture/Sources/SimlockCapture/Lifecycle.swift @@ -0,0 +1,431 @@ +// simlock's own: the lifecycle both capture programs run — the counters a stream reports with, the +// scaler, the encode loop, and the one way out. MIT-licensed; see NOTICE.md. +import CoreVideo +import Dispatch +import Foundation +import VideoToolbox + +/// Resolved once, by whichever way out arrives first, and read by the lifecycle's one waiter. A +/// class because a signal source, the stdin thread, a failed frame write and a source's own callback +/// all have to be able to reach it, and none of them runs on an actor. +public final class OneShot: @unchecked Sendable { + private let lock = NSLock() + private var settled = false + /// Held as a closure because `Value` may itself be optional, and a `Value?` would not say whether + /// the nil is the settled value or the absence of one. + private var produce: (() -> Value)? + private var waiters: [(Value) -> Void] = [] + + public init() {} + + /// Takes the first value offered and ignores the rest, so a SIGTERM that lands while a stop is + /// already running cannot ask for a second one. + public func set(_ offered: Value) { + lock.lock() + guard !settled else { + lock.unlock() + return + } + settled = true + produce = { offered } + let pending = waiters + waiters = [] + lock.unlock() + for waiter in pending { waiter(offered) } + } + + /// The settled value, or nil while nothing has offered one. + public func offered() -> Value? { + lock.lock() + defer { lock.unlock() } + return settled ? produce?() : nil + } + + public func wait() async -> Value { + await withCheckedContinuation { (continuation: CheckedContinuation) in + lock.lock() + if let settled = produce { + lock.unlock() + continuation.resume(returning: settled()) + return + } + waiters.append { resumed in continuation.resume(returning: resumed) } + lock.unlock() + } + } +} + +/// A captured frame, handed from the source's queue to the encode queue. +struct Pending: @unchecked Sendable { + let buffer: CVPixelBuffer + /// Monotonic nanoseconds (`DispatchTime.uptimeNanoseconds`), so a wall-clock jump cannot stall + /// the fps throttle. + let atNanos: UInt64 +} + +/// How a `start` call ended: running, or the one line it gave for why it could not start. A clean +/// refusal to begin because the caller had already asked to stop is `.stopped`, which is not a reason +/// anyone should be shown — it is the end that caller asked for. +enum StartOutcome { + case running + case refused(String) + case stopped +} + +/// What the frame path mutates, behind one lock. A class rather than an actor because the source's +/// callback is synchronous and stdout needs one writer at a time. +/// +/// why: `@unchecked Sendable` because the counts and the way out are reached from the source's +/// callback, the encode loop and the signal and stdin threads, and everything read here is either +/// under `lock` or written by the encode loop alone. +final class Session: @unchecked Sendable { + private let lock = NSLock() + private var writtenFrames = 0 + private var writtenBytes = 0 + private var _viewerGone = false + var scaler: Scaler? + var scaledSize = Dimensions(width: 0, height: 0) + /// The way out every end goes through: nil for a clean stop, else the one line to print last. + let exitRequest = OneShot() + /// How the `start` call ended, set last of the two so a lifecycle that waits on this one knows it + /// is reading the final word on the start. why: kept apart from `exitRequest` because a start that + /// fails and a caller that asks to stop arrive together more often than they sound like they + /// would, and the fact the caller acts on is the reason: a program that exits 0 after failing to + /// start leaves a viewer reading "wrote nothing" where it should read why nothing was written. + let startEnded = OneShot() + + init() {} + + var counts: (frames: Int, bytes: Int) { + lock.lock() + defer { lock.unlock() } + return (writtenFrames, writtenBytes) + } + + /// Set the moment a write to the video channel fails. The encode loop stops on it, so a stream + /// the far end hung up on goes out through the same door as one the daemon asked to stop. + var viewerGoneFlag: Bool { + lock.lock() + defer { lock.unlock() } + return _viewerGone + } + + func recordWritten(_ byteCount: Int) { + lock.lock() + writtenFrames += 1 + writtenBytes += byteCount + lock.unlock() + } + + func viewerGone() { + lock.lock() + _viewerGone = true + lock.unlock() + exitRequest.set(nil) + } +} + +// MARK: - Downscale +// +// Both sources hand over a picture at the device's native size (1179x2556 on an iPhone 16). A viewer +// several hops away through the gateway does not need every pixel, and each one is paid for twice on +// the uplink. VTPixelTransferSession scales on the GPU, so the saving is bandwidth without a CPU cost +// on the worker. + +/// Not thread-safe by itself: every call happens on the single encode queue. +final class Scaler { + private var transfer: VTPixelTransferSession? + private let pool: CVPixelBufferPool + + init(targetWidth: Int, targetHeight: Int) { + let attrs: [String: Any] = [ + kCVPixelBufferWidthKey as String: targetWidth, + kCVPixelBufferHeightKey as String: targetHeight, + kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA, + kCVPixelBufferIOSurfacePropertiesKey as String: [String: Any](), + ] + var poolRef: CVPixelBufferPool? + CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, attrs as CFDictionary, &poolRef) + self.pool = poolRef! + } + + /// Scales into a fresh buffer from the pool, or returns nil when the GPU transfer fails. + func transferFrom(_ source: CVPixelBuffer) -> CVPixelBuffer? { + if transfer == nil { + var created: VTPixelTransferSession? + guard VTPixelTransferSessionCreate( + allocator: kCFAllocatorDefault, + pixelTransferSessionOut: &created + ) == noErr, let created else { return nil } + transfer = created + } + guard let transfer else { return nil } + var target: CVPixelBuffer? + guard CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pool, &target) == noErr, + let target else { return nil } + guard VTPixelTransferSessionTransferImage(transfer, from: source, to: target) == noErr else { + return nil + } + return target + } +} + +/// Scales to the configured maximum and returns a buffer safe to encode. The encoder deep-copies what +/// it is given internally, so the recycled framebuffer IOSurface is never held. +func prepareForEncode( + _ buffer: CVPixelBuffer, + options: Options, + session: Session +) -> CVPixelBuffer? { + let width = CVPixelBufferGetWidth(buffer) + let height = CVPixelBufferGetHeight(buffer) + guard options.maxDimension > 0, max(width, height) > options.maxDimension else { + return buffer + } + // why: sized once, from the first picture — a rotation changes the long edge and the scale factor + // with it, and rebuilding the scaler mid-stream is out of scope for this task (see the module's + // known gaps). + if session.scaler == nil { + let factor = Double(options.maxDimension) / Double(max(width, height)) + // Even dimensions: H.264 4:2:0 rejects an odd chroma plane. + let targetWidth = max(2, Int(Double(width) * factor) & ~1) + let targetHeight = max(2, Int(Double(height) * factor) & ~1) + session.scaler = Scaler(targetWidth: targetWidth, targetHeight: targetHeight) + session.scaledSize = Dimensions(width: targetWidth, height: targetHeight) + log("scaling \(width)x\(height) -> \(targetWidth)x\(targetHeight)") + } + return session.scaler?.transferFrom(buffer) +} + +// MARK: - The encode loop + +/// Runs the encode loop until the viewer goes away. `newest(1)` on the stream means a slow encoder or +/// a relay that stops draining drops old frames instead of queueing them without bound, which is the +/// behaviour a live view needs. +/// +/// The first frame forces an IDR so the viewer that just asked for the stream can start decoding +/// immediately rather than waiting for the encoder's own keyframe interval; after that VideoToolbox +/// spaces IDRs itself (`MaxKeyFrameInterval = fps * 5`, from serve-sim's encoder). +func encodeLoop(_ frames: AsyncStream, options: Options, session: Session) async { + let intervalNanos = UInt64(1_000_000_000 / Double(options.fps)) + var lastSent: UInt64 = 0 + var started = false + var metaEmitted = false + let encoder = H264Encoder(fps: options.fps, bitrate: options.bitrate) + let jpegEncoder = VideoEncoder(quality: 0.7) + for await frame in frames { + if started, frame.atNanos &- lastSent < intervalNanos { continue } + lastSent = frame.atNanos + let forceKeyframe = !started + started = true + + guard let prepared = prepareForEncode(frame.buffer, options: options, session: session) else { + continue + } + // The meta frame rides the first picture because that is the first moment the encoded + // geometry (post-scale) is known. + if !metaEmitted { + metaEmitted = true + let meta: [String: Any] = [ + "udid": options.udid, + "codec": options.codec == "avcc" ? "h264" : "mjpeg", + "width": CVPixelBufferGetWidth(prepared), + "height": CVPixelBufferGetHeight(prepared), + "fps": options.fps, + "deviceSet": ProcessInfo.processInfo.environment["SIMLOCK_IOS_DEVICE_SET"] ?? "default", + ] + if let json = try? JSONSerialization.data(withJSONObject: meta), + let string = String(data: json, encoding: .utf8) { + log("ready \(string)") + Wire.emit(.meta, Data(string.utf8), countedBy: session) + } + } + if options.codec == "mjpeg" { + if let data = try? await jpegEncoder.encode(pixelBuffer: prepared) { + Wire.emit(.delta, data, countedBy: session) + } + } else if let encoded = try? await encoder.encode( + prepared, + forceKeyframe: forceKeyframe + ) { + if let description = encoded.description { + Wire.emit(.description, description, countedBy: session) + } + Wire.emit( + encoded.kind == .keyframe ? .keyframe : .delta, + encoded.avcc, + countedBy: session + ) + } + if session.viewerGoneFlag { return } + } +} + +// MARK: - The lifecycle + +/// Signals and stdin, on dispatch sources rather than C signal handlers so what runs there is +/// ordinary Swift and is allowed to stop a source. +final class ExitSources { + private var sources: [DispatchSourceSignal] = [] + private let queue = DispatchQueue(label: "exit-sources") + + init(onExit: @escaping @Sendable (_ reason: String?) -> Void) { + for sig in [SIGTERM, SIGINT, SIGHUP] { + signal(sig, SIG_IGN) + let source = DispatchSource.makeSignalSource(signal: sig, queue: queue) + source.setEventHandler { onExit(nil) } + source.resume() + sources.append(source) + } + // stdin is a liveness handle: the daemon that closes it is the daemon that asked for stop. A + // plain thread reads it because a blocked `read` has no async form. + Thread.detachNewThread { + var byte: UInt8 = 0 + while read(0, &byte, 1) > 0 {} + log("stdin closed") + onExit(nil) + } + } +} + +/// How long the lifecycle gives a `start` that is still in flight to report its last word once the +/// program is on its way out. +/// +/// why: the caller's own grace is 2 s and a source's `stop()` bounds itself to 1.5 s, so this leaves +/// the program inside the grace it was given: past it, staying alive buys nothing the caller is not +/// already willing to take by force. +let unwindBoundSeconds: TimeInterval = 0.4 + +/// Waits for `work` to finish, and gives up on it when `bound` has passed. +private func unwound(_ work: Task, within bound: TimeInterval) async { + // why: a one-shot rather than a task group, because a group waits for every child it started + // before its body ends — which would wait for the very call this bound exists to walk away from. + let ended = OneShot() + DispatchQueue.global().asyncAfter(deadline: .now() + bound) { ended.set(false) } + Task { + await work.value + ended.set(true) + } + _ = await ended.wait() +} + +/// Reports the running totals every few seconds, so a caller's log carries what a stream cost even +/// though a stream ended by a viewer going away has no chance to report at exit. +func watchStats(_ session: Session, startedAt: Date) { + Thread.detachNewThread { + var lastFrames = 0 + var lastBytes = 0 + while true { + Thread.sleep(forTimeInterval: 5) + let counted = session.counts + let secs = max(0.001, Date.now.timeIntervalSince(startedAt)) + log( + "STATS {\"fps\": \((Double(counted.frames - lastFrames) / 5 * 10).rounded() / 10), " + + "\"kbps\": \(Int(Double(counted.bytes - lastBytes) * 8 / 5 / 1000)), " + + "\"totalFrames\": \(counted.frames), \"totalBytes\": \(counted.bytes), " + + "\"avgKbps\": \(Int(Double(counted.bytes) * 8 / secs / 1000)), " + + "\"rssMb\": \(Int(residentBytes() / 1_048_576))}" + ) + lastFrames = counted.frames + lastBytes = counted.bytes + } + } +} + +/// Writes the `RESULT` line a `--duration` run promises, out of the counted totals. +func writeResultLine(_ session: Session, startedAt: Date) { + let secs = max(0.001, Date.now.timeIntervalSince(startedAt)) + let counted = session.counts + let stats: [String: Any] = [ + "seconds": (secs * 100).rounded() / 100, + "frames": counted.frames, + "bytes": counted.bytes, + "fps": (Double(counted.frames) / secs * 10).rounded() / 10, + "kbps": ((Double(counted.bytes) * 8 / secs) / 1000).rounded(), + "avgFrameBytes": counted.frames == 0 ? 0 : Int(Double(counted.bytes) / Double(counted.frames)), + "rssMb": Int(residentBytes() / 1_048_576), + ] + if let json = try? JSONSerialization.data(withJSONObject: stats) { + log("RESULT \(String(data: json, encoding: .utf8)!)") + } +} + +/// Runs one capture: start the source, encode what it hands over, and take the first way out that +/// arrives — always `stop()` first, then the exit it calls for. +/// +/// Every way out goes through here, because a capture that leaves its device-side state running is +/// the one thing this program must never do, and a bare `_exit` from a write path or a signal +/// handler skips exactly the cleanup that would have cleared it. +public func runCapture(options: Options, source: any ScreenSource) async -> Never { + let session = Session() + let signals = ExitSources(onExit: { reason in session.exitRequest.set(reason) }) + let startedAt = Date.now + watchStats(session, startedAt: startedAt) + if options.duration > 0 { + Task.detached { + try? await Task.sleep(nanoseconds: UInt64(options.duration * 1_000_000_000)) + writeResultLine(session, startedAt: startedAt) + session.exitRequest.set(nil) + } + } + + let (frames, sink) = AsyncStream.makeStream( + of: Pending.self, + bufferingPolicy: .bufferingNewest(1) + ) + let encoding = Task { await encodeLoop(frames, options: options, session: session) } + + // why: `start` runs on a task of its own rather than being awaited here. The way out most likely + // to arrive while it is still running is the caller's own — a stop closes stdin, and the caller + // SIGKILLs the program two seconds later — and a start awaited on this line swallows that stop + // until the source's own deadline passes, so a stream that was asked to stop dies by SIGKILL with + // the iPhone's media stream left running. Every way out still goes through the one door below. + let starting = Task { + do { + try await source.start( + deviceID: options.udid, + onFrame: { buffer in + sink.yield(Pending(buffer: buffer, atNanos: DispatchTime.now().uptimeNanoseconds)) + }, + onEnd: { reason in session.exitRequest.set(reason) } + ) + log("capture started") + session.startEnded.set(.running) + } catch is StoppedBeforeStart { + // The caller asked for this end before the source got going, which is the clean stop it + // asked for; naming it would turn a stop into a reason a viewer reads as a failure. + log("start stopped before it began") + session.startEnded.set(.stopped) + session.exitRequest.set(nil) + } catch { + // What the source opened is already closed again by the time it throws. The reason goes + // out as the program's last line, whatever else was already on its way out. + log("start failed: \(error)") + let message = "\(error)" + session.startEnded.set(.refused(message)) + session.exitRequest.set(message) + } + } + + let reason = await session.exitRequest.wait() + encoding.cancel() + // why: `stop()` is the whole way back, and is never skipped. It is what marks the stop a start in + // flight polls for, and it closes what the source has opened up to now. + await source.stop() + // why: then the start is given a bounded moment to land, because until it has the exit code this + // program is about to take is undecided — a start that fails as the caller asks to stop is the + // common case, not the rare one, and settling the code before it throws is a coin flip between + // the reason and a clean 0. Bounded because a caller SIGKILLs a program that outlives its grace + // and a CoreDevice call inside a start cannot be cancelled from here; past the bound this program + // goes out as a SIGKILL would, which the module's manual names as a known gap. + if session.startEnded.offered() == nil { + await unwound(starting, within: unwindBoundSeconds) + } + withExtendedLifetime(signals) {} + switch session.startEnded.offered() ?? .running { + case .refused(let message): fail(message) + case .running, .stopped: break + } + if let reason { fail(reason) } + exit(0) +} diff --git a/vendor/screen-capture/Sources/SimlockCapture/Options.swift b/vendor/screen-capture/Sources/SimlockCapture/Options.swift new file mode 100644 index 00000000..00c7c31d --- /dev/null +++ b/vendor/screen-capture/Sources/SimlockCapture/Options.swift @@ -0,0 +1,182 @@ +// simlock's capture programs share one library: the option parser, the frame envelope, the +// lifecycle, the stats line, the scaler and the encode loop. Only the `ScreenSource` differs between +// them, so the whole shape around a source lives here and each program's `main.swift` is the source +// it picks plus the gate that names it. +// +// This file is simlock's own, MIT-licensed; the encoders beside it are serve-sim's modified files — +// see NOTICE.md. +import CoreVideo +import Foundation + +/// The stream's settings, exactly as the daemon passes them. +public struct Options: Sendable { + public var udid = "" + public var codec = "avcc" + public var fps = 30 + public var bitrate = 2_000_000 + public var maxDimension = 720 + public var duration = 0.0 + + public init() {} +} + +/// nil for anything the parser cannot make sense of, which is a usage error (exit 64). +public func parseOptions(_ args: [String]) -> Options? { + var o = Options() + var positionals: [String] = [] + var i = 0 + while i < args.count { + let a = args[i] + func value() -> String? { + i += 1 + return i < args.count ? args[i] : nil + } + switch a { + case "--codec": guard let v = value() else { return nil }; o.codec = v + case "--fps": guard let v = value(), let n = Int(v) else { return nil }; o.fps = max(1, min(60, n)) + case "--bitrate": guard let v = value(), let n = Int(v) else { return nil }; o.bitrate = max(100_000, n) + case "--max-dimension": guard let v = value(), let n = Int(v) else { return nil }; o.maxDimension = max(0, n) + case "--duration": guard let v = value(), let s = Double(v) else { return nil }; o.duration = max(0, s) + default: + if a.hasPrefix("--") { return nil } + positionals.append(a) + } + i += 1 + } + guard positionals.count == 1 else { return nil } + o.udid = positionals[0] + guard o.codec == "avcc" || o.codec == "mjpeg" else { return nil } + return o +} + +/// One line of text a source cannot go beyond: it is what the daemon turns into the reason a viewer +/// reads in a 503, and what the program prints as its last stderr line. +public struct CaptureError: Error, CustomStringConvertible, Sendable { + public let message: String + public var description: String { message } + + public init(_ message: String) { + self.message = message + } +} + +/// What a source throws when it did not start because the caller had already asked it to stop. It is +/// not a reason a viewer needs: the caller asked for this, and it is the clean end it asked for. +public struct StoppedBeforeStart: Error { + public init() {} +} + +/// What hands decoded pictures to the encode loop. The simulator's framebuffer is one +/// implementation, CoreDevice's media stream is the second, and ADR 0031 names CoreMediaIO (the +/// QuickTime path) as the third. Each program picks one in its `main.swift`; there is no registry. +public protocol ScreenSource: AnyObject { + /// Brings the source up for `deviceID` and returns once it is running — before its first + /// picture, which is what the stream's first-frame bound measures to. + /// + /// Throws a `CaptureError` when it cannot start. + func start( + deviceID: String, + onFrame: @escaping @Sendable (CVPixelBuffer) -> Void, + onEnd: @escaping @Sendable (_ reason: String?) -> Void + ) async throws + + /// Idempotent, and back within 1.5 s whatever the device is doing — a close the daemon waits on + /// must not become the reason a lease outlives its viewer. + func stop() async +} + +/// serve-sim's AVCC envelope (`Sources/SimNative/StreamFormat.swift`) plus this tool's meta tag. +public enum Tag: UInt8 { + case description = 0x01 + case keyframe = 0x02 + case delta = 0x03 + case seed = 0x04 + case meta = 0x05 +} + +/// The video channel: a private duplicate of the stdout the program was started with. +public enum Wire { + /// Set by `claimStdout`, which every program's `main.swift` calls as its first statement. + public private(set) static var fd: Int32 = -1 + + /// Reserves the program's original stdout for the bitstream and points the inherited fd 1 at + /// stderr, so text that any code in this process writes to stdout — the copied capture code logs + /// with `print`, and frameworks do too — cannot cut a hole in the bitstream. + @discardableResult + public static func claimStdout() -> Int32 { + guard fd < 0 else { return fd } + let reserved = dup(STDOUT_FILENO) + guard reserved >= 0 else { + fatalError("could not duplicate stdout") + } + dup2(STDERR_FILENO, STDOUT_FILENO) + fd = reserved + return reserved + } + + /// Writes one envelope frame to the video channel: length prefix (covering the tag and payload), + /// tag, payload. A partial write would desynchronize every frame after it, so a short one is + /// retried until the frame is out; the only way a frame stops part-way is a failed write, which + /// ends the stream. Only the encode loop writes, which is why this is not `public`. + static func emit(_ tag: Tag, _ data: Data, countedBy session: Session) { + let total = UInt32(data.count + 1) + var out = Data(capacity: data.count + 5) + out.append(UInt8((total >> 24) & 0xff)) + out.append(UInt8((total >> 16) & 0xff)) + out.append(UInt8((total >> 8) & 0xff)) + out.append(UInt8(total & 0xff)) + out.append(tag.rawValue) + out.append(data) + session.recordWritten(out.count) + out.withUnsafeBytes { pointer in + var written = 0 + let base = pointer.baseAddress! + while written < out.count { + let n = write(fd, base + written, out.count - written) + if n <= 0 { + // EPIPE is the viewer going away, which is a normal end of a stream and the only + // way one can end without anyone asking. + session.viewerGone() + return + } + written += n + } + } + } +} + +public func log(_ message: String) { + FileHandle.standardError.write("[capture] \(message)\n".data(using: .utf8)!) +} + +/// One line on why the program is stopping, as the very last thing on stderr, then exit 1. The +/// daemon reads that line as the stream's reason. +public func fail(_ reason: String) -> Never { + fail(1, reason) +} + +/// As above, with the exit code the reason calls for. +/// +/// why: the daemon reads this program's *last* stderr line as the reason a viewer sees, and a +/// reason a framework handed back with newlines in it would arrive there as its last line only — +/// so the one line promised here is what goes out, with any inner break flattened. +public func fail(_ code: Int32, _ reason: String) -> Never { + let oneLine = reason + .split(whereSeparator: { $0.isNewline }) + .joined(separator: " / ") + FileHandle.standardError.write("\(oneLine)\n".data(using: .utf8)!) + exit(code) +} + +/// Resident set of this process, in bytes. This is the number that answers "what does one open +/// stream cost the worker", so it is measured rather than estimated. +public func residentBytes() -> UInt64 { + var info = mach_task_basic_info() + var count = mach_msg_type_number_t(MemoryLayout.size) / 4 + let status = withUnsafeMutablePointer(to: &info) { pointer -> kern_return_t in + pointer.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { + task_info(mach_task_self_, task_flavor_t(MACH_TASK_BASIC_INFO), $0, &count) + } + } + return status == KERN_SUCCESS ? info.resident_size : 0 +} diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/PixelBufferUtils.swift b/vendor/screen-capture/Sources/SimlockCapture/PixelBufferUtils.swift similarity index 97% rename from vendor/screen-capture/Sources/simlock-screen-capture/PixelBufferUtils.swift rename to vendor/screen-capture/Sources/SimlockCapture/PixelBufferUtils.swift index 1c138666..69f5ecc0 100644 --- a/vendor/screen-capture/Sources/simlock-screen-capture/PixelBufferUtils.swift +++ b/vendor/screen-capture/Sources/SimlockCapture/PixelBufferUtils.swift @@ -38,14 +38,13 @@ extension CVPixelBuffer { /// few allocations instead of creating one per frame. That is the only reason this /// holds state; the pool is rebuilt whenever the frame size changes, which happens on /// a rotation. -struct Photocopier { +public struct Photocopier { private var pool: CVPixelBufferPool? /// The frame size `pool` was built for, so a size change is noticed rather than /// silently copied into a buffer of the wrong dimensions. private var poolSize: Dimensions? - init() {} - + public init() {} /// The pool that fits `size`, rebuilding it when the one on hand does not. private mutating func pool(for size: Dimensions) -> CVPixelBufferPool? { if let pool, poolSize == size { @@ -70,7 +69,7 @@ struct Photocopier { /// A buffer of our own holding the same pixels as `source`, or nil when no copy /// could be made. - mutating func copy(_ source: CVPixelBuffer) -> CVPixelBuffer? { + public mutating func copy(_ source: CVPixelBuffer) -> CVPixelBuffer? { guard let pool = pool(for: source.dimensions) else { return nil } var fresh: CVPixelBuffer? guard CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pool, &fresh) == kCVReturnSuccess, diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/VideoEncoder.swift b/vendor/screen-capture/Sources/SimlockCapture/VideoEncoder.swift similarity index 100% rename from vendor/screen-capture/Sources/simlock-screen-capture/VideoEncoder.swift rename to vendor/screen-capture/Sources/SimlockCapture/VideoEncoder.swift diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/Xcode.swift b/vendor/screen-capture/Sources/SimlockCapture/Xcode.swift similarity index 97% rename from vendor/screen-capture/Sources/simlock-screen-capture/Xcode.swift rename to vendor/screen-capture/Sources/SimlockCapture/Xcode.swift index 77edf8bc..074fd23d 100644 --- a/vendor/screen-capture/Sources/simlock-screen-capture/Xcode.swift +++ b/vendor/screen-capture/Sources/SimlockCapture/Xcode.swift @@ -8,7 +8,7 @@ import Foundation /// The active developer directory — where this tool looks for the simulator's /// frameworks, and the one thing about the host machine it has to ask rather than know. -enum Xcode { +public enum Xcode { /// The directory `xcode-select` points at (`xcode-select -p`), e.g. /// `/Applications/Xcode.app/Contents/Developer`. /// @@ -17,7 +17,7 @@ enum Xcode { /// is a guess about somebody's machine. When the question cannot be asked at all — /// `xcode-select` is not where it is expected to be — the answer falls back to the /// standard install location, which is the more likely answer than no answer. - static func developerDir() -> String { + public static func developerDir() -> String { guard let output = run("/usr/bin/xcode-select", ["-p"]), let path = String(data: output, encoding: .utf8)?.trimmingCharacters(in: .whitespacesAndNewlines) else { diff --git a/vendor/screen-capture/Sources/SimlockMediaBridge/SimlockMediaBridge.m b/vendor/screen-capture/Sources/SimlockMediaBridge/SimlockMediaBridge.m new file mode 100644 index 00000000..89a010d6 --- /dev/null +++ b/vendor/screen-capture/Sources/SimlockMediaBridge/SimlockMediaBridge.m @@ -0,0 +1,221 @@ +#import "SimlockMediaBridge.h" +#import +#import +#import +#import +#import +#import +#import +#import + +// Two things happen here that a Swift file cannot do: reach AVConference's negotiator through the +// runtime, and override what that negotiator advertises about HEVC without touching a system +// framework. Both affect this process only. + +@interface NSObject (SimlockAVCMediaStreamNegotiator) +- (instancetype)initWithMode:(NSInteger)mode options:(NSDictionary *)options error:(NSError **)error; +- (id)offer; +- (BOOL)setAnswer:(NSData *)answer withError:(NSError **)error; +- (id)generateMediaStreamConfigurationWithError:(NSError **)error; +- (id)dictionary; +@end + +// Payload 100 is HEVC. DeviceHub advertises `FLS;VRAE:0;SW:1;` for it, and the idevice receiver +// documents torn pictures with that preset and advertises the string below for its standalone HEVC +// decoder instead. Only payload 100 is replaced; H.264 and every other part of the offer goes out as +// the framework built it. The string is from the MIT-licensed idevice project (see NOTICE.md). +static NSString *const kSimlockStandaloneHEVCFeatures = + @"FLS;MS:-1;LF:-1;LTR;CABAC;POS:0;EOD:1;HTS:2;RR:3;AR:16/9,5/8;XR:16/9,5/8;"; + +static IMP simlockOriginalScreenFeatures; + +static NSDictionary *simlockScreenDecoderFeatures(id self, SEL selector) { + NSDictionary *original = + ((NSDictionary *(*)(id, SEL))simlockOriginalScreenFeatures)(self, selector); + NSMutableDictionary *features = [original mutableCopy]; + if (features[@100]) { + features[@100] = kSimlockStandaloneHEVCFeatures; + } + return features; +} + +static BOOL simlockLTRPDisabled(id self, SEL selector) { + return NO; +} + +static void simlockDisableLTRPGetter(NSString *className) { + Method getter = class_getInstanceMethod( + NSClassFromString(className), @selector(ltrpEnabled)); + if (getter == NULL) { + NSLog(@"negotiator: %@ has no ltrpEnabled", className); + return; + } + method_setImplementation(getter, (IMP)simlockLTRPDisabled); +} + +// why: the offer is built inside -initWithMode:options:error: out of the settings preset and the +// local configuration, so both getters are overridden before the negotiator exists — and only for +// this process, which is the difference between this and modifying a system framework. +static BOOL simlockInstallNegotiatorHooks(void) { + static dispatch_once_t once; + __block BOOL installed = NO; + dispatch_once(&once, ^{ + Class settings = + NSClassFromString(@"AVCMediaStreamNegotiatorSettingsCoreDeviceScreenSharing"); + Method features = class_getInstanceMethod(settings, @selector(featureListString)); + if (features == NULL) { + NSLog(@"negotiator: HEVC capability setting unavailable"); + return; + } + simlockOriginalScreenFeatures = method_setImplementation(features, (IMP)simlockScreenDecoderFeatures); + installed = YES; + NSLog(@"negotiator: standalone-decoder HEVC capabilities enabled"); + simlockDisableLTRPGetter(@"AVCMediaStreamNegotiatorSettings"); + simlockDisableLTRPGetter(@"VCMediaNegotiatorLocalConfiguration"); + NSLog(@"negotiator: LTRP disabled for the offer"); + }); + return installed; +} + +NSObject *simlockNegotiator(NSDictionary *options, NSInteger mode) { + dlopen("/System/Library/PrivateFrameworks/AVConference.framework/AVConference", RTLD_NOW); + if (!simlockInstallNegotiatorHooks()) { + return nil; + } + Class negotiatorClass = NSClassFromString(@"AVCMediaStreamNegotiator"); + if (negotiatorClass == nil) { + NSLog(@"negotiator: AVCMediaStreamNegotiator is not present"); + return nil; + } + NSError *error = nil; + id negotiator = [(id)[negotiatorClass alloc] initWithMode:mode options:options error:&error]; + if (error != nil) { + NSLog(@"negotiator error: %@", error); + } + return negotiator; +} + +NSData *simlockOffer(NSObject *negotiator) { + id offer = [negotiator offer]; + return [offer isKindOfClass:NSData.class] ? offer : nil; +} + +NSDictionary *simlockAnswer(NSObject *negotiator, NSData *answer) { + NSError *error = nil; + if (![negotiator setAnswer:answer withError:&error]) { + NSLog(@"answer error: %@", error); + return nil; + } + id configuration = [negotiator generateMediaStreamConfigurationWithError:&error]; + if (error != nil) { + NSLog(@"configuration error: %@", error); + return nil; + } + id dictionary = [configuration dictionary]; + return [dictionary isKindOfClass:NSDictionary.class] ? dictionary : nil; +} + +// MARK: - The receiver socket + +// One socket receives at a time, and the one bound for the tunnel route is not it for long: that one +// exists to name the address and port `startMediaStream` is asked to send to, so it is put down once +// the stream has answered and `simlockAdopt` gives the receiver to the socket CoreDevice shares for the +// stream. `send` on a socket that was bound but never connected fails with EDESTADDRREQ, so while the +// reserved-port socket held the receiver every RTCP report, key-frame request and bye went nowhere. +static int simlockReceiverFd = -1; +static uint16_t simlockReceiverBoundPort = 0; + +// The iPhone sends to the host's own address on the tunnel's route, so that address is asked of the +// routing table rather than written down: a tunnel address is per-pairing and per-transport. +NSString *simlockBindForTunnel(NSString *tunnelIPAddress) { + int route = socket(AF_INET6, SOCK_DGRAM, 0); + struct sockaddr_in6 destination = {0}; + struct sockaddr_in6 source = {0}; + destination.sin6_len = sizeof(destination); + destination.sin6_family = AF_INET6; + destination.sin6_port = htons(9); + if (inet_pton(AF_INET6, tunnelIPAddress.UTF8String, &destination.sin6_addr) != 1) { + close(route); + return nil; + } + if (connect(route, (struct sockaddr *)&destination, sizeof(destination)) != 0) { + close(route); + return nil; + } + socklen_t length = sizeof(source); + getsockname(route, (struct sockaddr *)&source, &length); + close(route); + + char address[INET6_ADDRSTRLEN]; + inet_ntop(AF_INET6, &source.sin6_addr, address, sizeof(address)); + + simlockReceiverFd = socket(AF_INET6, SOCK_DGRAM, 0); + source.sin6_port = 0; + if (bind(simlockReceiverFd, (struct sockaddr *)&source, sizeof(source)) != 0) { + simlockCloseReceiver(); + return nil; + } + length = sizeof(source); + getsockname(simlockReceiverFd, (struct sockaddr *)&source, &length); + simlockReceiverBoundPort = ntohs(source.sin6_port); + int bufferSize = 4 * 1024 * 1024; + setsockopt(simlockReceiverFd, SOL_SOCKET, SO_RCVBUF, &bufferSize, sizeof(bufferSize)); + return @(address); +} + +uint16_t simlockReceiverPort(void) { + return simlockReceiverBoundPort; +} + +void simlockReleaseBoundSocket(void) { + if (simlockReceiverFd < 0) { + return; + } + close(simlockReceiverFd); + simlockReceiverFd = -1; +} + +NSData *simlockReceive(int milliseconds) { + if (simlockReceiverFd < 0) { + return nil; + } + struct pollfd waiter = {simlockReceiverFd, POLLIN, 0}; + if (poll(&waiter, 1, milliseconds) <= 0) { + return nil; + } + unsigned char buffer[65536]; + ssize_t received = recv(simlockReceiverFd, buffer, sizeof(buffer), 0); + return received > 0 ? [NSData dataWithBytes:buffer length:(NSUInteger)received] : nil; +} + +BOOL simlockSend(NSData *packet) { + if (simlockReceiverFd < 0) { + return NO; + } + return send(simlockReceiverFd, packet.bytes, packet.length, 0) == (ssize_t)packet.length; +} + +BOOL simlockAdopt(xpc_object_t sockets) { + xpc_dictionary_apply(sockets, ^bool(const char *key, xpc_object_t value) { + if (xpc_get_type(value) != XPC_TYPE_FD) { + return true; + } + int descriptor = xpc_fd_dup(value); + if (simlockReceiverFd < 0) { + simlockReceiverFd = descriptor; + int bufferSize = 4 * 1024 * 1024; + setsockopt(simlockReceiverFd, SOL_SOCKET, SO_RCVBUF, &bufferSize, sizeof(bufferSize)); + } else { + close(descriptor); + } + return true; + }); + return simlockReceiverFd >= 0; +} + +void simlockCloseReceiver(void) { + if (simlockReceiverFd >= 0) { + close(simlockReceiverFd); + simlockReceiverFd = -1; + } +} diff --git a/vendor/screen-capture/Sources/SimlockMediaBridge/include/SimlockMediaBridge.h b/vendor/screen-capture/Sources/SimlockMediaBridge/include/SimlockMediaBridge.h new file mode 100644 index 00000000..08b06379 --- /dev/null +++ b/vendor/screen-capture/Sources/SimlockMediaBridge/include/SimlockMediaBridge.h @@ -0,0 +1,50 @@ +#import +#import + +NS_ASSUME_NONNULL_BEGIN + +/// The parts of AVConference's media-stream negotiator the iPhone source needs, plus the socket it +/// receives the stream on. All of it is reached through the Objective-C runtime rather than a +/// header, because Apple ships none for this framework. +/// +/// It is a separate target because SwiftPM refuses a target that mixes Swift and Objective-C +/// sources. Only `simlock-device-capture` links it. + +/// Installs this process's own HEVC capability overrides (see the .m) and builds the negotiator in +/// `mode` (5 is the media-stream mode) from `options`. nil when AVConference cannot be loaded. +NSObject * _Nullable simlockNegotiator(NSDictionary *options, NSInteger mode); + +/// The negotiator's offer, the SDP-shaped blob `startMediaStream` is asked to send. +NSData * _Nullable simlockOffer(NSObject *negotiator); + +/// Applies the iPhone's answer and returns the resulting media-stream configuration, or nil when +/// the answer was refused. +NSDictionary * _Nullable simlockAnswer(NSObject *negotiator, NSData *answer); + +/// Binds a UDP socket on the host's own address on the route to the iPhone's tunnel, and returns +/// that address. Its port is the one `startMediaStream` is asked to send to. +NSString * _Nullable simlockBindForTunnel(NSString *tunnelIPAddress); + +/// The port `simlockBindForTunnel` landed on, 0 when no socket was bound. +uint16_t simlockReceiverPort(void); + +/// Puts down the socket bound for the tunnel route, keeping its port. `simlockAdopt` hands the receiver +/// to the socket CoreDevice shares for the stream, asked for on that same address and port. +void simlockReleaseBoundSocket(void); + +/// One datagram from the receiver, or nil when none arrived within `milliseconds`. +NSData * _Nullable simlockReceive(int milliseconds); + +/// Sends one RTCP packet on the receiver. +BOOL simlockSend(NSData *packet); + +/// Takes the first descriptor CoreDevice hands back for this stream as the receiver; the rest are +/// closed. NO when the dictionary carried none, which leaves this process with nothing to receive on. +/// The PoC streamed and reported over this socket with a plain `send`, so it is the one the iPhone's +/// sender answers. +BOOL simlockAdopt(xpc_object_t sockets); + +/// Closes the receiver, which is what unblocks a `simlockReceive` that is waiting. +void simlockCloseReceiver(void); + +NS_ASSUME_NONNULL_END diff --git a/vendor/screen-capture/Sources/simlock-device-capture/CoreDeviceVersion.swift b/vendor/screen-capture/Sources/simlock-device-capture/CoreDeviceVersion.swift new file mode 100644 index 00000000..db0e4a91 --- /dev/null +++ b/vendor/screen-capture/Sources/simlock-device-capture/CoreDeviceVersion.swift @@ -0,0 +1,66 @@ +// simlock's own, MIT-licensed (see NOTICE.md). +// +// The one place the CoreDevice version pin is enforced (architecture rule: one place enforces a +// rule). It runs before any CoreDevice call, because this program links the framework `weak` and a +// newer CoreDevice is exactly the case where calling into it is the thing that must not happen. +import Foundation + +enum CoreDeviceVersion { + /// The major this program's stubs were written against. A new Xcode major turns iPhone streams + /// off until `Stubs/` is checked against that release and this number moves with it. + static let supportedMajor = "642" + + static let requirement = + "Physical iPhone streams need Xcode 27 (CoreDevice \(supportedMajor).x); " + + static let infoPlist = + "/Library/Developer/PrivateFrameworks/CoreDevice.framework/Resources/Info.plist" + + /// The version string the framework reports, or why it could not be read. + enum Reading { + case version(String) + case absent + case unreadable + } + + /// `SIMLOCK_COREDEVICE_INFO_PLIST` exists for the slow lane, which cannot install a different + /// CoreDevice onto the Mac it runs on. The driver never passes it, and strips it from a daemon's + /// environment, so no operator's configuration can point the pin at another file. + static func read(environment: [String: String] = ProcessInfo.processInfo.environment) -> Reading { + let path = environment["SIMLOCK_COREDEVICE_INFO_PLIST"] ?? infoPlist + guard FileManager.default.fileExists(atPath: path) else { return .absent } + guard let contents = FileManager.default.contents(atPath: path), + let plist = try? PropertyListSerialization.propertyList(from: contents, format: nil), + let dictionary = plist as? [String: Any], + case let value? = dictionary["CFBundleShortVersionString"] as? String + else { + return .unreadable + } + return .version(value) + } + + /// The text before the first `.`, with surrounding spaces trimmed; the whole string when it + /// carries no dot. `6420.1` is therefore `6420`, not `642` — the comparison is on the major, not + /// on a prefix. + static func major(of version: String) -> String { + let trimmed = version.trimmingCharacters(in: .whitespacesAndNewlines) + guard let dot = trimmed.firstIndex(of: ".") else { return trimmed } + return String(trimmed[.. String? { + switch reading { + case .absent: + return requirement + "this Mac has no CoreDevice" + case .unreadable: + return requirement + "this Mac's CoreDevice version cannot be read" + case .version(let value): + guard major(of: value) == supportedMajor else { + return requirement + + "this Mac has CoreDevice \(value.trimmingCharacters(in: .whitespacesAndNewlines))" + } + return nil + } + } +} diff --git a/vendor/screen-capture/Sources/simlock-device-capture/DeviceScreenSource.swift b/vendor/screen-capture/Sources/simlock-device-capture/DeviceScreenSource.swift new file mode 100644 index 00000000..91ba9d7f --- /dev/null +++ b/vendor/screen-capture/Sources/simlock-device-capture/DeviceScreenSource.swift @@ -0,0 +1,753 @@ +// simlock-device-capture's source: one physical iPhone's screen, through CoreDevice's media stream. +// +// The steps, in the order the PoC proved them (ADR 0031): find the device, take a usage assertion so +// coredeviced loads its services, build an offer whose HEVC capabilities say "standalone decoder", +// start the stream to a local socket routed to the iPhone's tunnel, receive it as RTP, decode it with +// VideoToolbox. The deadline below is the only clock the start respects; every step shares it, so no +// step adds its own timeout on top of the stream's own first-frame bound. +// +// The shape of these calls is this repo's own PoC (docs/internal/adr/0031); the HEVC capability string +// is idevice's (MIT). See NOTICE.md. +import CoreDevice +import CoreVideo +import Foundation +import Network +import SimlockCapture +import SimlockMediaBridge + +/// What CoreDevice's usage assertion needs loaded before it will serve a media stream. DeviceHub takes +/// the same states for its own mirroring; without an assertion of our own the media stream's support +/// query fails on a Mac where DeviceHub is not already holding the device's services open. +let requiredDeviceStates: Set = [ + .developerModeEnabledIfRequired, + .coreDeviceServicesLoaded, + .extendedDeviceInfoLoaded, + .remoteServiceDiscoveryTrustedConnectivityAvailable, + .secureNetworkingAvailable, +] + +/// Launch to the first decoded picture. The stream module gives a capture 30 s to write its meta +/// frame, so this has to end inside that — 25 s leaves the encode loop 5 s to produce a picture, and a +/// reason that names the step that ran out of time is worth more to a user than "wrote nothing". +let startDeadlineSeconds: TimeInterval = 25 + +/// How often the device list is re-read, and the start call and the first picture are re-checked. +private let pollSeconds: TimeInterval = 0.25 + +/// The whole stop's bound. It is shorter than the daemon's own 2 s grace on purpose: the program that +/// gives up on an unreachable iPhone is the one that ends its stream cleanly, and a close that hangs +/// past the grace is SIGKILLed with the iPhone's stream left up. +private let stopBoundSeconds: TimeInterval = 1.5 + +/// This program's monotonic clock. A wall-clock jump must not extend or cut a stream's deadline. +func uptime() -> TimeInterval { ProcessInfo.processInfo.systemUptime } + +/// A CoreDevice call that has to be watched rather than awaited: awaiting its `Task` would let this +/// side's own deadline cancel a call that cannot be taken back, and a `startMediaStream` that completes +/// after its deadline started a stream whose id is the only thing that can stop it. +final class CallBox: @unchecked Sendable { + private let lock = NSLock() + private var result: Result? + + var settled: Result? { + lock.lock() + defer { lock.unlock() } + return result + } + + func complete(with offered: Result) { + lock.lock() + result = offered + lock.unlock() + } +} + +/// One physical iPhone's media stream, as a `ScreenSource`. +/// +/// A class behind one lock rather than an actor: the receive loop runs on its own thread for the life +/// of the stream, and a `stop()` that must be back inside 1.5 s cannot be built on a queue that may be +/// parked in the very CoreDevice call it is waiting for. Every lock use is in a synchronous helper, so +/// no async context ever holds the lock. +final class DeviceScreenSource: ScreenSource, @unchecked Sendable { + /// The one clock for launch → first decoded picture, shared by every step and by the receive loop. + let deadline: TimeInterval + + private let lock = NSLock() + private var stopping = false + private var byeWanted = false + private var device: RemoteDevice? + private var assertion: DeviceUsageAssertion? + private var streamID: UInt32? + private var startingCall: CallBox? + private var receiveLoopStarted = false + private var loopFinished = false + private var onEnd: ((String?) -> Void)? + + /// Signalled by the receive loop when it ends, whichever way. + private let loopEnded = DispatchSemaphore(value: 0) + + init() { + deadline = uptime() + startDeadlineSeconds + } + + // MARK: - ScreenSource + + func start( + deviceID: String, + onFrame: @escaping @Sendable (CVPixelBuffer) -> Void, + onEnd: @escaping @Sendable (_ reason: String?) -> Void + ) async throws { + setOnEnd(onEnd) + + do { + try refuseIfStopping() + + let device = try await findDevice(udid: deviceID) + try refuseIfStopping() + + let support = try await takeAssertionAndQuerySupport(of: device, deviceID: deviceID) + try refuseIfStopping() + + let tunnel = try tunnelAddress(of: device, deviceID: deviceID) + guard let local = simlockBindForTunnel(tunnel) else { + throw CaptureError("No route to iPhone \(deviceID)'s tunnel on this Mac") + } + try refuseIfStopping() + + let (negotiator, options, offer) = try makeOffer(for: device, deviceID: deviceID) + let response = try await startStream( + on: device, + offer: offer, + options: options, + support: support, + localAddress: local, + deviceID: deviceID + ) + try refuseIfStopping() + + let remoteSSRC = try applyAnswer(response, with: negotiator, deviceID: deviceID) + try await beginReceiving( + remoteSSRC: remoteSSRC, + answer: response, + localAddress: local, + onFrame: onFrame, + deviceID: deviceID + ) + } catch { + // Whatever this source opened is closed again before the reason goes out, so a start that + // was refused leaves nothing running on the iPhone. + await closeEverything() + throw error + } + } + + func stop() async { + if markStopping() { log("stopping the device capture") } + await closeEverything() + } + + // MARK: - The steps + + /// The iPhone's entry in CoreDevice's device list, re-read until it shows or the deadline passes. + /// An iPhone that is plugged in but not yet trusted appears only after a person answers on its + /// screen, so waiting is the correct behaviour and refusing at once is not. + private func findDevice(udid: String) async throws -> RemoteDevice { + let wanted = udid.lowercased() + while true { + try refuseIfStopping() + if let found = CoreDeviceCalls.allDevices().first(where: { + $0.deviceInfo.udid?.lowercased() == wanted + }) { + note(device: found) + return found + } + guard uptime() < deadline else { + throw CaptureError("iPhone \(udid) is not connected to this Mac") + } + try await nap(until: min(deadline, uptime() + pollSeconds)) + } + } + + /// The step everything else depends on: with no assertion of our own, coredeviced has no reason to + /// have this device's developer services loaded, and the media stream's support query then fails + /// with nothing in its message to say why. + private func takeAssertionAndQuerySupport( + of device: RemoteDevice, + deviceID: String + ) async throws -> DeviceMediaStream.SupportInfoResponse { + // The two awaits below carry no timeout of their own: the stream module's own 30 s + // first-frame bound ends a capture that never produces a picture, and whatever coredeviced + // answers after that finds the program already on its way out. + let assertion: DeviceUsageAssertion + do { + assertion = try await CoreDeviceCalls.acquireAssertion(of: device) + } catch { + throw CaptureError("Could not use iPhone \(deviceID): \(error)") + } + note(assertion: assertion) + watchInvalidation(of: assertion, deviceID: deviceID) + + do { + return try await CoreDeviceCalls.mediaStreamSupport(of: device) + } catch { + throw CaptureError("iPhone \(deviceID) does not offer a media stream: \(error)") + } + } + + private func tunnelAddress(of device: RemoteDevice, deviceID: String) throws -> String { + guard let tunnel = device.deviceInfo.tunnelIPAddressString, !tunnel.isEmpty else { + throw CaptureError("iPhone \(deviceID) has no paired tunnel on this Mac") + } + return tunnel + } + + /// The offer, built in this process by AVConference's own negotiator from CoreDevice's options for + /// this device, so the answer the iPhone sends back is one this side can apply. The same options go + /// back into `startMediaStream`. The HEVC feature string and the long-term reference pictures are + /// this process's own overrides, installed by the bridge before the negotiator exists. + private func makeOffer( + for device: RemoteDevice, + deviceID: String + ) throws -> (negotiator: NSObject, options: [String: CodableValue], offer: Data) { + var values = MediaStreamHelper.getNegotiatorOptions(for: device.deviceInfo) + values[DeviceMediaStream.kAVCMediaStreamOptionClientName] = "CoreDeviceScreenSharing" + values[DeviceMediaStream.kAVCMediaStreamOptionClientSessionID] = UUID() + values["avcMediaStreamOptionCallID"] = UUID().uuidString + guard let negotiator = simlockNegotiator(values, 5) else { + throw CaptureError("AVConference's media stream negotiator is not available on this Mac") + } + guard let offer = simlockOffer(negotiator) as Data? else { + throw CaptureError("Could not build a media offer for iPhone \(deviceID)") + } + // Built one at a time rather than with `compactMapValues`: the stubs leave CoreDevice's own + // enum cases out, and a `CodableValue` that cannot be built is the answer wanted here. + var options: [String: CodableValue] = [:] + for (key, value) in values { + if let coded = CodableValue(value) { options[key] = coded } + } + return (negotiator, options, offer) + } + + private func startStream( + on device: RemoteDevice, + offer: Data, + options: [String: CodableValue], + support: DeviceMediaStream.SupportInfoResponse, + localAddress: String, + deviceID: String + ) async throws -> DeviceMediaStream.StartResponse { + let request = DeviceMediaStream.StartRequest( + timeout: UInt(max(1, min(15, deadline - uptime()))), + type: .video, + direction: .output, + negotiatorOffer: offer, + clientSupportedFeatures: support.client.supportedFeatures, + receiverIP: localAddress, + receiverPort: simlockReceiverPort(), + options: options + ) + + let box = CallBox() + note(startingCall: box) + Task { + do { + box.complete(with: .success(try await CoreDeviceCalls.startStream(on: device, request))) + } catch { + box.complete(with: .failure(error)) + } + } + + // Watched with the deadline rather than cancelled: a call that returns after its deadline + // started a stream on the iPhone, and its answer carries the only id this process will ever + // have for it. + while true { + if let settled = box.settled { + clearStartingCall() + switch settled { + case .success(let response): + // why: the id is taken from the answer here rather than in `applyAnswer`, because a + // stop that lands the moment this call returns refuses the start before it gets + // there — and the refusal's way back stops what it knows the id of, or leaves this + // iPhone's stream up for CoreDevice to notice. + if let id = UInt32(exactly: response.connection.streamConfig.StreamInputID) { + note(streamID: id) + } + return response + case .failure(let error): + throw CaptureError("iPhone \(deviceID)'s stream did not start: \(error)") + } + } + guard uptime() < deadline else { + throw CaptureError("No video from iPhone \(deviceID) within \(Int(startDeadlineSeconds)) s") + } + try await nap(until: min(deadline, uptime() + pollSeconds)) + } + } + + /// Applies the iPhone's answer and takes from its configuration what this side needs. The stream's + /// id is recorded first, so the exit path every refusal below leads to stops the stream this + /// program just started rather than leaving it up. + private func applyAnswer( + _ response: DeviceMediaStream.StartResponse, + with negotiator: NSObject, + deviceID: String + ) throws -> UInt32 { + guard let streamID = UInt32(exactly: response.connection.streamConfig.StreamInputID) else { + throw CaptureError("iPhone \(deviceID)'s stream carried no usable id") + } + note(streamID: streamID) + + guard let configuration = simlockAnswer(negotiator, response.negotiatorAnswer as Data) else { + throw CaptureError("iPhone \(deviceID)'s media answer was refused by this Mac") + } + guard let suite = configuration["vcMediaStreamSRTPCipherSuite"] as? NSNumber, + suite.intValue == 0 + else { + throw CaptureError( + "iPhone \(deviceID) offered an encrypted media stream, which this program cannot read" + ) + } + // The iPhone's own RTP synchronization source, which every RTCP packet is addressed to. Read + // through `Encodable` rather than as a struct layout — the shape of a private type is the + // thing a hand-written stub is most likely to have wrong. The encoder asked is the + // connection, not the whole answer: the configuration is a part of the connection, so it is + // one level below the answer's own keys. + guard + let json = try? JSONSerialization.jsonObject(with: JSONEncoder().encode(response.connection)), + let object = json as? [String: Any], + let streamConfiguration = object["streamConfig"] as? [String: Any], + let remoteSSRC = (streamConfiguration["RemoteSSRC"] as? NSNumber)?.uint32Value + else { + throw CaptureError("iPhone \(deviceID)'s stream configuration was not readable") + } + return remoteSSRC + } + + /// Starts the receive loop. The source is running from here on, which is where `start` returns: + /// the first picture is what the start deadline measures to, and only the loop learns whether one + /// ever arrives, so the loop is what ends the source when none does. + private func beginReceiving( + remoteSSRC: UInt32, + answer response: DeviceMediaStream.StartResponse, + localAddress: String, + onFrame: @escaping @Sendable (CVPixelBuffer) -> Void, + deviceID: String + ) async throws { + // The port this program bound for the tunnel route only ever named the address `startMediaStream` + // was asked to send to, so it is put down now that the answer is in hand. What receives from here + // is the socket CoreDevice shares for the stream, which is the one the iPhone's sender is the + // peer of and the only one an RTCP report can be written from. + simlockReleaseBoundSocket() + guard let local = IPv6Address(localAddress), + let remote = IPv6Address(response.connection.sender.ipv6Address) + else { + throw CaptureError("iPhone \(deviceID)'s stream endpoints were not readable") + } + let adopted: Bool + do { + adopted = try NetworkUtils.withSharedSocketDictionary( + sourceAddress: local, + port: response.connection.receiver.port, + remoteAddress: remote, + remotePort: response.connection.sender.port + ) { sockets in + simlockAdopt(sockets) + } + } catch { + throw CaptureError("iPhone \(deviceID)'s stream socket was not shared: \(error)") + } + guard adopted else { + throw CaptureError("iPhone \(deviceID)'s stream shared no socket to receive on") + } + + let loop = ReceiveLoop(remoteSSRC: remoteSSRC, onFrame: onFrame, source: self) + markReceiveLoopStarted() + DispatchQueue(label: "device-capture-receive").async { loop.run(deviceID: deviceID) } + } + + // MARK: - What the loop and the lifecycle share + + var wantsBye: Bool { + lock.lock() + defer { lock.unlock() } + return byeWanted + } + + var isStopping: Bool { + lock.lock() + defer { lock.unlock() } + return stopping + } + + /// The loop's own end, reported once: nil for a clean end, else the reason. After the first picture + /// the source does not end on silence — a still iPhone sends nothing, and an iPhone that goes away + /// locks or is unplugged, which the driver's own watcher turns into the end of the lease. What ends + /// the source from here is its socket dying, or the assertion being invalidated. + func loopStopped(reason: String?) { + lock.lock() + let first = !loopFinished + loopFinished = true + let report = onEnd + lock.unlock() + loopEnded.signal() + guard first else { return } + report?(reason) + } + + // MARK: - The lock's own helpers + + private func setOnEnd(_ handler: @escaping (String?) -> Void) { + lock.lock() + onEnd = handler + lock.unlock() + } + + private func note(device found: RemoteDevice) { + lock.lock() + device = found + lock.unlock() + } + + private func note(assertion: DeviceUsageAssertion) { + lock.lock() + self.assertion = assertion + lock.unlock() + } + + private func note(streamID id: UInt32) { + lock.lock() + streamID = id + lock.unlock() + } + + private func note(startingCall box: CallBox) { + lock.lock() + startingCall = box + lock.unlock() + } + + private func clearStartingCall() { + lock.lock() + startingCall = nil + lock.unlock() + } + + private func takeStartingCall() -> CallBox? { + lock.lock() + defer { lock.unlock() } + return startingCall + } + + /// The handles the way back stops and releases, taken off the source as they are read. A `start` + /// that fails cleans what it opened, and a stop the caller asked for while that start was running + /// asks for the same way back; whoever takes a handle first is the one that closes it, so the same + /// assertion is never released and the same stream never stopped twice. + private func takeTargets() -> (device: RemoteDevice?, assertion: DeviceUsageAssertion?, streamID: UInt32?) { + lock.lock() + defer { lock.unlock() } + let opened = (device, assertion, streamID) + device = nil + assertion = nil + streamID = nil + return opened + } + + private func markReceiveLoopStarted() { + lock.lock() + receiveLoopStarted = true + lock.unlock() + } + + private var ownsReceiveLoop: Bool { + lock.lock() + defer { lock.unlock() } + return receiveLoopStarted && !loopFinished + } + + private func markStopping() -> Bool { + lock.lock() + defer { lock.unlock() } + let first = !stopping + stopping = true + byeWanted = true + return first + } + + private func refuseIfStopping() throws { + // why: `StoppedBeforeStart` rather than a reason — a stop is the caller asking for this end, + // and a program that prints a reason for the stop it was asked for hands a viewer a 503 about + // a stream nobody failed. + guard !isStopping else { throw StoppedBeforeStart() } + } + + // MARK: - The way back + + /// Every exit ends here: this program's own stream stopped by its own id, this program's own + /// assertion released, the socket closed. Nothing else on the iPhone is touched — DeviceHub's + /// stream for the same device is not this program's to stop. + private func closeEverything() async { + let bound = uptime() + stopBoundSeconds + + // A start that is still in flight is watched for its id, within the same bound. When it does + // not return in time this program exits without it, like a SIGKILL would; when it does, its + // stream id goes on the list of things to stop below. + if let pending = takeStartingCall() { + while pending.settled == nil, uptime() < bound { + try? await Task.sleep(nanoseconds: UInt64(pollSeconds * 1_000_000_000)) + } + if case let .success(response)? = pending.settled, + let id = UInt32(exactly: response.connection.streamConfig.StreamInputID) { + note(streamID: id) + } + clearStartingCall() + } + + // why: on a queue of its own rather than a `Task`, because everything below blocks and the + // CoreDevice call it waits for needs one of the pool's threads to answer — a blocking close + // on that pool is one thread short of the answer it is waiting for. The bound is what keeps + // the close the daemon is waiting on from turning into a SIGKILL. + await withCheckedContinuation { (continuation: CheckedContinuation) in + DispatchQueue(label: "device-capture-close").async { + self.closeBlockedParts(by: bound) + continuation.resume() + } + } + } + + /// The blocking half of the way back: the two waits, and the calls that may never be answered. + private func closeBlockedParts(by bound: TimeInterval) { + let opened = takeTargets() + if opened.device != nil || opened.assertion != nil || opened.streamID != nil { + let cleaned = DispatchSemaphore(value: 0) + Task.detached { + if let device = opened.device, let streamID = opened.streamID { + do { + // why: `stopAll: false` with this program's own id — the only thing on the + // iPhone this program is allowed to stop. + _ = try await CoreDeviceCalls.stopStream(on: device, id: streamID) + log("iPhone stream \(streamID) stopped") + } catch { + // An iPhone that is already gone cannot answer, which is the same end state as + // a stream that was stopped, and the lease is over either way. + log("iPhone stream \(streamID) did not stop cleanly: \(error)") + } + } + opened.assertion?.releaseAssertion() + cleaned.signal() + } + // Bounded by the same deadline: an iPhone that has gone away cannot answer + // `stopMediaStream`, and a close the daemon is waiting on must not become a SIGKILL. + if cleaned.wait(timeout: .now() + max(0, bound - uptime())) == .timedOut { + log("device capture cleanup gave up") + } + } + // Closing the receiver is also what unblocks a read the loop is parked in. Only a source that + // ever started one has a loop to wait for: a start refused before the socket opened would + // otherwise spend the whole stop bound waiting for a thread that was never created. + let waitingForLoop = ownsReceiveLoop + simlockCloseReceiver() + if waitingForLoop, loopEnded.wait(timeout: .now() + max(0, bound - uptime())) == .timedOut { + log("device capture receiver did not stop within \(stopBoundSeconds) s") + } + } + + /// coredeviced invalidates the assertion when the connection goes, and that is a fact this source + /// learns only from here. A release this source asked for itself arrives as a nil error. + private func watchInvalidation(of assertion: DeviceUsageAssertion, deviceID: String) { + assertion.registerInvalidationHandler(invokingOnQueue: .main) { [weak self] error in + guard let self, let error, !self.isStopping else { return } + self.loopStopped(reason: "iPhone \(deviceID)'s device connection was lost: \(error)") + } + } +} + +private func nap(until moment: TimeInterval) async throws { + let seconds = moment - uptime() + guard seconds > 0 else { return } + try await Task.sleep(nanoseconds: UInt64(seconds * 1_000_000_000)) +} + +// MARK: - The receive loop + +/// The loop that owns the socket: read, reorder, decode, hand over, and report back to the iPhone. +/// +/// One thread, because its parts have to stay in step with each other — a packet that is reordered, +/// dropped, or turned into a picture has to be seen by the RTCP report and by the key-frame request +/// exactly as the decoder saw it. +private final class ReceiveLoop: @unchecked Sendable { + private let remoteSSRC: UInt32 + private let onFrame: @Sendable (CVPixelBuffer) -> Void + private unowned let source: DeviceScreenSource + + private let parser = HEVCRTPReceiver() + private let decoder = HEVCDecoder() + /// The last decoded picture, re-emitted while nothing new arrives. The simulator source does the + /// same for the same reason: a still iPhone sends nothing, and a stream that sends nothing looks + /// dead from the viewer's side. + private var lastPicture: CVPixelBuffer? + private var lastEmitted = uptime() + private var lastReport = 0.0 + private var lastKeyframeRequest = 0.0 + private var decoded = 0 + + init( + remoteSSRC: UInt32, + onFrame: @escaping @Sendable (CVPixelBuffer) -> Void, + source: DeviceScreenSource + ) { + self.remoteSSRC = remoteSSRC + self.onFrame = onFrame + self.source = source + parser.onAccessUnit = { [weak self] unit in + self?.decoder.decode(unit.nals, timestamp: unit.timestamp) + } + parser.onLoss = { [weak self] in self?.decoder.invalidate() } + decoder.onFrame = { [weak self] pixel in self?.emit(pixel) } + decoder.onFailure = { message in log("HEVC: \(message)") } + decoder.onConfiguration = { message in log(message) } + } + + func run(deviceID: String) { + var timedOut = false + while true { + if source.wantsBye { + _ = simlockSend(parser.report(receiver: remoteSSRC, bye: true)) + break + } + if let datagram = simlockReceive(5) { + parser.receive(datagram as Data) + } + // why: the window is flushed and the report written on the clock rather than only when a + // packet arrives — the two moments a report is worth sending are the two where the iPhone + // has stopped sending: a picture held in the reorder window with nothing behind it, and a + // decoder still waiting for its first key frame. + parser.flushPending() + let now = uptime() + if reportIsDue(now: now) { report(now: now) } + if !keepAlive() { break } + // The one clock from launch to the first decoded picture, which `start` has already + // returned under: past it the stream would answer 200 and show nothing, so the source ends + // and names the step that ran out. + if decoded == 0, uptime() >= source.deadline { + timedOut = true + break + } + } + let reason: String? + if decoded > 0 || source.isStopping { + reason = nil + } else if timedOut { + reason = "No video from iPhone \(deviceID) within \(Int(startDeadlineSeconds)) s" + } else { + reason = "iPhone \(deviceID) sent no video this program could decode" + } + source.loopStopped(reason: reason) + } + + /// Re-emits the last picture at least every 200 ms, and says whether the loop should go on. + private func keepAlive() -> Bool { + let now = uptime() + if now - lastEmitted >= 0.2, let lastPicture { + lastEmitted = now + onFrame(lastPicture) + } + return !source.isStopping + } + + /// Whether a report is worth sending now: once a second, or sooner while the decoder still has no + /// picture to start from, because a key-frame request is at most twice a second (F20). Nothing is + /// reported before the iPhone's own synchronization source is known — a report speaks about + /// packets received from a source, and a picture loss indication has no one to ask. + private func reportIsDue(now: TimeInterval) -> Bool { + guard parser.source != nil else { return false } + return now - lastReport >= 1 || (decoder.needsKeyframe && now - lastKeyframeRequest >= 0.5) + } + + /// One receiver report, and a picture loss indication while the decoder still has no picture to + /// start from. That request is at most twice a second: an iPhone that cannot serve the key frame + /// is not helped by being asked harder. + private func report(now: TimeInterval) { + let requesting = decoder.needsKeyframe && now - lastKeyframeRequest >= 0.5 + guard simlockSend(parser.report(receiver: remoteSSRC, requestKeyframe: requesting)) else { + return + } + lastReport = now + if requesting { lastKeyframeRequest = now } + } + + /// Counts the pictures this loop has handed over, on the loop's own thread: the decode callback is + /// synchronous, so this and the first-picture check in `run` cannot disagree. + private func emit(_ pixel: CVPixelBuffer) { + decoded += 1 + lastPicture = pixel + lastEmitted = uptime() + onFrame(pixel) + } +} + +// MARK: - CoreDevice + +/// Every CoreDevice call, on the main thread under a run loop of this program's own. The framework's +/// client is a main-thread API in places and a command-line program has no UI run loop to hand it over, +/// so a call that wants one gets a hop to the main queue and its answer resumed from there. +enum CoreDeviceCalls { + private static let manager = DeviceManager(serviceConnection: CoreDeviceService.sharedConnection) + + static func allDevices() -> [RemoteDevice] { + if Thread.isMainThread { return manager.allDevices() } + return DispatchQueue.main.sync { manager.allDevices() } + } + + static func acquireAssertion(of device: RemoteDevice) async throws -> DeviceUsageAssertion { + try await onMain { + try await device.acquireDeviceUsageAssertion( + withReason: "simlock screen stream", + requirements: UsageAssertionRequirements( + deviceStates: requiredDeviceStates, + capabilities: [] + ) + ) + } + } + + static func mediaStreamSupport( + of device: RemoteDevice + ) async throws -> DeviceMediaStream.SupportInfoResponse { + try await onMain { try await device.mediaStreamSupportInfo } + } + + static func startStream( + on device: RemoteDevice, + _ request: DeviceMediaStream.StartRequest + ) async throws -> DeviceMediaStream.StartResponse { + try await onMain { try await device.startMediaStream(with: request) } + } + + static func stopStream( + on device: RemoteDevice, + id: UInt32 + ) async throws -> DeviceMediaStream.StopResponse { + try await onMain { + try await device.stopMediaStream( + with: DeviceMediaStream.StopRequest(stopAll: false, identifiers: [id]) + ) + } + } +} + +/// Runs an async CoreDevice call from the main thread and resumes wherever its answer arrives. +private func onMain( + _ work: @escaping () async throws -> Value +) async throws -> Value { + try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in + DispatchQueue.main.async { + Task { + do { + continuation.resume(returning: try await work()) + } catch { + continuation.resume(throwing: error) + } + } + } + } +} diff --git a/vendor/screen-capture/Sources/simlock-device-capture/HEVCDecoder.swift b/vendor/screen-capture/Sources/simlock-device-capture/HEVCDecoder.swift new file mode 100644 index 00000000..67fc7d78 --- /dev/null +++ b/vendor/screen-capture/Sources/simlock-device-capture/HEVCDecoder.swift @@ -0,0 +1,275 @@ +// simlock's own, MIT-licensed (see NOTICE.md). +// +// HEVCDecoder — turns the receiver's access units into pixel buffers with VideoToolbox. Everything +// here is public Apple API: by the time a packet reaches this file the private part of the path is +// over, and what arrives is an ordinary elementary stream. +import CoreMedia +import CoreVideo +import Foundation +import SimlockCapture +import VideoToolbox + +final class HEVCDecoder { + var onFrame: (CVPixelBuffer) -> Void = { _ in } + var onFailure: (String) -> Void = { _ in } + var onConfiguration: (String) -> Void = { _ in } + + /// No decoder can start mid-picture: until the first random-access point arrives, everything the + /// receiver hands over is a delta against a reference nobody holds. + private(set) var needsKeyframe = true + private var parameterSets: [UInt8: Data] = [:] + private var format: CMVideoFormatDescription? + private var session: VTDecompressionSession? + /// Whether the current GOP was entered at an open-GOP recovery point, whose leading pictures + /// reference the past that was discarded. + private var discardingLeadingPictures = false + + deinit { + if let session { VTDecompressionSessionInvalidate(session) } + } + + /// Drops the decoder's whole reference picture set and asks for a fresh key frame. Used on + /// transport loss, on a parameter-set change, and on any decode failure: after any of them the + /// session's references are no longer something this side can vouch for. + func invalidate() { + needsKeyframe = true + if let session { VTDecompressionSessionInvalidate(session) } + session = nil + } + + func decode(_ nals: [Data], timestamp: UInt32) { + var picture: [Data] = [] + var randomAccess = false + var parameterSetsChanged = false + var sliceTypes: [UInt8] = [] + + for nal in nals { + guard nal.count >= 2 else { continue } + let type = (nal[nal.startIndex] >> 1) & 63 + if (32...34).contains(type) { + // VPS, SPS, PPS. A changed set means the stream's shape moved under the session. + if parameterSets[type] != nal { + parameterSets[type] = nal + parameterSetsChanged = true + } + } else { + picture.append(nal) + if type < 32 { sliceTypes.append(type) } + // 16...21 are the IRAP types: the only pictures safe to start from. + if (16...21).contains(type) { randomAccess = true } + } + } + + if parameterSetsChanged { + invalidate() + format = nil + } + guard !sliceTypes.isEmpty else { return } + guard !needsKeyframe || randomAccess else { return } + // RASL pictures (types 8 and 9) following an open-GOP recovery point may reference the + // discarded past, so they pass by until the first non-RASL slice says the GOP is + // self-contained. + if discardingLeadingPictures, sliceTypes.contains(where: { $0 == 8 || $0 == 9 }) { return } + if needsKeyframe, randomAccess { + discardingLeadingPictures = true + } else if sliceTypes.contains(where: { $0 < 6 }) { + discardingLeadingPictures = false + } + + do { + if session == nil { try configure() } + guard let session, let format else { return } + let sample = try compressedSample( + picture: picture, + timestamp: timestamp, + format: format, + randomAccess: randomAccess + ) + try decodeSample(sample, with: session) + } catch { + invalidate() + onFailure("\(error)") + } + } + + // MARK: - Building a compressed sample + + /// The access unit in the length-prefixed form the format description was built for + /// (`nalUnitHeaderLength: 4`). + private func compressedSample( + picture: [Data], + timestamp: UInt32, + format: CMVideoFormatDescription, + randomAccess: Bool + ) throws -> CMSampleBuffer { + var bytes = Data() + for nal in picture { + bytes.append(contentsOf: HEVCRTPReceiver.word(UInt32(nal.count))) + bytes.append(nal) + } + + var block: CMBlockBuffer? + try checked( + CMBlockBufferCreateWithMemoryBlock( + allocator: kCFAllocatorDefault, + memoryBlock: nil, + blockLength: bytes.count, + blockAllocator: kCFAllocatorDefault, + customBlockSource: nil, + offsetToData: 0, + dataLength: bytes.count, + flags: 0, + blockBufferOut: &block + ), + "block allocation" + ) + guard let block else { throw CaptureError("no block buffer") } + try bytes.withUnsafeBytes { raw in + try checked( + CMBlockBufferReplaceDataBytes( + with: raw.baseAddress!, + blockBuffer: block, + offsetIntoDestination: 0, + dataLength: bytes.count + ), + "block copy" + ) + } + + // The RTP timestamp is 90 kHz for this stream; VideoToolbox only needs it to increase. + var timing = CMSampleTimingInfo( + duration: .invalid, + presentationTimeStamp: CMTime(value: Int64(timestamp), timescale: 90_000), + decodeTimeStamp: .invalid + ) + var sizes = [bytes.count] + var sample: CMSampleBuffer? + try checked( + CMSampleBufferCreateReady( + allocator: kCFAllocatorDefault, + dataBuffer: block, + formatDescription: format, + sampleCount: 1, + sampleTimingEntryCount: 1, + sampleTimingArray: &timing, + sampleSizeEntryCount: 1, + sampleSizeArray: &sizes, + sampleBufferOut: &sample + ), + "compressed sample" + ) + guard let sample else { throw CaptureError("no sample buffer") } + + // CoreMedia assumes sync samples unless told otherwise. A P or B picture depends on decoded + // references, and describing it as a random-access point is what makes a viewer show a + // picture it should have waited for. + if let attachments = CMSampleBufferGetSampleAttachmentsArray(sample, createIfNecessary: true) { + let dictionary = unsafeBitCast( + CFArrayGetValueAtIndex(attachments, 0), + to: CFMutableDictionary.self + ) + let dependsOnOthers = randomAccess ? kCFBooleanFalse! : kCFBooleanTrue! + CFDictionarySetValue( + dictionary, + Unmanaged.passUnretained(kCMSampleAttachmentKey_NotSync).toOpaque(), + Unmanaged.passUnretained(dependsOnOthers).toOpaque() + ) + CFDictionarySetValue( + dictionary, + Unmanaged.passUnretained(kCMSampleAttachmentKey_DependsOnOthers).toOpaque(), + Unmanaged.passUnretained(dependsOnOthers).toOpaque() + ) + } + return sample + } + + /// Synchronously, so decoded pictures and the reference state stay on the queue that fed the + /// decoder; the encode loop downstream of `onFrame` is the only hand-off. The callback's own + /// status is this session's only report of what happened to a picture, so it goes to the log. + private func decodeSample( + _ sample: CMSampleBuffer, + with session: VTDecompressionSession + ) throws { + let status = VTDecompressionSessionDecodeFrame( + session, + sampleBuffer: sample, + flags: [], + infoFlagsOut: nil + ) { status, _, image, _, _ in + if status != noErr { + log("HEVC: decode of one picture failed with OSStatus \(status)") + } + guard let image else { return } + self.needsKeyframe = false + self.onFrame(image) + } + try checked(status, "decode submission") + } + + // MARK: - The session + + private func configure() throws { + guard let vps = parameterSets[32], let sps = parameterSets[33], let pps = parameterSets[34] else { + throw CaptureError("no parameter sets yet") + } + let status = vps.withUnsafeBytes { v in + sps.withUnsafeBytes { s in + pps.withUnsafeBytes { p in + let pointers = [ + v.bindMemory(to: UInt8.self).baseAddress!, + s.bindMemory(to: UInt8.self).baseAddress!, + p.bindMemory(to: UInt8.self).baseAddress!, + ] + return CMVideoFormatDescriptionCreateFromHEVCParameterSets( + allocator: kCFAllocatorDefault, + parameterSetCount: 3, + parameterSetPointers: pointers, + parameterSetSizes: [vps.count, sps.count, pps.count], + nalUnitHeaderLength: 4, + extensions: nil, + formatDescriptionOut: &format + ) + } + } + } + try checked(status, "HEVC format") + guard let format else { throw CaptureError("no format description") } + + // Hardware is allowed, not required: this Mac may already be encoding several other streams, + // and VideoToolbox's own choice is the one that knows what the machine has left. + let specification: [CFString: Any] = [ + kVTVideoDecoderSpecification_EnableHardwareAcceleratedVideoDecoder: true, + ] + try checked( + VTDecompressionSessionCreate( + allocator: kCFAllocatorDefault, + formatDescription: format, + decoderSpecification: specification as CFDictionary, + imageBufferAttributes: [kCVPixelBufferIOSurfacePropertiesKey: [:]] as CFDictionary, + outputCallback: nil, + decompressionSessionOut: &session + ), + "decoder creation" + ) + guard let session else { throw CaptureError("no decompression session") } + VTSessionSetProperty(session, key: kVTDecompressionPropertyKey_RealTime, value: kCFBooleanTrue) + // why: `Unmanaged`, because the property is written through a `void *` out-parameter that this + // side owns after the call — an `Optional` slot takes the pointer as a bare address, + // which is what the compiler warns about, and nothing would ever release it. + var usingHardware: Unmanaged? + let hardwareStatus = VTSessionCopyProperty( + session, + key: kVTDecompressionPropertyKey_UsingHardwareAcceleratedVideoDecoder, + allocator: kCFAllocatorDefault, + valueOut: &usingHardware + ) + if hardwareStatus == 0, let enabled = usingHardware?.takeRetainedValue() as? Bool { + onConfiguration("HEVC decoder: \(enabled ? "hardware" : "software")") + } + } + + private func checked(_ status: OSStatus, _ operation: String) throws { + guard status != 0 else { return } + throw CaptureError("\(operation): OSStatus \(status)") + } +} diff --git a/vendor/screen-capture/Sources/simlock-device-capture/HEVCRTPReceiver.swift b/vendor/screen-capture/Sources/simlock-device-capture/HEVCRTPReceiver.swift new file mode 100644 index 00000000..9b66264a --- /dev/null +++ b/vendor/screen-capture/Sources/simlock-device-capture/HEVCRTPReceiver.swift @@ -0,0 +1,342 @@ +// simlock's own, MIT-licensed (see NOTICE.md). +// +// Turns the iPhone's RTP datagrams into whole HEVC access units, and builds the RTCP packets back to +// it. The payload format is RFC 7798 (single NAL, aggregation packet, fragmentation unit, without +// DONL). The receiver is written against what this one stream needs: one source, and a damaged +// picture dropped whole rather than decoded half-assembled. +import Foundation + +/// One RTP picture, as NAL units with the RTP timestamp it carried. +typealias AccessUnit = (nals: [Data], timestamp: UInt32) + +/// Reorders within a short window and assembles access units. Loss discards the affected unit. +final class HEVCRTPReceiver { + var onAccessUnit: (AccessUnit) -> Void = { _ in } + /// Called when the pictures already in flight can no longer be trusted, so the decoder can ask + /// for a fresh key frame instead of painting a torn one. + var onLoss: () -> Void = {} + + private(set) var source: UInt32? + private(set) var lossEvents = 0 + private(set) var reorderedPackets = 0 + private(set) var latePackets = 0 + + /// A packet that arrives out of order is held this long before the gap below it is accepted as + /// real. Long enough for ordinary reordering, short enough that a lost packet is noticed inside + /// one frame period. + private let reorderWindow: TimeInterval = 0.025 + /// A damaged unit is never handed over, so the byte ceiling is a runaway guard, not a policy. + private let maxUnitBytes = 8 * 1_024 * 1_024 + + private var pending: [UInt16: (data: Data, arrival: TimeInterval)] = [:] + private var nextExpected: UInt16? + private var timestamp: UInt32? + private var fragment = Data() + private var nals: [Data] = [] + private var unitBytes = 0 + private var damaged = false + + // Receiver-report bookkeeping (RFC 3550 §6.4.1). + private var receivedPackets: UInt32 = 0 + private var highestSequence: UInt32 = 0 + private var baseSequence: UInt32 = 0 + private var previousExpected: UInt32 = 0 + private var previousReceived: UInt32 = 0 + private var lastSenderReport: UInt32 = 0 + private var lastSenderReportAt: TimeInterval = 0 + private var transit: Double? + private var jitter = 0.0 + + private var now: TimeInterval { ProcessInfo.processInfo.systemUptime } + + func receive(_ datagram: Data) { + let header = Array(datagram.prefix(12)) + guard header.count >= 12, header[0] >> 6 == 2 else { return } + // RTCP keeps its own sequence space and must never enter the RTP reorder buffer. + guard header[1] & 127 == Self.hevcPayloadType else { + receiveInOrder(datagram) + return + } + let sender = Self.uint32(header, 8) + guard source == nil || source == sender else { return } + if source == nil { source = sender } + + let sequence = Self.uint16(header, 2) + if nextExpected == nil { nextExpected = sequence } + guard let expected = nextExpected else { return } + // Signed distance: a sequence number that wrapped is ahead, not behind. + let distance = Int16(bitPattern: sequence &- expected) + guard distance >= 0 else { + latePackets += 1 + return + } + guard pending[sequence] == nil else { return } + if distance > 0 { reorderedPackets += 1 } + pending[sequence] = (datagram, now) + flushPending() + } + + /// Releases anything the reorder window has outlived, so a lost packet stops holding the stream + /// hostage. Call it on every loop pass: time passing is itself an event here. + func flushPending() { + while let expected = nextExpected { + if let packet = pending.removeValue(forKey: expected) { + nextExpected = expected &+ 1 + receiveInOrder(packet.data) + continue + } + guard !pending.isEmpty else { return } + let oldest = pending.values.map(\.arrival).min()! + guard pending.count >= 256 || now - oldest >= reorderWindow else { return } + // The gap is real now; `receiveInOrder` will invalidate the unit it lands in. + nextExpected = pending.keys.min { ($0 &- expected) < ($1 &- expected) } + } + } + + // MARK: - Assembling one access unit + + private func receiveInOrder(_ datagram: Data) { + let bytes = Array(datagram) + guard bytes.count >= 4, bytes[0] >> 6 == 2 else { return } + + if bytes[1] >= 192, bytes[1] <= 223 { + noteSenderReports(bytes) + return + } + guard bytes.count >= 12, bytes[1] & 127 == Self.hevcPayloadType else { return } + let sender = Self.uint32(bytes, 8) + guard source == nil || source == sender else { return } + source = sender + + let sequence = Self.uint16(bytes, 2) + let timestamp = Self.uint32(bytes, 4) + var offset = 12 + Int(bytes[0] & 15) * 4 + guard offset <= bytes.count else { return } + if bytes[0] & 16 != 0 { + guard offset + 4 <= bytes.count else { return } + offset += 4 + Int(Self.uint16(bytes, offset + 2)) * 4 + } + var end = bytes.count + if bytes[0] & 32 != 0 { + let padding = Int(bytes.last!) + guard padding > 0, padding <= end - offset else { return } + end -= padding + } + guard offset + 2 <= end else { return } + + var sequenceGap = false + if let previous = lastSequence { + let distance = Int16(bitPattern: sequence &- previous) + guard distance > 0 else { return } + highestSequence &+= UInt32(distance) + if distance != 1 { sequenceGap = true } + } else { + highestSequence = UInt32(sequence) + baseSequence = highestSequence + } + lastSequence = sequence + receivedPackets &+= 1 + noteJitter(timestamp) + + if self.timestamp != timestamp { + // The previous unit ended without its marker bit, so whatever it was is unfinished. + if self.timestamp != nil, !nals.isEmpty || !fragment.isEmpty { + lossEvents += 1 + onLoss() + } + nals.removeAll(keepingCapacity: true) + fragment.removeAll(keepingCapacity: true) + unitBytes = 0 + damaged = false + self.timestamp = timestamp + } + // why: applied after the timestamp reset — a gap may have taken this picture's first slices, + // and marking the fresh unit damaged is the only way that reaches the decoder. + if sequenceGap { markDamaged() } + + let payload = Array(bytes[offset..> 1) & 63 + if !damaged { + switch kind { + case 0..<48: + if !fragment.isEmpty { markDamaged() } else { append(Data(payload)) } + case 48: + appendAggregated(payload) + case 49: + appendFragmented(payload) + default: + markDamaged() + } + } + + guard bytes[1] & 128 != 0 else { return } + if !fragment.isEmpty { markDamaged() } + // A decodable picture needs its first slice, including when joining mid-stream; a picture of + // later slices would decode against a reference the viewer never got. + let slices = nals.filter { ($0[$0.startIndex] >> 1) & 63 < 32 } + if !slices.isEmpty, !slices.contains(where: { $0.count >= 3 && $0[$0.startIndex + 2] & 128 != 0 }) { + markDamaged() + } + if !damaged, !nals.isEmpty { + onAccessUnit((nals: nals, timestamp: timestamp)) + } + nals.removeAll(keepingCapacity: true) + unitBytes = 0 + } + + private var lastSequence: UInt16? + + /// Aggregation packet: length-prefixed NAL units back to back. + private func appendAggregated(_ payload: [UInt8]) { + var index = 2 + while index < payload.count, !damaged { + guard index + 2 <= payload.count else { markDamaged(); return } + let length = Int(Self.uint16(payload, index)) + index += 2 + guard length >= 2, index + length <= payload.count else { markDamaged(); return } + append(Data(payload[index..= 4 else { markDamaged(); return } + let header = payload[2] + if header & 128 != 0 { + guard fragment.isEmpty else { markDamaged(); return } + fragment = Data([(payload[0] & 129) | ((header & 63) << 1), payload[1]]) + } + guard !fragment.isEmpty else { markDamaged(); return } + guard fragment.count + payload.count - 3 <= maxUnitBytes else { markDamaged(); return } + fragment.append(contentsOf: payload.dropFirst(3)) + if header & 64 != 0 { + let nal = fragment + fragment.removeAll(keepingCapacity: true) + append(nal) + } + } + + private func append(_ nal: Data) { + guard unitBytes + nal.count <= maxUnitBytes else { markDamaged(); return } + nals.append(nal) + unitBytes += nal.count + } + + private func markDamaged() { + fragment.removeAll(keepingCapacity: true) + nals.removeAll(keepingCapacity: true) + unitBytes = 0 + damaged = true + lossEvents += 1 + onLoss() + } + + // MARK: - RTCP + + /// One RTCP compound packet: the receiver report, a name, and a key-frame request or a bye when + /// either is called for. + func report( + receiver: UInt32, + requestKeyframe: Bool = false, + bye: Bool = false, + name: String = "simlock-device-capture" + ) -> Data { + var packet: [UInt8] + if let source { + let expected = highestSequence &- baseSequence &+ 1 + let interval = expected &- previousExpected + let received = receivedPackets &- previousReceived + let lost = max(Int64(expected) - Int64(receivedPackets), 0) + let fraction = interval > 0 + ? min(255, max(0, (Int64(interval) - Int64(received)) * 256 / Int64(interval))) + : 0 + let total = UInt32(min(lost, 0x7f_ffff)) + let delay = lastSenderReport == 0 + ? 0 + : UInt32(min((now - lastSenderReportAt) * 65_536, Double(UInt32.max))) + packet = [0x81, 201, 0, 7] + + Self.word(receiver) + + Self.word(source) + + [UInt8(fraction), + UInt8(truncatingIfNeeded: total >> 16), + UInt8(truncatingIfNeeded: total >> 8), + UInt8(truncatingIfNeeded: total)] + + Self.word(highestSequence) + + Self.word(UInt32(jitter)) + + Self.word(lastSenderReport) + + Self.word(delay) + previousExpected = expected + previousReceived = receivedPackets + } else { + packet = [0x80, 201, 0, 1] + Self.word(receiver) + } + + var nameChunk = Self.word(receiver) + [1, UInt8(name.utf8.count)] + Array(name.utf8) + [0] + while nameChunk.count % 4 != 0 { nameChunk.append(0) } + packet += [0x81, 202, 0, UInt8(nameChunk.count / 4)] + nameChunk + + // Picture loss indication (RFC 4585): the iPhone's encoder is the only one who can recover a + // picture this side dropped. + if requestKeyframe, let source { + packet += [0x81, 206, 0, 2] + Self.word(receiver) + Self.word(source) + } + if bye { + packet += [0x81, 203, 0, 1] + Self.word(receiver) + } + return Data(packet) + } + + /// Records the NTP-ish middle of each sender report so the report's own delay field can be + /// filled in, and ignores everything else they carry. + private func noteSenderReports(_ bytes: [UInt8]) { + var offset = 0 + while offset + 4 <= bytes.count { + let length = (Int(Self.uint16(bytes, offset + 2)) + 1) * 4 + guard length >= 4, offset + length <= bytes.count else { return } + if bytes[offset + 1] == 200, length >= 28 { + let sender = Self.uint32(bytes, offset + 4) + if source == nil || source == sender { + lastSenderReport = + (Self.uint32(bytes, offset + 8) << 16) | (Self.uint32(bytes, offset + 12) >> 16) + lastSenderReportAt = now + } + } + offset += length + } + } + + /// RFC 3550's inter-arrival jitter, in clock ticks, exponentially averaged. Differences rather + /// than absolutes, so a long session's timestamp wrap cannot move it. + private func noteJitter(_ timestamp: UInt32) { + let current = now * 90_000 - Double(timestamp) + if let previous = transit { + let delta = abs(current - previous) + if delta < 900_000 { jitter += (delta - jitter) / 16 } + } + transit = current + } + + static let hevcPayloadType = 100 + + static func uint16(_ bytes: [UInt8], _ at: Int) -> UInt16 { + UInt16(bytes[at]) << 8 | UInt16(bytes[at + 1]) + } + + static func uint32(_ bytes: [UInt8], _ at: Int) -> UInt32 { + UInt32(uint16(bytes, at)) << 16 | UInt32(uint16(bytes, at + 2)) + } + + static func word(_ value: UInt32) -> [UInt8] { + [ + UInt8(truncatingIfNeeded: value >> 24), + UInt8(truncatingIfNeeded: value >> 16), + UInt8(truncatingIfNeeded: value >> 8), + UInt8(truncatingIfNeeded: value), + ] + } +} + +/// A UDP datagram's length prefix helpers are in the receiver; the socket itself lives in the media +/// bridge, which is why this file never opens one. diff --git a/vendor/screen-capture/Sources/simlock-device-capture/main.swift b/vendor/screen-capture/Sources/simlock-device-capture/main.swift new file mode 100644 index 00000000..0c1cbd45 --- /dev/null +++ b/vendor/screen-capture/Sources/simlock-device-capture/main.swift @@ -0,0 +1,35 @@ +// simlock-device-capture — streams one physical iPhone's screen on stdout, through CoreDevice's +// media stream. +// +// Usage: +// simlock-device-capture [--codec avcc] [--fps N] [--bitrate BPS] +// [--max-dimension PX] [--duration SEC] +// +// The wire, the envelope, the lifecycle and the encoders are the same as simlock-screen-capture's; +// both live in SimlockCapture, and both end on 64, 1 or 0 — this program's version gate adds 69. +// What differs is where the pictures come from: CoreDevice asks the iPhone for an HEVC media stream, +// this program receives it as RTP, decodes it, and hands each picture to the same encode loop a +// simulator's framebuffer goes into. +// +// Version gate: the first thing this program does, before any CoreDevice call, is read CoreDevice's +// own version and refuse a Mac it was not written against (exit 69). The framework is linked `weak`, +// so the program starts on a Mac without it long enough to say so. +// +// SIMLOCK_COREDEVICE_INFO_PLIST names another Info.plist to read that version from, for the slow lane +// only; the driver never passes it. +import Foundation +import SimlockCapture + +// Claims the bitstream before anything can print, including the negotiator bridge's own logs. +Wire.claimStdout() + +guard let options = parseOptions(Array(CommandLine.arguments.dropFirst())) else { + log("usage: simlock-device-capture [--codec avcc|mjpeg] [--fps N] [--bitrate BPS] [--max-dimension PX] [--duration SEC]") + exit(64) +} + +if let refusal = CoreDeviceVersion.refusal(for: CoreDeviceVersion.read()) { + fail(69, refusal) +} + +await runCapture(options: options, source: DeviceScreenSource()) diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift b/vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift index b85255b8..3776ea87 100644 --- a/vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift +++ b/vendor/screen-capture/Sources/simlock-screen-capture/FrameCapture.swift @@ -5,6 +5,7 @@ // `NOTICE.md` prints the one command that diffs this file against pristine upstream at the commit // we took it from. Read that notice before editing this file. import Foundation +import SimlockCapture import CoreVideo import CoreMedia import CoreGraphics diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/SimFrameworks.swift b/vendor/screen-capture/Sources/simlock-screen-capture/SimFrameworks.swift index 5a61a3e3..5e806fe6 100644 --- a/vendor/screen-capture/Sources/simlock-screen-capture/SimFrameworks.swift +++ b/vendor/screen-capture/Sources/simlock-screen-capture/SimFrameworks.swift @@ -1,4 +1,5 @@ import Foundation +import SimlockCapture enum SimFrameworks { /// Loads the private simulator frameworks (CoreSimulator + SimulatorKit) into diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/SimulatorSource.swift b/vendor/screen-capture/Sources/simlock-screen-capture/SimulatorSource.swift new file mode 100644 index 00000000..35f37238 --- /dev/null +++ b/vendor/screen-capture/Sources/simlock-screen-capture/SimulatorSource.swift @@ -0,0 +1,36 @@ +// simlock's own, MIT-licensed (see NOTICE.md): the simulator's framebuffer as a `ScreenSource`. +// +// `FrameCapture` is serve-sim's file; this is the thin shape the shared lifecycle needs from it — +// start by UDID, hand each copied picture over, and stop. The 200 ms idle floor that re-emits the +// last picture when nothing new is painted is FrameCapture's own, and it is load-bearing here: a +// still simulator sends nothing, and a stream that sends nothing looks dead from the outside. +import CoreVideo +import Foundation +import SimlockCapture + +final class SimulatorSource: ScreenSource { + private let capture = FrameCapture() + private var stopped = false + + func start( + deviceID: String, + onFrame: @escaping @Sendable (CVPixelBuffer) -> Void, + onEnd: @escaping @Sendable (_ reason: String?) -> Void + ) async throws { + // The simulator source has no way to notice its own end: a simulator that shuts down stops + // calling back, and FrameCapture reports nothing else. The daemon notices the dead simulator + // through the device's own state and closes the capture, so `onEnd` stays unused here. + _ = onEnd + try await capture.start(deviceUDID: deviceID) { pixelBuffer, _ in + onFrame(pixelBuffer) + } + } + + func stop() async { + // why: idempotent — the lifecycle calls it on every way out, and a second `stop` would + // unregister callbacks this source no longer owns. + guard !stopped else { return } + stopped = true + await capture.stop() + } +} diff --git a/vendor/screen-capture/Sources/simlock-screen-capture/main.swift b/vendor/screen-capture/Sources/simlock-screen-capture/main.swift index 17f2b639..22cf47ce 100644 --- a/vendor/screen-capture/Sources/simlock-screen-capture/main.swift +++ b/vendor/screen-capture/Sources/simlock-screen-capture/main.swift @@ -28,359 +28,20 @@ // stream by closing stdin, and kills the process only if it is still running 2 s later, so "the // viewer pressed stop" and "the viewer went away" are the same event here, and no capture can // outlive the connection that asked for it. -import Foundation -import CoreMedia -import CoreVideo -import VideoToolbox - -// MARK: - Options - -struct Options { - var udid = "" - var codec = "avcc" - var fps = 30 - var bitrate = 2_000_000 - var maxDimension = 720 - var duration = 0.0 -} - -func parseOptions(_ args: [String]) -> Options? { - var o = Options() - var positionals: [String] = [] - var i = 0 - while i < args.count { - let a = args[i] - func value() -> String? { - i += 1 - return i < args.count ? args[i] : nil - } - switch a { - case "--codec": guard let v = value() else { return nil }; o.codec = v - case "--fps": guard let v = value(), let n = Int(v) else { return nil }; o.fps = max(1, min(60, n)) - case "--bitrate": guard let v = value(), let n = Int(v) else { return nil }; o.bitrate = max(100_000, n) - case "--max-dimension": guard let v = value(), let n = Int(v) else { return nil }; o.maxDimension = max(0, n) - case "--duration": guard let v = value(), let s = Double(v) else { return nil }; o.duration = max(0, s) - default: - if a.hasPrefix("--") { return nil } - positionals.append(a) - } - i += 1 - } - guard positionals.count == 1 else { return nil } - o.udid = positionals[0] - guard o.codec == "avcc" || o.codec == "mjpeg" else { return nil } - return o -} - -// MARK: - Framing - -/// serve-sim's AVCC envelope (`Sources/SimNative/StreamFormat.swift`) plus this tool's meta tag. -enum Tag: UInt8 { - case description = 0x01 - case keyframe = 0x02 - case delta = 0x03 - case seed = 0x04 - case meta = 0x05 -} - -/// What the frame path mutates, behind one lock. A class rather than an actor because the capture -/// callback is synchronous and stdout needs one writer at a time. -final class Session: @unchecked Sendable { - let lock = NSLock() - var scaler: Scaler? - var scaledSize = Dimensions(width: 0, height: 0) - var frames = 0 - var bytes = 0 - /// The scaler is sized from the first frame, so a resize mid-session is not tracked (a - /// rotation changes the framebuffer's long edge and the scale factor with it). The PoC - /// accepts the first frame's geometry; the fix is to rebuild the scaler on size change. - var scalerSize = Dimensions(width: 0, height: 0) -} - -let session = Session() - -/// The video channel: a private duplicate of the stdout we were started with. The inherited fd 1 -/// is redirected onto stderr below, so text that any code in this process writes to stdout — the -/// copied capture code logs with `print`, and frameworks do too — cannot reach the bitstream. -let videoFd: Int32 = { - let fd = dup(STDOUT_FILENO) - guard fd >= 0 else { - fatalError("could not duplicate stdout") - } - dup2(STDERR_FILENO, STDOUT_FILENO) - return fd -}() - -/// Writes one envelope frame to the video channel: length prefix (covering the tag and payload), -/// tag, payload. A partial write would desynchronize every frame after it, so the whole frame goes -/// out under one lock. -func emit(_ tag: Tag, _ data: Data) { - let total = UInt32(data.count + 1) - var out = Data(capacity: data.count + 5) - out.append(UInt8((total >> 24) & 0xff)) - out.append(UInt8((total >> 16) & 0xff)) - out.append(UInt8((total >> 8) & 0xff)) - out.append(UInt8(total & 0xff)) - out.append(tag.rawValue) - out.append(data) - session.lock.lock() - session.frames += 1 - session.bytes += out.count - session.lock.unlock() - out.withUnsafeBytes { pointer in - var written = 0 - let base = pointer.baseAddress! - while written < out.count { - let n = write(videoFd, base + written, out.count - written) - if n <= 0 { - // EPIPE is the viewer going away, which is the normal end of a stream. - _exit(0) - } - written += n - } - } -} - -func log(_ message: String) { - FileHandle.standardError.write("[capture] \(message)\n".data(using: .utf8)!) -} - -/// Resident set of this process, in bytes. This is the number that answers "what does one open -/// stream cost the worker", so it is measured rather than estimated. -func residentBytes() -> UInt64 { - var info = mach_task_basic_info() - var count = mach_msg_type_number_t(MemoryLayout.size) / 4 - let status = withUnsafeMutablePointer(to: &info) { pointer -> kern_return_t in - pointer.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { - task_info(mach_task_self_, task_flavor_t(MACH_TASK_BASIC_INFO), $0, &count) - } - } - return status == KERN_SUCCESS ? info.resident_size : 0 -} - -// MARK: - Downscale // -// serve-sim captures at the simulator's native framebuffer size (1179x2556 on an iPhone 16). A -// viewer several hops away through the gateway does not need every pixel, and each one is paid -// for twice on the uplink. VTPixelTransferSession scales on the GPU, so the saving is bandwidth -// without a CPU cost on the worker. - -/// Not thread-safe by itself: every call happens on the single encode queue. -final class Scaler: @unchecked Sendable { - private var transfer: VTPixelTransferSession? - private let pool: CVPixelBufferPool - - init(targetWidth: Int, targetHeight: Int) { - let attrs: [String: Any] = [ - kCVPixelBufferWidthKey as String: targetWidth, - kCVPixelBufferHeightKey as String: targetHeight, - kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA, - kCVPixelBufferIOSurfacePropertiesKey as String: [String: Any](), - ] - var poolRef: CVPixelBufferPool? - CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, attrs as CFDictionary, &poolRef) - self.pool = poolRef! - } - - /// Scales into a fresh buffer from the pool, or returns nil when the GPU transfer fails. - func transferFrom(_ source: CVPixelBuffer) -> CVPixelBuffer? { - if transfer == nil { - var created: VTPixelTransferSession? - guard VTPixelTransferSessionCreate( - allocator: kCFAllocatorDefault, - pixelTransferSessionOut: &created - ) == noErr, let created else { return nil } - transfer = created - } - guard let transfer else { return nil } - var target: CVPixelBuffer? - guard CVPixelBufferPoolCreatePixelBuffer(kCFAllocatorDefault, pool, &target) == noErr, - let target else { return nil } - guard VTPixelTransferSessionTransferImage(transfer, from: source, to: target) == noErr else { - return nil - } - return target - } -} - -// MARK: - Signal and stdin teardown - -func watchExitSources() { - for sig in [SIGTERM, SIGINT, SIGHUP] { - signal(sig) { _ in _exit(0) } - } - // stdin is a liveness handle: the daemon that closes it is the daemon that asked for stop. - Thread.detachNewThread { - var byte: UInt8 = 0 - while read(0, &byte, 1) > 0 {} - log("stdin closed") - _exit(0) - } -} - -/// Reports the running totals every few seconds, so a caller's log carries what a stream cost even -/// though a stream ended by a viewer going away has no chance to report at exit. -func watchStats() { - Thread.detachNewThread { - var lastFrames = 0 - var lastBytes = 0 - while true { - Thread.sleep(forTimeInterval: 5) - session.lock.lock() - let frames = session.frames - let bytes = session.bytes - session.lock.unlock() - let secs = max(0.001, Date.now.timeIntervalSince(startedAt)) - let window = max(0.001, secs) - log( - "STATS {\"fps\": \((Double(frames - lastFrames) / 5 * 10).rounded() / 10), " - + "\"kbps\": \(Int(Double(bytes - lastBytes) * 8 / 5 / 1000)), " - + "\"totalFrames\": \(frames), \"totalBytes\": \(bytes), " - + "\"avgKbps\": \(Int(Double(bytes) * 8 / window / 1000)), " - + "\"rssMb\": \(Int(residentBytes() / 1_048_576))}" - ) - lastFrames = frames - lastBytes = bytes - } - } -} +// Exit codes: 64 on bad usage, 1 when the source fails to start or stops with a reason, 0 on every +// clean end. The options, the envelope, the lifecycle and the encoders are SimlockCapture's; this +// file is the simulator source and the name of the program. +import Foundation +import SimlockCapture -// MARK: - Main +// Claims the bitstream before anything can print, including the source this program is about to +// build: FrameCapture logs with `print` while it starts. +Wire.claimStdout() guard let options = parseOptions(Array(CommandLine.arguments.dropFirst())) else { log("usage: simlock-screen-capture [--codec avcc|mjpeg] [--fps N] [--bitrate BPS] [--max-dimension PX] [--duration SEC]") exit(64) } -watchExitSources() - -let startedAt = Date.now -watchStats() -let capture = FrameCapture() -let encoder = H264Encoder(fps: options.fps, bitrate: options.bitrate) -let jpegEncoder = VideoEncoder(quality: 0.7) - -/// A captured frame, handed from the capture queue to the encode queue. -struct Pending: @unchecked Sendable { - let buffer: CVPixelBuffer - /// Monotonic nanoseconds (`DispatchTime.uptimeNanoseconds`), so a wall-clock jump cannot stall - /// the fps throttle. - let atNanos: UInt64 -} - -let (frames, sink) = AsyncStream.makeStream(of: Pending.self, bufferingPolicy: .bufferingNewest(1)) - -// Capture fires on SimulatorKit's render callbacks plus a 5 fps idle floor, so the cadence is the -// simulator's. Two things happen on the encode side: the fps throttle, then the encode. Dropped -// frames cost nothing — an H.264 P-frame carries the difference, so a jump from N to N+3 decodes -// as one slightly larger delta. - -/// Runs the encode loop. `newest(1)` means a slow encoder or a relay that stops draining drops -/// old frames instead of queueing them without bound, which is the behaviour a live view needs. -/// -/// The first frame forces an IDR so the viewer that just asked for the stream can start decoding -/// immediately rather than waiting for the encoder's own keyframe interval; after that -/// VideoToolbox spaces IDRs itself (`MaxKeyFrameInterval = fps * 5`, from serve-sim's encoder). -func encodeLoop() async { - let intervalNanos = UInt64(1_000_000_000 / Double(options.fps)) - var lastSent: UInt64 = 0 - var started = false - var metaEmitted = false - for await frame in frames { - if started, frame.atNanos &- lastSent < intervalNanos { continue } - lastSent = frame.atNanos - let forceKeyframe = !started - started = true - - guard let prepared = prepareForEncode(frame.buffer) else { continue } - if !metaEmitted { - metaEmitted = true - let meta: [String: Any] = [ - "udid": options.udid, - "codec": options.codec == "avcc" ? "h264" : "mjpeg", - "width": CVPixelBufferGetWidth(prepared), - "height": CVPixelBufferGetHeight(prepared), - "fps": options.fps, - "deviceSet": ProcessInfo.processInfo.environment["SIMLOCK_IOS_DEVICE_SET"] ?? "default", - ] - if let json = try? JSONSerialization.data(withJSONObject: meta), - let string = String(data: json, encoding: .utf8) { - log("ready \(string)") - emit(.meta, Data(string.utf8)) - } - } - if options.codec == "mjpeg" { - if let data = try? await jpegEncoder.encode(pixelBuffer: prepared) { - emit(.delta, data) - } - } else if let encoded = try? await encoder.encode( - prepared, - forceKeyframe: forceKeyframe - ) { - if let description = encoded.description { - emit(.description, description) - } - emit(encoded.kind == .keyframe ? .keyframe : .delta, encoded.avcc) - } - } -} - -/// Scales to the configured maximum and returns a buffer safe to encode. The encoder deep-copies -/// what it is given internally, so the recycled framebuffer IOSurface is never held. -func prepareForEncode(_ buffer: CVPixelBuffer) -> CVPixelBuffer? { - let width = CVPixelBufferGetWidth(buffer) - let height = CVPixelBufferGetHeight(buffer) - guard options.maxDimension > 0, max(width, height) > options.maxDimension else { - return buffer - } - session.lock.lock() - if session.scaler == nil { - let factor = Double(options.maxDimension) / Double(max(width, height)) - // Even dimensions: H.264 4:2:0 rejects an odd chroma plane. - let targetWidth = max(2, Int(Double(width) * factor) & ~1) - let targetHeight = max(2, Int(Double(height) * factor) & ~1) - session.scaler = Scaler(targetWidth: targetWidth, targetHeight: targetHeight) - session.scaledSize = Dimensions(width: targetWidth, height: targetHeight) - log("scaling \(width)x\(height) -> \(targetWidth)x\(targetHeight)") - } - let scaler = session.scaler - session.lock.unlock() - return scaler?.transferFrom(buffer) -} - -Task { - do { - try await capture.start(deviceUDID: options.udid) { pixelBuffer, _ in - sink.yield(Pending(buffer: pixelBuffer, atNanos: DispatchTime.now().uptimeNanoseconds)) - } - } catch { - log("start failed: \(error)") - _exit(1) - } - // The meta frame is written by the encode loop on the first frame, because that is the first - // moment the encoded geometry (post-scale) is known. - await encodeLoop() -} - -if options.duration > 0 { - try? await Task.sleep(nanoseconds: UInt64(options.duration * 1_000_000_000)) - let secs = max(0.001, Date.now.timeIntervalSince(startedAt)) - session.lock.lock() - let counted = (session.frames, session.bytes) - session.lock.unlock() - let stats: [String: Any] = [ - "seconds": (secs * 100).rounded() / 100, - "frames": counted.0, - "bytes": counted.1, - "fps": (Double(counted.0) / secs * 10).rounded() / 10, - "kbps": ((Double(counted.1) * 8 / secs) / 1000).rounded(), - "avgFrameBytes": counted.0 == 0 ? 0 : Int(Double(counted.1) / Double(counted.0)), - "rssMb": Int(residentBytes() / 1_048_576), - ] - if let json = try? JSONSerialization.data(withJSONObject: stats) { - log("RESULT \(String(data: json, encoding: .utf8)!)") - } - exit(0) -} - -dispatchMain() +await runCapture(options: options, source: SimulatorSource()) diff --git a/vendor/screen-capture/Stubs/CoreDevice.swiftmodule/arm64-apple-macos.swiftinterface b/vendor/screen-capture/Stubs/CoreDevice.swiftmodule/arm64-apple-macos.swiftinterface new file mode 100644 index 00000000..34fdc811 --- /dev/null +++ b/vendor/screen-capture/Stubs/CoreDevice.swiftmodule/arm64-apple-macos.swiftinterface @@ -0,0 +1,98 @@ +// swift-interface-format-version: 1.0 +// swift-compiler-version: Apple Swift version 6.4 (swiftlang-6.4.0.34.1 clang-2100.3.34.1) +// swift-module-flags: -target arm64-apple-macos26.0 -enable-library-evolution -swift-version 5 -module-name CoreDevice +// Hand-written stub for /Library/Developer/PrivateFrameworks/CoreDevice.framework (the client side +// of coredeviced). Apple ships no module interface for it, so this file declares only the members +// simlock-device-capture calls, with the signatures verified against the installed CoreDevice 642.x +// binary. The version pin in CoreDeviceVersion is what keeps this honest: an optimized client that +// guessed a wrong layout is the reason `-Onone` is load-bearing in Package.swift. +// +// A new Xcode major turns iPhone streams off until this file is checked against that release and the +// pin moves with it. +import Swift +import Foundation +import Dispatch +import CoreDeviceTypes +import CoreDeviceProtocols +import Network +import XPC + +public protocol CoreDeviceServiceConnection { +} + +public struct CoreDeviceService { + public static var sharedConnection: any CoreDevice.CoreDeviceServiceConnection { get } +} + +public class DeviceManager { + public init(serviceConnection: any CoreDevice.CoreDeviceServiceConnection) + final public func allDevices() -> [CoreDevice.RemoteDevice] +} + +public class RemoteDevice { + final public var id: Foundation.UUID { get } + final public var description: Swift.String { get } + public var deviceInfo: CoreDeviceProtocols.DeviceInfo { get } +} + +// ---- The media stream, reached through the same service connection ---- + +public enum CodableValue { public init?(_ value: Any) } +public struct MediaStreamHelper { public static func getNegotiatorOptions(for: CoreDeviceProtocols.DeviceInfo) -> [String: Any] } +public struct DeviceMediaStream { + public static var kAVCMediaStreamOptionClientName: String { get } + public static var kAVCMediaStreamOptionClientSessionID: String { get } + public enum StreamType { case audio; case video } + public enum Direction { case input; case output } + public struct SupportInfo { public var supportedFeatures: UInt64 { get }; public var description: String { get } } + public struct SupportInfoResponse { public var client: SupportInfo { get }; public var device: SupportInfo { get }; public var common: SupportInfo { get } } + public struct StartRequest { + public init(timeout: UInt, type: StreamType, direction: Direction, negotiatorOffer: Foundation.Data, clientSupportedFeatures: UInt64, receiverIP: String?, receiverPort: UInt16?, options: [String: CodableValue]) + } + public struct StartResponse : Encodable { public var connection: ConnectionInfo { get }; public var negotiatorAnswer: Foundation.Data { get }; public func encode(to: any Encoder) throws } + public struct ConnectionInfo : Encodable { public var receiver: NetworkInfo { get }; public var sender: NetworkInfo { get }; public var streamConfig: MediaStreamConfig { get }; public func encode(to: any Encoder) throws } + public struct NetworkInfo { public var ipv6Address: String { get }; public var port: UInt16 { get } } + public struct MediaStreamConfig { public var StreamInputID: Int { get } } + public struct StopRequest { public init(stopAll: Bool, identifiers: [UInt32]?) } + public struct StopResponse : Encodable { public func encode(to: any Encoder) throws } +} +extension RemoteDevice { + public var mediaStreamSupportInfo: DeviceMediaStream.SupportInfoResponse { get async throws } + public func startMediaStream(with: DeviceMediaStream.StartRequest) async throws -> DeviceMediaStream.StartResponse + public func stopMediaStream(with: DeviceMediaStream.StopRequest) async throws -> DeviceMediaStream.StopResponse +} + +public struct NetworkUtils { + public static func withSharedSocketDictionary(sourceAddress: Network.IPv6Address, port: UInt16, remoteAddress: Network.IPv6Address, remotePort: UInt16, body: (XPC.xpc_object_t) throws -> A) throws -> A +} + +// ---- The usage assertion that loads a device's services ---- + +// DeviceKit's device interaction model acquires these states before media use, and CoreDevice's own +// media stream needs the services loaded behind them. +public struct AssertableDeviceState : Swift.Hashable { + public static var developerModeEnabledIfRequired: CoreDevice.AssertableDeviceState { get } + public static var coreDeviceServicesLoaded: CoreDevice.AssertableDeviceState { get } + public static var extendedDeviceInfoLoaded: CoreDevice.AssertableDeviceState { get } + public static var remoteServiceDiscoveryTrustedConnectivityAvailable: CoreDevice.AssertableDeviceState { get } + public static var secureNetworkingAvailable: CoreDevice.AssertableDeviceState { get } + public static func == (lhs: CoreDevice.AssertableDeviceState, rhs: CoreDevice.AssertableDeviceState) -> Swift.Bool + public func hash(into hasher: inout Swift.Hasher) + public var hashValue: Swift.Int { get } +} +public struct UsageAssertionRequirements { + public init(deviceStates: Swift.Set, capabilities: Swift.Set) +} +public class DeviceUsageAssertion { + public func releaseAssertion() + public func registerInvalidationHandler(invokingOnQueue: Dispatch.DispatchQueue?, handler: @escaping (Swift.Error?) -> Swift.Void) +} +extension RemoteDevice { + public func acquireDeviceUsageAssertion(withReason: Swift.String, requirements: CoreDevice.UsageAssertionRequirements) async throws -> CoreDevice.DeviceUsageAssertion +} + +public struct Capability : Swift.Hashable { + public static func == (lhs: CoreDevice.Capability, rhs: CoreDevice.Capability) -> Swift.Bool + public func hash(into hasher: inout Swift.Hasher) + public var hashValue: Swift.Int { get } +} diff --git a/vendor/screen-capture/Stubs/CoreDeviceProtocols.swiftmodule/arm64-apple-macos.swiftinterface b/vendor/screen-capture/Stubs/CoreDeviceProtocols.swiftmodule/arm64-apple-macos.swiftinterface new file mode 100644 index 00000000..8732d8be --- /dev/null +++ b/vendor/screen-capture/Stubs/CoreDeviceProtocols.swiftmodule/arm64-apple-macos.swiftinterface @@ -0,0 +1,25 @@ +// swift-interface-format-version: 1.0 +// swift-compiler-version: Apple Swift version 6.4 (swiftlang-6.4.0.34.1 clang-2100.3.34.1) +// swift-module-flags: -target arm64-apple-macos26.0 -enable-library-evolution -swift-version 5 -module-name CoreDeviceProtocols +// Hand-written stub for the CoreDeviceProtocols module (its symbols live in CoreDevice.framework). +// Declares only what simlock-device-capture reads: the device's own identity, and the tunnel address +// the media stream is routed to. +import Swift +import Foundation +import CoreDeviceTypes + +public enum DeviceType { + public var description: Swift.String { get } + public var rawValue: Swift.String { get } +} + +public struct DeviceInfo { + public var name: Swift.String? { get } + public var udid: Swift.String? { get } + public var identifier: Foundation.UUID { get } + public var deviceType: CoreDeviceProtocols.DeviceType? { get } + public var transportType: CoreDeviceTypes.TransportType? { get } + public var productType: Swift.String? { get } + public var marketingName: Swift.String? { get } + public var tunnelIPAddressString: Swift.String? { get } +} diff --git a/vendor/screen-capture/Stubs/CoreDeviceTypes.swiftmodule/arm64-apple-macos.swiftinterface b/vendor/screen-capture/Stubs/CoreDeviceTypes.swiftmodule/arm64-apple-macos.swiftinterface new file mode 100644 index 00000000..9badb0ed --- /dev/null +++ b/vendor/screen-capture/Stubs/CoreDeviceTypes.swiftmodule/arm64-apple-macos.swiftinterface @@ -0,0 +1,14 @@ +// swift-interface-format-version: 1.0 +// swift-compiler-version: Apple Swift version 6.4 (swiftlang-6.4.0.34.1 clang-2100.3.34.1) +// swift-module-flags: -target arm64-apple-macos26.0 -enable-library-evolution -swift-version 5 -module-name CoreDeviceTypes -module-abi-name CoreDevice +// Hand-written stub for the CoreDeviceTypes module. Its symbols live in +// CoreDeviceUtilities.framework / CoreDevice.framework and no `.swiftinterface` ships with Xcode, so +// this declares only what simlock-device-capture reads. The `-module-abi-name` is what makes these +// declarations line up with the binary's mangled names. +import Swift +import Foundation + +public enum TransportType { + public var description: Swift.String { get } + public var rawValue: Swift.String { get } +}