Skip to content

Latest commit

 

History

History
 
 

native

Folders and files

README.md

Native capture helpers

macOS

macOS native recording will use a ScreenCaptureKit helper with the same process boundary as the Windows WGC helper:

  1. Electron resolves the selected source, output paths, and user-selected devices.
  2. The helper receives one structured JSON request.
  3. The helper owns ScreenCaptureKit/AVFoundation capture, timing, encoding, and muxing.
  4. Electron persists the resulting media/session manifest and reports helper errors explicitly.

The helper has one non-recording mode: openscreen-screencapturekit-helper --screen-access-status prints {"event":"screen-access","granted":<bool>} and exits. CGPreflightScreenCaptureAccess() caches its answer for the life of the calling process, so Electron's own read can never observe a Screen Recording grant made after launch — a helper spawned per read has no cache to be stale (electron/native-bridge/screen/macScreenAccess.ts).

Helper locations:

  1. OPENSCREEN_SCK_CAPTURE_EXE, for local development and diagnostics.
  2. electron/native/screencapturekit/build/openscreen-screencapturekit-helper, for locally built Swift output.
  3. electron/native/bin/darwin-arm64/openscreen-screencapturekit-helper or electron/native/bin/darwin-x64/openscreen-screencapturekit-helper, for packaged prebuilt helpers.

The macOS cursor-shape helper is resolved from OPENSCREEN_MAC_CURSOR_HELPER_EXE first, then the matching openscreen-macos-cursor-helper binary in the same local build and packaged electron/native/bin/darwin-${arch} directories.

Build the macOS helper with:

npm run build:native:mac

On non-macOS hosts this command exits successfully and does not affect Windows/Linux development. On macOS it builds the Swift package at electron/native/screencapturekit, writes the development binaries to electron/native/screencapturekit/build, and copies redistributable binaries to electron/native/bin/darwin-${arch}.

The current helper implementation supports display/window ScreenCaptureKit video capture, control-window exclusion through the request's optional excludedWindowIds and SCContentFilter(display:excludingWindows:), cursor exclusion through SCStreamConfiguration.showsCursor, H.264 encoding, MP4 muxing (with AVAssetWriter.movieFragmentInterval at 1s, so a helper that dies before finishWriting() still leaves a readable file — same reasoning as the Windows fragmented sink below), and ScreenCaptureKit system audio. Electron supplies only the HUD and an open Notes window as exclusions for a direct full-display capture, so ordinary OpenScreen windows remain recordable there. The helper adds Notification Center's windows on its own, and Finder's desktop-icon windows when the request sets hideDesktopIcons; a banner appears as a new window, so a direct capture re-reads the exclusion every 250 ms, while the system picker excludes Notification Center by bundle id. It excludes OpenScreen itself the same way, so a screen picked in the editor's Record mode still leaves out the HUD that only appears at Start: a picker capture records none of OpenScreen's windows, and window mode does not offer them. The helper reports this as excludesApp on picker-selected. The helper also attempts native ScreenCaptureKit microphone capture when the running macOS version exposes that capability, which is macOS 15 and later. Below that it records the take without the microphone and emits a microphone-unavailable warning; the app does not offer the microphone there (canRecordMicrophone in src/utils/platformUtils.ts). With no audio input listed it does the same and emits microphone-not-found: asked for a microphone anyway, ScreenCaptureKit starts that take and stalls the next one in the picker session (#995). Webcam recording currently stays as an Electron sidecar and is attached to the same recording session after the native screen capture stops.

Electron exposes is-native-mac-capture-available for capability probing. It resolves the same helper locations listed above and reports missing-helper until a Swift helper binary is present. When available, macOS recording routes screen/window capture through the native helper so editable cursor recordings do not bake the system cursor into the video. Cursor positions are sampled in Electron; when the cursor helper is available and Accessibility is granted, samples are also tagged with link/text cursor hints such as pointer.

See technical-documentation/architecture/recording.md for the contract, rollout phases, and SSOT rules.

Windows

Windows native recording is resolved from one of these locations:

  1. OPENSCREEN_WGC_CAPTURE_EXE, for local development and diagnostics.
  2. electron/native/wgc-capture/build/wgc-capture.exe, for a locally built Ninja helper.
  3. electron/native/wgc-capture/build/Release/wgc-capture.exe, for a locally built multi-config helper.
  4. electron/native/bin/win32-x64/wgc-capture.exe or electron/native/bin/win32-arm64/wgc-capture.exe, for packaged prebuilt helpers.

Build the Windows helper with:

# Builds for the host architecture by default (x64 on x64 hosts, arm64 on arm64 hosts).
npm run build:native:win

# Explicit target architecture (native on a matching host, or cross-compiled otherwise):
node scripts/build-windows-wgc-helper.mjs --arch arm64

The target architecture is resolved from --arch (or the OPENSCREEN_WIN_HELPER_ARCH env var), falling back to the host arch. Cross-compiling requires the matching MSVC component — "VS C++ ARM64/ARM64EC build tools" for arm64. The build writes the CMake output to electron/native/wgc-capture/build/wgc-capture.exe and copies the redistributable binary to electron/native/bin/win32-<arch>/wgc-capture.exe (e.g. win32-arm64).

Cross-compiling covers this helper, not the installer. npm run build:win:arm64 has to run on an ARM64 host: fetch-ffmpeg.mjs and fetch-onnxruntime.mjs provision for the host, and the ffmpeg licence check runs the downloaded binary. CI therefore builds the arm64 installer on a windows-11-arm runner.

The helper contract is process-based: the app starts the process with one JSON argument and sends commands on stdin. stop\n finalizes the recording. During migration the helper prints both newline-delimited JSON events and the legacy text messages Recording started / Recording stopped. Output path: <path>.

Current V2 JSON shape:

{
  "schemaVersion": 2,
  "recordingId": 123,
  "sourceType": "display",
  "sourceId": "screen:0:0",
  "displayId": 1,
  "windowHandle": null,
  "outputPath": "C:\\path\\recording-123.mp4",
  "videoWidth": 1920,
  "videoHeight": 1080,
  "fps": 60,
  "captureSystemAudio": false,
  "captureMic": false,
  "microphoneDeviceId": "default",
  "microphoneDeviceName": "Microphone (NVIDIA Broadcast)",
  "microphoneGain": 1.4,
  "webcamEnabled": true,
  "webcamDeviceId": "default",
  "webcamDeviceName": "Camera (NVIDIA Broadcast)",
  "webcamWidth": 1280,
  "webcamHeight": 720,
  "webcamFps": 30,
  "outputs": {
    "screenPath": "C:\\path\\recording-123.mp4"
  }
}

The current helper implementation supports display/window video capture, system audio loopback, selected-microphone capture, Media Foundation webcam capture, and a DirectShow webcam fallback for virtual cameras that are not exposed through Media Foundation. Webcam frames are currently composed into the primary MP4 as a bottom-right picture-in-picture overlay. Browser deviceId values do not always map to Media Foundation symbolic links or WASAPI endpoint IDs, so the renderer passes both browser IDs and user-visible device names. For microphones, the helper tries the requested WASAPI endpoint ID first, then resolves an active capture endpoint by microphoneDeviceName, then falls back to the default endpoint. For webcams, Electron resolves a matching DirectShow filter CLSID for the selected label; the helper uses Media Foundation first, then that exact DirectShow filter when the requested camera is absent from Media Foundation.

Container: recordings are written as fragmented MP4 (MFCreateFMPEG4MediaSink + MFCreateSinkWriterFromMediaSink, MF_MPEG4SINK_MIN_FRAGMENT_DURATION = 1s) rather than plain MP4. A plain MP4 has no index until IMFSinkWriter::Finalize() writes moov at the very end, so when the shutdown watchdog force-exits a wedged helper the file on disk holds every frame and no way to read them — that is why issues #252 / #292 / #327 cost the whole recording rather than the frozen tail of it. A fragmented MP4 writes its index up front and its samples in self-describing moof+mdat pairs, so the same kill leaves a file that plays up to the last complete fragment. This does not fix the freeze; it removes the data loss the freeze causes. Because the fragmented sink needs both output media types at construction, the sink writer is built from a media sink instead of from a URL, and the helper reads the video/audio stream positions back off the sink rather than assuming them. If any of that is unavailable on a machine, the helper retries with the plain container and says so — container in the encoder-selection event is fragmented-mp4 or mp4, and it reports what was used, not what was asked for.

Encoder selection: by default the helper keeps the existing sink-writer path first. If that path fails while setting up H.264, it retries with the Microsoft software H.264 encoder (mfh264enc.dll). The key of this retry is registering that encoder locally in the helper process via MFTRegisterLocalByCLSID, which makes a software H.264 encoder available even when the machine's hardware encoders are missing or broken; hardware transforms are disabled for the retry only as a secondary guard so the sink writer prefers the locally registered software encoder, not as the fallback mechanism itself. Set preferSoftwareEncoder: true in the helper JSON, or set OPENSCREEN_WGC_PREFER_SOFTWARE_ENCODER=true before launching Electron, to force the software path from the first attempt.

Frame input path: the helper feeds the encoder from the GPU when it can. On that path it copies the WGC frame across a keyed-mutex bridge to a second D3D11 device, converts BGRA to NV12 with the D3D11 video processor, and submits an allocator-owned DXGI sample to the hardware H.264 encoder, so no frame ever passes through system memory. The alternative is the original path: a staging texture, Map(D3D11_MAP_READ), and a row-by-row copy into an IMFMediaBuffer — which is where a driver stall costs a recording (issue #252). The GPU path is a preference, never a requirement: it is skipped outright for preferSoftwareEncoder and for inline webcam PiP (both need the frame in system memory), and it degrades to the CPU path on its own if the encoding device, the NV12 video processor, the shared bridge texture, the DXGI sample allocator, or the hardware sink writer is unavailable. The GPU path is OFF by default: it fixed #252 on the machine that reproduces it and broke recording outright in #336, and its fallbacks only cover failures during initialize(), not one that appears once frames are flowing. Set OPENSCREEN_WGC_ENABLE_DXGI_INPUT=1 to turn it on. Because the two paths land on different encoders and hardware MFTs default to constant bitrate, the GPU path asks for VBR through ICodecAPI; without it a static screen spends the full configured budget (measured 16.9 Mbps against 1.95 for the same desktop).

The helper reports the outcome through the encoder-selection stdout event (video is default, software-preferred, or software-fallback; videoInput is dxgi-nv12 or cpu-rgb32; container is fragmented-mp4 or mp4; all three report what the encoder settled on rather than what was asked for). On the GPU path the helper also prints one [frame-drops] gpu_bridge_contended=<n> line to stderr at stop: a frame the bridge was too busy to take is skipped rather than failing the recording, and a large count there is the first thing to look at in a report about missing frames. When the app sees software-fallback — the default encoder failed and the helper switched on its own — it shows a small dismissible notice in the recording HUD with a "Don't show again" option, because software encoding can raise CPU usage. An explicit software-preferred selection shows no notice, and the event stays available for diagnostics either way.

At startup the helper also emits capture-adapter, naming the GPU its D3D device landed on and the one actually driving the captured display, each with its LUID, plus one [adapters] line per enumerated adapter on stderr. createD3DDevice asks for the default adapter and nothing checks that it is the one driving the display; when they differ every frame crosses an adapter boundary before the caller touches it. The LUIDs are there because the descriptions are not enough to tell: an IddCx virtual display driver renders through the physical GPU and inherits its description string while being a separate DXGI adapter, so the configuration this diagnostic exists to catch is precisely the one where both names are identical and only the LUIDs differ (measured: NVIDIA Quadro RTX 4000 at LUID 0:24084 driving the display, the same string at 0:12889146 for the virtual adapter). monitorLookup says which of three things happened: ok, no-output-claims-it (the enumeration finished and nothing owns the captured monitor, which is what an active virtual display looks like), or unavailable (EnumOutputs refused, as it does in session 0 — the outputs were never inspected, so the absence means nothing about the hardware).

Desktop icons: "hideDesktopIcons": true on a display capture puts a borderless window, painted with the wallpaper by PaintDesktop, on the captured monitor directly above the window that hosts the icons (SHELLDLL_DefView's top-level parent) and below every app window (src/desktop_icon_cover.cpp). WGC cannot exclude another process's window, so covering is the only route that changes no setting. The window belongs to the helper, so it goes away with the recording or with a crashed helper; there is nothing to restore. Windows notifications cannot be kept out: toasts are drawn above every app window, and the only switch is Do Not Disturb, a user setting the helper does not touch.

Menus and popups in window recordings: a window source is captured per HWND, and a context menu or dropdown is a window of its own, so it used to be missing from the video (#894). On Windows 11 24H2 (build 26100) and later the helper sets GraphicsCaptureSession.IncludeSecondaryWindows (IGraphicsCaptureSession6) on window sources, and DWM draws the popups belonging to the captured window into the same texture: the frame size, DPI handling, occlusion immunity and the fixed frame pool are unchanged. Monitor sources never set it, since they already contain everything on the screen. Four things to know:

  • Popups are clipped at the window's edge. The texture is the window's size, so the part of a menu that overhangs it is not recorded (measured: a 276 px menu opened 138 px inside the right edge shows 138 px, cut at the last column).
  • It needs 24H2. Older runtimes keep capturing the window alone, as before. The helper reports the outcome as {"event":"secondary-windows","applied":true|false} (absent for monitor sources and when the option is disabled); the app logs it and shows nothing. Set OPENSCREEN_WGC_DISABLE_SECONDARY_WINDOWS=1 to turn the option off and get the window-only capture back.
  • What DWM lets in was measured, because Microsoft's rule is style and z-order, not process. It draws windows with WS_POPUP or WS_EX_TOOLWINDOW that overlap the window and have only other such windows between them and it. On build 26200, three unrelated TopMost popups from other processes (WS_POPUP, WS_POPUP plus WS_EX_TOOLWINDOW, and WS_EX_TOOLWINDOW alone), all overlapping the window with nothing else above it, stayed out of the recording, as did Explorer's own Win+X menu; the window's own owned menu was in. npm run test:wgc-window-popup:win reproduces the measurement (menu present, strangers absent, overhang clipped, kill switch removes the menu) and refuses to pass if a non-qualifying window sits above its target, since a stranger missing for that reason would prove nothing.
  • The menu shows up only when no ordinary window sits above the recorded window. If another window (neither a popup nor a tool window) overlaps it, the menu is missing from the first frame to the last, which is expected from how DWM selects secondary windows (#910). Measured on a second machine (Windows 11 build 26200, AMD, 46 runs of the fixture): the menu was captured in all 25 runs with nothing ordinary above the window, and absent in all 21 runs with one above it (Claude Desktop, Notepad); in the 20 valid runs at 2000 ms and 5000 ms no unrelated popup leaked (0 px). That is 0 leaks in 20 runs, which does not exclude a failure rate up to about 15 percent, and the fixture is synthetic WinForms, not a real application's menus.

Encoder diagnostic on final sink-writer failure: when the final sink-writer attempt fails (MFCreateSinkWriterFromMediaSink on the fragmented container, MFCreateSinkWriterFromURL on the plain one; the message names which), the helper logs the registered H.264 video encoder MFT count (via MFTEnumEx), the registered AAC encoder count when audio was requested, and the hex HRESULT. If no H.264 encoder is registered, it additionally emits the four-bullet actionable error (missing Media Feature Pack / GPU driver registration / empty HKLM:\SOFTWARE\Microsoft\Windows Media Foundation\Transforms / reboot). If an H.264 encoder IS registered but the sink writer still failed, it logs a hint pointing at invalid output path, missing MP4 mux, or GPU driver incompatibility. There is still no fail-fast pre-flight gate because MFTEnumEx and the sink writer can disagree about which H.264 encoders are available in non-interactive / Session 0 contexts.

Smoke-test the helper with:

npm run test:wgc-helper:win
npm run test:wgc-window:win
npm run test:wgc-window-popup:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:win

To validate a specific native webcam manually:

$env:OPENSCREEN_WGC_TEST_WEBCAM_DEVICE_NAME = "NVIDIA Broadcast"
npm run test:wgc-webcam:win
Remove-Item Env:OPENSCREEN_WGC_TEST_WEBCAM_DEVICE_NAME

To validate a specific native microphone manually:

$env:OPENSCREEN_WGC_TEST_MICROPHONE_DEVICE_NAME = "Microphone (NVIDIA Broadcast)"
npm run test:wgc-mic:win
Remove-Item Env:OPENSCREEN_WGC_TEST_MICROPHONE_DEVICE_NAME

Linux

Linux cursor recording is handled by openscreen-pipewire-helper (electron/native/pipewire-capture). Stage 1 emits cursor samples only — no video capture, no encoding.

It exists because screen.getCursorScreenPoint() returns {0,0} under Wayland. TelemetryRecordingSession therefore produced well-formed recordings whose every cursor sample sat in the top-left corner of the screen. The ScreenCast portal's METADATA cursor mode is the only source of a real pointer position on Wayland: the compositor keeps the cursor out of the captured pixels and attaches it to each frame as SPA_META_Cursor instead.

Helper locations, in resolution order:

  1. OPENSCREEN_LINUX_CURSOR_HELPER_EXE, for local development and diagnostics.
  2. electron/native/pipewire-capture/build/openscreen-pipewire-helper, for a locally built binary.
  3. electron/native/bin/linux-x64/openscreen-pipewire-helper (or linux-arm64), for packaged prebuilt helpers.

Build it with:

npm run build:native:linux

The build needs cargo and a C compiler and nothing else — in particular not libpipewire-0.3-dev, which Ubuntu does not ship by default and which needs root to install. The C shim compiles against headers vendored in electron/native/pipewire-capture/vendor/ and resolves libpipewire-0.3.so.0 with dlopen at runtime. On non-Linux hosts the command exits successfully.

The contract matches the other helpers: one JSON argument, newline-delimited JSON events on stdout, stop\n (or EOF) on stdin. Events are ready, stream-started, cursor-sample, warning, error and debug, each carrying "schemaVersion": 1. ready is emitted before the portal picker is raised, so the app's readiness timeout does not have to accommodate a human clicking a dialog; stream-started marks the point where samples begin.

Manual verification

This raises the GNOME/portal source picker and streams until you stop it.

OPENSCREEN_PIPEWIRE_DEBUG=1 electron/native/bin/linux-x64/openscreen-pipewire-helper '{"sampleIntervalMs":100}'

Pick a monitor in the dialog, move the mouse, and cursor-sample lines with real coordinates should stream out. Type stop and press Enter, or Ctrl-D, to end it.

OPENSCREEN_PIPEWIRE_DEBUG=1 additionally reports stream state transitions, the negotiated SPA buffer data type, the full list of metadata blocks that survived negotiation, and whether SPA_META_Cursor carries a sprite bitmap. Every line carries timestampMs, so a log shows how long the stream actually lived.

Field notes from the first real GNOME run

Two facts, and two bugs that run exposed. Both bugs are fixed; both are pinned by tests that run without a portal.

  • mutter negotiates MemFd, not DMA-BUF, even when the helper advertises MemPtr|MemFd|DmaBuf. Worth knowing for the video stage: the pixels arrive as a mappable fd, so a CPU path is viable and no DMA-BUF import is required.
  • SPA_META_Cursor was absent from every buffer. Producers declare SPA_PARAM_META_size as a fixed value — mutter 46.2 uses CURSOR_META_SIZE(384, 384) = 589872 bytes — and PipeWire intersects it with the consumer's declaration. The helper's range was capped at 256×256 = 262192 bytes, so the intersection was empty, the whole ParamMeta object was dropped, and the buffers arrived with no cursor metadata. There is no error for this: the stream negotiates and runs perfectly while reporting nothing. The ceiling is now 1024×1024, matching OBS. A range that does not contain the producer's constant is the failure mode to remember.
  • The stream was also dying on its own, reporting target not found. That string comes from WirePlumber's policy-node.lua, in the branch taken when node.dont-reconnect is set — which PW_STREAM_FLAG_DONT_RECONNECT sets. That branch destroys the node outright, turning a transient link failure into a permanent, silent end of capture. The flag is gone.

To exercise everything except the picker — dlopen, D-Bus, and the portal's cursor-mode check — run the non-interactive probe. This is also what npm run build:native:linux runs after a successful build:

electron/native/bin/linux-x64/openscreen-pipewire-helper '{"probeOnly":true}'

Known gaps

  • Mouse clicks need read access to evdev. Wayland exposes no portal for input events, so the helper reads left-button presses straight from evdev (/dev/input/event*). Those nodes are root:input, so a user with neither a udev uaccess rule for pointer devices (recommended) nor membership in the input group gets no readable device and every sample's interactionType stays "move" — the same as before. When a device is readable, the coinciding sample is tagged "click". Scope is deliberately narrow: BTN_LEFT only, never keystrokes (see pipewire-capture/src/input.rs), and OPENSCREEN_DISABLE_CLICK_CAPTURE=1 turns it off entirely. User setup: Mouse clicks on Wayland.
  • Window takes keep the client-side shadow as a black border, and a window take in which the pointer never enters the window has no cursor data. Both were observed on GNOME 46. The border comes from mutter's crop, which is the window's whole buffer, shadow included. The take with the pointer kept off the window logged cursorSamples: 0 and wrote no sidecar. The explanation for the cursor comes from the mutter 46.2 code: its window source attaches the cursor only while the window has the pointer. A take with the pointer inside the window has not been observed yet. Details and measurements: recording.md, Known gaps.
  • The user picks a source twice. Electron's desktopCapturer raises its own portal dialog for the video, and this helper raises a second one for the cursor. Collapsing them requires one portal session serving both, which is why the capture stage has to reuse this helper's session — SelectSources may only be called once per session.

STT helper

The speech-to-text helper (whisper-stt-server) is built separately from the capture helpers. See technical-documentation/architecture/transcription-and-captions.md for the architecture and scripts/build-whisper-stt.sh for the build command.