Skip to content

[Cocoa] Add a GPU process capture preview for WebKit's capture permission prompt - #78214

Open
eric-carlson wants to merge 1 commit into
WebKit:mainfrom
eric-carlson:capture-preview-gpu-process
Open

eric-carlson wants to merge 1 commit into
WebKit:mainfrom
eric-carlson:capture-preview-gpu-process

Conversation

@eric-carlson

@eric-carlson eric-carlson commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

934320c

[Cocoa] Add a GPU process capture preview for WebKit's capture permission prompt
https://bugs.webkit.org/show_bug.cgi?id=328783
rdar://190909434

Reviewed by NOBODY (OOPS!).

Working toward adding a live camera preview and a microphone level meter to WebKit's
getUserMedia permission prompt, so the user will be able see what they are about to
share. This patch adds the GPU process code necessary for to be able to capture audio
and/or video without an associated web process. The prompt itself will follow in a later
change.

- CapturePreviewManager captures from the chosen camera and microphone, hosts the camera frames in a
  layer the UI process can show, and reports the microphone level to the UI process.
- GPUProcess::startCapturePreview and stopCapturePreview run one preview at a time. A preview sets the
  process-wide capture attribution, TCC identity and media environment for its page, so stopping it
  restores them to whichever page was capturing before.
- GPUProcessProxy sends the camera, microphone and tccd sandbox extensions a preview needs. Unlike a
  granted capture's, they are revocable and are revoked when the preview stops.
- RemoteAudioSessionProxyManager counts a running preview as recording, so the audio session has the
  record category before the capture unit starts.
- Orientation monitoring counts capture and preview starts separately, and the start and stop messages
  say which with a RotationMonitoringReason. A preview stopping must not stop monitoring a camera that a
  capture still uses, and the reverse.

* Source/WebCore/platform/mediastream/ios/AVAudioSessionCaptureDeviceManager.h: Export
setPreferredMicrophoneID.
* Source/WebKit/CMakeLists.txt:
* Source/WebKit/DerivedSources.make:
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp:
(WebKit::GPUConnectionToWebProcess::reapplyCaptureEnvironment):
(WebKit::GPUConnectionToWebProcess::startMonitoringCaptureDeviceRotation):
(WebKit::GPUConnectionToWebProcess::stopMonitoringCaptureDeviceRotation):
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h:
* Source/WebKit/GPUProcess/GPUProcess.cpp:
(WebKit::GPUProcess::startCapturePreview):
(WebKit::GPUProcess::stopCapturePreview):
(WebKit::GPUProcess::stopCapturePreviewManager):
(WebKit::GPUProcess::didApplyCaptureEnvironment):
* Source/WebKit/GPUProcess/GPUProcess.h:
* Source/WebKit/GPUProcess/GPUProcess.messages.in:
* Source/WebKit/GPUProcess/cocoa/GPUConnectionToWebProcessCocoa.mm:
(WebKit::GPUConnectionToWebProcess::setCaptureAttributionString):
(WebKit::GPUConnectionToWebProcess::setTCCIdentity):
* Source/WebKit/GPUProcess/cocoa/GPUProcessCocoa.mm:
(WebKit::GPUProcess::setCaptureTCCIdentity):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.cpp:
(WebKit::RemoteAudioSessionProxyManager::hasActiveProxy):
(WebKit::RemoteAudioSessionProxyManager::beginCapturePreview):
(WebKit::RemoteAudioSessionProxyManager::endCapturePreview):
(WebKit::RemoteAudioSessionProxyManager::updateCategory):
(WebKit::RemoteAudioSessionProxyManager::tryToSetActiveForProcess):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.h:
* Source/WebKit/GPUProcess/webrtc/CapturePreviewManager.h: Added.
* Source/WebKit/GPUProcess/webrtc/CapturePreviewManager.mm: Added.
* Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.mm: Import Logging.h, which an
earlier file in its unified source bundle used to provide.
* Source/WebKit/GPUProcess/webrtc/UserMediaCaptureManagerProxy.cpp:
(WebKit::UserMediaCaptureManagerProxy::startProducingData):
(WebKit::UserMediaCaptureManagerProxy::applyCaptureEnvironment):
(WebKit::UserMediaCaptureManagerProxy::reapplyCaptureEnvironment):
* Source/WebKit/GPUProcess/webrtc/UserMediaCaptureManagerProxy.h:
* Source/WebKit/Shared/RotationMonitoringReason.h: Added.
* Source/WebKit/Shared/RotationMonitoringReason.serialization.in: Added.
* Source/WebKit/SourcesCocoa.txt:
* Source/WebKit/UIProcess/Cocoa/UserMediaPermissionRequestManagerProxy.mm:
(-[WKRotationCoordinatorObserver start:layer:reason:]):
(-[WKRotationCoordinatorObserver stop:reason:]):
(-[WKRotationCoordinatorObserver stopAllForCapturePreview]):
(WebKit::UserMediaPermissionRequestManagerProxy::startMonitoringCaptureDeviceRotation):
(WebKit::UserMediaPermissionRequestManagerProxy::stopMonitoringCaptureDeviceRotation):
(WebKit::UserMediaPermissionRequestManagerProxy::stopMonitoringCaptureDeviceRotationForCapturePreview):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.cpp:
(WebKit::GPUProcessProxy::startMonitoringCaptureDeviceRotation):
(WebKit::GPUProcessProxy::stopMonitoringCaptureDeviceRotation):
(WebKit::GPUProcessProxy::startCapturePreview):
(WebKit::GPUProcessProxy::capturePreviewAudioLevelChanged):
(WebKit::GPUProcessProxy::stopCapturePreview):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.h:
(WebKit::GPUProcessProxy::setCapturePreviewAudioLevelHandler):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.messages.in:
* Source/WebKit/UIProcess/UserMediaPermissionRequestManagerProxy.h:
* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::startMonitoringCaptureDeviceRotation):
(WebKit::WebPageProxy::stopMonitoringCaptureDeviceRotation):
(WebKit::WebPageProxy::stopMonitoringCaptureDeviceRotationForCapturePreview):
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:

c6a0807

Misc iOS, visionOS, tvOS & watchOS macOS Linux Windows Apple Internal
✅ 🧪 style ✅ 🛠 ios ✅ 🛠 mac ✅ 🛠 wpe ✅ 🛠 win ⏳ 🛠 ios-apple
✅ 🧪 bindings ✅ 🛠 ios-sim ✅ 🛠 mac-cmake-debug ✅ 🧪 wpe-wk2 ✅ 🧪 win-tests ⏳ 🛠 mac-apple
✅ 🧪 webkitperl ✅ 🧪 ios-wk2 ✅ 🧪 api-mac ✅ 🧪 api-wpe ⏳ 🛠 vision-apple
✅ 🧪 ios-wk2-wpt loading 🧪 api-mac-cmake-debug ⏳ 🧪 jsc-wpe
✅ 🧪 api-ios ✅ 🧪 mac-wk2 ✅ 🛠 gtk3-gcc
✅ 🛠 ios-safer-cpp ✅ 🧪 mac-cmake-debug-wk2 ✅ 🛠 gtk
✅ 🛠 vision ⏳ 🧪 mac-wk2-stress ✅ 🧪 gtk-wk2
✅ 🛠 vision-sim ⏳ 🧪 mac-intel-wk2 ✅ 🧪 api-gtk
✅ 🧪 vision-wk2 ✅ 🛠 mac-safer-cpp ✅ 🛠 playstation
✅ 🛠 tv ✅ 🧪 mac-site-isolation
✅ 🛠 tv-sim
✅ 🛠 watch
✅ 🛠 watch-sim

@eric-carlson
eric-carlson requested a review from cdumez as a code owner October 25, 2026 09:39
@eric-carlson eric-carlson self-assigned this Oct 25, 2026
@eric-carlson eric-carlson added the WebRTC For bugs in WebRTC label Oct 25, 2026
@webkit-ews-buildbot webkit-ews-buildbot added the merging-blocked Applied to prevent a change from being merged label Oct 25, 2026
@eric-carlson
eric-carlson force-pushed the capture-preview-gpu-process branch from 934320c to e5722fa Compare October 25, 2026 09:57
@webkit-ews-buildbot

Copy link
Copy Markdown
Collaborator

iOS Safer C++ Build #62501 (e5722fa)

❌ Found 1 failing file with 3 issues. Please address these issues before landing. See WebKit Guidelines for Safer C++ Programming.
(cc @rniwa)

…sion prompt

https://bugs.webkit.org/show_bug.cgi?id=328783
rdar://190909434

Reviewed by NOBODY (OOPS!).

Working toward adding a live camera preview and a microphone level meter to WebKit's
getUserMedia permission prompt, so the user will be able see what they are about to
share. This patch adds the GPU process code necessary for to be able to capture audio
and/or video without an associated web process. The prompt itself will follow in a later
change.

- CapturePreviewManager captures from the chosen camera and microphone, hosts the camera frames in a
  layer the UI process can show, and reports the microphone level to the UI process.
- GPUProcess::startCapturePreview and stopCapturePreview run one preview at a time. A preview sets the
  process-wide capture attribution, TCC identity and media environment for its page, so stopping it
  restores them to whichever page was capturing before.
- GPUProcessProxy sends the camera, microphone and tccd sandbox extensions a preview needs. Unlike a
  granted capture's, they are revocable and are revoked when the preview stops.
- RemoteAudioSessionProxyManager counts a running preview as recording, so the audio session has the
  record category before the capture unit starts.
- Orientation monitoring counts capture and preview starts separately, and the start and stop messages
  say which with a RotationMonitoringReason. A preview stopping must not stop monitoring a camera that a
  capture still uses, and the reverse.

* Source/WebCore/platform/mediastream/ios/AVAudioSessionCaptureDeviceManager.h: Export
setPreferredMicrophoneID.
* Source/WebKit/CMakeLists.txt:
* Source/WebKit/DerivedSources-input.xcfilelist:
* Source/WebKit/DerivedSources.make:
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.cpp:
(WebKit::GPUConnectionToWebProcess::reapplyCaptureEnvironment):
(WebKit::GPUConnectionToWebProcess::startMonitoringCaptureDeviceRotation):
(WebKit::GPUConnectionToWebProcess::stopMonitoringCaptureDeviceRotation):
* Source/WebKit/GPUProcess/GPUConnectionToWebProcess.h:
* Source/WebKit/GPUProcess/GPUProcess.cpp:
(WebKit::GPUProcess::startCapturePreview):
(WebKit::GPUProcess::stopCapturePreview):
(WebKit::GPUProcess::stopCapturePreviewManager):
(WebKit::GPUProcess::didApplyCaptureEnvironment):
* Source/WebKit/GPUProcess/GPUProcess.h:
* Source/WebKit/GPUProcess/GPUProcess.messages.in:
* Source/WebKit/GPUProcess/cocoa/GPUConnectionToWebProcessCocoa.mm:
(WebKit::GPUConnectionToWebProcess::setCaptureAttributionString):
(WebKit::GPUConnectionToWebProcess::setTCCIdentity):
* Source/WebKit/GPUProcess/cocoa/GPUProcessCocoa.mm:
(WebKit::GPUProcess::setCaptureTCCIdentity):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.cpp:
(WebKit::RemoteAudioSessionProxyManager::hasActiveProxy):
(WebKit::RemoteAudioSessionProxyManager::beginCapturePreview):
(WebKit::RemoteAudioSessionProxyManager::endCapturePreview):
(WebKit::RemoteAudioSessionProxyManager::updateCategory):
(WebKit::RemoteAudioSessionProxyManager::tryToSetActiveForProcess):
* Source/WebKit/GPUProcess/media/RemoteAudioSessionProxyManager.h:
* Source/WebKit/GPUProcess/webrtc/CapturePreviewManager.h: Added.
* Source/WebKit/GPUProcess/webrtc/CapturePreviewManager.mm: Added.
* Source/WebKit/GPUProcess/webrtc/RemoteSampleBufferDisplayLayer.mm: Import Logging.h, which an
earlier file in its unified source bundle used to provide.
* Source/WebKit/GPUProcess/webrtc/UserMediaCaptureManagerProxy.cpp:
(WebKit::UserMediaCaptureManagerProxy::startProducingData):
(WebKit::UserMediaCaptureManagerProxy::applyCaptureEnvironment):
(WebKit::UserMediaCaptureManagerProxy::reapplyCaptureEnvironment):
* Source/WebKit/GPUProcess/webrtc/UserMediaCaptureManagerProxy.h:
* Source/WebKit/Shared/RotationMonitoringReason.h: Added.
* Source/WebKit/Shared/RotationMonitoringReason.serialization.in: Added.
* Source/WebKit/SourcesCocoa.txt:
* Source/WebKit/UIProcess/Cocoa/UserMediaPermissionRequestManagerProxy.mm:
(-[WKRotationCoordinatorObserver start:layer:reason:]):
(-[WKRotationCoordinatorObserver stop:reason:]):
(-[WKRotationCoordinatorObserver stopAllForCapturePreview]):
(WebKit::UserMediaPermissionRequestManagerProxy::startMonitoringCaptureDeviceRotation):
(WebKit::UserMediaPermissionRequestManagerProxy::stopMonitoringCaptureDeviceRotation):
(WebKit::UserMediaPermissionRequestManagerProxy::stopMonitoringCaptureDeviceRotationForCapturePreview):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.cpp:
(WebKit::GPUProcessProxy::startMonitoringCaptureDeviceRotation):
(WebKit::GPUProcessProxy::stopMonitoringCaptureDeviceRotation):
(WebKit::GPUProcessProxy::startCapturePreview):
(WebKit::GPUProcessProxy::capturePreviewAudioLevelChanged):
(WebKit::GPUProcessProxy::stopCapturePreview):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.h:
(WebKit::GPUProcessProxy::setCapturePreviewAudioLevelHandler):
* Source/WebKit/UIProcess/GPU/GPUProcessProxy.messages.in:
* Source/WebKit/UIProcess/UserMediaPermissionRequestManagerProxy.h:
* Source/WebKit/UIProcess/WebPageProxy.cpp:
(WebKit::WebPageProxy::startMonitoringCaptureDeviceRotation):
(WebKit::WebPageProxy::stopMonitoringCaptureDeviceRotation):
(WebKit::WebPageProxy::stopMonitoringCaptureDeviceRotationForCapturePreview):
* Source/WebKit/UIProcess/WebPageProxy.h:
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
@eric-carlson
eric-carlson force-pushed the capture-preview-gpu-process branch from e5722fa to c6a0807 Compare October 25, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merging-blocked Applied to prevent a change from being merged WebRTC For bugs in WebRTC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants