Adopt PacedMediaSamplesDecoder in AudioVideoRendererRemote - #78227

8f93d68

Adopt PacedMediaSamplesDecoder in AudioVideoRendererRemote
https://bugs.webkit.org/show_bug.cgi?id=327832
rdar://190225014

Reviewed by NOBODY (OOPS!).

AudioVideoRendererRemote converted every MediaSample to a MediaSamplesBlock and sent it, still
compressed, to the GPU process, which was the only place media was decoded. When no hardware decoder
could be had, WebCoreDecompressionSession decoded video in software in the GPU process instead:
through a VideoToolbox software decoder, as it only enabled the hardware decoder rather than
requiring it, or through VideoDecoder.

With the new DecoderInContentProcessEnabled preference on, a track asks
MediaSamplesDecoder::canDecode() on its first sample and on every format change. When the format
is supported, the track creates a PacedMediaSamplesDecoder at the next sync sample and sends the
decoded blocks to the GPU process. On a format change the previous decoder is drained, and the output
of the new mode is held until the draining decoders are done, so blocks reach the GPU process in
order. Video blocks are decoded with the minimum upcoming presentation time the client provides,
and expectMinimumUpcomingPresentationTime() isn't forwarded to the GPU process while video is
decoded in the content process. As isReadyForMoreSamples() can be called on any thread and the
decoder only on the renderer's queue, the renderer keeps a copy of the decoder's readiness, updated
after each decode and flush and whenever the decoder's requestMediaDataWhenReady() settles. The
decoder is told when the current time changes (rate change, play, seek, stall cap). Protected
content and formats the content process can't decode keep the compressed path, except encoded
unprotected audio on Cocoa (the new mustDecodeInContentProcess()), for which this is an error.

A decoder that fails to be created, to decode or to drain is reported through the error callback as
a VideoDecodingError or an AudioDecodingError; the GPU track is flushed and the track moves to
TrackState::Mode::Failed, which drops every sample enqueued afterwards. There is no fallback to
decoding in the GPU process.

The preference reaches the renderer as VideoRendererPreference::DecoderInContentProcess and is
forwarded to the GPU process. There, the new WebCoreDecompressionSession::DecodingFlag::HardwareOnly,
set by VideoMediaSampleRenderer when that preference is present, requires a hardware
VTDecompressionSession for the formats the content process can decode in software (VP8, VP9 profile
0 at 8 bits and AV1) and rejects with kVTCouldNotFindVideoDecoderErr instead of falling back to
VideoDecoder. Other formats, such as H.264, HEVC or MPEG-4 Part 2, are still decoded in software in
the GPU process when no hardware decoder is available. AudioVideoRendererAVFObjC reports the failure
as the new PlatformMediaError::HardwareDecoderUnavailable; AudioVideoRendererRemote then flushes the
GPU track, asks the client to re-enqueue, and decodes that codec in the content process using the
new MediaSamplesDecoder::Filter::ForceSoftware, which allows decoding in software the formats
MediaSamplesDecoder otherwise leaves to a hardware decoder.

DecoderInContentProcessEnabled is shared with the GPU process, which uses its own value for the
DecoderInContentProcess renderer preference and rejects a content process whose
DecoderInContentProcess doesn't match it. With it set, the GPU process treats audio that
mustDecodeInContentProcess() says the content process must decode, and any sample that fails
conversion, as coming from a compromised content process, and RemoteAudioVideoRendererProxyManager
rejects them with a MESSAGE_CHECK.

MediaPlayerPrivateWebM::load() calls setPreferences() a second time, which replaces the preferences
set by the first call; that call now keeps DecoderInContentProcess, which the GPU process checks.

* LayoutTests/media/media-source/content/test-flac-manifest.json: Added.
* LayoutTests/media/media-source/content/test-flac.mp4: Added.
* LayoutTests/media/media-source/media-mp4-flac-expected.txt: Added.
* LayoutTests/media/media-source/media-mp4-flac.html: Added.
* Source/WTF/Scripts/Preferences/UnifiedWebPreferences.yaml:
(DecoderInContentProcessEnabled): Added, shared with the GPU process.
* Source/WebCore/html/HTMLMediaElement.cpp:
(WebCore::HTMLMediaElement::loadResource): Set VideoRendererPreference::DecoderInContentProcess
when the preference is enabled.
* Source/WebCore/platform/MediaSamplesBlock.cpp:
(WebCore::MediaSamplesBlock::toMediaSample const): Remove the assertion, the GPU process validates the
result.
* Source/WebCore/platform/MediaSamplesDecoder.cpp:
(WebCore::decoderConfiguration): Accept hardware decodable formats with Filter::ForceSoftware.
* Source/WebCore/platform/MediaSamplesDecoder.h:
* Source/WebCore/platform/MediaStrategy.cpp:
(WebCore::MediaStrategy::createAudioVideoRenderer const):
* Source/WebCore/platform/MediaStrategy.h:
* Source/WebCore/platform/PlatformMediaError.cpp:
(WebCore::convertEnumerationToString):
* Source/WebCore/platform/PlatformMediaError.h: Export convertEnumerationToString() for the WebKit log.
* Source/WebCore/platform/cocoa/VideoToolboxSoftLink.cpp:
* Source/WebCore/platform/cocoa/VideoToolboxSoftLink.h:
* Source/WebCore/platform/graphics/MediaPlayer.cpp:
(WebCore::MediaPlayer::videoRendererPreferences const): Added.
* Source/WebCore/platform/graphics/MediaPlayer.h:
* Source/WebCore/platform/graphics/MediaPlayerEnums.h:
* Source/WebCore/platform/graphics/avfoundation/AudioVideoRendererAVFObjC.mm:
(WebCore::AudioVideoRendererAVFObjC::setVideoRenderer): Report kVTCouldNotFindVideoDecoderErr as
HardwareDecoderUnavailable when DecoderInContentProcess is set.
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.h:
* Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateMediaSourceAVFObjC.mm:
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::createRenderer):
(WebCore::MediaPlayerPrivateMediaSourceAVFObjC::MediaPlayerPrivateMediaSourceAVFObjC):
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.h:
* Source/WebCore/platform/graphics/cocoa/MediaPlayerPrivateWebM.mm:
(WebCore::MediaPlayerPrivateWebM::createRenderer):
(WebCore::MediaPlayerPrivateWebM::MediaPlayerPrivateWebM):
(WebCore::MediaPlayerPrivateWebM::load): Keep DecoderInContentProcess in the second setPreferences() call.
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.h:
* Source/WebCore/platform/graphics/cocoa/VideoMediaSampleRenderer.mm:
(WebCore::VideoMediaSampleRenderer::requiresHardwareDecoder const): Added.
(WebCore::VideoMediaSampleRenderer::decodeNextSampleIfNeeded): Set DecodingFlag::HardwareOnly.
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.h:
* Source/WebCore/platform/graphics/cocoa/WebCoreDecompressionSession.mm:
(WebCore::contentProcessCanDecodeInSoftware): Added.
(WebCore::requiresHardwareDecoder): Added.
(WebCore::WebCoreDecompressionSession::ensureDecoderForSample): Require a hardware decoder with
DecodingFlag::HardwareOnly for the formats the content process can decode in software.
(WebCore::WebCoreDecompressionSession::decodeSampleInternal): Reject those formats instead of
falling back to VideoDecoder.
(WebCore::WebCoreDecompressionSession::decodeSampleSync):
* Source/WebCore/platform/video-codecs/cocoa/VideoDecoderVTBSession.h:
* Source/WebCore/platform/video-codecs/cocoa/VideoDecoderVTBSession.mm:
(WebCore::VideoDecoderVTBSession::create): Use
kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder with HardwareDecoder::Required.
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererDecodingPolicy.h: Added.
(WebKit::mustDecodeInContentProcess):
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.cpp:
(WebKit::RemoteAudioVideoRendererProxyManager::createManager): Set DecoderInContentProcess from the
GPU process preference.
(WebKit::RemoteAudioVideoRendererProxyManager::setPreferences): Check that DecoderInContentProcess
matches the GPU process preference.
(WebKit::RemoteAudioVideoRendererProxyManager::isDecoderInContentProcessEnabled const): Added.
(WebKit::RemoteAudioVideoRendererProxyManager::shouldHaveBeenDecodedInContentProcess const): Added.
(WebKit::RemoteAudioVideoRendererProxyManager::newTrackInfoForTrack): Reject audio the content process
must decode.
(WebKit::RemoteAudioVideoRendererProxyManager::enqueueSample): Ditto, and reject samples that fail
conversion when DecoderInContentProcessEnabled is set.
* Source/WebKit/GPUProcess/media/RemoteAudioVideoRendererProxyManager.h:
* Source/WebKit/Shared/WebCoreArgumentCodersMedia.serialization.in:
* Source/WebKit/WebKit.xcodeproj/project.pbxproj:
* Source/WebKit/WebProcess/GPU/media/AudioVideoRendererRemote.cpp:
(WebKit::AudioVideoRendererRemote::create):
(WebKit::AudioVideoRendererRemote::AudioVideoRendererRemote):
(WebKit::AudioVideoRendererRemote::~AudioVideoRendererRemote):
(WebKit::AudioVideoRendererRemote::shutdown): Release the track states and decoders.
(WebKit::AudioVideoRendererRemote::expectMinimumUpcomingPresentationTime): Don't forward the
minimum while video is decoded in the content process.
(WebKit::AudioVideoRendererRemote::prepareToSeek): Tell the decoders the time changed.
(WebKit::AudioVideoRendererRemote::finishSeek): Ditto.
(WebKit::AudioVideoRendererRemote::removeTrack): Release the track state.
(WebKit::AudioVideoRendererRemote::enqueueSample): Hand the sample to processSample().
(WebKit::AudioVideoRendererRemote::trackState):
(WebKit::AudioVideoRendererRemote::flushDecoders):
(WebKit::AudioVideoRendererRemote::activeConnection):
(WebKit::AudioVideoRendererRemote::processSample):
(WebKit::AudioVideoRendererRemote::selectMode):
(WebKit::contentProcessDecoderFilters):
(WebKit::AudioVideoRendererRemote::beginMode): Fail audio that must be decoded in the content process
but can't be.
(WebKit::AudioVideoRendererRemote::didDrainDecoder):
(WebKit::AudioVideoRendererRemote::didCreateDecoder):
(WebKit::AudioVideoRendererRemote::dispatchBlock):
(WebKit::AudioVideoRendererRemote::resumeWaitingBlocks):
(WebKit::AudioVideoRendererRemote::dropPendingBlocks):
(WebKit::AudioVideoRendererRemote::emitBlock):
(WebKit::AudioVideoRendererRemote::sendHeldBlock):
(WebKit::AudioVideoRendererRemote::sendHeldBlocks):
(WebKit::AudioVideoRendererRemote::releaseDrainedOutput):
(WebKit::AudioVideoRendererRemote::deliverMinimumUpcomingTime):
(WebKit::AudioVideoRendererRemote::decrementPendingSamples):
(WebKit::AudioVideoRendererRemote::sendBlock):
(WebKit::AudioVideoRendererRemote::sendDecodedBlock):
(WebKit::AudioVideoRendererRemote::didDecode):
(WebKit::AudioVideoRendererRemote::decoderFailed):
(WebKit::AudioVideoRendererRemote::hardwareDecoderUnavailable):
(WebKit::AudioVideoRendererRemote::requestVideoReenqueue):
(WebKit::AudioVideoRendererRemote::willFlushTrack):
(WebKit::AudioVideoRendererRemote::releaseTrackState):
(WebKit::AudioVideoRendererRemote::releaseAllDecoders):
(WebKit::AudioVideoRendererRemote::notifyDecodersOfTimeChange):
(WebKit::AudioVideoRendererRemote::updateDecoderReadiness): Added.
(WebKit::AudioVideoRendererRemote::isReadyForMoreSamples):
(WebKit::AudioVideoRendererRemote::currentTime const):
(WebKit::AudioVideoRendererRemote::playbackPosition const): Added.
(WebKit::AudioVideoRendererRemote::notifyTimeReachedAndStall): Tell the decoders the time changed.
(WebKit::AudioVideoRendererRemote::cancelTimeReachedAction): Ditto.
(WebKit::AudioVideoRendererRemote::flushTrack): Flush the decoders before the GPU track.
(WebKit::AudioVideoRendererRemote::sendFlushTrack): Split out of flushTrack().
(WebKit::AudioVideoRendererRemote::readyForMoreDataStateIfExists): Added.
(WebKit::AudioVideoRendererRemote::gpuProcessConnectionDidClose): Release the decoders.
(WebKit::AudioVideoRendererRemote::MessageReceiver::effectiveRateChanged): Tell the decoders the time changed.
(WebKit::AudioVideoRendererRemote::MessageReceiver::errorOccurred): Hand HardwareDecoderUnavailable
to hardwareDecoderUnavailable().
(WebKit::AudioVideoRendererRemote::MessageReceiver::readyForMoreMediaData):
* Source/WebKit/WebProcess/GPU/media/AudioVideoRendererRemote.h:
* Source/WebKit/WebProcess/GPU/media/WebMediaStrategy.cpp:
(WebKit::WebMediaStrategy::createAudioVideoRenderer const):
* Source/WebKit/WebProcess/GPU/media/WebMediaStrategy.h:
* Tools/TestWebKitAPI/Tests/WebCore/cocoa/MediaSamplesDecoderTests.mm:
(TestWebKitAPI::TEST(MediaSamplesDecoder, CanDecodeVP9WhenHardwareDecoderIsUnavailable)): Added.

8f93d68

Misc	iOS, visionOS, tvOS & watchOS	macOS	Linux	Windows	Apple Internal
✅ 🧪 style	✅ 🛠 ios	✅ 🛠 mac	✅ 🛠 wpe	✅ 🛠 win	⏳ 🛠 ios-apple
✅ 🧪 bindings	✅ 🛠 ios-sim	✅ 🛠 mac-cmake-debug	✅ 🧪 wpe-wk2	✅ 🧪 win-tests	⏳ 🛠 mac-apple
✅ 🧪 webkitperl	✅ 🧪 ios-wk2	✅ 🧪 api-mac	✅ 🧪 api-wpe		⏳ 🛠 vision-apple
	✅ 🧪 ios-wk2-wpt	✅ 🧪 api-mac-cmake-debug	🧪 jsc-wpe
✅ 🧪 jsc-x86-64	✅ 🧪 api-ios	❌ 🧪 mac-wk2	✅ 🛠 gtk3-gcc
✅ 🛠 🧪 jsc-debug-arm64	✅ 🛠 ios-safer-cpp	❌ 🧪 mac-cmake-debug-wk2	✅ 🛠 gtk
	✅ 🛠 vision	❌ 🧪 mac-wk2-stress	✅ 🧪 gtk-wk2
	✅ 🛠 vision-sim	🧪 mac-intel-wk2	✅ 🧪 api-gtk
	✅ 🧪 vision-wk2	✅ 🛠 mac-safer-cpp	✅ 🛠 playstation
	✅ 🛠 tv	✅ 🧪 mac-site-isolation
	✅ 🛠 tv-sim
	✅ 🛠 watch
	✅ 🛠 watch-sim