Page MenuHomePhabricator

Fix: Proper handling of AV1 codec string in TimedMediaHandler
Closed, ResolvedPublicBUG REPORT

Description

Problem:
Currently, TimedMediaHandler incorrectly handles AV1 codec strings in WebMHandler.php. The codec string for AV1 is passed as lowercase (av1), which is not compliant with browser and VideoJS requirements. AV1 codec strings are case-sensitive and must follow the format defined by the AV1 specification, such as av01.<profile>.<level><tier>.<bitDepth>.

This issue prevents original AV1 video files uploaded to MediaWiki from being played through the VideoJS player. Instead, users see the error:
"No compatible source was found for this media."

Steps to Reproduce:

  1. Upload an AV1 video file
  2. Check the MIME type generated by getWebType() in WebMHandler.php.

Current output: video/webm; codecs="av1, opus"
Correct output: video/webm; codecs="av01.0.05M.08, opus"

  1. Attempt to play the video in VideoJS or a browser.

Without this fix, playback fails with the error: "No compatible source was found for this media.". There is also no way to watch the video source in Videojs.

Proposed Solution (How I solved the problem in my local wiki:):
The following changes have been made to the WebMHandler.php file:

  1. Preserve case sensitivity for AV1 codec strings:

A custom function (getAV1CodecString) was implemented to dynamically generate the correct codec string based on the video metadata.

  1. Dynamic generation of AV1 codec string:

The getAV1CodecString function analyzes video metadata (e.g., resolution, bit depth, frame rate) to construct a codec string that complies with the AV1 specification.
For example: A 1080p video at 30fps, 8-bit depth is assigned av01.0.05M.08. A 4K video at 60fps, 10-bit depth is assigned av01.0.06H.10 etc

  1. Case-sensitive handling in getWebType:

Only the AV1 codec string retains its case-sensitive format. All other codecs (e.g., VP8, VP9, Opus) are converted to lowercase for consistency.

	/**
	 * Returns the MIME type for the file, including codecs information.
	 *
	 * @param File $file The file object.
	 * @return string The MIME type including the codecs.
	 */
	public function getWebType( $file ) {
		// Determine the base type (audio or video) based on file dimensions
		$baseType = ( $file->getWidth() === 0 && $file->getHeight() === 0 ) ? 'audio' : 'video';

		// Get the stream types (codecs) from the file metadata
		$streams = $this->getStreamTypes( $file );
		if ( !$streams ) {
			// Return the base type with WebM format if no streams are found
			return $baseType . '/webm';
		}

		// Process codecs: Keep AV1 in original case, convert others to lowercase
		$processedStreams = array_map( function ( $codec ) {
			return stripos( $codec, 'av01.' ) === 0 ? $codec : strtolower( $codec );
		}, $streams );

		// Combine processed streams into a single codec string
		$codecs = implode( ', ', $processedStreams );
		return $baseType . '/webm; codecs="' . $codecs . '"';
	}
	
	/**
	 * Generates a codec string for AV1 based on the metadata.
	 *
	 * @param array $metadata Video metadata.
	 * @return string The codec string for AV1.
	 */
	private function getAV1CodecString( $metadata ) {
		// Default values in case metadata is incomplete
		$profile = '0'; // Default: Main Profile
		$level = '05';  // Default: Level 5.0
		$tier = 'M';    // Default: Main Tier
		$bitDepth = '08'; // Default: 8-bit

		// Check for bit depth
		if ( isset( $metadata['video']['bit_depth'] ) ) {
			if ( $metadata['video']['bit_depth'] === 10 ) {
				$bitDepth = '10'; // 10-bit depth
			} elseif ( $metadata['video']['bit_depth'] === 12 ) {
				$bitDepth = '12'; // 12-bit depth (rare)
			}
		}

		// Analyze resolution and adjust level
		if ( isset( $metadata['video']['resolution_x'] ) && isset( $metadata['video']['resolution_y'] ) ) {
			$width = $metadata['video']['resolution_x'];
			$height = $metadata['video']['resolution_y'];

			// Determine level based on resolution and frame rate
			if ( isset( $metadata['video']['frame_rate'] ) ) {
				$frameRate = $metadata['video']['frame_rate'];
			} else {
				$frameRate = 30; // Assume a default frame rate if not provided
			}

			$pixelsPerSecond = $width * $height * $frameRate;

			// Assign level based on the AV1 specification (simplified mapping)
			if ( $pixelsPerSecond <= 829440 ) { // <= 720p@30fps
				$level = '02';
			} elseif ( $pixelsPerSecond <= 2073600 ) { // <= 1080p@30fps
				$level = '03';
			} elseif ( $pixelsPerSecond <= 3686400 ) { // <= 1080p@60fps or 1440p@30fps
				$level = '04';
			} elseif ( $pixelsPerSecond <= 7372800 ) { // <= 4K@30fps
				$level = '05';
			} else {
				$level = '06'; // Higher resolutions or frame rates
			}
		}

		// Check for tier, assume Main Tier by default
		if ( isset( $metadata['video']['tier'] ) && $metadata['video']['tier'] === 'high' ) {
			$tier = 'H'; // High Tier
		}

		// Construct the codec string
		return "av01.$profile.$level$tier.$bitDepth";
	}

	/**
	 * @param File $file
	 * @return string[]|false
	 */
	public function getStreamTypes( $file ) {
		$streamTypes = [];
		$metadata = $this->unpackMetadata( $file->getMetadata() );
		if ( !$metadata || isset( $metadata['error'] ) ) {
			return false;
		}
		// id3 gives 'V_VP8' for what we call VP8
		if ( isset( $metadata['video'] ) ) {
			if ( $metadata['video']['dataformat'] === 'vp8' ) {
				$streamTypes[] = 'VP8';
			} elseif ( $metadata['video']['dataformat'] === 'vp9'
				|| $metadata['video']['dataformat'] === 'V_VP9'
			) {
				// Currently getID3 calls it V_VP9. That will probably change to vp9
				// once getID3 actually gets support for the codec.
				$streamTypes[] = 'VP9';
			} elseif ( $metadata['video']['dataformat'] === 'V_AV1' ) {
				$streamTypes[] = $this->getAV1CodecString( $metadata );
			}
		}
		if ( isset( $metadata['audio'] ) ) {
			if ( $metadata['audio']['dataformat'] === 'vorbis' ) {
				$streamTypes[] = 'Vorbis';
			} elseif ( $metadata['audio']['dataformat'] === 'opus'
				|| $metadata['audio']['dataformat'] === 'A_OPUS'
			) {
				// Currently getID3 calls it A_OPUS. That will probably change to 'opus'
				// once getID3 actually gets support for the codec.
				$streamTypes[] = 'opus';
			}
		}

		return $streamTypes;
	}

This issue might appear like a feature request, but it is fundamentally a bug since it breaks playback for AV1 files uploaded to MediaWiki. Correct MIME types are a requirement for browser and VideoJS compatibility.

Event Timeline

@PeaceDeadC do you want to submit a patch ?
https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker
https://www.mediawiki.org/wiki/Gerrit/Tutorial

If not, i might do it later, but can you then share how you would like to be credited (pseudonym or real name, with ur email or without an email, etc)

@PeaceDeadC do you want to submit a patch ?
https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker
https://www.mediawiki.org/wiki/Gerrit/Tutorial

If not, i might do it later, but can you then share how you would like to be credited (pseudonym or real name, with ur email or without an email, etc)

I posted here because I am currently having issues logging into Gerrit. Unfortunately, I am unable to log in or reset my password there. Until this issue is resolved, I can only suggest that someone else submits the patch on my behalf. You can credit me using my pseudonym.

I posted here because I am currently having issues logging into Gerrit. Unfortunately, I am unable to log in or reset my password there.

https://idp.wikimedia.org should allow you to reset your LDAP password used for Gerrit. If there is an issue, please elaborate. Thanks!

Change #1104397 had a related patch set uploaded (by PeaceDeadC; author: PeaceDeadC):

[mediawiki/extensions/TimedMediaHandler@master] Fix: Proper handling of AV1 codec string in WebMHandler

https://gerrit.wikimedia.org/r/1104397

https://idp.wikimedia.org should allow you to reset your LDAP password used for Gerrit. If there is an issue, please elaborate. Thanks!

I finally managed to reset my password, and afterward, I was able to log in to Gerrit. I'm not sure what the issue was, but it wasn't working as of yesterday. I've successfully submitted a patch for review. Thank you for your assistance!

Thank you, I'll review somewhere this week hopefully.

Ok we left off finding that some of the detailed stuff isn't available in metadata we get from GetID3, so we'll want to follow up on the upstream side later to process codec data for VP8, VP9, and AV1 at least (and maybe H.264/H.265 for good measure though we don't _yet_ allow those codecs).

I'll spend some time today cleaning it up to leave good "fixme" points to improve when upstream has the info, and skip the test cases we can't pass yet. I'll add a separate task for going after the upstream fixes and getting that integrated.

Comments were pretty good so I'm just merging the last version, rebased, with the failing test case commented out until the fixmes are resolved with upstream. ;D

Change #1104397 merged by jenkins-bot:

[mediawiki/extensions/TimedMediaHandler@master] Fix: Proper handling of AV1 codec string in WebMHandler

https://gerrit.wikimedia.org/r/1104397

I'm going to provisionally close this out as we've merged and it'll go out with future updates. If any surprises, reopen or file new issue with specific updates. :D Thanks everybody for your contributions! We're gonna catch up on this tech debt or else :D