-
Notifications
You must be signed in to change notification settings - Fork 171
linux: Implement libiamf-based IAMF decoder (#4221) #7851
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Implements a libiamf-based audio decoder. Decoder specific code is guarded behind a GN flag `enable_iamf_decode,` which controls the ENABLE_IAMF_DECODE macro. `enable_iamf_decode` is off by default, so the new code doesn't build. b/341792042
|
/gemini reviw |
|
/gemini review |
|
It looks like you're trying to request a code review. To initiate a new code review, please create a new top-level comment on the pull request with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a libiamf-based IAMF audio decoder for Linux, controlled by the enable_iamf_decode build flag. The changes are comprehensive, covering the decoder implementation, integration into the media pipeline, build system updates, and support checks. However, my review identified several critical issues that will prevent the code from compiling, including commented-out code in header files and incorrect use of const qualifiers. I have also pointed out a high-severity issue related to potential integer overflow and medium-severity issues regarding code style and performance. Addressing these points will be necessary to ensure the new decoder is correct and robust.
| #include "starboard/shared/starboard/player/decoded_audio_internal.h" | ||
| #include "starboard/shared/starboard/player/filter/audio_decoder_internal.h" | ||
| #include "starboard/shared/starboard/player/job_queue.h" | ||
| // #include "third_party/libiamf/source/code/include/IAMF_decoder.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This include is necessary for the declaration of IAMF_DecoderHandle and the IAMF decoder functions used in the implementation. It appears to have been commented out by mistake, which will cause a compilation failure.
| // #include "third_party/libiamf/source/code/include/IAMF_decoder.h" | |
| #include "third_party/libiamf/source/code/include/IAMF_decoder.h" |
| bool ConfigureDecoder(const IamfBufferInfo* info, | ||
| int64_t timestamp, | ||
| std::string* error_message) const; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ConfigureDecoder method modifies the state of the IAMF decoder via IAMF_decoder_set_* functions. Therefore, it should not be declared as a const method, as this will result in a compilation error.
| bool ConfigureDecoder(const IamfBufferInfo* info, | |
| int64_t timestamp, | |
| std::string* error_message) const; | |
| bool ConfigureDecoder(const IamfBufferInfo* info, | |
| int64_t timestamp, | |
| std::string* error_message); |
| OutputCB output_cb_; | ||
| ErrorCB error_cb_; | ||
|
|
||
| // IAMF_DecoderHandle decoder_ = nullptr; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| std::vector<uint8_t> config_obus; | ||
| size_t config_obus_size; | ||
| std::vector<uint8_t> data; | ||
| const scoped_refptr<InputBuffer> input_buffer; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The input_buffer member is declared as const but is not initialized in a constructor, and the struct does not have one. This will cause a compilation error. To fix this, you can remove the const qualifier, which will allow it to be assigned a value after the IamfBufferInfo object is created.
| const scoped_refptr<InputBuffer> input_buffer; | |
| scoped_refptr<InputBuffer> input_buffer; |
| return false; | ||
| } | ||
|
|
||
| int next_obu_pos = reader->pos() + obu_size; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The variable next_obu_pos is calculated from reader->pos() which returns a size_t and obu_size which is a uint32_t. Storing the result in a signed int can lead to data loss, truncation, or signed overflow issues on 64-bit platforms or with large buffers. It should be declared as size_t to match the type of reader->pos() and to correctly handle large buffer sizes.
| int next_obu_pos = reader->pos() + obu_size; | |
| size_t next_obu_pos = reader->pos() + obu_size; |
| IAMF_decoder_decode(decoder_, info.data.data(), info.data.size(), nullptr, | ||
| const_cast<uint8_t*>(decoded_audio->data())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The const_cast here is unnecessary. decoded_audio is a non-const scoped_refptr<DecodedAudio>, so decoded_audio->data() resolves to the non-const overload uint8_t* DecodedAudio::data(). Using const_cast when not needed can be confusing and may hide actual const-correctness issues.
| IAMF_decoder_decode(decoder_, info.data.data(), info.data.size(), nullptr, | |
| const_cast<uint8_t*>(decoded_audio->data())); | |
| IAMF_decoder_decode(decoder_, info.data.data(), info.data.size(), nullptr, | |
| decoded_audio->data()); |
| info->config_obus.assign(reader.buf(), reader.buf() + info->config_obus_size); | ||
| info->data.assign(reader.buf() + info->config_obus_size, | ||
| reader.buf() + reader.size()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These assign calls copy data from the input buffer into new vectors. This is inefficient as it involves memory allocation and copying. Consider modifying IamfBufferInfo to use a span-like type (e.g., base::span<const uint8_t>) instead of std::vector<uint8_t> for config_obus and data. This would allow you to create views into the original input_buffer's data without copying. The IamfBufferInfo::input_buffer member can be used to keep the underlying data alive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request introduces a libiamf-based IAMF decoder for Linux. The changes look good overall, adding the necessary build configurations, decoder implementation, and support checks. However, I've found several critical issues that will prevent the code from compiling or cause runtime errors, including a potential security vulnerability due to an out-of-bounds read. Please see my detailed comments for suggestions on how to address these issues.
| #include "starboard/shared/starboard/player/decoded_audio_internal.h" | ||
| #include "starboard/shared/starboard/player/filter/audio_decoder_internal.h" | ||
| #include "starboard/shared/starboard/player/job_queue.h" | ||
| // #include "third_party/libiamf/source/code/include/IAMF_decoder.h" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The header IAMF_decoder.h that defines IAMF_DecoderHandle is commented out. This will cause compilation errors as the type IAMF_DecoderHandle (used on line 66) will be undefined. This line should be uncommented.
| // #include "third_party/libiamf/source/code/include/IAMF_decoder.h" | |
| #include "third_party/libiamf/source/code/include/IAMF_decoder.h" |
| OutputCB output_cb_; | ||
| ErrorCB error_cb_; | ||
|
|
||
| // IAMF_DecoderHandle decoder_ = nullptr; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| str->clear(); | ||
| str->resize(max_bytes_to_read); | ||
|
|
||
| int bytes_read = ::starboard::strlcpy( | ||
| str->data(), reinterpret_cast<const char*>(buf), max_bytes_to_read); | ||
| if (bytes_read == max_bytes_to_read) { | ||
| // Ensure that the string is null terminated. | ||
| if (buf[bytes_read] != '\0') { | ||
| return -1; | ||
| } | ||
| } | ||
| str->resize(bytes_read); | ||
|
|
||
| // Account for null terminator. | ||
| return ++bytes_read; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current implementation of ReadStringInternal has a potential out-of-bounds read. If the source string has a length equal to max_bytes_to_read, the check buf[bytes_read] accesses memory outside the intended bounds, which is a security risk. A safer and cleaner implementation would be to first locate the null terminator within the given buffer limit before copying the string.
const void* null_terminator = memchr(buf, '\0', max_bytes_to_read);
if (!null_terminator) {
// Not null-terminated within max_bytes_to_read.
return -1;
}
int len = static_cast<const uint8_t*>(null_terminator) - buf;
str->assign(reinterpret_cast<const char*>(buf), len);
return len + 1;|
|
||
| int samples_decoded = | ||
| IAMF_decoder_decode(decoder_, info.data.data(), info.data.size(), nullptr, | ||
| const_cast<uint8_t*>(decoded_audio->data())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using const_cast to get a writable buffer from decoded_audio->data() is unsafe and not best practice. The DecodedAudio class provides a writable_data() method for this purpose, which should be used instead to ensure type safety and clarity.
| const_cast<uint8_t*>(decoded_audio->data())); | |
| decoded_audio->writable_data()); |
| std::optional<uint32_t> binaural_audio_element_id; | ||
| std::optional<uint32_t> surround_audio_element_id; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The variables binaural_audio_element_id and surround_audio_element_id are local to ParseDescriptorOBU. Since ParseDescriptorOBU is called in a loop within ParseInputBuffer, any parsed audio element information is lost between calls. This can lead to incorrect mix presentation selection if the AudioElement OBU and MixPresentation OBU are not parsed in the same call.
To fix this, these variables should be declared in ParseInputBuffer and passed as pointers to ParseDescriptorOBU to preserve their state across multiple OBU parsings within a single input buffer.
| std::vector<uint8_t> config_obus; | ||
| size_t config_obus_size; | ||
| std::vector<uint8_t> data; | ||
| const scoped_refptr<InputBuffer> input_buffer; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and you may need fix the build failure.
2025-11-04T00:28:04.4024028Z 54 | int GetMaxChannelCount() {
2025-11-04T00:28:04.4024429Z | ^~~~~~~~~~~~~~~~~~
2025-11-04T00:28:04.4024778Z 1 error generated.
| #if ENABLE_IAMF_DECODE | ||
| if (audio_stream_info.codec == kSbMediaAudioCodecIamf) { | ||
| // IAMF mixes audio signals to the highest available speaker layout. | ||
| // TODO: Set the number of channels below Starboard. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: bug number?
| @@ -0,0 +1,327 @@ | |||
| // Copyright 2024 The Cobalt Authors. All Rights Reserved. | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: 2025?
|
|
||
| void IamfAudioDecoder::Initialize(const OutputCB& output_cb, | ||
| const ErrorCB& error_cb) { | ||
| SB_DCHECK(BelongsToCurrentThread()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SB_CHECK on new code
| decoder_is_configured_ = true; | ||
| } | ||
|
|
||
| scoped_refptr<DecodedAudio> decoded_audio = new DecodedAudio( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can't we use unique_ptr instead?
Reference-counted objects make it difficult to understand ownership and destruction order, especially when multiple threads are involved.
Cherry-picks the C25 PR that implemented a Linux libiamf-based IAMF decoder. Minor changes were made to correct build errors/bring the code to more modern standards. The original commit message follows:
Implements a libiamf-based audio decoder.
Decoder specific code is guarded behind a GN flag
enable_iamf_decode,which controls the ENABLE_IAMF_DECODE macro.enable_iamf_decodeis off by default, so the new code doesn't build.b/341792042
Bug: 376312767