Playlist: display fallback seekbar style waveform for random waveforms - #81028

👋 Thanks for your first Pull Request and for helping build the future of Gutenberg and WordPress, @Intenzi! In case you missed it, we'd love to have you join us in our Slack community.

If you want to learn more about WordPress development in general, check out the Core Handbook full of helpful information.

Thanks for this fix @Intenzi - looks good to me. Can you resolve the merge conflicts and we can get this fix in?

@jeryj 👋🏻 just doing some light triage and wondering if this fix is still required. Cheers!

I believe @scruffian is familiar with this bug. Could you take a look?

Gave this a test locally and it does what it says on the tin. One thing worth sorting out is the version bump, which trunk has already moved past.

I had Claude set up the repro and run it both ways, here is what came back:

Tested on a local wp-env build with a two track Playlist: track 1 pointed at https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3 (which serves no Access-Control-Allow-Origin header), track 2 a local upload. Same post, same page, built twice - once with trunk's waveform-utils.js and once with this branch.

Before, the CORS blocked track draws a convincing waveform that has nothing to do with the audio:

After, it falls back to the flat seekbar:

The local track still renders real bars, and switching between the two flips the style correctly in both directions:

Eyeballing two waveforms is not much of a test, so the canvas pixels were measured as well - standard deviation of the per column bar heights. Before: 64.22 on the CORS blocked track vs 63.68 on the local one, ie. statistically indistinguishable, which is the misleading part. After: 0.00 on the CORS blocked track, 63.68 on the local one.

Note on the dependency bump: trunk is already on @arraypress/waveform-player@^1.26.0, past the ^1.24.0 here, and that is what the merge conflict is. Dropping c779b68 and keeping just the waveform-utils.js change should clear it. 1.26.0 still adds waveform-is-placeholder on fetch failure and still calls options.onLoad, so nothing here depends on 1.24 specifically.

Also worth noting there is no visible flash of the placeholder waveform before the seekbar replaces it - the library runs drawWaveform(), renderMarkers() and onLoad() in the same tick, so the redraw lands before the browser paints.

Can you merge trunk and drop the bump commit? I'm happy to re-test once that is in.

I'm not very keen on this idea. I think if users choose bars then we should honour that choice, even if they aren't accurate. I think this will feel like a more broken experience. Instead, why don't we try saving the waveform data in the block?

Instead, why don't we try saving the waveform data in the block?

Thats a great idea, maybe we can try this.

I'm not very keen on this idea. I think if users choose bars then we should honour that choice, even if they aren't accurate. I think this will feel like a more broken experience.

Has the user chosen bars in this case? I understood it was caused by a CORS error when the Audio URL was on a remote server that didn't send CORS headers. Maybe @Intenzi can clarify.

Thankyou for the response, I haven't been receiving any mails on this. I'll follow up with a response and required clarifications for the same by today.

To answer @adamsilverstein 's question: Yes, when a user has chosen any playstyle (mirror/dots/etc.) for the track, it will revert to a flat seekbar to visibly display that the playlist block was unable to get the waveform for this track.

I completely agree with @scruffian 's thought process. "If a user wants to display a dot pattern on their song, it should do that and not suddenly show a flat seekbar." is a very valid concern.

There are two technical limitations to why I have not approached storing the waveform into the block locally or in a database:-
a> It doesn't resolve the existing issue: #80534 i.e. any audio file that is stored in externally hosted platform (such as Amazon S3, Cloudflare or CDNs) will face the same issue. We won't be able to obtain the waveform to then be looking towards storing it into the post/block.
As:

• Playing an audio file requires no permissions (can be done through tag without CORS restrictions)
• Analyzing the waveform of an audio requires a download of the raw audio file:

export async function generateWaveform(url, samples = DEFAULT_SAMPLES, shouldDetectBPM = false) {
  let audioContext;
  try {
      const AudioCtx = window.AudioContext || window.webkitAudioContext;
      audioContext = new AudioCtx();
      const response = await fetch(url);
      const arrayBuffer = await response.arrayBuffer();
      const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);

      let peaks = extractPeaks(audioBuffer, samples);
      ...

• This gets blocked and thus we won't obtain the waveform to then be able to followup with an approach of saving it.

b> If we decide to save the waveform data somehow (which is by default an array of 1800 float values per track) into the post content, it will lead to a much larger initial payload size when anyone requests for a page with multiple tracks in the playlist block.

1. The crux UX reason for the present solution is to prevent any fake waveform from showing up as it is intentionally misleading to a user.

2. Imagine a track that is completely silent in the beginning but still shows visual peaks. It becomes significantly more manual to identify that there is an issue with obtaining the waveform. This leads to an audio-visual discrepancy for the viewers.

3. The current solution thus makes it visible in the editor that the track is unable to render the chosen waveform style because the waveform peaks cannot be extracted from the remote audio source they have provided. As a possible enhancement: we could also display the same via text to show an editor-only warning notice to make this clear to the author, while the frontend simply renders the clean seekbar without showing any error notices to visitors.

I looked a bit further, as an industrial precedent : SoundCloud also follows the UX of falling back to a flat seekbar when no metadata could be obtained/waveform couldn't be analysed.

I am open to any clarifications and suggestions, improvements we can utilise to make the experience better for the end user. Please comment if any further questions arise for the PR and the topic.

@Intenzi you make some great points. Can we explore both options - save the waveform locally if possible (as demonstrated in #83086) and then still fallback to showing the seekbar if not possible?

🤖 PR meta 🤖

🎉 Props

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: Intenzi <intenzi@git.wordpress.org>
Co-authored-by: adamsilverstein <adamsilverstein@git.wordpress.org>
Co-authored-by: ramonjd <ramonopoly@git.wordpress.org>
Co-authored-by: jeryj <jeryj@git.wordpress.org>
Co-authored-by: scruffian <scruffian@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Updated as activity occurs, without notifying anyone named here. Add the props-bot label to refresh.

👋 @scruffian I've gone through the PR #83086 , I think its a good optimisation towards caching waveforms. Great work there!

I understand that it avoids having to download and decode the full audio file on the front end everytime just to generate the waveform.
Like I mentioned in my earlier comment, I am not very keen on having waveforms being stored directly in the page itself, it feels a little sub-optimised as its only scoped to the page in particular whereas this should be an audio file level optimisation.

I just went through #84712 and I think it makes sense if we can move the saving of the waveform to the database separately as attachment metadata, as proposed there. Would you be willing to look into that approach? I think it's a great middle ground that avoids post bloat and keeps the data reusable across other blocks.

Regardless, the solution in this PR is separate and complimentary :) It targets scenarios where an audio file is missing, deleted or blocked by CORS. Ensuring it cleanly falls back to a flat seekbar rather than a fake waveform. This fallback can be used as the safety net to build further on top the optimisation PR you are working towards 😄 .