Repository navigation
Add JPEG XL support - #77584
Add JPEG XL support#77584adamsilverstein wants to merge 67 commits into
Conversation
Add client-side JPEG XL support without growing the vips worker bundle. The 3 MB vips-jxl.wasm module is split into its own @wordpress/vips/jxl-wasm script module that is only fetched the first time a JXL image is processed. How it works: - packages/vips/src/jxl-wasm.ts is a tiny new script module whose only job is to export the base64 data URL for vips-jxl.wasm. - @wordpress/vips/jxl-wasm is registered as a new wpScriptModuleExports entry so the build emits build/modules/vips/jxl-wasm.min.js (~3 MB) as a separately loadable module. - vips-worker.ts adds vipsEnsureJxlSupport(), which on first call dynamic-imports '@wordpress/vips/jxl-wasm' to get the data URL, then RPC-passes it to the worker via setJxlWasmUrl(). - The worker (packages/vips/src/index.ts) stores the URL, adds vips-jxl.wasm to dynamicLibraries on the next getVips() call, and returns the URL from locateFile(). If vips was already initialized without JXL, the existing instance is discarded so the reinit picks up JXL support. - packages/upload-media calls vipsEnsureJxlSupport() from prepareItem whenever the input or output type is image/jxl. image/jxl is added to CLIENT_SIDE_SUPPORTED_MIME_TYPES and 'jxl' to VALID_IMAGE_FORMATS and the ImageFormat type. The vipsConvertImageFormat wrapper MIME union is widened accordingly. - JXL encoding uses effort=3 (libvips default 7 is too slow for interactive use). Size impact: - worker.min.js: unchanged (~13.1 MB, same as trunk). - jxl-wasm.min.js: new separate module (~3.0 MB), fetched only when a JXL image is encountered. Alternative to #77570, which bundles vips-jxl.wasm directly into the worker (+3 MB on every editor page load). Opened so the size bot can compare. Refs #76981.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
Size Change: +850 kB (+10.74%) Total Size: 8.76 MB 📦 View Changed
|
|
Flaky tests detected in 4ce3bac. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/33117672777 Navigates the items list via UP/DOWN arrow keys in
|
Resolve conflict in packages/upload-media/src/store/private-actions.ts: trunk moved image format transcoding from client-side prepareItem to server-driven thumbnail generation. Adopt the new pipeline and keep the JXL WASM lazy-load, but trigger it in two places: prepareItem for JXL input files, and transcodeImageItem for server-requested JXL output.
|
A few notes from a review pass: Bundle size / lighter-weight JXL options (re #76981)
Minor code note
|
JXL is not broadly web-compatible: most browsers (including Chrome) cannot display it and the server cannot read it (GD/Imagick have no JXL decoder, and fileinfo reports it as image/x-jxl). Uploading JXL as-is was rejected outright — by the editor's allowed-MIME check and by core's wp_check_filetype_and_ext() — and even when allowed produced an undisplayable attachment with no dimensions or sub-sizes. Decode JXL to JPEG client-side with vips (the JXL WASM module is already lazy-loaded on demand) and upload the JPEG, mirroring how HEIC is handled. The original .jxl is preserved as a companion file in $metadata['original'] so no data is lost. The editor and front end now use the portable JPEG, with real dimensions and the full set of sub-sizes. - Register image/jxl as an allowed upload MIME type, and restore the type during validation via a magic-byte-checked wp_check_filetype_and_ext filter so the sideloaded original passes despite the image/x-jxl finfo mismatch. - Add original-jxl sideload handling to the REST controller, skipping the dimension read since JXL cannot be measured server-side. - Generalize the HEIC companion delete hook to clean up JXL originals too. - Add an e2e test and JXL asset covering the conversion and companion.
# Conflicts: # packages/upload-media/src/store/private-actions.ts
|
I did some testing on this feature, made some small fixes and got it working, it now properly handles uploaded JXL files. I created a JXL using Squoosh for testing: Because JPEG XL (JXL) is not supported in Chromium where client-side media is active, and generally to provide a web-safe format for users, I decided to treat JXL uploads similar to HEIC uploads. So we do the following:
The output format should also honor the See screencast: jxl.to.jpeg.on.upload.mp4 |
|
I will work on a core backport once the core client-side media feature is restored. |
# Conflicts: # packages/vips/CHANGELOG.md # packages/vips/README.md
# Conflicts: # packages/upload-media/CHANGELOG.md # packages/upload-media/src/store/private-actions.ts # packages/upload-media/src/store/test/actions.ts # packages/vips/CHANGELOG.md # packages/vips/src/index.ts
Trunk converted the private actions tests to TypeScript, and the merge brought this test in untyped, which failed the typecheck. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
The JXL flag only flipped once the init resolved, so a second JXL operation arriving mid-init mistook the pending JXL instance for a stale one and shut it down under the first. Record the flag when the init starts, and defer shutting down a retired instance until no operation is still using it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
The recycle guard only counts ResizeCrop and Rotate, so another item's completed resize could terminate the worker between sending it the JXL library and decoding, or mid-decode. The convert then ran on a fresh worker without JXL support and a valid file failed with a decode error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Core also empties the type when image/jxl is not an allowed MIME type, so restoring it whenever the type was empty let JXL uploads through on sites that removed it, such as a network whose upload file types leave it out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
The JXL to JPEG conversion passed only the quality, so it always stripped metadata, even on sites that filter image_strip_meta to keep it, and every sub-size made from that JPEG lost it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Every JXL was converted to JPEG, which has no alpha channel, so a transparent JXL came out with its transparent areas filled in. Check for transparency first and convert those to PNG, as the PNG to JPEG path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
The cached download was a copy of the module's own bytes, so once a JXL was processed the main thread held two copies of the ~3 MB library for the rest of the session. Cache the module's bytes and copy only when sending them to a worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Its only consumer excluded JXL again straight away, since JXL takes its own conversion branch, so listing it did nothing but read a header that never yields dimensions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
The test graph check rejected the untyped mock against the generic signature, and the private actions tests never reach the JXL branch that calls it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Cover JXL uploads that the client-side pipeline mishandles: several at once, transparent ones, sites that keep image metadata, and sites that do not allow JXL at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
Without generate_sub_sizes set to false, the server's check for image types it cannot resize rejected the upload before the file type check the test is about could run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
7.2 is in beta, so the core backport targets the next release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AcYRm3egtiB9yfX1VFaatb
# Conflicts: # packages/upload-media/CHANGELOG.md # packages/vips/CHANGELOG.md
Keeping JXL uploads in their own format means the server has to record their width and height and accept their sub-size sideloads, but getimagesize() cannot read JXL. Parse the codestream SizeHeader instead, for the attachment metadata and the sideload endpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
Chrome 155 and Firefox 158 now display JPEG XL, so there is no longer a reason to convert JXL uploads to JPEG and keep the original on the side. Treat JXL like any other vips-supported image: upload it as-is and generate its sub-sizes as JXL on the client. The vips wrappers load JXL support whenever they read or write JXL, holding the worker so a recycle cannot drop it in between. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
The e2e tests asserted the old JPEG conversion and companion original. Check instead that the upload and its sub-sizes are JXL, that HDR and gain-map uploads are stored byte-for-byte, and that transparency carries into the sub-sizes. The test browser cannot display JXL yet, so these tests no longer wait for the image to render. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
|
Browser support for JXL has moved a lot since I wrote #77584 (comment), enough that converting JXL uploads to JPEG no longer makes sense. I've switched this PR to treat JXL like any other image: the upload and all of its sub-sizes stay JXL. Claude pulled together where browsers stand today, and what changed in the PR:
The core backport in WordPress/wordpress-develop#12006 will need the same change. |
The JXL upload_mimes filter ran after core's check_upload_mimes(), so it added JXL back on networks whose upload file types leave it out. Register it first. Also accept untyped arguments in the filetype filter, since core does not type them and some callers pass null, and reuse the codestream reader for the JXL sniff instead of a second copy of the signatures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
The recycle guard only knew about ResizeCrop, Rotate and JXL work, so a transcode, transparency check or UltraHDR probe could have its worker terminated mid-call and never settle. Route every vips wrapper through withVipsWorkerHeld instead of special-casing JXL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
The recipient was only recorded after the RPC resolved, so several JXL operations starting together each copied and transferred the ~3 MB library. Share the pending hand-off, and drop it if it fails so the next call retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
A failed JXL-enabled init kept the JXL bytes, so every later operation, JPEG and PNG included, retried the broken library. Retired instances were only shut down once no operation at all was running, so a busy batch kept the old WASM heap alive next to the new one. And encodePixelsAsJpeg never held its instance, so it could be shut down under it. Lease an instance per operation instead: each instance counts its own users and shuts down as soon as it is retired and idle. Only JXL work retires a plain instance, and after a failed JXL init other work starts a plain instance while JXL work keeps retrying. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
The test plugin removed JXL at priority 20, after Gutenberg's filter whatever its priority, so it could not catch Gutenberg adding JXL back after check_upload_mimes(). Use the default priority, as core does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
Add a REST test for the sideload endpoint reading a JXL sub-size's dimensions. The empty-container test went through wp_check_filetype_and_ext(), where fileinfo already recognises the JXL ftyp box, so check gutenberg_is_jxl_file() directly instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
The big image threshold check decoded the source with createImageBitmap, which rejects for JXL in browsers without JXL support. The rejection stranded the upload in the queue, so every JXL e2e test timed out. Use the dimensions the server read from the JXL header instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uBiB3PfgBJpX6dbD1mN4B
Summary
Adds JPEG XL (JXL) support to client-side media processing, without making every editor page download a JXL decoder it will probably never use.
A JXL upload is treated like any other image the browser can process: the
.jxlitself becomes the attachment, and vips generates its sub-sizes as JXL in the browser, since the server cannot read JXL without GD/Imagick JXL support. Chrome 155 and Firefox 158 display JXL by default and Safari has partial support since 17, so there is no longer a reason to convert it to JPEG the way HEIC is. JXL also becomes an available output format, so an uploaded JPEG/PNG/WebP can be transcoded to JXL on the way in.The JXL WASM decoder (~850 kB gzipped) lives in its own script module chunk that the browser only fetches the first time a JXL image is actually processed. Editor sessions that never touch JXL transfer no extra bytes.
Fixes #76981.
Core backport: WordPress/wordpress-develop#12006
How it works
The lazy chunk.
packages/vips/src/jxl-wasm.tsis a tiny script module that exports the inlined bytes ofwasm-vips/vips-jxl.wasm. It is listed inwpScriptModuleExportsso the build emits a standalonebuild/modules/vips/jxl-wasm.min.js, and@wordpress/vips/workerdeclares it as a dynamicmodule_dependenciesentry so WordPress's import map can resolve it at runtime.Getting the bytes to the worker.
vipsEnsureJxlSupport()on the main thread dynamic-imports that module - which is what triggers the download - and hands the bytes to the worker through a newsetJxlWasm()RPC. It transfers a bareArrayBufferrather than the typed array: comctx only fast-pathsArrayBufferandArrayand walks anything else key by key, which for a multi-megabyte typed array means millions of main-thread allocations per call. The download is cached, and the RPC repeats only when the worker has been recycled since the last call.Worker-side reinit. The worker stores the bytes and adds
vips-jxl.wasmtodynamicLibrarieson the nextgetVips(). If vips was already running without JXL, that instance is shut down first (it holds a large WASM heap that would otherwise be stranded for the life of the worker) and recreated with JXL support.locateFilewraps the bytes in a Blob URL when vips asks forvips-jxl.wasm. JXL encoding useseffort=3- libvips' default of 7 is too slow to sit through interactively.Upload path.
image/jxljoins the vips-processable MIME types, so a JXL takes the same path as a WebP or AVIF: the original is uploaded withgenerate_sub_sizesoff andgenerateThumbnails()sideloads JXL sub-sizes. A.jxlwhoseFile.typecomes back empty, as it does on a lot of systems, is re-labelledimage/jxlfirst. The upload-media vips wrappers (resize, rotate, convert, compress) callvipsEnsureJxlSupport()whenever they read or write JXL, and hold the worker while they do, so a recycle cannot drop the JXL library in between. That also covers transcoding to JXL, where the server picks the output format after the upload response comes back.Server side.
lib/media/load.phpregistersimage/jxlinupload_mimes; core has no JXL entry, so without it the allowed-types check rejects the file. fileinfo reports JXL asimage/x-jxl, so awp_check_filetype_and_extfilter sniffs the magic bytes - both the naked codestream and the ISOBMFF container - and restores the expected extension and type.getimagesize()cannot read JXL either, sogutenberg_get_jxl_info()decodes the width and height from the codestream SizeHeader (swapping them for transposing orientations). Awp_generate_attachment_metadatafilter uses it to record the upload's dimensions, which core needs to report the missing sub-sizes, and the sideload endpoint falls back to it when checking each JXL sub-size.Bundle size
From the size bot on the latest run:
build/modules/vips/jxl-wasm.min.jsbuild/modules/vips/worker.min.jsbuild/scripts/upload-media/index.min.jsThe always-loaded cost is about 720 bytes. The 850 kB is paid once, on demand, and then cached.
Note the chunk is the
vips-jxl.wasmdynamic library inlined into a JS module, so it transfers somewhat heavier than the raw.wasmwould. See the discussion below on lighter-weight alternatives.Screencast
jxl.to.jpeg.on.upload.mp4
Testing instructions
jxl-wasm.min.jsat all..jxlimage.jxl-wasm.min.jsis fetched once, and the block shows the image (in Chrome 155+, Firefox 158+ or Safari 17+; older browsers cannot display JXL)..jxland then another JPEG in the same session. No second fetch, and both process normally on the JXL-enabled vips instance./wp-json/wp/v2/media/<id>should reportmime_typeofimage/jxl, the realwidthandheight, and every entry inmedia_details.sizesshould be a.jxlwithmime_typeofimage/jxl..jxland its sub-sizes are removed from disk.image_editor_output_format) and upload a JPEG. The resulting attachment should be a.jxl.build/modules/vips/worker.min.asset.phpdeclares a dynamicmodule_dependenciesentry on@wordpress/vips/jxl-wasm.Test files: any
.jxlwill do, or the three fixtures intest/e2e/assets/(200x150_e2e_test_image_decode.jxl,_hdr.jxl,_gainmap.jxl).Tests
client-side-media-processing.spec.js: a JXL upload and its sub-sizes stay JXL with the right dimensions; a 16-bit HDR JXL and a gain-map JXL are stored byte for byte; a transparent JXL keeps its alpha in the sub-sizes; several JXLs uploaded together alongside a JPEG all process. The test browser (Chromium 153) cannot display JXL, so these do not wait for the image to render.phpunit/media/gutenberg-get-jxl-info-test.phpcovers the dimension parser (naked codestream, container, div8 sizes, orientation, non-JXL input) and the attachment metadata filter.packages/vips/src/test/cover the worker reinit (omit until bytes arrive, rebuild with JXL, reuse, recover from a failed init) andvipsEnsureJxlSupport(download once, skip the repeat RPC, re-send after a worker recycle, retry rather than cache a failed download).packages/upload-media/src/store/test/cover the JXL upload path, the emptyFile.typecase and JXL sub-size generation, andpackages/upload-media/src/store/utils/test/jxl-support.tschecks the wrappers load JXL support only when JXL is involved.History
This was the third of three approaches tried for #76981, and the one that stuck. The other two are closed:
vips-jxl.wasmstraight into the worker, which added the whole decoder to every editor load whether or not anyone touched a JXL.wp-vips-jxl), which keeps the bytes out of core but puts an install step in front of anyone who wants JXL, and leaves non-admins without it.Splitting the decoder into its own on-demand chunk gets the "no cost unless you use it" property of the plugin approach without the install step.
AI Use
Code and description both written with 🤖 Claude Code. I will review and test.