Test failed on Tests_Image_Editor_Imagick::test_resizes_are_small_for_16bit_images #237

Activity

1. RamyAllam

   added

   bug

   on Jan 9, 2025
2. kittenkamala

   added

   ContributorDay

   on Aug 26, 2025
3. kittenkamala

   added

   WCUS26

   on Jun 3, 2026
4. kittenkamala

   moved this to In progress in Hosting Team WCUS 2026on Jul 30, 2026
5. kittenkamala

   added this to Hosting Team WCUS 2026on Jul 30, 2026
6. kittenkamala

   moved this from In progress to Ready in Hosting Team WCUS 2026on Aug 12, 2026
7. kittenkamala commented on Sep 16, 2026

   

   kittenkamala

   on Sep 16, 2026

   Collaborator

   More actions

   Hmm. The runner should not exit in the case of a failure like this but rather should report it.
8. softglazee commented on Oct 24, 2026

   

   softglazee

   on Oct 24, 2026

   More actions

   Reproduction Report

   Environment

   • WordPress: trunk, r64693
   • PHP: 8.4.27
   • Server: Docker image built from this repository's docker/Dockerfile with PHP_VERSION=8.4, tests run through prepare.php, test.php, report.php and cleanup.php
   • Database: MariaDB 11.8.9
   • Browser: n/a (command line test)
   • OS: Linux (Docker Desktop, WSL2 kernel 6.18.33)
   • Theme: the test suite's default theme
   • MU Plugins: None
   • Plugins: None
   • ImageMagick 7.1.1-43 Q16, imagick 3.8.1, PHPUnit 9.6.38

   Steps taken

   1. Ran the full suite with php test.php.
   2. Ran only this test through test.php, with WPT_PHPUNIT_CMD set to the default command plus --filter test_resizes_are_small_for_16bit_images.
   3. Replaced src/wp-includes/class-wp-image-editor-imagick.php with the version from r60245, the last revision before r60246 changed the indexed PNG handling. From r59589 (the commit named in the report) to r60245 that file only differs in whitespace. Ran step 2 again.
   4. ❌ Bug is not occurring: the cloudflare-status data set passes in all three runs.

   ....S                                                               5 / 5 (100%)
   
   There was 1 skipped test:
   
   1) Tests_Image_Editor_Imagick::test_resizes_are_small_for_16bit_images with data set "test8" ('/tmp/wp-test-runner/tests/php...t8.png')
   ImageMagick 7 is unable to optimize grayscale images with 1-bit transparency.
   
   OK, but incomplete, skipped, or risky tests!
   Tests: 5, Assertions: 4, Skipped: 1.
   

   Expected behavior

   • The resized PNG is smaller than the original, and the test passes.

   Additional Notes

   • File sizes in bytes, with the same steps as the test (load, resize to 50 percent, save), on this ImageMagick build:

   Data set	Original	Editor at r60245	Editor at r60246	Trunk r64693
   cloudflare-status	24940	23500	12223	23500
   deskcat8	105705	90507	42138	90507
   17-c3-duplicate-entries	24841	16879	12812	16879
   rabbit-time-paletted	200581	82915	52242	82915
   test8	45231	11026	13148	99853

   • The report shows 26899 for cloudflare-status, larger than the original. With the same editor code (r60245) I get 23500 here. So the output size depends on the ImageMagick build, not only on the code. On trunk the margin here is 1440 bytes, about 6 percent.
   • The same kind of difference is already handled in core for test8: it is not smaller on ImageMagick 7 (99853 above), and core skips it there since r60736 (core ticket #63932).
   • Host results for r64692 show no failure in this test from Hostinger (ImageMagick 7.1.1-47, PHP 8.1) or Digitoimisto Dude (ImageMagick 6.9.12-98, PHP 8.3). Their failures are in other tests.
   • About the runner exiting on a failure: test.php exits with 1 when PHPUnit reports a failure, and prints Error: Failed to perform operation: .... The result still gets reported, because report.php is a separate step that reads junit.xml. I checked this with a temporary test that fails (the test8 check without the ImageMagick 7 skip):

   Failed asserting that 99853 is less than 45231.
   ...
   FAILURES!
   Tests: 1, Assertions: 1, Failures: 1.
   Error: Failed to perform operation: cd /tmp/wp-test-runner && php ./vendor/phpunit/phpunit/phpunit --dont-report-useless-tests --verbose --filter Tests_Zz_Size_Probe.
   test.php exit=1
   ...
   [+] TEST RESULTS
   
   {"tests":"1","failures":"1","errors":"0","time":"0.237035","testsuites":{"Tests_Zz_Size_Probe":{"name":"Tests_Zz_Size_Probe","tests":"1","failures":"1","errors":"0","testcases":{"test_test8_resize_is_smaller_without_the_imagemagick_7_skip":{"name":"test_test8_resize_is_smaller_without_the_imagemagick_7_skip","failure":"Tests_Zz_Size_Probe::test_test8_resize_is_smaller_without_the_imagemagick_7_skip\nThe resized image file size is not smaller than the original file size.\nFailed asserting that 99853 is less than 45231.\n\n\/tmp\/wp-test-runner\/tests\/phpunit\/tests\/image\/zz-size-probe.php:19"}}}}}
   

   With WPT_REPORT_API_KEY set, that is the result report.php uploads. The README script, docker/entrypoint.sh and the GitHub workflow all run report.php after a failed test step.

   Suggestion

   I do not think this needs a change in the runner or its documentation. The failure came from the file size check in core's test, which depends on the ImageMagick build, and it does not occur on trunk here or on the two hosts above. I think this can be closed. If it comes back on a host, it belongs in core Trac (Media component), with the ImageMagick version from env.json.

   AI Assisted: I used an AI assistant to help investigate and draft this; I ran and checked every test and result myself.