@tstarling: This task has been assigned to you a while ago. Is this still a valid (open) task which requires more work, or do you assume this got fixed by T389734?

All Excimer segfault bugs are now suspected duplicates of T389734, i.e. glibc timer aliasing, glibc#32833.

Found the issue and resolved it. I ran composer run --timeout=0 phpunit:entrypoint -- --group Database --exclude-group Broken,ParserFuzz,Stub,Standalone --debug (with --debug) to see which test causes this which narrowed things down.

Test 'ObjectCacheTest::testNewAnythingAccel' started
Segmentation fault
Script phpunit handling the phpunit event returned with error code 139
Script @phpunit was called via phpunit:entrypoint
``

From the logs, your code triggers a segfault when running PHP, which is probably an upstream bug in PHP rather than a bug in your code per se.

You would the Linux Kernel to generate a core dump when php crashes. Running the jobs with ulimit -c unlimited should enable that and once a core file has been acquired, remove the setting to avoid filing disk.

jforrester closed https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/29

docker-registry.discovery.wmnet/dev/buster-swift53 failed cause of an outdated gpg key.

Most likely opcache corruption. Errors started only 1ms after the request timeout at 12:36:03.013, and then there was a flood of segfaults over the next few hundred milliseconds, finishing with "[WARNING] failed processes threshold (40 in 60 sec) is reached, initiating reload" at 12:36:03.800.

This is fixed and released in LuaSandbox 4.1.1.

I have published the result of this debugging as Blog Post: Investigate a PHP segmentation fault

And eventually it is fixed:

16:30:26 <wmf-insecte> Project beta-update-databases-eqiad build #69029: FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69029/

Mentioned in SAL (#wikimedia-releng) [2023-07-26T15:29:52Z] <jnuche> reenabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/ Should be fixed now! https://phabricator.wikimedia.org/T342769

Possibly caused by errant user input that wasn't stopped.

Mentioned in SAL (#wikimedia-releng) [2023-07-26T15:04:22Z] <jnuche> disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/ after debugging is done. See https://phabricator.wikimedia.org/T342769

After I have remembered PHP provides some gdb helpers ( .gdbinit in php-7.4.30 branch), I have added it to my home dir and loaded it with gdb> source /home/hashar/gdbinit.

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-schemata/-/merge_requests/29

So Instead of messing up with systemd / coredump / ulimit I went to invoke the php command under GDB based on a previous issue T296539#7531235

Mentioned in SAL (#wikimedia-releng) [2023-07-26T14:02:48Z] <hashar> deployment-prep: on deployment-deploy03, installed php / libpcre debugging symbols for T342769: sudo apt-get install php7.4-common-dbgsym php7.4-cli-dbgsym libpcre2-dbg

I have changed /etc/systemd/coredump.conf:

- KeepFree=20%
+ KeepFree=4G

But that comes from Puppet. I have thus disabled the Puppet agent for now and have set the value to 4G. The core file should then be stored under /var/lib/systemd/coredump.

Thanks, this is great!

I filed https://github.com/php/php-src/issues/11734

It would seem that there's a critical section between zend_object_std_init() and the assignment of handlers, during which time it's unsafe to call emalloc().

Change 938925 merged by jenkins-bot:

[mediawiki/php/luasandbox@master] Fix segmentation fault when memory limit is exceeded in LuaSandbox init

https://gerrit.wikimedia.org/r/938925

Change 938925 had a related patch set uploaded (by TK-999; author: TK-999):

[mediawiki/php/luasandbox@master] Fix segmentation fault when memory limit is exceeded in LuaSandbox init

https://gerrit.wikimedia.org/r/938925

I think this is not a PHP 8.0 specific issue, it'd likely happen on earlier versions as well. We just happened to notice it because we were on the lookout for core dumps while investigating unrelated JIT issues.

Thanks a lot, Tim :)- using your advice, I inspected a more recent dump and found that it was a request where PHP was exceeding its configured memory limit while it was parsing a rather complex page. Going by that, I managed to narrow it down to a consistent reproducer:

--TEST--
Memory limit exceeded during sandbox init
--INI--
memory_limit=2M
--FILE--
<?php
$buf = str_repeat('a', 1000000);
$sandboxes = [];
for ($i = 0; $i < 100; $i++) {
    $sandboxes[] = new LuaSandbox();
}
?>
--EXPECTF--
Fatal error: Allowed memory size of 2097152 bytes exhausted%s(tried to allocate %d bytes) in %s on line %d

Given the previous comment, is there any reason to keep this task protected at this point? I'm not seeing one.

Change 936327 merged by jenkins-bot:

[mediawiki/core@master] docker: update to latest published images

https://gerrit.wikimedia.org/r/936327