[ Intl ] Fix Intl feature inside Blueprints in Playground - #3073

I found the root cause. The above blueprint will run successfully if I comment the proxyFileSystem(...) function in playground-worker-endpoint.ts :

if (!isPrimary) {
	const pathsToShareBetweenPhpInstances = [
		'/tmp',
		requestHandler.documentRoot,
		'/internal/shared',
		'/internal/symlinks',
	];
	const pathsToProxy = pathsToShareBetweenPhpInstances.filter(
		(path) => !isPathToSharedFS(php, path)
	);

	// TODO: Document that this shift is a breaking change.
	// Proxy the filesystem for all secondary PHP instances to
	// the primary one.
	proxyFileSystem(
		await requestHandler.getPrimaryPhp(),
		php,
		pathsToProxy
	);
}

This means something is happening to the /internal/shared/icudt74l.dat file during the proxy.

If I add these lines in the proxyFileSystem(...) function :

+ if( path !== '/internal/shared' ){
	// @ts-ignore
	replica[__private__symbol].FS.mount(
		// @ts-ignore
		replica[__private__symbol].PROXYFS,
		{
			root: path,
			// @ts-ignore
			fs: sourceOfTruth[__private__symbol].FS,
		},
		path
	);
+ }

It runs correctly. I wonder if mounting a file used by PHP internals like the intl extension will corrupt it.

@adamziel I created a new directory named /internal/private which will be used to store everything related to dynamic extensions. This directory shouldn't be proxied between PHP instances since there seems to be file corruption with the icu.dat file when it is done in /internal/shared.

I tagged you because I wanted to know if /internal/private is a meaningful path to you.

Maybe /internal/libs ?

Hm, not mounting that file worries me. First, what corrupts that file? Could it affect other files and corrupt other data? Second, the system is wired for sharing files between php instances and I worry about the consequences of not sharing them.

The reason we have these mounts is to share a set of files between all the php instances - they're created once in the primary env and then every other php instance sources them from that primary php's filesystem. If we stop mounting, what would happen to icu.dat in non-primary php instances or after the php env is rotated?

@adamziel Here’s my current understanding of why we were still hitting the Intl issue and how it relates to filesystem sharing.

Each PHP instance is initialized via loadWebRuntime. During that phase, the instance loads and links against its intl.so extension and the corresponding icu.dat file [ which is fetched and memoized at runtime ]. Those links are established during initialization, before any filesystem proxying happens.

Later on, when proxyFileSystem is called, we mount /internal/shared from the primary PHP instance onto the secondary ones using FS.mount. Although the icu.dat file is still visible after the mount, this operation effectively replaces the underlying filesystem for that path. As a result, the ICU runtime inside the PHP instance can no longer use the icu.dat file it was originally linked to during loadWebRuntime, and Intl starts failing [ e.g. Collator constructor errors ].

Even though I can’t say exactly what happens internally inside FS.mount, the observed behavior is consistent : after proxying /internal/shared, PHP’s intl extension can no longer access ICU data reliably. This strongly suggests that remounting that directory breaks the early filesystem bindings created during PHP initialization.

To fix this, my proposal is to isolate native extensions .so and their required runtime assets [ like icu.dat ] into a dedicated, instance-local directory, such as : /internal/private, /internal/libraries or /internal/libs

Each PHP instance would load its extensions from this directory during startup, and that directory would never be proxied or remounted. This keeps ICU and other native extensions stable and instance-safe.

In contrast, /internal/shared should only be used for files that are not tightly coupled to a specific PHP instance and are safe to share across instances.

What do you think ?

@mho22 Were you able to verify your theory by doing a checksum of the file with and without the mounting?

@akirk The checksum is always the same. I was wrong about data corruption. But I think I found out what was the issue :

This is the proxy-file-system.ts proxyFileSystem modified function :

export async function proxyFileSystem(
	sourceOfTruth: PHP,
	replica: PHP,
	paths: string[]
) {
	const __private__symbol = Object.getOwnPropertySymbols(sourceOfTruth)[0];
	for (const path of paths) {
		if (!replica.fileExists(path)) {
			replica.mkdir(path);
		}
		if (!sourceOfTruth.fileExists(path)) {
			sourceOfTruth.mkdir(path);
		}


+		async function checksumSHA256(data: Uint8Array): Promise<string>
+		{
+			const hashBuffer = await crypto.subtle.digest( 'SHA-256', data as BufferSource );
+
+			const hashArray = Array.from(new Uint8Array(hashBuffer));
+
+			return hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
+		}


+		if( replica.fileExists( '/internal/shared' ) ) console.log( 'BEFORE', await checksumSHA256( replica.readFileAsBuffer( '/internal/shared/icudt74l.dat' ) ) );

		// @ts-ignore
		replica[__private__symbol].FS.mount(
			// @ts-ignore
			replica[__private__symbol].PROXYFS,
			{
				root: path,
				// @ts-ignore
				fs: sourceOfTruth[__private__symbol].FS,
			},
			path
		);

+		if( replica.fileExists( '/internal/shared' ) ) console.log( 'AFTER ', await checksumSHA256( replica.readFileAsBuffer( '/internal/shared/icudt74l.dat' ) ) );
	}
}

These are the logs when the icudt74l.dat file is in the /internal/shared/ directory.

BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad

PHP Fatal error:  Uncaught IntlException: Constructor failed in /internal/eval.php:5
Stack trace:
#0 /internal/eval.php(5): Collator->__construct('en_US')
#1 {main}
  thrown in /internal/eval.php on line 5

And these are the logs when the icudt74l.dat file is in the /internal/private/ directory. [ Not proxied ].

BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
BEFORE dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad
AFTER  dc778b9ffe18ed319ad3fb70754f80e51cf7b6dbfff38fc0c0a5f27bb5463dad

There is no Uncaught IntlException anymore. This is certainly not enough to verify my theory. So this time I focused on where and how that icu file was reached and it looks like, when I replace the logs with this in the proxy-file-system.ts file :

// @ts-ignore
const FS1 = replica[__private__symbol].FS;
const node1 = FS1.lookupPath('/internal/shared/icudt74l.dat').node;

console.log('is MEMFS?', node1.mount.type === FS1.filesystems.MEMFS);
console.log('is PROXYFS?', node1.mount.type === FS1.filesystems.PROXYFS);

I'll get these logs :

is MEMFS? true
is PROXYFS? false
is MEMFS? true
is PROXYFS? false
is MEMFS? false
is PROXYFS? true
is MEMFS? false
is PROXYFS? true

So at one point in time, we reach data from PROXYFS instead of MEMFS and, based on what GPT says, this could be the reason why Intl crashes.

MEMFS owns the bytes; PROXYFS forwards the bytes.
Intl (ICU) needs ownership, not forwarding.

This Blueprint:

<?php 

require_once '/wordpress/wp-load.php';

try {
  $collator = new Collator( get_locale() );
} catch(Exception $e) {
  var_dump([
     'msg' => intl_get_error_message(),
     'code' => intl_get_error_code(),
  ]);
  throw $e;
}

Gave me some more info:

array(2) {
["msg"]=> string(65) "collator_create: unable to open ICU collator: U_FILE_ACCESS_ERROR"
["code"]=> int(4)
}

It seems like something fundamental with file access is broken in PROXYFS. I'd rather not work around that but pinpoint the exact problem. I've built a bunch of tests to see how PROXYFS deals with different FS operations and found that this test fails with errno 8:

beforeEach(async () => {
	phpA = new PHP(await loadNodeRuntime(phpVersion));
	phpB = new PHP(await loadNodeRuntime(phpVersion));

	// Mount /internal/shared from phpA to phpB using PROXYFS
	proxyFileSystem(phpA, phpB, [sharedPath]);
});

it('should open a file in read-write mode (O_RDWR)', () => {
	const testFile = `${sharedPath}/rdwr-test.txt`;
	phpA.writeFile(testFile, 'initial content');

	const fsB = getFS(phpB);
	const stream = fsB.open(testFile, O_RDWR);

	try {
		// Read from the file
		const readBuffer = new Uint8Array(8);
		fsB.read(stream, readBuffer, 0, 8, 0);
		expect(new TextDecoder().decode(readBuffer)).toBe('initial ');

		// Write to the file
		const writeContent = 'new';
		const writeBuffer = new TextEncoder().encode(writeContent);
		fsB.write(stream, writeBuffer, 0, writeBuffer.length, 8); // Write after "initial "

		fsB.close(stream);

		expect(phpA.readFileAsText(testFile)).toBe('initial newnt'); // 'content' becomes 'newnt'
	} finally {
		fsB.close(stream);
	}
});

There could be more problems in there.

This one also seems to be failing:

it('should change file access and modification times through PROXYFS', () => {
	const testFile = `${sharedPath}/utime-test.txt`;
	phpA.writeFile(testFile, 'utime test');

	const fsB = getFS(phpB);
	const oldStat = fsB.stat(testFile);

	// Set new access and modification times (e.g., 1 day later)
	const newAtime = (oldStat.atime.getTime() / 1000) + 86400;
	const newMtime = (oldStat.mtime.getTime() / 1000) + 86400;

	fsB.utime(testFile, newAtime, newMtime);

	const newStat = fsB.stat(testFile);
	// Emscripten's FS.utime expects seconds, but stat returns milliseconds.
	// Compare with a small tolerance due to potential floating point inaccuracies.
	expect(newStat.atime.getTime() / 1000).toBeCloseTo(newAtime);
	expect(newStat.mtime.getTime() / 1000).toBeCloseTo(newMtime);
});

Back to square one :(

• The read-write mode test failure was a red herring, it was just calling close() twice.
• The utime test failuer was a red herring. It works if we avoid / 1000.
• The chown test failure is legitimate, but MEMFS also doesn't support changing owner and group.

Some additional info:

I've instrumented all of the stream ops functions in PROXYFS as follows:

		stream_ops: {
			open(stream) {
				if (stream.path.endsWith('.dat')) {
					console.log('stream open', stream);
				}
 

This gave me a stream open log line right before the error. I've set a breakpoint on it and followed the execution step by step. Here's the trace in case we ever need it again:

Some interesting source links:

• ucol_open
• udata_openChoice
• uprv_mapFile

A possible source of the error:

    /* data not found */
    if(U_SUCCESS(*pErrorCode)) {
        if(U_SUCCESS(subErrorCode)) {
            /* file not found */
            *pErrorCode=U_FILE_ACCESS_ERROR;
        } else {
            /* entry point not found or rejected */
            *pErrorCode=subErrorCode;
        }
    }

@mho22 is that helpful and puts you in a good place to continue the search?

@adamziel Yes, thank you! I am on it. Let's find out why it returns U_FILE_ACCESS_ERROR.

@adamziel @akirk I think I fixed the issue. Everything was related to an unexistent mmap function in PROXYFS. I will explain how I debugged that here. This can help for our future selves.

Thanks to Adam's first check, I was able to find my way to the different files starting with Collator's __construct to the PROXYFS node_stream's open function. I documented that here :

• PROXYFS.open
• FS.open
• __syscall_openat
• $open -> uprv_mapfile
  $uprv_mapFile_74 -> openCommonData
• $func14532 -> openCommonData
  -> extendICUData
  -> doLoadFromCommonData
• $func14524 -> doOpenChoice
• $func14522 -> udata_openChoice
• $udata_openChoice_74 -> load
• $_ZN6icu_7413CollationRoot4loadEPKcR10UErrorCode -> getRootCacheEntry
• $ZN6icu_7413umtx_initOnceIPKcEEvRNS_9UInitOnceEPFvT_R10UErrorCodeES5_S7 -> getRootCacheEntry
• wrapper
• stubs
• $_ZN6icu_7413CollationRoot17getRootCacheEntryER10UErrorCode -> loadTailoring
• $_ZN6icu_7415CollationLoader13loadTailoringERKNS_6LocaleER10UErrorCode -> makeInstance
• $_ZN6icu_748Collator12makeInstanceERKNS_6LocaleER10UErrorCode -> createInstance
• $_ZN6icu_748Collator14createInstanceERKNS_6LocaleER10UErrorCode -> Collator::createInstance
• $ucol_open_74 -> ucol_open
• $func16048 -> collator_ctor
• $zim_Collator___construct -> Collator::__construct

This gave me the basis to debug the way back after PROXYFS has opened the file. And first stop : open() in uprv_mapfile [ And fortunately it was the only stop I had to do ].

I found out the following part of the file was called [ since there are 4 uprv_mapFile declarations in the umapFile.cpp file ] on line 207 :

#elif MAP_IMPLEMENTATION==MAP_POSIX
    U_CFUNC UBool
    uprv_mapFile(UDataMemory *pData, const char *path, UErrorCode *status) {
        int fd;
        int length;
        struct stat mystat;
        void *data;

        if (U_FAILURE(*status)) {
            return false;
        }

        UDataMemory_init(pData); /* Clear the output struct.        */

        /* determine the length of the file */
        if(stat(path, &mystat)!=0 || mystat.st_size<=0) {
            return false;
        }
        length=mystat.st_size;

        /* open the file */
        fd=open(path, O_RDONLY);
        if(fd==-1) {
            return false;
        }

        /* get a view of the mapping */
#if U_PLATFORM != U_PF_HPUX
        data=mmap(nullptr, length, PROT_READ, MAP_SHARED, fd, 0);
#else
        data=mmap(nullptr, length, PROT_READ, MAP_PRIVATE, fd, 0);
#endif
        close(fd); /* no longer needed */
        if(data==MAP_FAILED) {
            // Possibly check the errno value for ENOMEM, and report U_MEMORY_ALLOCATION_ERROR?
            return false;
        }

        pData->map = (char *)data + length;
        pData->pHeader=(const DataHeader *)data;
        pData->mapAddr = data;
        pData->length = length;
#if U_PLATFORM == U_PF_IPHONE || U_PLATFORM == U_PF_ANDROID
    // Apparently supported from Android 23 and higher:
    //   https://github.com/ggml-org/llama.cpp/pull/3631
    // Checking for the flag itself is safer than checking for __ANDROID_API__.
#   ifdef POSIX_MADV_RANDOM
        posix_madvise(data, length, POSIX_MADV_RANDOM);
#   endif
#endif
        return true;
    }

open was called successfully, and close too, so I first focused on code between these two function calls :

        /* get a view of the mapping */
#if U_PLATFORM != U_PF_HPUX
        data=mmap(nullptr, length, PROT_READ, MAP_SHARED, fd, 0);
#else
        data=mmap(nullptr, length, PROT_READ, MAP_PRIVATE, fd, 0);
#endif

And yup! PROXYFS does not have a mmap function declaration in php_8_5.js. First clue. I decided to copy paste a valid mmap function declaration inside PROXYFS, and after adding the same console.log as Adam in his previous comment I found out it called the new PROXYFS mmap function. But : it still returned the U_FILE_ACCESS_ERROR exception. Obsviously. But that was a confirmation.

I had to find a way to shims that mmap call into something possible for PROXYFS. So : next step, logging in ICU.

First of all, how to modify that file. Well, by copying my own file into the intl Dockerfile :

packages/php-wasm/compile/shared/intl on line 115 :

# Build ICU libraries
RUN set -euxo pipefail; \
	wget https://github.com/unicode-org/icu/releases/download/release-74-2/icu4c-74_2-src.tgz; \
	tar -xvf icu4c-74_2-src.tgz; \
	rm -rf /root/icu/source/data/; \
	rm icu4c-74_2-src.tgz; \

	wget https://github.com/unicode-org/icu/releases/download/release-74-2/icu4c-74_2-data.zip; \
	unzip icu4c-74_2-data.zip -d /root/icu/source; \
	rm icu4c-74_2-data.zip; \

	mkdir -p /root/native; \
	cd /root/native; \
	/root/icu/source/runConfigureICU Linux; \
	make -j1; \
-       make install; \
+	make install;
+
+ COPY ./shared/intl/umapfile.cpp /root/icu/source/common/umapfile.cpp
+
+ RUN	cd /root/icu/source; \
-       cd /root/icu/source; \
	mkdir -p /root; \
	source /root/emsdk/emsdk_env.sh; \
	export CFLAGS="-fPIC -D__x86_64__"; \
	export CXXFLAGS="-fPIC -D__x86_64__"; \

	emconfigure ./configure \
		--host=i386-unknown-freebsd \
		--prefix=/root \
		--with-cross-build=/root/native \
		--with-data-packaging=archive \
		--disable-extras \
		--enable-static \
		--disable-shared; \
	EMCC_FLAGS="-D__x86_64__" emmake make -j1; \
	EMCC_FLAGS="-D__x86_64__" emmake make install;

Then, run emscripten enabled logs. Easy :

#include <emscripten/emscripten.h>

emscripten_log(EM_LOG_CONSOLE, "ICU uprv_mapFile path=%s size=%d", path, length);

These logs helped me understand what was missing. I finally replaced the mmap path with another code :

-         /* get a view of the mapping */
- #if U_PLATFORM != U_PF_HPUX
-        data=mmap(nullptr, length, PROT_READ, MAP_SHARED, fd, 0);
- #else
-         data=mmap(nullptr, length, PROT_READ, MAP_PRIVATE, fd, 0);
- #endif
-         close(fd); /* no longer needed */
-         if(data==MAP_FAILED) {
-             // Possibly check the errno value for ENOMEM, and report U_MEMORY_ALLOCATION_ERROR?
-             return false;
-        }
+        // --- allocate ---
+        data = malloc(length);
+        if (!data) {
+            close(fd);
+            *status = U_MEMORY_ALLOCATION_ERROR;
+            return false;
+        }
+
+        emscripten_log(EM_LOG_CONSOLE, "allocate done");
+
+        // --- read ---
+        r = pread(fd, data, length, 0);
+        close(fd);
+
+        if (r != length) {
+            free(data);
+            return false;
+        }
+
+        emscripten_log(EM_LOG_CONSOLE, "ICU map OK: %s (%d bytes)", path, length);

And it worked! No more U_FILE_ACCESS_ERROR Exception! I looked into malloc and found out that a uprv_malloc call was done in another declaration of uprv_mapFile but in the MAP_IMPLEMENTATION==MAP_STDIO option.

So I decided to use a clean umapfile.cpp file and add these 4 lines instead of replacing mmap :

#include "umapfile.h"

+ #if defined(__EMSCRIPTEN__)
+ #  undef MAP_IMPLEMENTATION
+ #  define MAP_IMPLEMENTATION MAP_STDIO
+ #endif

/* memory-mapping base definitions ------------------------------------------ */

It still worked. Nice. Next : /root/replace.sh that thing :

packages/php-wasm/compile/shared/intl on line 133 :

- COPY ./shared/intl/umapfile.cpp /root/icu/source/common/umapfile.cpp
+ RUN /root/replace.sh 's/#include \"umapfile.h\"/#include "umapfile.h"\n\n#if defined(__EMSCRIPTEN__)\n#  undef MAP_IMPLEMENTATION\n#  define MAP_IMPLEMENTATION MAP_STDIO\n#endif\n\n/' /root/icu/source/common/umapfile.cpp;

And that's it. 🎉

I will now update the pull request with it. I should also find a way to reproduce the issue in a test in php-dynamic-loading.spec.ts and recompile the Intl extensions in Web and Node.

@mho22 Great debugging! Can we leave the CPP code unchanged and just implement mmap in PROXYFS? Otherwise things will break the next time another C/CPP library calls mmap on one of those files.

Here's some prior art on overriding Emscripten core FS methods:

https://github.com/articles-adamziel-com/wordpress-playground/blob/33f450256b8536d374eade7b7031e58fccbc10fc/packages/php-wasm/node/src/lib/load-runtime.ts#L199

https://github.com/articles-adamziel-com/wordpress-playground/blob/33f450256b8536d374eade7b7031e58fccbc10fc/packages/php-wasm/node/src/lib/load-runtime.ts#L120

The last commit implements a version of mmap for PROXYFS. The new test I implemented to test if my mmap function fixes this issue passed locally for node. We'll see if this doesn't crash other tests in CI.

However, I don't like the following part of the function but it solves the 7.3 and 7.2 crash :

if (phpRuntime.phpVersion.major === 7 && phpRuntime.phpVersion.minor <= 3) {
	const path = phpRuntime.FS.getPath(stream.node);

	if (!path.endsWith(".dat")) {
		const fs = stream.node.mount.opts.fs;
		const stat = fs.fstat(stream.nfd);
		length = stat.size >>> 0;
	}
}

This has to be more generic. ini files and php files can't be part of that if statement.

Edit: it still doesn't work for Web JSPI...

merged via #3126