ext/json: replace packed charmap bitmap with a byte-indexed lookup table by adapik · Pull Request #23682 · php/php-src

adapik · 2026-09-13T17:45:34Z

Summary

php_json_escape_string()'s scalar loop tests every byte of every string with ZEND_BIT_TEST(charmap, byte) — a packed 256-bit bitmap decoded as charmap[byte/32] >> (byte%32) & 1. That's a real dependency chain: compute a shift+mask to get the word index, then load, then shift the loaded word to extract the bit. This runs on every character passed to json_encode(), regardless of content.

This PR replaces it with a direct 256-byte lookup table indexed by the byte value itself — table[byte]. No address computation needed before the load; the byte value is the index.

How this was found

While investigating a SIMD fast-path #23675 for the same function, I found this classification test wasn't part of the SIMD work at all — it's the pre-existing scalar loop every build already runs, untouched since long before any vectorization. That made it worth optimizing on its own, independent of any SIMD story, since it benefits every build on every architecture.

My first version generated the 256-entry table at compile time via a couple of helper macros expanding a readable predicate into a static const initializer. It worked and benchmarked identically, but I found the macro too ugly and verbose for what it bought. In this PR we fill the table with a single, ordinary for loop, called once from PHP_MINIT_FUNCTION(json), it's much easy to understand, test and process. It runs once per process, before any request — the overhead is one 256-iteration loop at startup, which is negligible against the millions of json_encode() calls it then speeds up over the process's lifetime.

Benchmark

Docker, AMD Ryzen 5 7500F, gcc 13.3, default release flags), 7 trials/corpus, master vs. this change:

Corpus	small	medium	large
clean_ascii	+8%	+10%	+6%
html_heavy	+3%	+4%	+5%
mixed	+3%	+18%	+18%
non_ascii	+5%	+6%	+4%
url_heavy	+1%	+1%	+6%

No regression on any corpus beyond two 4–5 byte boundary cases at −5%. Every substantive corpus improves, including non-ASCII and URL-heavy content, which are typically the hardest to move without a trade-off elsewhere. Biggest win: +18% on mixed content (medium/large).

I used the same test suite as in #23675.

Reproducing these numbers

curl -fsSL https://raw.githubusercontent.com/adapik/php-src/perf/json-sse2-escape-scripts/scripts/json/cross_cpu_bench.sh -o cross_cpu_bench.sh
chmod +x cross_cpu_bench.sh
FORK_REF=perf/json-escape-dirty-table ./cross_cpu_bench.sh

Testing

30/30 corpora pass byte-for-byte correctness against unpatched output, across both default and full-flags (UNESCAPED_SLASHES|UNESCAPED_UNICODE|HEX_*) configurations.
Table contents independently verified bit-exact against the original charmap bitmap for all 256 byte values.
New test (json_encode_byte_classification.phpt) sweeps all 256 byte values against an independently-written reference implementation of the escaping rules, across four option combinations, plus verifies every lone high-bit byte correctly reports JSON_ERROR_UTF8.
No behavior change — output is identical for every input; this is purely an internal implementation change to an existing function.

Girgias · 2026-09-14T15:46:34Z


 static const char digits[] = "0123456789abcdef";

+static uint8_t php_json_escape_dirty_table[256];


This could be a const static table of bool, we have other such static tables in ext/mbstring.

Might make sense to have a small PHP file generate a C header file that contains just the table. :)

adapik · 2026-09-14T17:23:52Z

I actually started with a hand-written static map, but found it hard to read and not self-explanatory. Now I followed the well-established ext/mbstring pattern instead - script ext/json/gen_json_escape_table.php generates php_json_escape_table.h (a static const bool[256] table).

@Girgias thanks for the tip-off

Benchmarks did'nt move (obviously).

TimWolla · 2026-09-14T17:40:00Z

I actually started with a hand-written static map, but found it hard to read and not self-explanatory.

I believe that by using designated initializers and comments we can get the best of both worlds: Readability and no moving parts (the generated file should be set up with https://github.com/php/php-src/blob/master/.github/actions/verify-generated-files/action.yml and the Makefile).

Inverting the meaning of the entries and then doing the following:

static const bool php_json_escape_clean_table[256] = {
[' '] = 1,
['!'] = 1,
['#'] = 1,
['$'] = 1,
['%'] = 1,
['('] = 1,
[')'] = 1,
['*'] = 1,
['+'] = 1,
[','] = 1,
['-'] = 1,
['.'] = 1,
['0'] = 1,
['1'] = 1,
['2'] = 1,
['3'] = 1,
['4'] = 1,
['5'] = 1,
['6'] = 1,
['7'] = 1,
['8'] = 1,
['9'] = 1,
[':'] = 1,
[';'] = 1,
['='] = 1,
['?'] = 1,
['@'] = 1,
['A'] = 1,
['B'] = 1,
['C'] = 1,
['D'] = 1,
['E'] = 1,
['F'] = 1,
['G'] = 1,
['H'] = 1,
['I'] = 1,
['J'] = 1,
['K'] = 1,
['L'] = 1,
['M'] = 1,
['N'] = 1,
['O'] = 1,
['P'] = 1,
['Q'] = 1,
['R'] = 1,
['S'] = 1,
['T'] = 1,
['U'] = 1,
['V'] = 1,
['W'] = 1,
['X'] = 1,
['Y'] = 1,
['Z'] = 1,
['['] = 1,
[']'] = 1,
['^'] = 1,
['_'] = 1,
['`'] = 1,
['a'] = 1,
['b'] = 1,
['c'] = 1,
['d'] = 1,
['e'] = 1,
['f'] = 1,
['g'] = 1,
['h'] = 1,
['i'] = 1,
['j'] = 1,
['k'] = 1,
['l'] = 1,
['m'] = 1,
['n'] = 1,
['o'] = 1,
['p'] = 1,
['q'] = 1,
['r'] = 1,
['s'] = 1,
['t'] = 1,
['u'] = 1,
['v'] = 1,
['w'] = 1,
['x'] = 1,
['y'] = 1,
['z'] = 1,
['{'] = 1,
['|'] = 1,
['}'] = 1,
['~'] = 1,
[127] = 1,
};

will safely initialize the remaining entries to 0.

Side note: Should 0x7f really not be escaped? That looks like a pre-existing bug.

adapik · 2026-09-14T18:35:09Z

@TimWolla Good point, but I don't think this needs the CI/Makefile editing. The table content is fixed (it encodes JSON's escaping rules, which aren't expected to change), so gen_json_escape_table.php isn't a build-time dependency — it's there as a documentation/maintainability aid. In practice no one is going to read the 256-entry table itself; if someone does need to read or change it, they'll follow the header comment back to the generator script, which is short, concise, and self-explanatory.

For what it's worth, the mbstring generators this PR follows (ucgendat.php → eaw_table.h, gen_rare_cp_bitvec.php → rare_cp_bitvec.h) aren't wired into verify-generated-files or the Makefile either, so this keeps parity with that existing pattern.

DEL (0x7F) — checked against RFC 8259. Everything from 0x5D up — DEL included — is legal unescaped. So not escaping it is spec-compliant, not a bug. This PR preserves that existing behavior.

TimWolla · 2026-09-14T18:39:39Z

aren't wired into verify-generated-files

That would be a bug then. The CI job exists to verify generated files actually are in sync (to prevent accidental or malicious changes from slipping into generated files).

DEL (0x7F) — checked against RFC 8259. Everything from 0x5D up — DEL included — is legal unescaped. So not escaping it is spec-compliant, not a bug. This PR preserves that existing behavior.

Surprising (given that it's practically non-printable), but it seems that node.js agrees, so no change needed.

adapik · 2026-09-14T18:58:47Z

@TimWolla No worries, not against it in general - just figured it wasn't needed since the table's static. But yeah, fair point about what the CI job's actually for, so I went ahead and added it.

I wouldn't mind going through the rest of the scripts and opening a follow-up PR for those later on.

I wouldn't trust nodejs :) just saying, only stick to RFC

json: replace packed charmap bitmap with a byte-indexed lookup table

18fe1fe

adapik requested a review from bukka as a code owner September 13, 2026 17:45

github-actions Bot added the Extension: json label Sep 13, 2026

adapik marked this pull request as draft September 13, 2026 18:43

adapik marked this pull request as ready for review September 14, 2026 05:39

adapik changed the title ~~json: replace packed charmap bitmap with a byte-indexed lookup table~~ ext/json: replace packed charmap bitmap with a byte-indexed lookup table Sep 14, 2026

Girgias reviewed Sep 14, 2026

View reviewed changes

static table + php script for generation

71446a2

adapik requested a review from Girgias September 14, 2026 17:24

adapik requested a review from TimWolla as a code owner September 14, 2026 18:53

github-actions Bot added the Category: CI label Sep 14, 2026

script generation check - CI and makefile added

5114f86

adapik force-pushed the perf/json-escape-dirty-table branch from 53fd066 to 5114f86 Compare September 14, 2026 20:15

Navigation Menu

Uh oh!

ext/json: replace packed charmap bitmap with a byte-indexed lookup table - #23682

ext/json: replace packed charmap bitmap with a byte-indexed lookup table#23682
adapik wants to merge 3 commits into
php:masterfrom
adapik:perf/json-escape-dirty-table

adapik commented Sep 13, 2026 •

edited

Loading

Uh oh!

Girgias Sep 14, 2026

Uh oh!

adapik commented Sep 14, 2026

Uh oh!

TimWolla commented Sep 14, 2026

Uh oh!

adapik commented Sep 14, 2026

Uh oh!

TimWolla commented Sep 14, 2026

Uh oh!

adapik commented Sep 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		static const char digits[] = "0123456789abcdef";

		static uint8_t php_json_escape_dirty_table[256];

Uh oh!

Conversation

adapik commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How this was found

Benchmark

Testing

Uh oh!

Girgias Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

adapik commented Sep 14, 2026

Uh oh!

TimWolla commented Sep 14, 2026

Uh oh!

adapik commented Sep 14, 2026

Uh oh!

TimWolla commented Sep 14, 2026

Uh oh!

adapik commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adapik commented Sep 13, 2026 •

edited

Loading

adapik commented Sep 14, 2026 •

edited

Loading