gh-161145: Test more bytes simultaneously in find_first_nonascii - #161147

See #122902 which is similar.

Undrafting, #122902 has landed. I reran the perf numbers and this still improves performance of these cases.

tl;dr I tested reading 16 words per branch instead of 8 words per branch: in short, it's not way faster, it is slower overall.

find_max_char() reads 16 words (128 bytes on 64-bit systems) at once and then computes the maximum characters from the or accumulator to build the result.

Your PR reads 8 reads (64 bytes on 64-bit) at once, but it falls back on the loop reading 1 word (8 bytes on 64-bit) at once to detect which byte is non-ASCII.

I was curious to know if it would be more efficient to also process 16 words at once in find_first_nonascii(). I replaced the main loop while (end - p >= 8 * SIZEOF_SIZE_T) { with code adapted from find_max_char():

        // Long ASCII runs: test 16 words (128 bytes on 64-bit) per branch.
        for (; end - p >= 16 * SIZEOF_SIZE_T; p += 16 * SIZEOF_SIZE_T) {
            /* Four independent accumulators so the loop does not serialize on
               one register when the compiler does not vectorize it. */
            size_t v0 = 0, v1 = 0, v2 = 0, v3 = 0;
            for (int i = 0; i < 16 * SIZEOF_SIZE_T; i += 4 * SIZEOF_SIZE_T) {
                const size_t *w = (const size_t *)(p + i);
                v0 |= w[0];
                v1 |= w[1];
                v2 |= w[2];
                v3 |= w[3];
            }
            size_t block = v0 | v1 | v2 | v3;
            if (block & ASCII_CHAR_MASK) {
                break;
            }
        }

In short, (on my laptop), reading 16 words at once is not way faster than reading 8 words: overall, it's 1.01x slower.

Benchmark results:

This table is full of numbers, it's hard to read :-( Let's compare 16words to 8words instead:

Benchmark hidden because not significant (3): (b'a' * 8).decode(), (b'a' * 16 + 'é'.encode() + b'a' * 128).decode(), (b'a' * 500 + 'é'.encode() + b'a' * 128).decode()

Updated benchmark script:

import pyperf

runner = pyperf.Runner()

for n in (8, 100, 200, 4096):
    runner.timeit(f"(b'a' * {n}).decode()", "s.decode()",
                  globals={"s": b'a' * n})
runner.timeit(f"(b'a' * 1 MiB).decode()", "s.decode()",
              globals={"s": b'a' * (1024 * 1024)})
runner.timeit(f"(b'a' * 16 MiB).decode()", "s.decode()",
              globals={"s": b'a' * (16 * 1024 * 1024)})

for tail in (10, 128):
    for n in (16, 100, 136, 200, 500):
        s = b'a' * n + 'é'.encode() + b'a' * tail
        runner.timeit(f"(b'a' * {n} + 'é'.encode() + b'a' * {tail}).decode()",
                      "s.decode()", globals={"s": s})
    s = b'a' * (1024 * 1024) + 'é'.encode() + b'a' * tail
    runner.timeit(f"(b'a' * 1 MiB + 'é'.encode() + b'a' * {tail}).decode()",
                  "s.decode()", globals={"s": s})
    s = b'a' * (16 * 1024 * 1024) + 'é'.encode() + b'a' * tail
    runner.timeit(f"(b'a' * 16 MiB + 'é'.encode() + b'a' * {tail}).decode()",
                  "s.decode()", globals={"s": s})

I checked the x86-64 assembly code of find_first_nonascii().

On the main branch, it uses one 64-bit and instruction which reads 64-bit from memory (and r8,QWORD PTR [rax]) per branch:

   0x0000000000788120 <+80>:	add    rax,0x8
   0x0000000000788124 <+84>:	cmp    r9,rax
   0x0000000000788127 <+87>:	jb     0x788220 <find_first_nonascii+336>
   0x000000000078812d <+93>:	mov    r8,rdx
   0x0000000000788130 <+96>:	and    r8,QWORD PTR [rax]
   0x0000000000788133 <+99>:	je     0x788120 <find_first_nonascii+80>

With this change, it uses 4 x 128-bit reads (ex: movdqu xmm0,XMMWORD PTR [rax+0x10]) + 4 x 128-bit por:

   0x00000000007880e0 <+16>:	movabs r8,0x8080808080808080
   ...

   0x0000000000788110 <+64>:	add    rax,0x40
   0x0000000000788114 <+68>:	mov    rdx,rsi
   0x0000000000788117 <+71>:	sub    rdx,rax
   0x000000000078811a <+74>:	cmp    rdx,0x3f
   0x000000000078811e <+78>:	jle    0x788156 <find_first_nonascii+134>

   0x0000000000788120 <+80>:	movdqu xmm0,XMMWORD PTR [rax+0x10]
   0x0000000000788125 <+85>:	movdqu xmm2,XMMWORD PTR [rax+0x20]
   0x000000000078812a <+90>:	movdqu xmm1,XMMWORD PTR [rax+0x30]
   0x000000000078812f <+95>:	movdqu xmm3,XMMWORD PTR [rax]
   0x0000000000788133 <+99>:	por    xmm0,xmm2
   0x0000000000788137 <+103>:	por    xmm1,xmm3
   0x000000000078813b <+107>:	por    xmm0,xmm1
   0x000000000078813f <+111>:	movdqa xmm1,xmm0
   0x0000000000788143 <+115>:	psrldq xmm1,0x8
   0x0000000000788148 <+120>:	por    xmm0,xmm1
   0x000000000078814c <+124>:	movq   rdx,xmm0
   0x0000000000788151 <+129>:	test   rdx,r8
   0x0000000000788154 <+132>:	je     0x788110 <find_first_nonascii+64>

Updated to the different loop form. Agreed the slower is less common. For #161146 in particular it goes from "always exactly one byte" to multiple bytes at once most the time, even the slowdown cases will be faster for the tokenizer.

I did look at more generalized SIMD libraries a bit, there are cool tricks to use to get even faster but need to go to platform specific intrinsics which don't seem worth it; this is fast enough and stays fairly legible. Non-obvious why 8 is the right width, but legible what the code is doing.

I did look at more generalized SIMD libraries a bit, there are cool tricks to use to get even faster but need to go to platform specific intrinsics which don't seem worth it

I agree. I prefer "regular C code" designed in a way so that C compilers can vectorize it, what you did, rather than specialized SIMD code which be less portable and more annoying to maintain.

In CPython, we try to keep the code easy to maintain, and so avoid too crazy optimizations hacks 😁