On avx2, I measure about 1% speedup.

$ ~/chess/pyshbench/pyshbench.py ./stockfish.master ./stockfish.patch 10
run       base       test     diff
  1    2105092    2126002   +20910
  2    2110419    2144096   +33677
  3    2118367    2114663    -3704
  4    2092706    2118924   +26218
  5    2121526    2142196   +20670
  6    2093430    2134822   +41392
  7    2108027    2116699    +8672
  8    2209317    2228042   +18725
  9    2120596    2147904   +27308
 10    2197075    2223127   +26052

Result of  10 runs
==================
base (...kfish.master) =    2127656  +/- 25473
test (...ckfish.patch) =    2149648  +/- 25822
diff                   =     +21992  +/- 7793

speedup        = +0.0103
P(speedup > 0) =  1.0000

as a minor nit, on avx2 some functions are unused:

nnue/../nnue/architectures/../layers/affine_transform.h:110:12: warning: variable ‘m256_hadd’ set but not used [-Wunused-but-set-variable]
  110 |       auto m256_hadd = [](__m256i sum, int bias) -> int {
      |            ^~~~~~~~~
nnue/../nnue/architectures/../layers/affine_transform.h:130:12: warning: variable ‘m128_hadd’ set but not used [-Wunused-but-set-variable]
  130 |       auto m128_hadd = [](__m128i sum, int bias) -> int {
      |            ^~~~~~~~~
nnue/../nnue/architectures/../layers/affine_transform.h:136:12: warning: variable ‘m128_haddx4’ set but not used [-Wunused-but-set-variable]
  136 |       auto m128_haddx4 = [](__m128i sum0, __m128i sum1, __m128i sum2, __m128i sum3, __m128i bias) -> __m128i {

I profiled the vnni512 version with the Intel Software Developement Emulator. If the results are to be believed they show that the most costly affine transform is almost twice as fast (or at least instruction-wise) in the PR version.
Master: https://pastebin.com/UKLWjVMG in particular https://pastebin.com/UdyXvcaJ
PR: https://pastebin.com/adCYuC4X in particular https://pastebin.com/RgvsvtGV

test here https://tests.stockfishchess.org/tests/view/5f9d516f6a2c112b60691da3

but considering that majority of the workers are AVX2 I would still appreciate testing on vnni256, avx512, vnni512

The test passed. Is LTC required for speedup patches? If not I'll just fix the unused variables warnings and prepare a PR.

As far as I understand, LTC is not required for pure speed ups.

I think computing 8 inner-products in parallel should be possible on AVX2 without register spilling, but I haven't tried it yet.

How to do the horizontal adding most efficiently is an interesting question, especially on AVX512.
One possible approach is to first fold each 512-bit vector of partial sums into a 256-bit vector of partial sums by adding the upper part to the lower part. I don't know if that is faster than @Sopel97 's implementation.

It may now also make sense to clip the 32-bit sums immediately to 8-bit ints. This may be difficult to do in SF, though.

LTC is not required for a speedup patch.

This is a pretty big patch, mostly because it covers several architectures and cases. I would appreciate if @Sopel97 spents some time/effort to make sure it is written optimally from a maintenance/generality point of view. My current thinking is that for the NNUE core machinery it makes sense to optimize, in a machine specific way if needed, the code. Yet, we should keep in mind that the code should remain modifiable as new network architectures or hardware architectures appear.

I would appreciate if somebody with intrinsics programming experience can review the patch (@syzygy1 or @mstembera ?)

Sure, I'll document it and cover the cases that are currently exit(1)

I tried 8x unroll at godbolt. With clang it looks very clean, with gcc not so much though and spills a few registers.
https://godbolt.org/z/hfdT9G
Might be worth for AVX512 if we find a nice way to do 8x horizontal additions.

Just throwing this here for the future. I won't be changing this now.

I've made some changes to make the code cleaner, esured to cover all possible cases, and addressed the stuff syzygy pointed out. When it passes CI and gets @vondele 's approval I will put the final version on fishtest and then make a new PR.

I gave avx512 compiles (g++ 10.2, without PGO) for old and new versions to ipmanchess and he kindly ran both.
https://cdn.discordapp.com/attachments/771338720028393472/772192457290285086/Stockfish_avx512_no-pgo_old-new_benches.PNG

With Cfish, I found that the compiled code looks better if you replace the row0/1/2/3 pointers with a single pointer (looks like the difference between your clang and gcc output).

(I guess it's not so easy to look at the SF assembly code because SF relies heavily on LTO.)

Nice AVX512 speedup :)

I'm tinkering on godbolt. GCC seems to have #pragma GCC unroll 4 (#pragma unroll 4 for clang) which helps https://godbolt.org/z/41PsGa. Maybe such enforced unrolling could be forced further (you can even force it to unroll the whole thing, though that would likely trash I$), though this is for another PR.

At least in Cfish the unrolling happens automatically after profiling.

It would be nice if one could make gcc produce assembly output even with LTO. I don't think it is possible, but I could easily be wrong. But you can still execute the g++ evaluate_nnue.cpp -S (with all the right flags) yourself and leave out -flto. You may have to prevent the Makefile from removing the .gcda files.

I think I'll look into using XED next time for such stuff (it comes with SDE). It seems to be east to find the assembly listings for the affine transforms. In the real stockfish compilation it's unrolled better than at godbolt, 2x and uses rax for indexing with constant offsets.

on linux you could do CXXFLAGS="-g" make -j ARCH=x86-64-avx2 profile-build && objdump -dlS ./stockfish to get the assembly, with source and line numbers intermixed.

I've done a few tests, seemingly a little faster now on avx2 as well. Compiled a couple of crosses etc. So looks good for a final test if you feel it is final.

You can keep this PR, just force push a version with a full commit message including the testing results.

BTW, to test vnni etc, I have used cloud resources in the past, that's quite doable, but obviously no requirement. It is nice you've been able to check correctness on the emulator (please check again for the final version).

So a new net was merged and appveyor failed because it compares the bench to master. I've rebased and changed the bench in the commit message to match new master.