Thank you for the pull request. I guess that one usage of this format is to distribute training data between net developers. In that case, we could need to support shuffle at some timing.

Could you please take a look at CI? https://travis-ci.org/github/nodchip/Stockfish/builds/725750282

Yes, I'm slowly trying to get CI working. May have to come back to it tomorrow as it's already late for me.

I was talking with noob on discord and he says that shuffling over large distances is not needed to ensure enough randomness (because two games differ on average the same regardless of how far they are). Only shuffling within a batch that's loaded in memory is enough. Persisting the shuffled output is also not needed. Even if some people would still want to shuffle I think .bin for that is the best we can do.

If my memory is correct, someone in Discrod said that we can gain more slos with learn shuffle. Are there experiment results for it?

It was only a test between completely no shuffle and full shuffle. I don't think anyone did tests on partial shuffles over small blocks.

The following compiles on linux, but I didn't check functionality:

diff --git a/src/extra/nnue_data_binpack_format.h b/src/extra/nnue_data_binpack_format.h
index c86a55c2a..e6cd7ad20 100644
--- a/src/extra/nnue_data_binpack_format.h
+++ b/src/extra/nnue_data_binpack_format.h
@@ -41,8 +41,9 @@ THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 #include <array>
 #include <immintrin.h>
 
-#ifdef linux
+#ifdef __linux__
 #include <x86intrin.h>
+#include <climits>
 #else
 #include <intrin.h>
 #endif
@@ -2700,7 +2701,7 @@ namespace chess
         return Bitboard::square(sq0) | sq1;
     }
 
-    [[nodiscard]] constexpr Bitboard operator""_bb(std::uint64_t bits)
+    [[nodiscard]] constexpr Bitboard operator""_bb(unsigned long long bits)
     {
         return Bitboard::fromBits(bits);
     }
@@ -7295,4 +7296,4 @@ namespace binpack
             std::cout << "Processed " << (cur - base) << " bytes and " << numProcessedPositions << " positions.\n";
         }
     }
-}
\ No newline at end of file
+}

some comments:

• I think __linux__ might be the wrong define for this (this might be specific to gcc). Can one, instead of #include <intrin.h> be using the same approach as nnue_common.h.
• unsigned long long bits there is only a small list of parameters that are allowed in this case, std::uint64_tis not part of this. https://en.cppreference.com/w/cpp/language/user_literal

Okay, checks passed now

nice... btw, I see you make this optional. If it works well, I would consider making this the default & only option. While there is already some data generated, there will be even more generated in the future, and the size of these files is an obstacle to sharing them and having fewer options might be a path to making progress quicker.

Thank you for sending and fixing the pull request. I merged it.