I have read through this, looks OK.

Independent from this PR, should we integrate the permutation code (as mentioned official-stockfish/Stockfish#4620 (comment)) in the trainer somehow ?

Independent from this PR, should we integrate the permutation code (as mentioned official-stockfish/Stockfish#4620 (comment)) in the trainer somehow ?

We could maybe apply it during serialization if we have a good enough solution. Right now we should work on a separate tool. I made some initial scaffolding here https://github.com/Sopel97/nnue-pytorch/blob/ftperm/ftperm.py

I think the greedy permutation that is used in the SF PR mentioned is likely a near-optimal solution to do this.

Computing the permutation needs some 50M positions, which in principle we can do by just using that many from a binpack and doing inference. Right now this is some modified version of SF collecting that data, but that's not ideal.

anyway, docs seem complete. Ready for a merge?

yes