Thanks. Would it be feasible to regenerate the current best datasets with the score alteration way of removing the positions? If not then I could try doing it by comparing the filtered and unfiltered datasets, if available.

Just a note, personally I'd like to name the filtering function less generically, for example by appending the hash of the first master network trained with the filtered data, this way we would have a connection established and it would be harder to make a mistake in the future. Not sure what others think.

Would it be feasible to regenerate the current best datasets with the score alteration way of removing the positions?

It would be difficult to exactly regenerate the best datasets with score alteration instead of position removal, since the current best datasets were generated imprecisely - it's a rough mix of filtered data pieces mixed together.

It'd be easier to generate a different version of the best datasets with score alteration instead of removal. However, I'm guessing this would be slightly weaker than the best datasets, all else being equal, since maximizing binpack compression has a tradeoff - binpack fragment granularity would be at the per-training-game level vs. many fragments per training game with position removal. Score alteration improves compression at the cost of less randomness after shuffling. Needs some testing. It may also take a few weeks to regenerate.

Just a note, personally I'd like to name the filtering function less generically, for example by appending the hash of the first master network trained with the filtered data, this way we would have a connection established and it would be harder to make a mistake in the future. Not sure what others think.

Is the intent to document an exact way to filter the data for the first recent master network? If so, the difference between the exact filtering method used and this PR is:

• startpos was removed from all filtered games
• position removal instead of score alteration. trades off binpack compression for more randomness
• hardcoded to depth6 multipv2

Is the intent

The intent is to have versioning for filtering functions in a way that prevents misuse.

The PR is still marked as draft. Are there plans to further develop it?

The PR is still marked as draft. Are there plans to further develop it?

Yea, we're close to finalizing how this should work. I'll get the PR into a reviewable state after some clarifications.

The intent is to have versioning for filtering functions in a way that prevents misuse.

Makes sense, how about something like this?

transform filter-335a9b2d8a80 input_file in.binpack output_file out.binpack

• uses score alteration by default to maximize binpack compression
• filters out all startpos and positions where either move is a capture at a hardcoded depth6 mulitipv2 search

transform filter-335a9b2d8a80 alter_score 0 input_file in.binpack output_file out.binpack

• enables removing positions entirely, to replicate the exact dataset preparation method used for training nn-335a9b2d8a80.nnue. with the default random-fen-skipping 3 value, this may have slightly higher elo due to trading some compression for more shuffling randomness (unconfirmed).

Sounds good.

Ready for review. I realized if more binpack shuffling randomness is desired, it's easy to remove positions with score VALUE_NONE after filtering. The computationally expensive part of filtering is during search. As such, I don't think there's a strong need for an alter_score 0 option to remove positions during filtering. This way binpacks will stay compressed by default.

I'll plan to re-filter the training data later to maintain binpack compression but it may take a while to complete.

Looks good to me.