Skip to content

Commit e8d64af

Browse files
Sopel97vondele
Sopel97
authored and
vondele
committed
New NNUE architecture and net
Introduces a new NNUE network architecture and associated network parameters, as obtained by a new pytorch trainer. The network is already very strong at short TC, without regression at longer TC, and has potential for further improvements. https://tests.stockfishchess.org/tests/view/60a159c65085663412d0921d TC: 10s+0.1s, 1 thread ELO: 21.74 +-3.4 (95%) LOS: 100.0% Total: 10000 W: 1559 L: 934 D: 7507 Ptnml(0-2): 38, 701, 2972, 1176, 113 https://tests.stockfishchess.org/tests/view/60a187005085663412d0925b TC: 60s+0.6s, 1 thread ELO: 5.85 +-1.7 (95%) LOS: 100.0% Total: 20000 W: 1381 L: 1044 D: 17575 Ptnml(0-2): 27, 885, 7864, 1172, 52 https://tests.stockfishchess.org/tests/view/60a2beede229097940a03806 TC: 20s+0.2s, 8 threads LLR: 2.93 (-2.94,2.94) <0.50,3.50> Total: 34272 W: 1610 L: 1452 D: 31210 Ptnml(0-2): 30, 1285, 14350, 1439, 32 https://tests.stockfishchess.org/tests/view/60a2d687e229097940a03c72 TC: 60s+0.6s, 8 threads LLR: 2.94 (-2.94,2.94) <-2.50,0.50> Total: 45544 W: 1262 L: 1214 D: 43068 Ptnml(0-2): 12, 1129, 20442, 1177, 12 The network has been trained (by vondele) using the https://github.com/glinscott/nnue-pytorch/ trainer (started by glinscott), specifically the branch https://github.com/Sopel97/nnue-pytorch/tree/experiment_56. The data used are in 64 billion positions (193GB total) generated and scored with the current master net d8: https://drive.google.com/file/d/1hOOYSDKgOOp38ZmD0N4DV82TOLHzjUiF/view?usp=sharing d9: https://drive.google.com/file/d/1VlhnHL8f-20AXhGkILujnNXHwy9T-MQw/view?usp=sharing d10: https://drive.google.com/file/d/1ZC5upzBYMmMj1gMYCkt6rCxQG0GnO3Kk/view?usp=sharing fishtest_d9: https://drive.google.com/file/d/1GQHt0oNgKaHazwJFTRbXhlCN3FbUedFq/view?usp=sharing This network also contains a few architectural changes with respect to the current master: Size changed from 256x2-32-32-1 to 512x2-16-32-1 ~15-20% slower ~2x larger adds a special path for 16 valued ClippedReLU fixes affine transform code for 16 inputs/outputs, buy using InputDimensions instead of PaddedInputDimensions this is safe now because the inputs are processed in groups of 4 in the current affine transform code The feature set changed from HalfKP to HalfKAv2 Includes information about the kings like HalfKA Packs king features better, resulting in 8% size reduction compared to HalfKA The board is flipped for the black's perspective, instead of rotated like in the current master PSQT values for each feature the feature transformer now outputs a part that is fowarded directly to the output and allows learning piece values more directly than the previous network architecture. The effect is visible for high imbalance positions, where the current master network outputs evaluations skewed towards zero. 8 PSQT values per feature, chosen based on (popcount(pos.pieces()) - 1) / 4 initialized to classical material values on the start of the training 8 subnetworks (512x2->16->32->1), chosen based on (popcount(pos.pieces()) - 1) / 4 only one subnetwork is evaluated for any position, no or marginal speed loss A diagram of the network is available: https://user-images.githubusercontent.com/8037982/118656988-553a1700-b7eb-11eb-82ef-56a11cbebbf2.png A more complete description: https://github.com/glinscott/nnue-pytorch/blob/master/docs/nnue.md closes #3474 Bench: 3806488
1 parent f90274d commit e8d64af

File tree

13 files changed

+265
-173
lines changed

13 files changed

+265
-173
lines changed

‎src/Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ endif
4141
SRCS = benchmark.cpp bitbase.cpp bitboard.cpp endgame.cpp evaluate.cpp main.cpp \
4242
material.cpp misc.cpp movegen.cpp movepick.cpp pawns.cpp position.cpp psqt.cpp \
4343
search.cpp thread.cpp timeman.cpp tt.cpp uci.cpp ucioption.cpp tune.cpp syzygy/tbprobe.cpp \
44-
nnue/evaluate_nnue.cpp nnue/features/half_kp.cpp
44+
nnue/evaluate_nnue.cpp nnue/features/half_ka_v2.cpp
4545

4646
OBJS = $(notdir $(SRCS:.cpp=.o))
4747

‎src/evaluate.cpp‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ namespace Eval {
120120

121121
if (filename.has_value())
122122
actualFilename = filename.value();
123-
else
123+
else
124124
{
125125
if (eval_file_loaded != EvalFileDefaultName)
126126
{
@@ -1116,10 +1116,8 @@ Value Eval::evaluate(const Position& pos) {
11161116
// Scale and shift NNUE for compatibility with search and classical evaluation
11171117
auto adjusted_NNUE = [&]()
11181118
{
1119-
int material = pos.non_pawn_material() + 4 * PawnValueMg * pos.count<PAWN>();
1120-
int scale = 580
1121-
+ material / 32
1122-
- 4 * pos.rule50_count();
1119+
1120+
int scale = 903 + 28 * pos.count<PAWN>() + 28 * pos.non_pawn_material() / 1024;
11231121

11241122
Value nnue = NNUE::evaluate(pos) * scale / 1024 + Time.tempoNNUE;
11251123

@@ -1134,7 +1132,7 @@ Value Eval::evaluate(const Position& pos) {
11341132
Value psq = Value(abs(eg_value(pos.psq_score())));
11351133
int r50 = 16 + pos.rule50_count();
11361134
bool largePsq = psq * 16 > (NNUEThreshold1 + pos.non_pawn_material() / 64) * r50;
1137-
bool classical = largePsq || (psq > PawnValueMg / 4 && !(pos.this_thread()->nodes & 0xB));
1135+
bool classical = largePsq;
11381136

11391137
// Use classical evaluation for really low piece endgames.
11401138
// One critical case is the draw for bishop + A/H file pawn vs naked king.
@@ -1151,8 +1149,7 @@ Value Eval::evaluate(const Position& pos) {
11511149
&& !lowPieceEndgame
11521150
&& ( abs(v) * 16 < NNUEThreshold2 * r50
11531151
|| ( pos.opposite_bishops()
1154-
&& abs(v) * 16 < (NNUEThreshold1 + pos.non_pawn_material() / 64) * r50
1155-
&& !(pos.this_thread()->nodes & 0xB))))
1152+
&& abs(v) * 16 < (NNUEThreshold1 + pos.non_pawn_material() / 64) * r50)))
11561153
v = adjusted_NNUE();
11571154
}
11581155

‎src/evaluate.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ namespace Eval {
3939
// The default net name MUST follow the format nn-[SHA256 first 12 digits].nnue
4040
// for the build process (profile-build and fishtest) to work. Do not change the
4141
// name of the macro, as it is used in the Makefile.
42-
#define EvalFileDefaultName "nn-62ef826d1a6d.nnue"
42+
#define EvalFileDefaultName "nn-8a08400ed089.nnue"
4343

4444
namespace NNUE {
4545

‎src/nnue/evaluate_nnue.cpp‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ namespace Stockfish::Eval::NNUE {
3535
LargePagePtr<FeatureTransformer> featureTransformer;
3636

3737
// Evaluation function
38-
AlignedPtr<Network> network;
38+
AlignedPtr<Network> network[LayerStacks];
3939

4040
// Evaluation function file name
4141
std::string fileName;
@@ -83,7 +83,8 @@ namespace Stockfish::Eval::NNUE {
8383
void initialize() {
8484

8585
Detail::initialize(featureTransformer);
86-
Detail::initialize(network);
86+
for (std::size_t i = 0; i < LayerStacks; ++i)
87+
Detail::initialize(network[i]);
8788
}
8889

8990
// Read network header
@@ -92,7 +93,7 @@ namespace Stockfish::Eval::NNUE {
9293
std::uint32_t version, size;
9394

9495
version = read_little_endian<std::uint32_t>(stream);
95-
*hashValue = read_little_endian<std::uint32_t>(stream);
96+
*hashValue = read_little_endian<std::uint32_t>(stream);
9697
size = read_little_endian<std::uint32_t>(stream);
9798
if (!stream || version != Version) return false;
9899
desc->resize(size);
@@ -117,7 +118,8 @@ namespace Stockfish::Eval::NNUE {
117118
if (!read_header(stream, &hashValue, &netDescription)) return false;
118119
if (hashValue != HashValue) return false;
119120
if (!Detail::read_parameters(stream, *featureTransformer)) return false;
120-
if (!Detail::read_parameters(stream, *network)) return false;
121+
for (std::size_t i = 0; i < LayerStacks; ++i)
122+
if (!Detail::read_parameters(stream, *(network[i]))) return false;
121123
return stream && stream.peek() == std::ios::traits_type::eof();
122124
}
123125

@@ -126,7 +128,8 @@ namespace Stockfish::Eval::NNUE {
126128

127129
if (!write_header(stream, HashValue, netDescription)) return false;
128130
if (!Detail::write_parameters(stream, *featureTransformer)) return false;
129-
if (!Detail::write_parameters(stream, *network)) return false;
131+
for (std::size_t i = 0; i < LayerStacks; ++i)
132+
if (!Detail::write_parameters(stream, *(network[i]))) return false;
130133
return (bool)stream;
131134
}
132135

@@ -154,10 +157,15 @@ namespace Stockfish::Eval::NNUE {
154157
ASSERT_ALIGNED(transformedFeatures, alignment);
155158
ASSERT_ALIGNED(buffer, alignment);
156159

157-
featureTransformer->transform(pos, transformedFeatures);
158-
const auto output = network->propagate(transformedFeatures, buffer);
160+
const std::size_t bucket = (pos.count<ALL_PIECES>() - 1) / 4;
159161

160-
return static_cast<Value>(output[0] / OutputScale);
162+
const auto [psqt, lazy] = featureTransformer->transform(pos, transformedFeatures, bucket);
163+
if (lazy) {
164+
return static_cast<Value>(psqt / OutputScale);
165+
} else {
166+
const auto output = network[bucket]->propagate(transformedFeatures, buffer);
167+
return static_cast<Value>((output[0] + psqt) / OutputScale);
168+
}
161169
}
162170

163171
// Load eval, from a file stream or a memory stream

src/nnue/features/half_kp.cpp renamed to src/nnue/features/half_ka_v2.cpp

Lines changed: 12 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -16,32 +16,32 @@
1616
along with this program. If not, see <http://www.gnu.org/licenses/>.
1717
*/
1818

19-
//Definition of input features HalfKP of NNUE evaluation function
19+
//Definition of input features HalfKAv2 of NNUE evaluation function
2020

21-
#include "half_kp.h"
21+
#include "half_ka_v2.h"
2222

2323
#include "../../position.h"
2424

2525
namespace Stockfish::Eval::NNUE::Features {
2626

2727
// Orient a square according to perspective (rotates by 180 for black)
28-
inline Square HalfKP::orient(Color perspective, Square s) {
29-
return Square(int(s) ^ (bool(perspective) * 63));
28+
inline Square HalfKAv2::orient(Color perspective, Square s) {
29+
return Square(int(s) ^ (bool(perspective) * 56));
3030
}
3131

3232
// Index of a feature for a given king position and another piece on some square
33-
inline IndexType HalfKP::make_index(Color perspective, Square s, Piece pc, Square ksq) {
33+
inline IndexType HalfKAv2::make_index(Color perspective, Square s, Piece pc, Square ksq) {
3434
return IndexType(orient(perspective, s) + PieceSquareIndex[perspective][pc] + PS_NB * ksq);
3535
}
3636

3737
// Get a list of indices for active features
38-
void HalfKP::append_active_indices(
38+
void HalfKAv2::append_active_indices(
3939
const Position& pos,
4040
Color perspective,
4141
ValueListInserter<IndexType> active
4242
) {
4343
Square ksq = orient(perspective, pos.square<KING>(perspective));
44-
Bitboard bb = pos.pieces() & ~pos.pieces(KING);
44+
Bitboard bb = pos.pieces();
4545
while (bb)
4646
{
4747
Square s = pop_lsb(bb);
@@ -52,7 +52,7 @@ namespace Stockfish::Eval::NNUE::Features {
5252

5353
// append_changed_indices() : get a list of indices for recently changed features
5454

55-
void HalfKP::append_changed_indices(
55+
void HalfKAv2::append_changed_indices(
5656
Square ksq,
5757
StateInfo* st,
5858
Color perspective,
@@ -63,23 +63,22 @@ namespace Stockfish::Eval::NNUE::Features {
6363
Square oriented_ksq = orient(perspective, ksq);
6464
for (int i = 0; i < dp.dirty_num; ++i) {
6565
Piece pc = dp.piece[i];
66-
if (type_of(pc) == KING) continue;
6766
if (dp.from[i] != SQ_NONE)
6867
removed.push_back(make_index(perspective, dp.from[i], pc, oriented_ksq));
6968
if (dp.to[i] != SQ_NONE)
7069
added.push_back(make_index(perspective, dp.to[i], pc, oriented_ksq));
7170
}
7271
}
7372

74-
int HalfKP::update_cost(StateInfo* st) {
73+
int HalfKAv2::update_cost(StateInfo* st) {
7574
return st->dirtyPiece.dirty_num;
7675
}
7776

78-
int HalfKP::refresh_cost(const Position& pos) {
79-
return pos.count<ALL_PIECES>() - 2;
77+
int HalfKAv2::refresh_cost(const Position& pos) {
78+
return pos.count<ALL_PIECES>();
8079
}
8180

82-
bool HalfKP::requires_refresh(StateInfo* st, Color perspective) {
81+
bool HalfKAv2::requires_refresh(StateInfo* st, Color perspective) {
8382
return st->dirtyPiece.piece[0] == make_piece(perspective, KING);
8483
}
8584

src/nnue/features/half_kp.h renamed to src/nnue/features/half_ka_v2.h

Lines changed: 26 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,8 @@
1818

1919
//Definition of input features HalfKP of NNUE evaluation function
2020

21-
#ifndef NNUE_FEATURES_HALF_KP_H_INCLUDED
22-
#define NNUE_FEATURES_HALF_KP_H_INCLUDED
21+
#ifndef NNUE_FEATURES_HALF_KA_V2_H_INCLUDED
22+
#define NNUE_FEATURES_HALF_KA_V2_H_INCLUDED
2323

2424
#include "../nnue_common.h"
2525

@@ -32,33 +32,34 @@ namespace Stockfish {
3232

3333
namespace Stockfish::Eval::NNUE::Features {
3434

35-
// Feature HalfKP: Combination of the position of own king
36-
// and the position of pieces other than kings
37-
class HalfKP {
35+
// Feature HalfKAv2: Combination of the position of own king
36+
// and the position of pieces
37+
class HalfKAv2 {
3838

3939
// unique number for each piece type on each square
4040
enum {
4141
PS_NONE = 0,
42-
PS_W_PAWN = 1,
43-
PS_B_PAWN = 1 * SQUARE_NB + 1,
44-
PS_W_KNIGHT = 2 * SQUARE_NB + 1,
45-
PS_B_KNIGHT = 3 * SQUARE_NB + 1,
46-
PS_W_BISHOP = 4 * SQUARE_NB + 1,
47-
PS_B_BISHOP = 5 * SQUARE_NB + 1,
48-
PS_W_ROOK = 6 * SQUARE_NB + 1,
49-
PS_B_ROOK = 7 * SQUARE_NB + 1,
50-
PS_W_QUEEN = 8 * SQUARE_NB + 1,
51-
PS_B_QUEEN = 9 * SQUARE_NB + 1,
52-
PS_NB = 10 * SQUARE_NB + 1
42+
PS_W_PAWN = 0,
43+
PS_B_PAWN = 1 * SQUARE_NB,
44+
PS_W_KNIGHT = 2 * SQUARE_NB,
45+
PS_B_KNIGHT = 3 * SQUARE_NB,
46+
PS_W_BISHOP = 4 * SQUARE_NB,
47+
PS_B_BISHOP = 5 * SQUARE_NB,
48+
PS_W_ROOK = 6 * SQUARE_NB,
49+
PS_B_ROOK = 7 * SQUARE_NB,
50+
PS_W_QUEEN = 8 * SQUARE_NB,
51+
PS_B_QUEEN = 9 * SQUARE_NB,
52+
PS_KING = 10 * SQUARE_NB,
53+
PS_NB = 11 * SQUARE_NB
5354
};
5455

5556
static constexpr IndexType PieceSquareIndex[COLOR_NB][PIECE_NB] = {
5657
// convention: W - us, B - them
5758
// viewed from other side, W and B are reversed
58-
{ PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_NONE, PS_NONE,
59-
PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_NONE, PS_NONE },
60-
{ PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_NONE, PS_NONE,
61-
PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_NONE, PS_NONE }
59+
{ PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_KING, PS_NONE,
60+
PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_KING, PS_NONE },
61+
{ PS_NONE, PS_B_PAWN, PS_B_KNIGHT, PS_B_BISHOP, PS_B_ROOK, PS_B_QUEEN, PS_KING, PS_NONE,
62+
PS_NONE, PS_W_PAWN, PS_W_KNIGHT, PS_W_BISHOP, PS_W_ROOK, PS_W_QUEEN, PS_KING, PS_NONE }
6263
};
6364

6465
// Orient a square according to perspective (rotates by 180 for black)
@@ -69,17 +70,17 @@ namespace Stockfish::Eval::NNUE::Features {
6970

7071
public:
7172
// Feature name
72-
static constexpr const char* Name = "HalfKP(Friend)";
73+
static constexpr const char* Name = "HalfKAv2(Friend)";
7374

7475
// Hash value embedded in the evaluation file
75-
static constexpr std::uint32_t HashValue = 0x5D69D5B8u;
76+
static constexpr std::uint32_t HashValue = 0x5f234cb8u;
7677

7778
// Number of feature dimensions
7879
static constexpr IndexType Dimensions =
7980
static_cast<IndexType>(SQUARE_NB) * static_cast<IndexType>(PS_NB);
8081

81-
// Maximum number of simultaneously active features. 30 because kins are not included.
82-
static constexpr IndexType MaxActiveDimensions = 30;
82+
// Maximum number of simultaneously active features.
83+
static constexpr IndexType MaxActiveDimensions = 32;
8384

8485
// Get a list of indices for active features
8586
static void append_active_indices(
@@ -107,4 +108,4 @@ namespace Stockfish::Eval::NNUE::Features {
107108

108109
} // namespace Stockfish::Eval::NNUE::Features
109110

110-
#endif // #ifndef NNUE_FEATURES_HALF_KP_H_INCLUDED
111+
#endif // #ifndef NNUE_FEATURES_HALF_KA_V2_H_INCLUDED

0 commit comments

Comments
 (0)