Skip to content

bench covering both NNUE and classical, PGO, CI #2931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vondele wants to merge 3 commits into from
Closed

bench covering both NNUE and classical, PGO, CI #2931

vondele wants to merge 3 commits into from

Conversation

@vondele
Copy link
Member

@vondele vondele commented Aug 7, 2020 

bench covering both NNUE and classical, PGO, CI

Add additional options to the bench command

classical, NNUE, mixed.

Default mixed, examples:

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

Requires the default net to be present,
download automatically if needed for profile-build.

requires fishtest work, and presumably travis work, before commit.

Bench: 3607702

@gvreuls gvreuls mentioned this pull request Aug 7, 2020
Closed
@vondele vondele mentioned this pull request Aug 7, 2020
Closed
@mstembera mstembera mentioned this pull request Aug 7, 2020
Closed
This was referenced Aug 8, 2020
Closed
Closed
Closed
Closed
@vondele vondele force-pushed the hybridBenchCI branch 4 times, most recently from 40c42b7 to b61fd4f Compare August 10, 2020 11:25
@gvreuls @vondele
Download the default NNUE net in AppVeyor
4436bc5 
This is a partial PR concerning NNUE CI and PGO builds.
It modifies the appveyor script so it downloads the default
NNUE net before the stockfish bench is run.

No functional change.
@vondele vondele force-pushed the hybridBenchCI branch 5 times, most recently from fc6b7ee to 0a47bad Compare August 10, 2020 20:55
vondele added 2 commits August 10, 2020 23:31
@vondele
Download net in travis CI and adjust tests
ab92c6d 
Adjust travis to cover more archs, speedup instrumented testing,
and download net before building to be able to compute hybrid bench.

in preparation for better NNUE integration in CI.

No functional change.
@vondele
bench covering both NNUE and classical, PGO, CI
21ff092 
Add additional options to the bench command

classical, NNUE, mixed.

requires the default net to be present, can be obtained with

```
make net
```

Default bench is mixed, further examples:

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

Requires the default net to be present,
download automatically if needed for profile-build.

PGO gives a nice speedup on fishtest,
passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569

requires fishtest work, and travis work, before commit.

Bench: 4290577
vondele pushed a commit to vondele/Stockfish that referenced this pull request Aug 11, 2020
@gvreuls @vondele
This commit enables a mixed bench, to improve CI and allow for PGO (p…
19ff8c3 
…rofile-build) of the NNUE part of the code.

Joint work gvreuls / vondele

* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:

classical, NNUE, mixed.

mixed (default) and NNUE require the default net to be present,
which can be obtained with

```
make net
```

Further examples (first is equivalent to `./stockfish bench`):

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)

PGO gives a nice speedup on fishtest:

passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569

passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c

closes official-stockfish#2931

fixes official-stockfish#2907

requires fishtest updates before commit

Bench: 4290577
@vondele vondele added the to be merged Will be merged shortly label Aug 11, 2020
@vondele vondele closed this in ea6220f Aug 11, 2020
joergoster pushed a commit to joergoster/Stockfish-old that referenced this pull request Aug 11, 2020
@gvreuls @joergoster
This commit enables a mixed bench, to improve CI and allow for PGO (p…
f6f528a 
…rofile-build) of the NNUE part of the code.

Joint work gvreuls / vondele

* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:

classical, NNUE, mixed.

mixed (default) and NNUE require the default net to be present,
which can be obtained with

```
make net
```

Further examples (first is equivalent to `./stockfish bench`):

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)

PGO gives a nice speedup on fishtest:

passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569

passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c

closes official-stockfish/Stockfish#2931

fixes official-stockfish/Stockfish#2907

requires fishtest updates before commit

Bench: 4290577
lucabrivio pushed a commit to lucabrivio/Stockfish that referenced this pull request Aug 11, 2020
@gvreuls @lucabrivio
This commit enables a mixed bench, to improve CI and allow for PGO (p…
45ca4c2 
…rofile-build) of the NNUE part of the code.

Joint work gvreuls / vondele

* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:

classical, NNUE, mixed.

mixed (default) and NNUE require the default net to be present,
which can be obtained with

```
make net
```

Further examples (first is equivalent to `./stockfish bench`):

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)

PGO gives a nice speedup on fishtest:

passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569

passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c

closes official-stockfish/Stockfish#2931

fixes official-stockfish/Stockfish#2907

requires fishtest updates before commit

Bench: 4290577
noobpwnftw pushed a commit to noobpwnftw/Stockfish that referenced this pull request Aug 15, 2020
@gvreuls @noobpwnftw
This commit enables a mixed bench, to improve CI and allow for PGO (p…
cf9833c 
…rofile-build) of the NNUE part of the code.

Joint work gvreuls / vondele

* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:

classical, NNUE, mixed.

mixed (default) and NNUE require the default net to be present,
which can be obtained with

```
make net
```

Further examples (first is equivalent to `./stockfish bench`):

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)

PGO gives a nice speedup on fishtest:

passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569

passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c

closes official-stockfish#2931

fixes official-stockfish#2907

requires fishtest updates before commit

Bench: 4290577
Dantist added a commit to Dantist/Stockfish that referenced this pull request Dec 22, 2020
@Dantist
[PART OF] This commit enables a mixed bench, to improve CI and allow …
f0861d0 
…for PGO (profile-build) of the NNUE part of the code.

Joint work gvreuls / vondele

* Download the default NNUE net in AppVeyor
* Download net in travis CI `make net`
* Adjust tests to cover more archs, speedup instrumented testing
* Introduce 'mixed' bench as default, with further options:

classical, NNUE, mixed.

mixed (default) and NNUE require the default net to be present,
which can be obtained with

```
make net
```

Further examples (first is equivalent to `./stockfish bench`):

```
./stockfish bench 16 1 13 default depth mixed
./stockfish bench 16 1 13 default depth classical
./stockfish bench 16 1 13 default depth NNUE
```

The net is now downloaded automatically if needed for `profile-build`
(usual `build` works fine without net present)

PGO gives a nice speedup on fishtest:

passed STC:
LLR: 2.93 (-2.94,2.94) {-0.50,1.50}
Total: 3360 W: 469 L: 343 D: 2548
Ptnml(0-2): 20, 246, 1030, 356, 28
https://tests.stockfishchess.org/tests/view/5f31b5499081672066537569

passed LTC:
LLR: 2.97 (-2.94,2.94) {0.25,1.75}
Total: 8824 W: 609 L: 502 D: 7713
Ptnml(0-2): 8, 430, 3438, 519, 17
https://tests.stockfishchess.org/tests/view/5f31c87b908167206653757c

closes official-stockfish#2931

fixes official-stockfish#2907

requires fishtest updates before commit

Bench: 4290577
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

to be merged Will be merged shortly

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vondele @gvreuls