Skip to content

ci: Run tests 3 times to reduce flakes - #20

Merged
kylecarbs merged 4 commits into
mainfrom
peerflake
Jan 13, 2022
Merged

kylecarbs merged 4 commits into
mainfrom
peerflake

Conversation

@kylecarbs

Copy link
Copy Markdown
Member

No description provided.

@kylecarbs kylecarbs self-assigned this Jan 13, 2022
@codecov

codecov Bot commented Jan 13, 2022 •

Copy link
Copy Markdown

Codecov Report

Merging #20 (f61d9ea) into main (92710ed) will increase coverage by 0.17%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #20      +/-   ##
==========================================
+ Coverage   70.26%   70.44%   +0.17%     
==========================================
  Files          18       18              
  Lines        1130     1130              
==========================================
+ Hits          794      796       +2     
+ Misses        266      264       -2     
  Partials       70       70              
Flag Coverage Δ
macos-latest 62.62% <ø> (+1.01%) ⬆️
ubuntu-latest 70.08% <ø> (+1.50%) ⬆️
windows-latest 62.42% <ø> (+1.93%) ⬆️
Impacted Files Coverage Δ
peer/conn.go 73.98% <0.00%> (+0.31%) ⬆️
peer/channel.go 88.05% <0.00%> (+0.62%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92710ed...f61d9ea. Read the comment docs.

Comment thread .github/workflows/coder.yaml Outdated
- run:
gotestsum --jsonfile="gotests.json" --packages="./..." --
-covermode=atomic -coverprofile="gotests.coverage" -timeout=1m
-covermode=atomic -coverprofile="gotests.coverage" -timeout=3m -count=10 -race

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's very cool that gotestsum has a count option. Having the multiple-iteration run should go a long way in helping us catch these intermittent failures!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plus we're actually get a 30x run out of this since it is running 10x in parallel on Windows/mac/linux

@bryphe-coder

Copy link
Copy Markdown
Contributor

The error in the PR here is really strange:

WARNING: DATA RACE
Read at 0x00c0003c44f0 by goroutine 123:
  github.com/pion/ice/v2.(*Agent).Close()
      /home/runner/go/pkg/mod/github.com/pion/ice/v2@v2.1.18/agent.go:899 +0xde
  github.com/pion/webrtc/v3.(*ICEGatherer).Close()
      /home/runner/go/pkg/mod/github.com/pion/webrtc/v3@v3.1.13/icegatherer.go:184 +0xd9
  github.com/pion/webrtc/v3.(*ICETransport).Stop()
      /home/runner/go/pkg/mod/github.com/pion/webrtc/v3@v3.1.13/icetransport.go:197 +0x176
  github.com/pion/webrtc/v3.(*PeerConnection).Close()
      /home/runner/go/pkg/mod/github.com/pion/webrtc/v3@v3.1.13/peerconnection.go:1905 +0x724
  github.com/coder/coder/peer.(*Conn).closeWithError()
      /home/runner/work/coder/coder/peer/conn.go:494 +0x414
  github.com/coder/coder/peer.(*Conn).CloseWithError()
      /home/runner/work/coder/coder/peer/conn.go:460 +0x51
  github.com/coder/coder/peer_test.TestConn.func9()
      /home/runner/work/coder/coder/peer/conn_test.go:201 +0xb6
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.6/x64/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /opt/hostedtoolcache/go/1.17.6/x64/src/testing/testing.go:1306 +0x47

Haven't seen it before in any of our runs. I wonder if the --count option is doing something undesirable, like running individual tests in parallel?

@bryphe-coder

Copy link
Copy Markdown
Contributor

An alternative is we could consider having a helper script that runs the gotestsum command multiple times, like from this stackoverflow:

run() {
    number=$1
    shift
    for i in `seq $number`; do
      $@
    done
}

@kylecarbs

Copy link
Copy Markdown
Member Author

The count is actually from the Go test CLI. It essentially runs the command multiple times.

Unfortunately, we'll have to use my fork of pion/ice, until my PR gets merged.

@kylecarbs kylecarbs changed the title ci: Run tests 10 times to reduce flakes ci: Run tests 3 times to reduce flakes Jan 13, 2022
@kylecarbs
kylecarbs merged commit 550c4fb into main Jan 13, 2022
@kylecarbs
kylecarbs deleted the peerflake branch January 13, 2022 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants