Skip to content

fix(cli)!: protect client Logger and refactor cli scaletest tests - #8317

Merged
johnstcn merged 16 commits into
mainfrom
cj/scaletest-createworkspaces-flake
Jul 6, 2023
Merged

johnstcn merged 16 commits into
mainfrom
cj/scaletest-createworkspaces-flake

Conversation

@johnstcn

@johnstcn johnstcn commented Jul 5, 2023 •

Copy link
Copy Markdown
Member

This attempts to address the test flakes reported in #4942 -- sadly, the runner logs have expired and are no longer available so I've done a best-effort fix here.

  • (breaking?) Protects Logger and LogBodies fields of codersdk.Client with its mutex. This addresses a data race in cli/scaletest.
  • Fillets the existing cli/createworkspaces unit test and moves the testing logic there into the tests under scaletest/createworkspaces.
  • Adds testutil.RaceEnabled bool const and conditionaly skips previously-skipped tests under scaletest/ if the race detector is enabled. This is unfortunate and sad, but I would prefer to have these tests at least running without the race detector than not running at all.
  • Adds IgnoreErrors option to fake in-memory agent loggers; having the agents fail the test immediately when they encounter any sort of error isn't really helpful.

@johnstcn johnstcn self-assigned this Jul 5, 2023
Comment thread cli/scaletest_test.go
t.Skipf("This test is flakey. See https://github.com/coder/coder/issues/4942")
t.Parallel()

// This test does a create-workspaces scale test with --no-cleanup, checks

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review: already tested in scaletest/createworkspaces/run_test.go. No point duplicating.

Comment thread codersdk/client.go Outdated
Comment on lines +106 to +111
// Logger is optionally provided to log requests.
// logger is optionally provided to log requests.
// Method, URL, and response code will be logged by default.
Logger slog.Logger
logger slog.Logger

// LogBodies can be enabled to print request and response bodies to the logger.
LogBodies bool
// logBodies can be enabled to print request and response bodies to the logger.
logBodies bool

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review: these fields were being concurrently accessed in a number of places. Protecting them with a mutex for the moment. This treatment could theoretically be extended to other fields, but some fields (like URL) are very frequently accessed, so such a change has a large blast radius.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This treatment could theoretically be extended to other fields, but some fields (like URL) are very frequently accessed, so such a change has a large blast radius.

I think the URL can do without a mutex, it's set in New and shouldn't change after the fact. We probably should pass logger in New as well since it's a bit weird that it changes later.

PS. I wouldn't be too worried about a mutex and frequent field access, the client isn't performance critical and the delays seem negligible to me.

Comment thread scaletest/createworkspaces/run_test.go Outdated
Comment on lines +85 to +87
// Since the runner creates the workspace on it's own, we have to keep
// listing workspaces until we find it, then wait for the build to
// finish, then start the agents.
go func() {
var workspace codersdk.Workspace
for {
res, err := client.Workspaces(ctx, codersdk.WorkspaceFilter{})
if !assert.NoError(t, err) {
return
}
workspaces := res.Workspaces

if len(workspaces) == 1 {
workspace = workspaces[0]
break
}

time.Sleep(100 * time.Millisecond)
}

coderdtest.AwaitWorkspaceBuildJob(t, client, workspace.LatestBuild.ID)

agentClient := agentsdk.New(client.URL)
agentClient.SetSessionToken(authToken)
agentCloser := agent.New(agent.Options{
Client: agentClient,
Logger: slogtest.Make(t, nil).Named("agent").Leveled(slog.LevelWarn),
})
t.Cleanup(func() {
_ = agentCloser.Close()
})

coderdtest.AwaitWorkspaceAgents(t, client, workspace.ID)
}()
go eventuallyStartFakeAgent(t, ctx, client, authToken)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review: extracted to test helper

require.Len(t, workspaces.Workspaces, 0)
})

t.Run("NoCleanup", func(t *testing.T) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review: this is mostly copy-pasta that only serves to validate that the NoCleanup option is respected.

Comment thread testutil/race_on.go
Comment on lines +1 to +7
//go:build race

package testutil

const (
RaceEnabled = true
)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: this may be controversial, but conditionally skipping tests that we know are flaky and slow under the race detector is preferable to just leaving these tests languishing in a perma-skipped state.

@johnstcn johnstcn changed the title chore(cli): address test flake in scaletest/createworkspaces chore(cli): un-skip tests in scaletest Jul 5, 2023
@johnstcn johnstcn changed the title chore(cli): un-skip tests in scaletest chore(cli): refactor cli scaletest tests Jul 5, 2023
@johnstcn
johnstcn requested review from mafredri and mtojek July 5, 2023 11:03
@johnstcn
johnstcn marked this pull request as ready for review July 5, 2023 11:03
@mafredri

mafredri commented Jul 5, 2023

Copy link
Copy Markdown
Member

I didn't review yet, but

(breaking?) Protects Logger and LogBodies fields of codersdk.Client with its mutex. This addresses a data race in cli/scaletest.

I think PR might be better tagged as test (vs chore, which is essentially "non-code changes"), but I think we should avoid a breaking change in a PR that's tagged either (test/chore). Should we break this up into two PRs? Or perhaps the title could be something along the lines of refactor: protect codersdk logger and refactor scaletest tests.

@johnstcn

johnstcn commented Jul 5, 2023

Copy link
Copy Markdown
Member Author

I didn't review yet, but

(breaking?) Protects Logger and LogBodies fields of codersdk.Client with its mutex. This addresses a data race in cli/scaletest.

I think PR might be better tagged as test (vs chore, which is essentially "non-code changes"), but I think we should avoid a breaking change in a PR that's tagged either (test/chore). Should we break this up into two PRs? Or perhaps the title could be something along the lines of refactor: protect codersdk logger and refactor scaletest tests.

I'll update with the fix prefix instead.

@johnstcn johnstcn changed the title chore(cli): refactor cli scaletest tests fix(cli)!: refactor cli scaletest tests Jul 5, 2023
@github-actions github-actions Bot added the release/breaking This label is applied to PRs to detect breaking changes as part of the release process label Jul 5, 2023
@johnstcn johnstcn changed the title fix(cli)!: refactor cli scaletest tests fix!(cli): refactor cli scaletest tests Jul 5, 2023
@github-actions github-actions Bot removed the release/breaking This label is applied to PRs to detect breaking changes as part of the release process label Jul 5, 2023
@johnstcn johnstcn changed the title fix!(cli): refactor cli scaletest tests fix!(cli): protect client Logger and refactor cli scaletest tests Jul 5, 2023
Comment thread codersdk/client.go Outdated
Comment on lines +106 to +111
// Logger is optionally provided to log requests.
// logger is optionally provided to log requests.
// Method, URL, and response code will be logged by default.
Logger slog.Logger
logger slog.Logger

// LogBodies can be enabled to print request and response bodies to the logger.
LogBodies bool
// logBodies can be enabled to print request and response bodies to the logger.
logBodies bool

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This treatment could theoretically be extended to other fields, but some fields (like URL) are very frequently accessed, so such a change has a large blast radius.

I think the URL can do without a mutex, it's set in New and shouldn't change after the fact. We probably should pass logger in New as well since it's a bit weird that it changes later.

PS. I wouldn't be too worried about a mutex and frequent field access, the client isn't performance critical and the delays seem negligible to me.

Comment thread testutil/race_off.go Resolved
Comment thread scaletest/createworkspaces/run_test.go Resolved
@johnstcn johnstcn changed the title fix!(cli): protect client Logger and refactor cli scaletest tests fix(cli)!: protect client Logger and refactor cli scaletest tests Jul 5, 2023
@github-actions github-actions Bot added the release/breaking This label is applied to PRs to detect breaking changes as part of the release process label Jul 5, 2023

@mtojek mtojek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather set logger and logBoddies once during object creation, but I guess that this is beyond the scope of this PR.

Comment thread codersdk/client.go

// Logger returns the logger for the client.
func (c *Client) Logger() slog.Logger {
c.mu.RLock()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a comment to mu sync.RWMutex, which describes what exactly you want to guard. I have a feeling that one-day future somebody will try to remove it and misses its point.

@johnstcn

johnstcn commented Jul 5, 2023

Copy link
Copy Markdown
Member Author

I'd rather set logger and logBoddies once during object creation, but I guess that this is beyond the scope of this PR.

Agreed, I'd honestly push for all the fields of codersdk.Client to be immutable upon creation. That's a larger refactor than I'd be happy to take on at the moment though.

Comment thread scaletest/createworkspaces/run_test.go Resolved
@mafredri

mafredri commented Jul 5, 2023 •

Copy link
Copy Markdown
Member

I'd rather set logger and logBoddies once during object creation, but I guess that this is beyond the scope of this PR.

To be fair, logBodies seems useful enough to be able to change at runtime. Let's say you do an API request, it fails, then you retry with log bodies. (Looking at this from a consumer perspective.)

Then again, we could use functional options instead, the aforementioned option is useful on a per-request basis. Essentially client.FetchThing(ctx, id, codersdk.WithLogBodies()).

@johnstcn
johnstcn merged commit 7fcf319 into main Jul 6, 2023
@johnstcn
johnstcn deleted the cj/scaletest-createworkspaces-flake branch July 6, 2023 08:43
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

release/breaking This label is applied to PRs to detect breaking changes as part of the release process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants