Skip to content

feat(scaletest/dashboard): integrate chromedp - #9927

Merged
johnstcn merged 12 commits into
mainfrom
cj/scaletest-dashboard-add-chromedp
Oct 2, 2023
Merged

johnstcn merged 12 commits into
mainfrom
cj/scaletest-dashboard-add-chromedp

Conversation

@johnstcn

@johnstcn johnstcn commented Sep 29, 2023 •

Copy link
Copy Markdown
Member

Fixes #9131

This PR replaces the existing implementation of dashboard load test generation with chromedp.

The 10,000 foot view:

  • Open a browser
  • Generate a session token
  • Visit client.URL
  • Pick one of the elements that's visible on the page
  • Click it and wait for a predefined element to appear
  • Repeat until context is done

Notes/future work:

I initially tried to make a tight "state machine" and model the valid transitions from one page to another. It quickly became evident that this would break quickly, so I refactored into the more "stateless" form that you see here.

There is definitely some room for more "intelligent" choice of elements to click, as opposed to completely random.
Edit: you can specify a seed with --rand-seed to get a deterministic test.

This commit adds a set of actions to automatically
interact with a Coder instance using chromedp.
This commit integrates the chromedp actions in the previous commit
into the scaletest dashboard command, and re-enables the previously
disabled unit test.

Note that this unit test does not actually run headless chrome, nor
does it test the actual scaletest actions yet, as coderdtest only
exposes an API and does not expose the actual site.
@johnstcn johnstcn self-assigned this Sep 29, 2023
@johnstcn
johnstcn marked this pull request as ready for review September 29, 2023 10:19
@johnstcn
johnstcn requested review from mafredri and mtojek September 29, 2023 10:20
Comment thread scaletest/dashboard/run.go Resolved
Comment thread scaletest/dashboard/config.go Resolved
{
Label: "template_files",
ClickOn: `a[href^="/templates/"][href$="/docs"]:not([aria-current])`,
WaitFor: `.monaco-editor`,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we wrap with an extra coder-editor div or .monaco-editor introduced by us?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, it seems to work for the moment?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's leave it as is 👍

@Emyrk Emyrk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

This is pretty neat. Just curious, would this be crazy to use for some e2e style tests? I think we use nodejs and some browser automation for that, but I like it being in Go 😆.

I ask for some of the OIDC stuff which has a lot of cookie interactions.

Comment thread scaletest/dashboard/chromedp.go Resolved
@johnstcn

Copy link
Copy Markdown
Member Author

Just curious, would this be crazy to use for some e2e style tests? I think we use nodejs and some browser automation for that, but I like it being in Go 😆.

Under the hood, this probably does a very similar thing to what Playwright does.
I can definitely see it being useful for e2e-style tests. I consciously tried to stay away from performing any actual e2e-style testing in this actual change, but I definitely think there's a use-case here. It would mean re-writing a lot of the e2e stuff though.

Comment thread cli/exp_scaletest.go Outdated
Flag: "max-wait",
Env: "CODER_SCALETEST_DASHBOARD_MAX_WAIT",
Default: "1s",
Default: "10s",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance we may be able to eliminate these options? It'd be nice if you didn't need to adjust these parameters to have a working load test, and that it'd continue working even if coderd was under too much load, etc.

We could instead track user experience impact when things slow down to a crawl.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This more just controls the interval between actions. Between 1 and 10 seconds I think is a reasonable default. I'll rename it to better reflect its semantics.

Comment thread go.mod
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/chromedp/cdproto v0.0.0-20230802225258-3cf4e6d46a89 // indirect
github.com/chromedp/chromedp v0.9.2 // indirect
github.com/chromedp/sysutil v1.0.0 // indirect

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These new imports add 5MB to the slim binary, perhaps we need to consider breaking off exp scaletest from slim 🤔

@johnstcn johnstcn Sep 29, 2023 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do this here 👍

Edit: better done in a follow-up PR

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread scaletest/dashboard/chromedp.go Resolved
@johnstcn
johnstcn merged commit 1c48610 into main Oct 2, 2023
@johnstcn
johnstcn deleted the cj/scaletest-dashboard-add-chromedp branch October 2, 2023 09:40
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

scaletest: improve dashboard test

4 participants