You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
don't rely on global state to rate-limit AcquireJob requests #9428
in an effort to prevent the database from getting lots of queries from lots of provisioner daemons, a "debounce" mechanism was implemented. However, this mechanism relies on global state, which causes flaky tests whenever it is enabled.
I originally thought that we could just solve this issue by moving the debounce state out of globals and into an explicit struct that won't be shared by tests.
However, I think there are bigger problems with the debouncing strategy for external provisioners with different tags:
Debouncing maintains the last time any daemon tried to acquire a job and failed, and then doesn't allow new requests for some defined period afterward. But just because we failed to get jobs with a specific set of tags, doesn't mean we will fail for a different set. Our current debouncing strategy would make it so that one daemon that mostly doesn't have any jobs (e.g. a local provisioner for a specific user) will slow down job acquisition for other daemons that do have jobs.
Instead of an architecture where we poll periodically on a timer, I think provisioner daemon should "long poll." That is, if it attempts to acquire a job and none are available, that RPC should block until a job is available.
On the coderd side, instead of periodically polling, we should use the pubsub to get async notifications about available jobs. Coderd still has to run the SQL query to attempt to acquire the job, since there are potentially other coderds that could also acquire it and lock it, but when it fails to acquire a job it should just wait until it gets a notification over the pubsub about a new job matching what it's looking for before querying again, rather than just retrying periodically.
All this needs to be aware of tags and provisioner types (echo/terraform) to avoid one set of provisioners blocking another.
@kylecarbs@bpmct I'd like your thoughts here including priority relative to the other external provisionerd work.
I can't speak to the architecture. Regarding prioritization, could I have help understanding impact of doing this? e.g. "Jobs are acquired instantly instead of every 3 seconds" or "Aims on reducing coderd load by a significant margin," or "no real user impact, but cleaner architecture which will help with phases 3 and 4 of the RFC."
With no action, if you create a bunch of external provisioner daemons, they each poll 1 per second. E.g. if you create 30 external provisioner daemons, you add 30 qps to your idle database load. Jobs can take up to a second to be acquired even when idle.
If we build what I'm proposing, provisioner daemons (internal and external) will not contribute to idle database load, and jobs should be acquired within a few DB round-trip times (when idle).
Ah I see! Will defer to @kylecarbs here. Seems like a good performance optimization but I'd be open to doing it after the other stages in the provisioners RFC
Activity
spikecurtis commentedon Aug 30, 2023
I originally thought that we could just solve this issue by moving the debounce state out of globals and into an explicit struct that won't be shared by tests.
However, I think there are bigger problems with the debouncing strategy for external provisioners with different tags:
Debouncing maintains the last time any daemon tried to acquire a job and failed, and then doesn't allow new requests for some defined period afterward. But just because we failed to get jobs with a specific set of tags, doesn't mean we will fail for a different set. Our current debouncing strategy would make it so that one daemon that mostly doesn't have any jobs (e.g. a local provisioner for a specific user) will slow down job acquisition for other daemons that do have jobs.
spikecurtis commentedon Aug 30, 2023
Instead of an architecture where we poll periodically on a timer, I think provisioner daemon should "long poll." That is, if it attempts to acquire a job and none are available, that RPC should block until a job is available.
On the coderd side, instead of periodically polling, we should use the pubsub to get async notifications about available jobs. Coderd still has to run the SQL query to attempt to acquire the job, since there are potentially other coderds that could also acquire it and lock it, but when it fails to acquire a job it should just wait until it gets a notification over the pubsub about a new job matching what it's looking for before querying again, rather than just retrying periodically.
All this needs to be aware of tags and provisioner types (echo/terraform) to avoid one set of provisioners blocking another.
@kylecarbs @bpmct I'd like your thoughts here including priority relative to the other external provisionerd work.
bpmct commentedon Aug 30, 2023
I can't speak to the architecture. Regarding prioritization, could I have help understanding impact of doing this? e.g. "Jobs are acquired instantly instead of every 3 seconds" or "Aims on reducing coderd load by a significant margin," or "no real user impact, but cleaner architecture which will help with phases 3 and 4 of the RFC."
spikecurtis commentedon Aug 31, 2023
With no action, if you create a bunch of external provisioner daemons, they each poll 1 per second. E.g. if you create 30 external provisioner daemons, you add 30 qps to your idle database load. Jobs can take up to a second to be acquired even when idle.
If we build what I'm proposing, provisioner daemons (internal and external) will not contribute to idle database load, and jobs should be acquired within a few DB round-trip times (when idle).
bpmct commentedon Aug 31, 2023
Ah I see! Will defer to @kylecarbs here. Seems like a good performance optimization but I'd be open to doing it after the other stages in the provisioners RFC