Skip to content

don't rely on global state to rate-limit AcquireJob requests #9428

Closed

Description

@spikecurtis

in an effort to prevent the database from getting lots of queries from lots of provisioner daemons, a "debounce" mechanism was implemented. However, this mechanism relies on global state, which causes flaky tests whenever it is enabled.

Activity

  1. added
    s3Bugs that confuse, annoy, or are purely cosmetic
    on Aug 30, 2023
  2. self-assigned this
    on Aug 30, 2023
  3. spikecurtis commented on Aug 30, 2023

    @spikecurtis
    CollaboratorAuthor

    I originally thought that we could just solve this issue by moving the debounce state out of globals and into an explicit struct that won't be shared by tests.

    However, I think there are bigger problems with the debouncing strategy for external provisioners with different tags:

    Debouncing maintains the last time any daemon tried to acquire a job and failed, and then doesn't allow new requests for some defined period afterward. But just because we failed to get jobs with a specific set of tags, doesn't mean we will fail for a different set. Our current debouncing strategy would make it so that one daemon that mostly doesn't have any jobs (e.g. a local provisioner for a specific user) will slow down job acquisition for other daemons that do have jobs.

  4. spikecurtis commented on Aug 30, 2023

    @spikecurtis
    CollaboratorAuthor

    Instead of an architecture where we poll periodically on a timer, I think provisioner daemon should "long poll." That is, if it attempts to acquire a job and none are available, that RPC should block until a job is available.

    On the coderd side, instead of periodically polling, we should use the pubsub to get async notifications about available jobs. Coderd still has to run the SQL query to attempt to acquire the job, since there are potentially other coderds that could also acquire it and lock it, but when it fails to acquire a job it should just wait until it gets a notification over the pubsub about a new job matching what it's looking for before querying again, rather than just retrying periodically.

    All this needs to be aware of tags and provisioner types (echo/terraform) to avoid one set of provisioners blocking another.

    @kylecarbs @bpmct I'd like your thoughts here including priority relative to the other external provisionerd work.

  5. bpmct commented on Aug 30, 2023

    @bpmct
    Member

    I can't speak to the architecture. Regarding prioritization, could I have help understanding impact of doing this? e.g. "Jobs are acquired instantly instead of every 3 seconds" or "Aims on reducing coderd load by a significant margin," or "no real user impact, but cleaner architecture which will help with phases 3 and 4 of the RFC."

  6. spikecurtis commented on Aug 31, 2023

    @spikecurtis
    CollaboratorAuthor

    With no action, if you create a bunch of external provisioner daemons, they each poll 1 per second. E.g. if you create 30 external provisioner daemons, you add 30 qps to your idle database load. Jobs can take up to a second to be acquired even when idle.

    If we build what I'm proposing, provisioner daemons (internal and external) will not contribute to idle database load, and jobs should be acquired within a few DB round-trip times (when idle).

  7. bpmct commented on Aug 31, 2023

    @bpmct
    Member

    Ah I see! Will defer to @kylecarbs here. Seems like a good performance optimization but I'd be open to doing it after the other stages in the provisioners RFC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

s3Bugs that confuse, annoy, or are purely cosmetic

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions