Skip to content

User-local provisioner daemons #909

Closed

Description

@kconley-sq

Our team's developers need to access their local USB devices in their Coder workspaces via USB/IP, and to minimize the latency between a USB/IP server running on a developer's local machine (which serves the USB hardware they have connected to their machine) and the USB/IP client running in a Coder workspace (which attaches that USB hardware remotely over the network), we'd like our developers to be able to provision Coder workspaces to run on their local machines (e.g. using the Docker Terraform provider).

I was describing this use case to @kylecarbs on Slack, and he mentioned:

Coder v2 has the concept of a provisioner daemon. It accepts provision operations (like workspace start/stop) and performs them. Developers could run a provisioner daemon on their local machine, and then Terraform would execute against it. This would enable Terraform to run locally, so you could use unix:///var/run/docker.sock. Our architecture certainly enables this.

We'll have to work with y'all to get the UX nice, but we can certainly provide a flow like:

  1. Download the Coder CLI.
  2. Run "coder provisioner" in the background.
  3. Have the workspace provisioned on your local machine, using that provisioner daemon.

We'll have to add some targeting to provisionerd, but that should be simple enough!

That sounds exactly like what we were hoping for, so I've created this GitHub issue to formally request this feature in Coder. 😃

Building on what @kylecarbs suggested above, I think enabling something like the following would meet our requirements:

  1. A developer can run a Coder CLI command (e.g. coder provisioner) to launch a local provisioner that is associated/authenticated with coderd using the credentials/URL the developer previously provided via coder login.
  2. A developer can specify that workspace operations (e.g. creation, stopping, etc.) should use a particular provisioner daemon so that they can ensure that a workspace will be created/provisioned/stopped on their local machine as opposed to other machines that would not immediately have the ability to provision/control Terraform resources on the developer's local machine.
    • For the CLI, maybe this could be specified via an option when running coder workspace create like --provisioner-daemon <PROVISIONER_DAEMON_ID> where a developer specifies the name/reference to the provisioner daemon of interest via <PROVISIONER_DAEMON_ID>?
    • For the web UI, maybe this could be a drop-down on the workspace creation webpage from which the user could select a provisioner daemon from a list of provisioner daemons that they are authorized to use (see below)?
  3. A developer can restrict other Coder users from creating workspaces using a provisioner daemon they are running on their local machine. I think this would help prevent one developer from accidentally (or perhaps maliciously) creating a workspace on a different developer's machine as well as avoid having too many options for selecting a provisioner daemon in the new workspace creation web UI (see above about the proposed drop-down option) when many developers maintain their own provisioner daemons.
    • I think something that would work well for our team would be if the default behavior of Coder would be to give a developer the ability to create workspaces using a provisioner daemon they've spawned themselves but restrict other users from utilizing it unless the provisioner daemon has been associated/registered to be shared throughout a Coder organization. And maybe that association/registration could be performed by the developer via a CLI command or web UI page?

What do you think?

cc @sunilvenkatraman

Activity

  1. self-assigned this
    on Apr 12, 2022
  2. kylecarbs commented on Apr 12, 2022

    @kylecarbs
    Member

    @kconley-sq I was able to get this working. It's not scoped to a specific user right now, but we are able to use the Docker Terraform Provider to spin up a workspace. This is a basic template that lets you specify an image. Pretty neat!

    terraform {
      required_providers {
        coder = {
          source  = "coder/coder"
          version = "~> 0.3.1"
        }
        docker = {
          source  = "kreuzwerker/docker"
          version = "~> 2.16.0"
        }
      }
    }
    
    provider "coder" {
      url = "http://host.docker.internal:3000/"
    }
    
    provider "docker" {
      host = "unix:///var/run/docker.sock"
    }
    
    resource "coder_agent" "dev" {
      arch = "amd64"
      os   = "linux"
    }
    
    resource "docker_image" "go" {
      name = "mcr.microsoft.com/vscode/devcontainers/go:1"
    }
    
    resource "docker_container" "go" {
      image      = docker_image.go.latest
      name       = "go"
      entrypoint = ["sh", "-c", coder_agent.dev.init_script]
      user       = "vscode"
      host {
        host = "host.docker.internal"
        ip   = "host-gateway"
      }
      env = [
        "CODER_TOKEN=${coder_agent.dev.token}"
      ]
    }
  3. bpmct commented on May 5, 2022

    @bpmct
    Member

    @arivictor would something like this work for your use case of local machines?

  4. tjcran commented on May 16, 2022

    @tjcran

    I am adding this one into Community MVP milestone. It unblocks specific use cases we've heard from multiple people now.

  5. added this to the Community MVP milestone on May 16, 2022
  6. misskniss commented on May 17, 2022

    @misskniss
  7. 7 remaining items

  8. removed this from the Post-MVP milestone on Jul 22, 2022
  9. dwahler commented on Jul 28, 2022

    @dwahler
    Contributor

    It sounds like there are a few prerequisites to being able to implement this as more than just a demo:

    First of all, we need to decide on a notion of identity for provisioners, so that we can distinguish between different instances of the Terraform provisioner. Simplest thing is probably just to assign each of them a UUID, but we might also want to consider namespacing them, so that you'd have e.g. global and dwahler/local.

    Secondly, we need to think about security (authn/authz). #44 covers authentication between coderd and provisionerd. For authorization, @kconley-sq's proposed defaults sound great, but I can think of some other potential things to consider:

    • Should every user be allowed to run local provisioners?
    • If they have a local provisioner, does that automatically imply the ability to create their own templates?
    • Do we need to restrict which templates can run on which provisioners, to avoid leaking sensitive template parameters?
    • Should we require some kind of local user confirmation before running arbitrary Terraform code that we receive from the remote coderd?

    And then there are lifecycle issues. It probably doesn't make sense to delete a provisioner instance if there are existing workspaces associated with it. But if the provisioner goes away unexpectedly (e.g. your laptop dies) then we probably need a way to clean up its state.

  10. github-actions commented on Oct 22, 2022

    @github-actions
    Contributor

    This issue is becoming stale. In order to keep the tracker readable and actionable, I'm going close to this issue in 7 days if there isn't more activity.

  11. assigned and unassigned on Oct 26, 2022
  12. kylecarbs commented on Nov 7, 2022

    @kylecarbs
    Member
  13. kylecarbs commented on Nov 9, 2022

    @kylecarbs
    Member

    Provisioner daemons and template versions will gain the concept of tags. Tags must match for a provision job to execute on the daemon.

    Tags will be defined in the frontmatter of a template's README:

    ---
    tags:
    - scope: organization
    - environment: staging
    ---
    
    # Title
    
    Some additional information...

    When a provisioner job is queued (like template creation or workspace build), it will be clear to the user that the job is pending on a set of tags. If provisioner: user is specified, we'll make a unique exception in the UI directing the user to start a provisioner daemon. All users can create provisioner daemons, but they are useless unless templates target them.

    # Registers and starts a new provisioner daemon with a set of tags
    $ coder provisionerd start --tag environment=staging --tag scope=organization

    The scope tag specifies whether the provisioner daemon is registered for the organization or the user. If not provided, the default value is user. If the user doesn't have permission to create an organization-wide provisioner daemon, the request to start will fail.

    The following tags will be specified by default but can be overridden by the user:

    • arch: Specifies the architecture of the host system.
    • os: Specifies the operating system of the host system.
  14. kylecarbs commented on Dec 14, 2022

    @kylecarbs
    Member

    This is done!

  15. matifali commented on Jan 4, 2024

    @matifali
    Contributor

    @bpmct @kylecarbs, We currently do not have the ability to run a workspace on the developer's local docker daemon (as requested originally in the issue); I think we should reopen this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

apiArea: HTTP API

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions