Skip to content

Resource Metadata + Secrets #2989

Closed

Description

@ammario

Problem statement

Developers want to see additional metadata about resources in Coder. This could be an instance type, the number of cores an instance has, or a link to the cloud resource itself (say in a GCP dashboard).

As a developer, I want to see information about my workspace, both information that is dynamic (e.g external IP) and the most relevant aspects of the template (no. CPUs).

Definition of done

We should add arbitrary key/value pairs that allow template authors to expose additional metadata. This should be added in our Terraform Provider, allowing customers to expose arbitrary values from existing resources that are provisioned. The key/value pairs would be available from the CLI and the dashboard.


Implementation notes (@bpmct)

Dashboard (figma):

Nested resources (3)

Template:

resource "null_resource" "about" {}

resource "coder_metadata" "about_info" {
  resource_id = null_resource.about.id
  item {
    key = "type"
    value = null
  }
  item {
    key = "quicklinks"
    value = "[onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)"
  }
}

resource "kubernetes_pod" "main-dev" {
  # ...
}

resource "coder_metadata" "dev_info" {
  resource_id = kubernetes_pod.main-dev.id
  item {
    key = "api_key"
    value = "asdfjkl"
    sensitive: true
  }
}

CLI:

$ coder metadata my-workspace
RESOURCE_NAME      KEY                 VALUE
about              workspace-id        acf2a70d-6347-4b8c
about              quicklinks          [onboarding guide](https://mywiki.com/onboarding), [troubleshooting](https://mywiki.com/troubleshooting)
main-dev           type             kubernetes_pod
main-dev           api_key             (hidden)

Some values are sensitive. Use `coder metadata myworkspace <key>` to fetch individual values.

In the future, perhaps resource_id = "global" or global = true could be used to display metadata workspace-wide in another panel if the hack is widely used.


Related:

Activity

  1. changed the title [-]Resource Metadata[/-] [+]Resource Metadata + Secrets[/+] on Jul 14, 2022
  2. ketang commented on Jul 14, 2022

    @ketang
    Contributor

    I'd like to give customers more help here. Are there certain common items we can discover automatically? Or perhaps naming conventions that tell us to pull out particular variables?

    Also a limitation (perhaps) of the above is around units/types and naming. For instance we measure RAM and disk in GB, but they're separate concepts. And they may want to measure CPU in a strict count.

  3. ammario commented on Jul 14, 2022

    @ammario
    MemberAuthor

    I don't think we can easily assume which parts of the resource object are most relevant to the user.

  4. ketang commented on Jul 14, 2022

    @ketang
    Contributor

    I wouldn't assume in terms of limitations on what can be relevant. But if we make it zero effort to see how much e.g. S3 space is in use, I think that'd be good. And we should be able to do it with no coupling worth noting.

  5. ammario commented on Jul 14, 2022

    @ammario
    MemberAuthor

    We can't really make arbitrary API calls (like getting the s3 bucket used) to these clouds. That would couple us heavily with the provisioner environment. So, we're limited to what terraform is exposing as variables.

    If we want some "default metadata" to expose, we can just put those blocks in our example templates. That way the user can easily discover how to disable them and how to expose more metadata.

  6. ketang commented on Jul 14, 2022

    @ketang
    Contributor

    Sorry, "use" was the wrong word. I meant "allocated," which is something we can know from just our local information. We're targeting quotas to work on workspaces not in them, so that should be fine.

    I don't want to give up on actual usage one day; I think it's pretty doable, but that's not what I meant above.

  7. ammario commented on Jul 14, 2022

    @ammario
    MemberAuthor

    Sorry, "use" was the wrong word. I meant "allocated," which is something we can know from just our local information. We're targeting quotas to work on workspaces not in them, so that should be fine.

    Ok, in that case I think we should include default metadata blocks in our examples.

  8. assigned and unassigned on Jul 19, 2022
  9. ketang commented on Jul 19, 2022

    @ketang
    Contributor

    I think the way we should provide access to the secrets in the workspace is via a command coder env get <key name> (run in the workspace that has access to the values). That is flexible enough to work on multiple operating systems in a consistent way and gives users a starting point where they can then store the values in anything else they want.

    I believe this can be done in a satisfactory manner with no web UI. The template files and the above coder command should do it pretty well. Perhaps there's future work that would justify a web UI, but I think to start we don't need any of CLI, API, or web UI.

  10. ammario commented on Jul 19, 2022

    @ammario
    MemberAuthor

    Are you suggesting that on top of the original post's suggestion or instead of?

    Also environment secrets can already be injected without any new feature, which is why I left the environment injection outside of my suggestion:

    resource "twilio_iam_api_key" "api_key" {
      account_sid   = "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      friendly_name = "Test API Key"
    }
    
    resource "coder_agent" "dev" {
      # ...
      env = {
        TWILIO_API_SECRET = "${twilio_iam_api_key.api_key.secret}"
      }
    }
  11. ammario commented on Jul 19, 2022

    @ammario
    MemberAuthor

    If you're suggesting that we add a coder env command in addition to environment variables, check out how Vercel approaches the problem.

    TL;DR they don't have a vercel env get command, probably because it leads to an anti-pattern in scripts where you call out to vercel instead of the environment. This breaks the repo when running in CI or another environment without the coder command present. Also: there is no compatibility risk with scripts? All shell interpreters and most programming languages abstract environment variables to a portable interface.

  12. 31 remaining items

  13. bpmct commented on Jul 20, 2022

    @bpmct
    Member

    To my knowledge, we haven't had any user requests for this feature. It does unlock some valuable use cases:

    • display CPU cores to workspace user
    • display links to relevant docs to workspace user
    • display necessary values from Terraform-generated resources (e.g API key) for consumption elsewhere
    • hide resource type (unnecessary complexity in dashboard)
    • display the instance type to the workspace user
    • down the road: custom icons for resources

    I think it's still valuable to define the feature & schema but I suggest we don't prioritize this in coming weeks, especially as we focus on "blocking" issues. For now, I consider it a "nice-to-have"

    cc @misskniss @tjcran

  14. ketang commented on Jul 20, 2022

    @ketang
    Contributor

    Where a subset of this may relate to demonstrated customer needs is with respect to resource quotas.

  15. bpmct commented on Jul 21, 2022

    @bpmct
    Member

    After a bit more research, I think metadata is still the best name for this feature. As Ammar mentioned, it is a broad term but the feature itself will likely expand to support more than simply key/value pairs in future iterations so the term is "encompassing".

    I also think it is distinct enough from parameters.

    Also spoke with @kylecarbs some more and since it's on the roadmap let's keep it moving! Over the next week or so, we might adjust as the roadmap changes but spoke with @dwahler and @BrunoQuaresma and seems like we're good to go.

  16. ammario commented on Jul 21, 2022

    @ammario
    MemberAuthor

    Thanks @bpmct. What is the expected timeline for an MVP into main?

  17. dwahler commented on Jul 21, 2022

    @dwahler
    Contributor

    For the initial backend implementation, here's what I'm thinking:

    • To keep things simple, for now let's assume that metadata is always associated with a resource, which might be a null_resource
    • Metadata values are always either strings or null (integers etc. would be supported in templates, but stringified)
    • The provisioner API will not directly return coder_metadata resources in job results (since they're an implementation detail of the Terraform provisioner) and instead extracts their key/value data and returns it along with the "target" resource
    • Metadata is stored in a new workspace_resource_metadata DB table and included as a field of the WorkspaceResource object that we return from /api/v2/workspacebuilds/{id}/resources:
    @@ -28,6 +28,12 @@ type WorkspaceResource struct {
            Type       string                      `json:"type"`
            Name       string                      `json:"name"`
            Agents     []WorkspaceAgent            `json:"agents,omitempty"`
    +       Metadata   []WorkspaceResourceMetadata `json:"metadata,omitempty"`
     }
     
    +type WorkspaceResourceMetadata struct {
    +       Key       string `json:"key"`
    +       Value     string `json:"value"`
    +       Sensitive bool   `json:"sensitive,omitempty"`
    +	// in future: Icon? Order? Type? LastUpdated?
    +}
    • The API response includes both "explicit" metadata fields returned from the provisioner, and "implicit" ones like type, with explicit taking precedence and null-valued fields omitted from the response

    How does that sound? If I don't run into any unexpected snags in the implementation, I expect to be able to get the backend and CLI parts of this done within the next week.

  18. kylecarbs commented on Jul 21, 2022

    @kylecarbs
    Member

    That seems perfect to me!

  19. tjcran commented on Jul 25, 2022

    @tjcran

    @ammario @bpmct @kylecarbs we now just need to weigh this against the other roadmap items for August, but this seems like a solid candidate.

  20. bpmct commented on Jul 26, 2022

    @bpmct
    Member

    @tjcran This is currently in this sprint, the July roadmap, and being worked on. We can consider prioritizing against the August roadmap if necessary

  21. dwahler commented on Jul 27, 2022

    @dwahler
    Contributor

    I've gotten the core of this (provisioner and API only) implemented in PRs coder/terraform-provider-coder#34 and #3242.

    Unfortunately, I hit a noticeable snag, which is that the syntax described in the requirements:

    resource "coder_metadata" "dev_info" {
      resource_id = kubernetes_pod.main-dev.id
      # ...
    }
    

    may not actually be valid Terraform code. If the target resource is something like a pod whose lifecycle is tied to the workspace start/stop, then it has a count attribute which may be either 0 or 1, and you get an error like this when trying to directly reference its id field:

    Error: Missing resource instance key
    Because kubernetes_pod.main-dev has "count" set, its attributes must be accessed on specific instances.

    The error is from Terraform itself, not from our provider, and I haven't yet found anything in the language that would allow us to make the original syntax work. As a workaround, you can do this instead, but it's kind of ugly:

    resource "coder_metadata" "dev_info" {
      resource_id = try(kubernetes_pod.main-dev[0].id, "")
      # ...
    }
    

    Any suggestions for better ways to deal with this are welcome.

  22. bpmct commented on Jul 27, 2022

    @bpmct
    Member

    @dwahler that seems reasonable to me. Template admins could add a count to the metadata to avoid the try function 🤷🏼

    # main.tf
    resource "aws-instance" "dev" {
        count = count = data.coder_workspace.me.start_count
        # ... 
    }
    
    resource "coder_metadata" "dev_info" {
    +  count = data.coder_workspace.me.start_count
    
    -   resource_id = try(kubernetes_pod.main-dev[0].id, "")
    +   resource_id = kubernetes_pod.main-dev[0].id
      # ...
    }

    Another thing that comes to mind is using Terraform's depends_on to establish the relation, but I think that introduces some more problems:

    • user may try to add multiple resources to depends_on or legitimately need it to depend on a resource
    • to my knowledge, depends_on works for the resource as a whole. no way to have separate metadata for a unique "count" of a resource
  23. bpmct commented on Jul 28, 2022

    @bpmct
    Member

    Note: I'll write the docs for this once the frontend comes in and close the issue once that is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions