Skip to content

Simplify identification of orphaned resources with better workspace ID #3048

Closed as not planned

Description

@ammario

Currently, the workspace ID is generated from a UUID.

As an infrastructure admin, occasionally I have a workspace that fails to delete or the state is otherwise corrupted. In such cases, I may want to use a script to delete the orphaned resources. Since the workspace ID should be used in the persistent resource ID (see #3000 (comment)), we should include some identifying information in the workspace ID.

Here's an example of a possible format:

  • template ID fragment: the first 8 hex characters of the template ID
  • user ID fragment: the first 8 hex characters of the user ID
  • random: 10 random hexadecimal chars (40 bits of entropy)

This leads to an ID that looks like:

abababab-cdcdcdcd-efefefefefefefefefefefefef

Which would have a 50% chance of collision if there were 1 million workspaces under the same template and same user.

Whoever implements this should run the birthday problem calculations as there's only a few bits of entropy in the template ID and user ID.

Activity

  1. mafredri commented on Jul 20, 2022

    @mafredri
    Member

    Could an alternative option be to use the API to list the deleted resources (shouldn't be a problem since we usually soft-delete entries)? Especially if used in a script, then doing the API call makes sense to me.

    I'm just wondering how keeping part of the workspace ID static will help with discerning between deleted and non-deleted workspaces for a user? Since we're using so little entropy from workspace (and user) IDs, it's also not unthinkable that two different templates share the same prefix.

  2. ketang commented on Jul 20, 2022

    @ketang
    Contributor

    Could an alternative option be to use the API to list the deleted resources

    I think the perspective that I have here is that we should consider that sometimes our understanding of the state of the world is inaccurate, and something needs to be fixed using means external to Coder.

  3. mafredri commented on Jul 20, 2022

    @mafredri
    Member

    I think the perspective that I have here is that we should consider that sometimes our understanding of the state of the world is inaccurate, and something needs to be fixed using means external to Coder.

    Thanks, that makes sense. For that case, I'd think the reverse of my suggestion could be applied. Get all resources from provider and query Coder if it knows about them, if not => delete?

    That's not to say I don't see value in the suggested, though, and it'd work even if Coder is down. But I do worry a bit about the potential collisions.

    For example, I did the following quick test:

    var m = map[string]bool{}
    
    func TestUUID(t *testing.T) {
    	dupes := 0
    	for i := 0; i < 1000000; i++ {
    		u := uuid.New().String()[:8]
    		if _, ok := m[u]; ok {
    			dupes++
    		}
    		m[u] = true
    	}
    	t.Logf("%d duplicates", dupes)
    }

    And the result was:

    === RUN   TestUUID
        uuid_test.go:20: 130 duplicates
    

    From this, it doesn't seem too far-fetched that two templates could share the same prefix. Changing the parameters around a bit produce some interesting results, like 10,000 = 0 dupes, 100,000 = 2 dupes and 1,000,000 = 482 dupes. Then again, who's going to create >10,000 templates (or users)?

  4. ammario commented on Jul 20, 2022

    @ammario
    MemberAuthor

    Thanks, that makes sense. For that case, I'd think the reverse of my suggestion could be applied. Get all resources from provider and query Coder if it knows about them, if not => delete?

    A big design goal is to avoid coupling with any particular cloud, leaving all of that interaction to Terraform. Maybe long-term we can build out first class cloud integrations, but I think for the scope of this issue and short-term product development, we shouldn't consider such things.

    That's not to say I don't see value in the suggested, though, and it'd work even if Coder is down. But I do worry a bit about the potential collisions.

    To work this out I found this handy calculator. Use the log2 mode and then set $D$ to the number of random hex chars multiplied by 4. In your example that would be $D=32$. If we have 16 hex chars in the random portion, then even with 1 billion random IDs for the same user/template there would only be a 1% chance of a collision, which I think is acceptably low.

  5. ketang commented on Jul 20, 2022

    @ketang
    Contributor

    it doesn't seem too far-fetched that two templates could share the same prefix.

    It's true, but our goal is not to make it impossible to have duplication; it's to make it much easier in most cases to track down orphaned resources. If we don't catch all the orphans 0.013% of the time, that means we are doing so 99.987% of the time. I'm pretty satisfied with that, especially in a status quo where it's 0% of the time.

    A big design goal is to avoid coupling with any particular cloud, leaving all of that interaction to Terraform. Maybe long-term we can build out first class cloud integrations, but I think for the scope of this issue and short-term product development, we shouldn't consider such things.

    I believe those are good guidelines in general. I think the above is a more absolute statement than necessary, though. I believe we can support some valuable functionality with a limited set of clouds in a decoupled way. We should only do so where there's clear and significant return on investment, but I can think of a few cases where one might wish to have narrow and proprietary support for introspection into certain types of resource.

  6. mafredri commented on Jul 21, 2022

    @mafredri
    Member

    I was of the mind that a false-positive would be bad, if relied upon. But now I understand better the purpose and use-case, I agree that the risk seems acceptable here. Thanks for expanding @ammario and @ketang.

  7. bpmct commented on Jul 22, 2022

    @bpmct
    Member

    Having a practical workspace ID is a good suggestion. It doesn't need to address all cases.

    If an admin needs a more powerful option, GCP, AWS, and Azure all support arbitrary key/value pairs on resources for identification purposes. All the data doesn't have to be in the name.

    Tags could be added to the resource (via the template) such as organization, template version, workspace last started for easy identification within the cloud provider.

  8. ketang commented on Jul 31, 2022

    @ketang
    Contributor

    Tags could be added to the resource (via the template) such as organization, template version, workspace last started for easy identification within the cloud provider.

    We'll want to tread carefully here lest we clutter up the namespaces with noise.

  9. ammario commented on Aug 1, 2022

    @ammario
    MemberAuthor

    Tags could be added to the resource (via the template) such as organization, template version, workspace last started for easy identification within the cloud provider.

    We'll want to tread carefully here lest we clutter up the namespaces with noise.

    This is something the user does within their template as code, not Coder.

  10. deansheather commented on Aug 24, 2022

    @deansheather
    Collaborator

    Why not just encourage people to set metadata fields on objects in their terraform and add such fields to our example templates? We could also make it easier to lookup workspaces by ID, and since we soft-delete workspaces admins would be able to easily identify which template/user is responsible for the orphaned resources.

  11. ammario commented on Aug 25, 2022

    @ammario
    MemberAuthor

    Why not just encourage people to set metadata fields on objects in their terraform and add such fields to our example templates? We could also make it easier to lookup workspaces by ID, and since we soft-delete workspaces admins would be able to easily identify which template/user is responsible for the orphaned resources.

    Your suggestion is the simplest short-term solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions