You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Simplify identification of orphaned resources with better workspace ID #3048
Currently, the workspace ID is generated from a UUID.
As an infrastructure admin, occasionally I have a workspace that fails to delete or the state is otherwise corrupted. In such cases, I may want to use a script to delete the orphaned resources. Since the workspace ID should be used in the persistent resource ID (see #3000 (comment)), we should include some identifying information in the workspace ID.
Here's an example of a possible format:
template ID fragment: the first 8 hex characters of the template ID
user ID fragment: the first 8 hex characters of the user ID
random: 10 random hexadecimal chars (40 bits of entropy)
This leads to an ID that looks like:
abababab-cdcdcdcd-efefefefefefefefefefefefef
Which would have a 50% chance of collision if there were 1 million workspaces under the same template and same user.
Whoever implements this should run the birthday problem calculations as there's only a few bits of entropy in the template ID and user ID.
Could an alternative option be to use the API to list the deleted resources (shouldn't be a problem since we usually soft-delete entries)? Especially if used in a script, then doing the API call makes sense to me.
I'm just wondering how keeping part of the workspace ID static will help with discerning between deleted and non-deleted workspaces for a user? Since we're using so little entropy from workspace (and user) IDs, it's also not unthinkable that two different templates share the same prefix.
Could an alternative option be to use the API to list the deleted resources
I think the perspective that I have here is that we should consider that sometimes our understanding of the state of the world is inaccurate, and something needs to be fixed using means external to Coder.
I think the perspective that I have here is that we should consider that sometimes our understanding of the state of the world is inaccurate, and something needs to be fixed using means external to Coder.
Thanks, that makes sense. For that case, I'd think the reverse of my suggestion could be applied. Get all resources from provider and query Coder if it knows about them, if not => delete?
That's not to say I don't see value in the suggested, though, and it'd work even if Coder is down. But I do worry a bit about the potential collisions.
From this, it doesn't seem too far-fetched that two templates could share the same prefix. Changing the parameters around a bit produce some interesting results, like 10,000 = 0 dupes, 100,000 = 2 dupes and 1,000,000 = 482 dupes. Then again, who's going to create >10,000 templates (or users)?
Thanks, that makes sense. For that case, I'd think the reverse of my suggestion could be applied. Get all resources from provider and query Coder if it knows about them, if not => delete?
A big design goal is to avoid coupling with any particular cloud, leaving all of that interaction to Terraform. Maybe long-term we can build out first class cloud integrations, but I think for the scope of this issue and short-term product development, we shouldn't consider such things.
That's not to say I don't see value in the suggested, though, and it'd work even if Coder is down. But I do worry a bit about the potential collisions.
To work this out I found this handy calculator. Use the log2 mode and then set $D$ to the number of random hex chars multiplied by 4. In your example that would be $D=32$. If we have 16 hex chars in the random portion, then even with 1 billion random IDs for the same user/template there would only be a 1% chance of a collision, which I think is acceptably low.
it doesn't seem too far-fetched that two templates could share the same prefix.
It's true, but our goal is not to make it impossible to have duplication; it's to make it much easier in most cases to track down orphaned resources. If we don't catch all the orphans 0.013% of the time, that means we are doing so 99.987% of the time. I'm pretty satisfied with that, especially in a status quo where it's 0% of the time.
A big design goal is to avoid coupling with any particular cloud, leaving all of that interaction to Terraform. Maybe long-term we can build out first class cloud integrations, but I think for the scope of this issue and short-term product development, we shouldn't consider such things.
I believe those are good guidelines in general. I think the above is a more absolute statement than necessary, though. I believe we can support some valuable functionality with a limited set of clouds in a decoupled way. We should only do so where there's clear and significant return on investment, but I can think of a few cases where one might wish to have narrow and proprietary support for introspection into certain types of resource.
I was of the mind that a false-positive would be bad, if relied upon. But now I understand better the purpose and use-case, I agree that the risk seems acceptable here. Thanks for expanding @ammario and @ketang.
Having a practical workspace ID is a good suggestion. It doesn't need to address all cases.
If an admin needs a more powerful option, GCP, AWS, and Azure all support arbitrary key/value pairs on resources for identification purposes. All the data doesn't have to be in the name.
Tags could be added to the resource (via the template) such as organization, template version, workspace last started for easy identification within the cloud provider.
Tags could be added to the resource (via the template) such as organization, template version, workspace last started for easy identification within the cloud provider.
We'll want to tread carefully here lest we clutter up the namespaces with noise.
Tags could be added to the resource (via the template) such as organization, template version, workspace last started for easy identification within the cloud provider.
We'll want to tread carefully here lest we clutter up the namespaces with noise.
This is something the user does within their template as code, not Coder.
Why not just encourage people to set metadata fields on objects in their terraform and add such fields to our example templates? We could also make it easier to lookup workspaces by ID, and since we soft-delete workspaces admins would be able to easily identify which template/user is responsible for the orphaned resources.
Why not just encourage people to set metadata fields on objects in their terraform and add such fields to our example templates? We could also make it easier to lookup workspaces by ID, and since we soft-delete workspaces admins would be able to easily identify which template/user is responsible for the orphaned resources.
Your suggestion is the simplest short-term solution.
Activity
mafredri commentedon Jul 20, 2022
Could an alternative option be to use the API to list the deleted resources (shouldn't be a problem since we usually soft-delete entries)? Especially if used in a script, then doing the API call makes sense to me.
I'm just wondering how keeping part of the workspace ID static will help with discerning between deleted and non-deleted workspaces for a user? Since we're using so little entropy from workspace (and user) IDs, it's also not unthinkable that two different templates share the same prefix.
ketang commentedon Jul 20, 2022
I think the perspective that I have here is that we should consider that sometimes our understanding of the state of the world is inaccurate, and something needs to be fixed using means external to Coder.
mafredri commentedon Jul 20, 2022
Thanks, that makes sense. For that case, I'd think the reverse of my suggestion could be applied. Get all resources from provider and query Coder if it knows about them, if not => delete?
That's not to say I don't see value in the suggested, though, and it'd work even if Coder is down. But I do worry a bit about the potential collisions.
For example, I did the following quick test:
And the result was:
From this, it doesn't seem too far-fetched that two templates could share the same prefix. Changing the parameters around a bit produce some interesting results, like
10,000 = 0 dupes,100,000 = 2 dupesand1,000,000 = 482 dupes. Then again, who's going to create >10,000 templates (or users)?ammario commentedon Jul 20, 2022
A big design goal is to avoid coupling with any particular cloud, leaving all of that interaction to Terraform. Maybe long-term we can build out first class cloud integrations, but I think for the scope of this issue and short-term product development, we shouldn't consider such things.
To work this out I found this handy calculator. Use the log2 mode and then set$D$ to the number of random hex chars multiplied by 4. In your example that would be $D=32$ . If we have 16 hex chars in the random portion, then even with 1 billion random IDs for the same user/template there would only be a 1% chance of a collision, which I think is acceptably low.
ketang commentedon Jul 20, 2022
It's true, but our goal is not to make it impossible to have duplication; it's to make it much easier in most cases to track down orphaned resources. If we don't catch all the orphans 0.013% of the time, that means we are doing so 99.987% of the time. I'm pretty satisfied with that, especially in a status quo where it's 0% of the time.
I believe those are good guidelines in general. I think the above is a more absolute statement than necessary, though. I believe we can support some valuable functionality with a limited set of clouds in a decoupled way. We should only do so where there's clear and significant return on investment, but I can think of a few cases where one might wish to have narrow and proprietary support for introspection into certain types of resource.
mafredri commentedon Jul 21, 2022
I was of the mind that a false-positive would be bad, if relied upon. But now I understand better the purpose and use-case, I agree that the risk seems acceptable here. Thanks for expanding @ammario and @ketang.
bpmct commentedon Jul 22, 2022
Having a practical workspace ID is a good suggestion. It doesn't need to address all cases.
If an admin needs a more powerful option, GCP, AWS, and Azure all support arbitrary key/value pairs on resources for identification purposes. All the data doesn't have to be in the name.
Tags could be added to the resource (via the template) such as
organization,template version,workspace last startedfor easy identification within the cloud provider.ketang commentedon Jul 31, 2022
We'll want to tread carefully here lest we clutter up the namespaces with noise.
ammario commentedon Aug 1, 2022
This is something the user does within their template as code, not Coder.
deansheather commentedon Aug 24, 2022
Why not just encourage people to set metadata fields on objects in their terraform and add such fields to our example templates? We could also make it easier to lookup workspaces by ID, and since we soft-delete workspaces admins would be able to easily identify which template/user is responsible for the orphaned resources.
ammario commentedon Aug 25, 2022
Your suggestion is the simplest short-term solution.