You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Difficult to debug failures of agent init script in Docker provider #1544
Clear log should be provided to surface errors so users do not have to go down several rabbit holes to try and understand what went wrong.
One acceptable outcome suggested in backend variety:
the Docker agent init script could log errors and then sleep forever if it hits a problem, instead of exiting. This would keep the container from exiting, and so the end user would see the build succeed, but the agent failed to connect. They could then docker logs ... to see what's up.
alternatively, we just fix kreuzwerker/docker to (optionally?) not delete containers after 15s
What
If the init script fails, there is very little way to debug the root cause (e.g. with Docker).
For example, if you set the Coder access URL to something invalid in the template, then all Docker containers will exit immediately e.g.
In the above example, the error "container exited immediately" is actually coming from kreuzwerker/docker itself -- see kreuzwerker/terraform-provider-docker#92
To actually dig out the root cause of the error, you have to e.g. turn on debug logging in the Docker daemon and check the daemon logs to actually see what's going wrong.
This isn't great usability, and we should have a better way of surfacing various kinds of errors from the agent init script.
There are quite a few possibilities for things to go wrong, including:
curl does not exist
failed to create temporary directory
failed to fetch agent binary
failed to chmod binary
coder agent failed
...
Why
Make it easier for users to figure out why their container won't start.
I think as part of this, we could make the init script more robust. We can test for curl, wget and busybox wget, I think this should cover all of the most common scenarios.
Additionally, mktemp is not currently busybox compatible, requires the addition of a single X (mktemp -d -t tmp.coderXXXXX => mktemp -d -t tmp.coderXXXXXX).
We might add set -x to the script for better debug output (only helps with Docker logging enabled or a brute-force approach like: while :; do docker logs -f coder-developer-dev-root; sleep 0.1; done).
changed the title [-]Bug: Difficult to debug failures of agent init script in Docker provider[/-][+]Difficult to debug failures of agent init script in Docker provider[/+]on Jun 7, 2022
From discussion in Backend variety: the Docker agent init script could log errors and then sleep forever if it hits a problem, instead of exiting. This would keep the container from exiting, and so the end user would see the build succeed, but the agent failed to connect. They could then docker logs ... to see what's up.
Activity
mafredri commentedon May 18, 2022
I think as part of this, we could make the init script more robust. We can test for
curl,wgetandbusybox wget, I think this should cover all of the most common scenarios.Additionally,
mktempis not currentlybusyboxcompatible, requires the addition of a singleX(mktemp -d -t tmp.coderXXXXX=>mktemp -d -t tmp.coderXXXXXX).We might add
set -xto the script for better debug output (only helps with Docker logging enabled or a brute-force approach like:while :; do docker logs -f coder-developer-dev-root; sleep 0.1; done).I could submit a PR for these improvements.
fix: More robust provisionersdk agent init scripts
fix: More robust provisionersdk agent init scripts (#1551)
johnstcn commentedon May 24, 2022
Good suggestion from @dwahler :
#1198 (comment)
tjcran commentedon May 30, 2022
@johnstcn is this one worth keeping open as #1198 is closed? or is this one still reproducible?
johnstcn commentedon May 31, 2022
@tjcran we should keep it open; having the wrong agent URL set will still cause this to happen.
[-]Bug: Difficult to debug failures of agent init script in Docker provider[/-][+]Difficult to debug failures of agent init script in Docker provider[/+]spikecurtis commentedon Jun 17, 2022
From discussion in Backend variety: the Docker agent init script could log errors and then sleep forever if it hits a problem, instead of exiting. This would keep the container from exiting, and so the end user would see the build succeed, but the agent failed to connect. They could then
docker logs ...to see what's up.johnstcn commentedon Jun 17, 2022
(Also from Backend variety): alternatively, we just fix
kreuzwerker/dockerto (optionally?) not delete containers after 15sMerge branch 'main' into cj/gh-1544/agent-script-wait
johnstcn commentedon Jul 14, 2022
#2922 should address this.
Please reopen if this is not the case.