Skip to content

Difficult to debug failures of agent init script in Docker provider #1544

Closed

Description

@johnstcn

Definition of Done

Clear log should be provided to surface errors so users do not have to go down several rabbit holes to try and understand what went wrong.

One acceptable outcome suggested in backend variety:

the Docker agent init script could log errors and then sleep forever if it hits a problem, instead of exiting. This would keep the container from exiting, and so the end user would see the build succeed, but the agent failed to connect. They could then docker logs ... to see what's up.

alternatively, we just fix kreuzwerker/docker to (optionally?) not delete containers after 15s

What

If the init script fails, there is very little way to debug the root cause (e.g. with Docker).
For example, if you set the Coder access URL to something invalid in the template, then all Docker containers will exit immediately e.g.

image

In the above example, the error "container exited immediately" is actually coming from kreuzwerker/docker itself -- see kreuzwerker/terraform-provider-docker#92

To actually dig out the root cause of the error, you have to e.g. turn on debug logging in the Docker daemon and check the daemon logs to actually see what's going wrong.

This isn't great usability, and we should have a better way of surfacing various kinds of errors from the agent init script.

There are quite a few possibilities for things to go wrong, including:

  • curl does not exist
  • failed to create temporary directory
  • failed to fetch agent binary
  • failed to chmod binary
  • coder agent failed
  • ...

Why

Make it easier for users to figure out why their container won't start.

Activity

  1. mafredri commented on May 18, 2022

    @mafredri
    Member

    I think as part of this, we could make the init script more robust. We can test for curl, wget and busybox wget, I think this should cover all of the most common scenarios.

    Additionally, mktemp is not currently busybox compatible, requires the addition of a single X (mktemp -d -t tmp.coderXXXXX => mktemp -d -t tmp.coderXXXXXX).

    We might add set -x to the script for better debug output (only helps with Docker logging enabled or a brute-force approach like: while :; do docker logs -f coder-developer-dev-root; sleep 0.1; done).

    I could submit a PR for these improvements.

  2. johnstcn commented on May 24, 2022

    @johnstcn
    MemberAuthor

    Good suggestion from @dwahler :
    #1198 (comment)

  3. tjcran commented on May 30, 2022

    @tjcran

    Good suggestion from @dwahler : #1198 (comment)

    @johnstcn is this one worth keeping open as #1198 is closed? or is this one still reproducible?

  4. johnstcn commented on May 31, 2022

    @johnstcn
    MemberAuthor

    @tjcran we should keep it open; having the wrong agent URL set will still cause this to happen.

  5. changed the title [-]Bug: Difficult to debug failures of agent init script in Docker provider[/-] [+]Difficult to debug failures of agent init script in Docker provider[/+] on Jun 7, 2022
  6. spikecurtis commented on Jun 17, 2022

    @spikecurtis
    Collaborator

    From discussion in Backend variety: the Docker agent init script could log errors and then sleep forever if it hits a problem, instead of exiting. This would keep the container from exiting, and so the end user would see the build succeed, but the agent failed to connect. They could then docker logs ... to see what's up.

  7. johnstcn commented on Jun 17, 2022

    @johnstcn
    MemberAuthor

    (Also from Backend variety): alternatively, we just fix kreuzwerker/docker to (optionally?) not delete containers after 15s

  8. added a commit that references this issue on Jul 13, 2022
  9. johnstcn commented on Jul 14, 2022

    @johnstcn
    MemberAuthor

    #2922 should address this.

    Please reopen if this is not the case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions