Skip to content

Improve error message when SSH connection drops #1369

Closed

Description

@ericpaulsen

Problem Summary

On a spotty network that resulted in a dropped connection between client and v2-provisioned EC2 instance, the error returned was: wait: remote command exited without exit status or exit signal

this message is a bit confusing and does not clearly explain that the connection was dropped.

Definition of Done

Check for this error and translate this to "Lost connection to agent."
Test for lost connection produces this error

Activity

  1. tjcran commented on May 16, 2022

    @tjcran

    @kylecarbs what is returning this error message? Is this being returned from coderd to the CLI?

  2. added this to the Community MVP milestone on May 16, 2022
  3. tjcran commented on May 16, 2022

    @tjcran

    This is confusing and makes it harder to troubleshoot, but is not a breaking experience. It's a nice to have for the community launch right now.

  4. 2 remaining items

  5. ketang commented on Jun 7, 2022

    @ketang
    Contributor
  6. dwahler commented on Jul 22, 2022

    @dwahler
    Contributor

    I did some investigation into how the golang.org/x/crypto/ssh package handles unexpectedly dropped connections:

    • The library's example code (using a direct TCP connection) returns the same remote command exited without exit status or exit signal message, which as @ketang pointed out is the string representation of an ssh.ExitMissingError. It's possible to retrieve the underlying I/O error (e.g. a timeout) by calling client.Close(); client.Wait().
    • With our WebRTC backend, client.Wait() returns an io.EOF error regardless of whether the connection was closed cleanly or uncleanly. I haven't figured out yet what's causing this difference in behavior.
    • With our Wireguard backend, I couldn't reproduce the issue. Even disabling the physical network interface doesn't seem to drop the SSH connection -- it just hangs and comes back when the connection is available again.

    So in the case of WebRTC, I don't see a way to do much better than just replacing the error message with a better one (#3131).

    I think the Wireguard behavior of keeping the connection alive is better overall. But it might still be worth trying to give some indication (maybe a desktop notification?) while it's interrupted, especially if we can determine that the problem is with the user's internet connection rather than the remote end.

  7. ketang commented on Jul 31, 2022

    @ketang
    Contributor

    I suspect the desktop notification could easily become a pest in certain conditions. I think we should kick that one down the road until we get into the system tray business.

  8. bpmct commented on Aug 1, 2022

    @bpmct
    Member

    Once #2779 is merged, will we likely see a different behavior/error pattern?

  9. bpmct commented on Aug 8, 2022

    @bpmct
    Member

    @dwahler is this fixed by #3131?

  10. dwahler commented on Aug 9, 2022

    @dwahler
    Contributor

    Depends on your interpretation. If the goal of this ticket is to improve that one specific error message, then yes. If the goal is to reliably give a useful message when an SSH connection drops, then it's not really done, because #3131 only affects the behavior with WebRTC, so it'll become moot when #2779 lands and we switch to Tailscale for client connections.

    I figured it made sense to keep this ticket open in order to not split the discussion, but I don't mind closing it if others feel differently.

  11. tjcran commented on Aug 10, 2022

    @tjcran

    Depends on your interpretation. If the goal of this ticket is to improve that one specific error message, then yes. If the goal is to reliably give a useful message when an SSH connection drops, then it's not really done, because #3131 only affects the behavior with WebRTC, so it'll become moot when #2779 lands and we switch to Tailscale for client connections.

    I figured it made sense to keep this ticket open in order to not split the discussion, but I don't mind closing it if others feel differently.

    I would suggest that we close this ticket as the original requirements of this ticket are met by improving that specific error message.

    But we should also have an epic or something to track the general improvements to SSH drop messages.

  12. removed this from the Community MVP milestone on Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

apiArea: HTTP APIcliArea: CLI

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions