Skip to content

coder port-forward does not work as expected. Requests are hanging #3733

Closed

Description

@jaulz

Is there a manual way to port forward into a specific agent? Right now, I booted pgadmin in a separate docker container and I can access it the very first time. However, as soon as I hit a POST request it doesn't work anymore and it seems to be in an infinite loop. If there is no way, is it possible to turn on some logging to see what's happening? Thanks a lot!

Activity

  1. bpmct commented on Aug 29, 2022

    @bpmct
  2. jaulz commented on Aug 30, 2022

    @jaulz
    Author

    Thanks for your response! So I just compared these two approaches:

    coder port-forward xyz.pgadmin --tcp 1235:5433

    and

    ssh -L 1234:127.0.0.1:5433 coder.xyz.pgadmin

    It turns out that the second one is much more reliable. The first one usually works immediately after I entered the command and I can load the first page in the browser. Though, any following action ends in an infinite loop.

    These are the logs when I start the port forward:

    2022-08-30 06:27:02.764 [DEBUG]	<./peer/conn.go:486>	(*Conn).dialChannel	creating data channel	{"label": "ping", "opts": "\u0026{ID:1 Negotiated:true Protocol: Unordered:false OpenOnDisconnect:true}"}
    2022-08-30 06:27:02.766 [DEBUG]	<./peer/conn.go:486>	(*Conn).dialChannel	creating data channel	{"label": "echo", "opts": "\u0026{ID:2 Negotiated:true Protocol: Unordered:false OpenOnDisconnect:true}"}
    2022-08-30 06:27:02.766 [DEBUG]	<./peer/conn.go:285>	(*Conn).negotiate	negotiating
    2022-08-30 06:27:02.767 [DEBUG]	<./peer/conn.go:321>	(*Conn).negotiate	awaiting remote description...
    2022-08-30 06:27:02.767 [DEBUG]	<./peer/conn.go:327>	(*Conn).negotiate	setting remote description
    2022-08-30 06:27:02.768 [DEBUG]	<./peer/conn.go:237>	(*Conn).init.func4	signaling state updated	{"state": "have-remote-offer"}
    2022-08-30 06:27:02.770 [DEBUG]	<./peer/conn.go:183>	(*Conn).init.func2	ice gathering state updated	{"state": "gathering"}
    2022-08-30 06:27:02.770 [DEBUG]	<./peer/conn.go:237>	(*Conn).init.func4	signaling state updated	{"state": "stable"}
    2022-08-30 06:27:02.770 [DEBUG]	<./peer/conn.go:350>	(*Conn).negotiate	sending answer	{"answer": {"type": "answer", "sdp": "v=0\r\no=- 5193156003228223931 1861840822 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=fingerprint:sha-256 09:49:3F:92:16:FD:CA:3F:B3:4F:FE:68:B7:92:B2:E6:B7:0D:5D:CA:39:05:41:42:27:56:A7:3D:3E:66:76:FC\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:active\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:IoRKrUhtsmnMThWQ\r\na=ice-pwd:EpyqfKuwPZgLaRHMqDXgKDhtIJTIjFJl\r\n"}}
    2022-08-30 06:27:02.771 [DEBUG]	<./peer/conn.go:356>	(*Conn).negotiate	sent answer
    2022-08-30 06:27:02.771 [DEBUG]	<./peer/conn.go:164>	(*Conn).init.func1	ice connection state updated	{"state": "checking"}
    2022-08-30 06:27:02.772 [DEBUG]	<./peer/conn.go:375>	(*Conn).AddRemoteCandidate.func1	accepting candidate	{"candidate": "candidate:3967065881 1 udp 2130706431 192.168.1.31 59354 typ host"}
    2022-08-30 06:27:02.774 [DEBUG]	<./peer/conn.go:255>	(*Conn).init.func7.1	sending local candidate	{"candidate": "candidate:3002019262 1 udp 2130706431 172.25.0.3 50000 typ host"}
    2022-08-30 06:27:02.796 [DEBUG]	<./peer/conn.go:255>	(*Conn).init.func7.1	sending local candidate	{"candidate": "candidate:825276374 1 udp 1694498815 167.235.129.2 42612 typ srflx raddr 0.0.0.0 rport 42612"}
    2022-08-30 06:27:02.809 [DEBUG]	<./peer/conn.go:375>	(*Conn).AddRemoteCandidate.func1	accepting candidate	{"candidate": "candidate:643436868 1 udp 1694498815 31.159.54.8 64644 typ srflx raddr 0.0.0.0 rport 64644"}
    2022-08-30 06:27:02.815 [DEBUG]	<./peer/conn.go:255>	(*Conn).init.func7.1	sending local candidate	{"candidate": "candidate:2847320077 1 udp 16777215 127.0.0.1 54695 typ relay"}
    2022-08-30 06:27:02.906 [DEBUG]	<./peer/conn.go:241>	(*Conn).init.func5	dtls transport state updated	{"state": "connecting"}
    2022-08-30 06:27:02.906 [DEBUG]	<./peer/conn.go:245>	(*Conn).init.func6	selected candidate pair changed	{"local": {"foundation": "825276374", "priority": 1694498815, "address": "167.121.129.2", "protocol": 1, "port": 42612, "type": 2, "component": 1, "relatedAddress": "0.0.0.0", "relatedPort": 42612, "tcpType": ""}, "remote": {"foundation": "643436868", "priority": 1694498815, "address": "31.159.25.8", "protocol": 1, "port": 64644, "type": 2, "component": 1, "relatedAddress": "0.0.0.0", "relatedPort": 64644, "tcpType": ""}}
    2022-08-30 06:27:02.907 [DEBUG]	<./peer/conn.go:164>	(*Conn).init.func1	ice connection state updated	{"state": "connected"}
    2022-08-30 06:27:02.909 [DEBUG]	<./peer/conn.go:375>	(*Conn).AddRemoteCandidate.func1	accepting candidate	{"candidate": "candidate:2847320077 1 udp 16777215 127.0.0.1 56291 typ relay"}
    2022-08-30 06:27:02.975 [DEBUG]	<./peer/conn.go:241>	(*Conn).init.func5	dtls transport state updated	{"state": "connected"}
    2022-08-30 06:27:02.976 [DEBUG]	<./peer/conn.go:201>	(*Conn).init.func3.1	rtc connection updated	{"state": "connected"}
    2022-08-30 06:27:02.997 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 1, "label": "ping"}
    2022-08-30 06:27:02.997 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 2, "label": "echo"}
    

    And then once I trigger the first HTTP request via this connection:

    2022-08-30 06:27:07.775 [DEBUG]	<./peer/conn.go:183>	(*Conn).init.func2	ice gathering state updated	{"state": "complete"}
    2022-08-30 06:27:33.892 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 5, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:33.893 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 3, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.219 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 7, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.220 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 9, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.220 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 11, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.221 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 13, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.254 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 15, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.312 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 11, "label": "tcp://127.0.0.1:5433"} ...
      "error": connection was closed:
                   github.com/coder/coder/peer.init
                       /home/runner/work/coder/coder/peer/conn.go:28
                 - EOF
    2022-08-30 06:27:34.314 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 3, "label": "tcp://127.0.0.1:5433"} ...
      "error": connection was closed:
                   github.com/coder/coder/peer.init
                       /home/runner/work/coder/coder/peer/conn.go:28
                 - EOF
    2022-08-30 06:27:34.330 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 7, "label": "tcp://127.0.0.1:5433"} ...
      "error": connection was closed:
                   github.com/coder/coder/peer.init
                       /home/runner/work/coder/coder/peer/conn.go:28
                 - EOF
    2022-08-30 06:27:34.350 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 13, "label": "tcp://127.0.0.1:5433"} ...
      "error": connection was closed:
                   github.com/coder/coder/peer.init
                       /home/runner/work/coder/coder/peer/conn.go:28
                 - EOF
    2022-08-30 06:27:34.410 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 9, "label": "tcp://127.0.0.1:5433"} ...
      "error": connection was closed:
                   github.com/coder/coder/peer.init
                       /home/runner/work/coder/coder/peer/conn.go:28
                 - EOF
    2022-08-30 06:27:34.481 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 17, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.483 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 19, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.484 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 21, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.485 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 23, "label": "tcp://127.0.0.1:5433"}
    2022-08-30 06:27:34.488 [DEBUG]	<./peer/channel.go:128>	(*Channel).init.func3	datachannel opening	{"id": 25, "label": "tcp://127.0.0.1:5433"}
    

    And eventually when I already noticed that the network requests are pending forever I see that the channels are closed with an error (though error is null?):

    2022-08-30 06:29:06.623 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 21, "label": "tcp://127.0.0.1:5433", "error": null}
    2022-08-30 06:29:36.707 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 15, "label": "tcp://127.0.0.1:5433", "error": null}
    2022-08-30 06:30:05.897 [DEBUG]	<./peer/channel.go:277>	(*Channel).closeWithError	datachannel closing with error	{"id": 17, "label": "tcp://127.0.0.1:5433", "error": null}
    
  3. sharkymark commented on Aug 30, 2022

    @sharkymark
    Contributor

    @bpmct @kylecarbs Is this related? #2658 Also I continue to experience this on occasion, so I'd appreciate if you all would re-open it until resolved.

  4. bpmct commented on Sep 23, 2022

    @bpmct
    Member

    Looks like the same issue as #2658. I was able to reproduce this.

    Screen.Recording.2022-09-23.at.3.24.36.PM.mov

    The ssh -L works fine for me, though. Doesn't prompt for a password.

  5. changed the title [-]Manual way to port forward or debug the same?[/-] [+]`coder port-forward` does not work as expected. Requests are hanging[/+] on Sep 23, 2022
  6. added this to the EE milestone on Sep 23, 2022
  7. coadler commented on Sep 23, 2022

    @coadler
    Contributor

    I was playing with this and noticed I was able to get the same behavior reloading the React page over and over as well. The same doesn't seem to happen with python3 -m http.server.

  8. added
    s0Major regression, all-hands-on-deck to fix
    on Oct 6, 2022
  9. Akantor47 commented on Oct 17, 2022

    @Akantor47

    Hello there,

    coder port-forward does stop working for me after a few request.
    and ssh -L doesn't even start probably due to some Windows SSH Problems.

    But once I setup a port forwarding via ssh connected visual studio code, it works like a charm...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

s0Major regression, all-hands-on-deck to fix

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions