Fix Python 3.8 file operations #431

brianmay · 2020-05-14T21:58:27Z

Under Python 3.8 we can not wrap a File in a Sock.

Note this currently requires Python >= 3.5

brianmay · 2020-05-14T21:59:18Z

sshuttle/ssnet.py


    def __init__(self, mux, channel):
-        SockWrapper.__init__(self, mux.rsock, mux.wsock)
+        SockWrapper.__init__(self, mux.rfile, mux.wfile)


This is a bit weird that SockWrapper needs to be able to work with files and socks.

outime · 2020-05-20T12:49:14Z

@brianmay are you planning to merge this soon?

brianmay · 2020-05-20T21:02:08Z

@outime I am anticipating people will get upset when they realize I dropped support for Python 2.4. See #56.

But I somewhat think it is time to drop support for Python 2.x. Maybe apply the scream test. As in apply this change and see if anyone screams about it...

Anybody able to test this with Python 3.8? I only have Python 3.7 on my servers. I think it should work...

brianmay · 2020-05-20T21:12:01Z

This should fix #381

thesurlydev · 2020-05-21T00:03:14Z

@outime I am anticipating people will get upset when they realize I dropped support for Python 2.4. See #56.

But I somewhat think it is time to drop support for Python 2.x. Maybe apply the scream test. As in apply this change and see if anyone screams about it...

Anybody able to test this with Python 3.8? I only have Python 3.7 on my servers. I think it should work...

I just tried this with 3.8 and still getting the following:

Starting sshuttle proxy.
firewall manager: Starting firewall with Python version 3.8.2
firewall manager: ready method name nat.
IPv6 enabled: None
UDP enabled: False
DNS enabled: True
User enabled: False
Binding redirector: 12300
TCP redirector listening on ('127.0.0.1', 12300).
TCP redirector listening with <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 12300)>.
Binding DNS: 12300 12299
DNS listening on ('127.0.0.1', 12299).
DNS listening with <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('127.0.0.1', 12299)>.
Starting client with Python version 3.8.2
c : connecting to server...
c : executing: ['ssh', '-p', '22', 'Shane Witbeck@laptop', '--', '\'/Users/switbe/.pyenv/versions/3.7.5/bin/python\' -c \'import sys, os; verbosity=3; sys.stdin = os.fdopen(0, "rb"); exec(compile(sys.stdin.read(1275), "assembler.py", "exec"))\'']
c :  > channel=0 cmd=PING len=7 (fullness=0)
-c:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
server: assembling 'sshuttle' (7 bytes)
server: assembling 'sshuttle.cmdline_options' (66 bytes)
server: assembling 'sshuttle.helpers' (955 bytes)
server: assembling 'sshuttle.ssnet' (5716 bytes)
server: assembling 'sshuttle.hostwatch' (2467 bytes)
server: assembling 'sshuttle.server' (3786 bytes)
Starting server with Python version 3.7.5
 s: latency control setting = True
 s:  > channel=0 cmd=PING len=7 (fullness=0)
 s: auto-nets:False
 s:  > channel=0 cmd=ROUTES len=0 (fullness=7)
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=7/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 15/15
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=7/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 8/8
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=7/0)
c : Connected.
c : Waiting: 3 r=[5, 7, 9] w=[9] x=[] (fullness=7/0)
c :   Ready: 3 r=[9] w=[9] x=[]
c : <  channel=0 cmd=PING len=7
c :  > channel=0 cmd=PONG len=7 (fullness=7)
c : <  channel=0 cmd=ROUTES len=0
firewall manager: Got subnets: [(2, 0, False, '0.0.0.0', 0, 0)]
firewall manager: Got partial nslist: [(2, '127.0.0.53')]
firewall manager: Got nslist: [(2, '127.0.0.53')]
firewall manager: Got ports: 0,12300,0,12299
firewall manager: Got udp: False, user: None
firewall manager: setting up.
firewall manager: setting up IPv4.
>> iptables -t nat -N sshuttle-12300
>> iptables -t nat -F sshuttle-12300
>> iptables -t nat -I OUTPUT 1 -j sshuttle-12300
>> iptables -t nat -I PREROUTING 1 -j sshuttle-12300
>> iptables -t nat -A sshuttle-12300 -j RETURN -m addrtype --dst-type LOCAL ! -p udp
>> iptables -t nat -A sshuttle-12300 -j RETURN -m addrtype --dst-type LOCAL -p udp ! --dport 53
>> iptables -t nat -A sshuttle-12300 -j REDIRECT --dest 0.0.0.0/0 -p tcp --to-ports 12300 -m ttl ! --ttl 42
>> iptables -t nat -A sshuttle-12300 -j REDIRECT --dest 127.0.0.53/32 -p udp --dport 53 --to-ports 12299 -m ttl ! --ttl 42
c : mux wrote: 15/15
c : mux wrote: 15/15
client_loop: send disconnect: Broken pipe
c : Waiting: 3 r=[5, 7, 9] w=[] x=[] (fullness=14/0)
c :   Ready: 3 r=[9] w=[] x=[]
firewall manager: undoing changes.
firewall manager: undoing IPv4 changes.
>> iptables -t nat -D OUTPUT -j sshuttle-12300
>> iptables -t nat -D PREROUTING -j sshuttle-12300
>> iptables -t nat -F sshuttle-12300
>> iptables -t nat -X sshuttle-12300
firewall manager: undoing /etc/hosts changes.
c : fatal: server died with error code 255
shane@beefcake:~$ sshuttle --version
0.78.6.dev45+gf674aac.d20200520

brianmay · 2020-05-21T00:14:02Z

Shane Witbeck <notifications@github.com> writes:

I just tried this with 3.8 and still getting the following:

Not good. Can I please verify if this works for you? * With Python 3.7 with this change? * With Python 3.7 without this change? -- Brian May <brian@linuxpenguins.xyz> https://linuxpenguins.xyz/brian/

outime · 2020-05-21T00:17:47Z

Shane Witbeck notifications@github.com writes:
I just tried this with 3.8 and still getting the following:
Not good. Can I please verify if this works for you? * With Python 3.7 with this change? * With Python 3.7 without this change? -- Brian May brian@linuxpenguins.xyz https://linuxpenguins.xyz/brian/

Not Shane but earlier today I ran the pip release with the latest Python 3.7 and worked fine. Haven't tried with this change.

thesurlydev · 2020-05-21T15:37:00Z

Not good. Can I please verify if this works for you?

With Python 3.7 with this change?

With Python 3.7 without this change?

I get the same result (not working with and without change) using Python 3.7.7 client with 3.7.5 server:

Starting sshuttle proxy.
firewall manager: Starting firewall with Python version 3.7.7
firewall manager: ready method name nat.
IPv6 enabled: None
UDP enabled: False
DNS enabled: True
User enabled: False
Binding redirector: 12300
TCP redirector listening on ('127.0.0.1', 12300).
TCP redirector listening with <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 12300)>.
Binding DNS: 12300 12299
DNS listening on ('127.0.0.1', 12299).
DNS listening with <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_DGRAM, proto=0, laddr=('127.0.0.1', 12299)>.
Starting client with Python version 3.7.7
c : connecting to server...
c : executing: ['ssh', '-p', '22', 'Shane Witbeck@laptop', '--', '\'/Users/switbe/.pyenv/versions/3.7.5/bin/python\' -c \'import sys, os; verbosity=3; sys.stdin = os.fdopen(0, "rb"); exec(compile(sys.stdin.read(1275), "assembler.py", "exec"))\'']
c :  > channel=0 cmd=PING len=7 (fullness=0)
-c:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
server: assembling 'sshuttle' (7 bytes)
server: assembling 'sshuttle.cmdline_options' (66 bytes)
server: assembling 'sshuttle.helpers' (955 bytes)
server: assembling 'sshuttle.ssnet' (5712 bytes)
server: assembling 'sshuttle.hostwatch' (2468 bytes)
server: assembling 'sshuttle.server' (3786 bytes)
Starting server with Python version 3.7.5
 s: latency control setting = True
 s:  > channel=0 cmd=PING len=7 (fullness=0)
 s: auto-nets:False
 s:  > channel=0 cmd=ROUTES len=0 (fullness=7)
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=7/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 15/15
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=7/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 8/8
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=7/0)
c : Connected.
c : Waiting: 3 r=[5, 7, 9] w=[9] x=[] (fullness=7/0)
c :   Ready: 3 r=[9] w=[9] x=[]
c : <  channel=0 cmd=PING len=7
c :  > channel=0 cmd=PONG len=7 (fullness=7)
c : <  channel=0 cmd=ROUTES len=0
firewall manager: Got subnets: [(2, 0, False, '0.0.0.0', 0, 0)]
firewall manager: Got partial nslist: [(2, '127.0.0.53')]
firewall manager: Got nslist: [(2, '127.0.0.53')]
firewall manager: Got ports: 0,12300,0,12299
firewall manager: Got udp: False, user: None
firewall manager: setting up.
firewall manager: setting up IPv4.
>> iptables -t nat -N sshuttle-12300
>> iptables -t nat -F sshuttle-12300
>> iptables -t nat -I OUTPUT 1 -j sshuttle-12300
>> iptables -t nat -I PREROUTING 1 -j sshuttle-12300
>> iptables -t nat -A sshuttle-12300 -j RETURN -m addrtype --dst-type LOCAL ! -p udp
>> iptables -t nat -A sshuttle-12300 -j RETURN -m addrtype --dst-type LOCAL -p udp ! --dport 53
>> iptables -t nat -A sshuttle-12300 -j REDIRECT --dest 0.0.0.0/0 -p tcp --to-ports 12300 -m ttl ! --ttl 42
>> iptables -t nat -A sshuttle-12300 -j REDIRECT --dest 127.0.0.53/32 -p udp --dport 53 --to-ports 12299 -m ttl ! --ttl 42
c : mux wrote: 15/15
c : mux wrote: 15/15
c : Waiting: 3 r=[5, 7, 9] w=[] x=[] (fullness=14/0)
client_loop: send disconnect: Broken pipe
c :   Ready: 3 r=[9] w=[] x=[]
firewall manager: undoing changes.
firewall manager: undoing IPv4 changes.
>> iptables -t nat -D OUTPUT -j sshuttle-12300
>> iptables -t nat -D PREROUTING -j sshuttle-12300
>> iptables -t nat -F sshuttle-12300
>> iptables -t nat -X sshuttle-12300
firewall manager: undoing /etc/hosts changes.
c : fatal: server died with error code 255

thesurlydev · 2020-05-27T15:22:38Z

Please disregard my comment above. My issue seems not related to Python version and was able to get things working again. See: #434 (comment)

brianmay · 2020-05-27T23:45:44Z

OK, that makes more sense, thanks for the update.

Not that I really trust this PR. I tested it for about 10 seconds, it it appeared to work fine for me...

Under Python 3.8 we can not wrap a File in a Sock. Note this currently requires Python >= 3.5

* Drop 2.7 * Add 3.7 and 3.8

brianmay · 2020-05-28T21:42:26Z

This does seem to work for me. I am going to use the scream test as a reliable way of testing if (a) this works for everybody and (b) people are happy dropping Python 2.7 support. Merging...

andrewdunndev · 2020-06-04T13:08:44Z

Sorry to bump a merged PR, curious though, when will this hit in a release?

brianmay · 2020-06-04T22:35:35Z

Version 1.0.1 now released.

brianmay commented May 14, 2020

View reviewed changes

brianmay mentioned this pull request May 14, 2020

Fails to connect to endpoints with Python 3.8 #381

Closed

brianmay mentioned this pull request May 20, 2020

Unable to connect after upgrading client to Ubuntu 20.04 #434

Open

brianmay force-pushed the fix_python_38 branch from 90ae84c to d4f8987 Compare May 20, 2020 22:19

brianmay added 2 commits May 29, 2020 07:39

Fix Python 3.8 file operations

9c87357

Under Python 3.8 we can not wrap a File in a Sock. Note this currently requires Python >= 3.5

Updated supported Python versions

620b30d

* Drop 2.7 * Add 3.7 and 3.8

brianmay force-pushed the fix_python_38 branch from d4f8987 to 620b30d Compare May 28, 2020 21:42

brianmay merged commit 9f27c19 into master May 28, 2020

brianmay deleted the fix_python_38 branch May 28, 2020 21:44

This was referenced Jun 3, 2020

ref: replace usage of deprecated imp #449

Merged

chore: remove branching related to python < 3.5 #450

Merged

brianmay mentioned this pull request Jun 7, 2020

Version 0.78.6.dev55+gec2018d+ Traceback for tunnel-in-tunnel #454

Closed

slashdd mentioned this pull request Aug 11, 2020

PR#431 breaks connecting to Python <=3.5 hosts. #503

Closed

Fix Python 3.8 file operations #431

Fix Python 3.8 file operations #431

Uh oh!

Conversation

brianmay commented May 14, 2020

Uh oh!

brianmay May 14, 2020

Choose a reason for hiding this comment

Uh oh!

outime commented May 20, 2020

Uh oh!

brianmay commented May 20, 2020

Uh oh!

brianmay commented May 20, 2020

Uh oh!

thesurlydev commented May 21, 2020

Uh oh!

brianmay commented May 21, 2020 via email

Uh oh!

outime commented May 21, 2020

Uh oh!

thesurlydev commented May 21, 2020

Uh oh!

thesurlydev commented May 27, 2020

Uh oh!

brianmay commented May 27, 2020

Uh oh!

brianmay commented May 28, 2020

Uh oh!

andrewdunndev commented Jun 4, 2020

Uh oh!

brianmay commented Jun 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants