Repository navigation
urllib/urllib2: HTTPS over (Squid) Proxy fails #42870
Description
| BPO | 1424152 |
|---|---|
| Nosy | @doko42, @facundobatista, @gpshead, @mjpieters, @jcea, @orsenthil, @matejcik, @benjaminp, @djc, @cameron-simpson, @bitdancer, @vadmium |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/orsenthil'
closed_at = <Date 2009-07-26.12:41:18.494>
created_at = <Date 2006-02-04.17:50:19.000>
labels = ['type-bug', 'library']
title = 'urllib/urllib2: HTTPS over (Squid) Proxy fails'
updated_at = <Date 2015-07-09.23:55:46.460>
user = 'https://bugs.python.org/kxroberto'bugs.python.org fields:
activity = <Date 2015-07-09.23:55:46.460>
actor = 'martin.panter'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2009-07-26.12:41:18.494>
closer = 'orsenthil'
components = ['Library (Lib)']
creation = <Date 2006-02-04.17:50:19.000>
creator = 'kxroberto'
dependencies = ['24599']
files = ['9753', '11332', '11333', '11334', '11335', '11454', '14339', '31201']
hgrepos = []
issue_num = 1424152
keywords = ['patch']
message_count = 54.0
messages = ['27434', '27435', '27436', '27437', '63317', '63318', '64022', '64223', '67083', '67146', '68235', '68825', '68861', '68863', '69000', '69211', '69412', '69520', '71137', '72267', '72268', '72954', '76855', '76856', '79573', '88113', '88114', '88117', '88118', '88119', '88120', '88121', '88257', '88275', '89611', '89617', '89621', '89689', '89694', '89779', '89806', '90164', '90598', '90603', '90622', '90626', '90633', '90909', '90926', '90938', '194704', '197530', '197554', '246533']
nosy_count = 31.0
nosy_names = ['doko', 'facundobatista', 'gregory.p.smith', 'jjlee', 'mjpieters', 'jcea', 'orsenthil', 'kxroberto', 'vila', 'matejcik', 'benjamin.peterson', 'djc', 'mwilck', 'chrisl', 'mihalis68', 'dpeterson', 'poeml', 'cameron', 'matb', 'nfl', 'AndrewTrick', 'ckassab', 'r.david.murray', 'hozn', 'statik', 'piranha', 'jamesbroadhead', 'cmcqueen1975', 'jarek.jpa', 'martin.panter', 'l']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1424152'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']
Activity
kxroberto commentedon Feb 4, 2006
py2.4.2/win32
The proxy mechanism of python fails on https and does
work completely wrong (not using the CONNECT scheme).
(after urlopen some minute(s) freeze then EOF error)
no CONNECT even appears in the squid proxy access log.
Robert
kxroberto commentedon Feb 11, 2006
Logged In: YES
user_id=972995
Meanwhile I wrote my own CONNECT quick hack. As indeed
this hack works correct for all proxied environments tested
up to now (>30) I wonder how open_https (in urllib and
urllib2) ever in the past managed to come through a proxy,
because there is some differentiation in open_https for the
case, that there is a proxy!? Who has written that
if..else's? Are there proxies which really do
SSL-handshaking directly and make an extra connection to the
target server? I guess that would even make certificate
handling very strange... I cannot immagine and never saw
one. But maybe such proxies exist. I am not a real expert
for such networking questions, but I guess CONNECT is widely
used and in my own proxies I can see in the log file, that
all common browsers use a HTTP CONNECT request for https
proxying. CONNECT should at least be implemented as an
option in urllibX
Robert
doko42 commentedon Aug 14, 2007
seen with urllib2 as well:
https://bugs.launchpad.net/ubuntu/+source/python2.5/+bug/122551
orsenthil commentedon Aug 14, 2007
Please verify if this recipe is of any help:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/456195
mwilck commentedon Mar 6, 2008
The recipe in its current form doesn't work with urllib2 in python 2.5
The reason it fails is that the HTTPConnection.request() method isn't
passed the request itself (with the proxy host and port info). No easy
solution to be seen.
mwilck commentedon Mar 6, 2008
I think this is a major issue because urllib2 is widely used, and any
application using it is useless behind a HTTPS proxy. A Prominent
example in the Linux world is yum, the Fedora Linux package management
tool. HTTPS proxies are a extermely common in the corporate world.
The cookbook recipe shows that it is possible to implement a CONNECT
proxy (with quite a few lines of code, actually), but because it's not
part of urllib2 it has been broken by python 2.5.
chrisl commentedon Mar 19, 2008
Hi,
I am working on a patch to implement the https proxy support for
urllib2. It works fine for me, but feel free to change the patch.
Can any one take a look please?
Thanks
chrisl commentedon Mar 20, 2008
In cast it is not obvious. I already attached the purpose patch
"http-tunnel-urllib" in the bug.
cameron-simpson commentedon May 20, 2008
Chris, I'm trying your patch out now. My quick reading of it looks ok.
cameron-simpson commentedon May 21, 2008
Well I've happily done lots of SOAP transations, admittedly all to the
same webservice, via HTTPS-over-proxy-CONNECT using ChrisL's patch. It
seems to work just fine. Whom do we petition to get this into the
mainline python sources?
matb commentedon Jun 15, 2008
Just supporting, that proxy support over https is widely used and
important. I am wondering, why this request is open for more than 2 years.
Mathias
mjpieters commentedon Jun 27, 2008
I've created a python 2.4 compatible distribution of the python 2.5
urllib2 and httplib modules with the patch attached to this issue. This
has proven especially useful in zc.buildout or other setuptools
environments:
http://pypi.python.org/pypi/httpsproxy_urllib2
nfl commentedon Jun 27, 2008
Hi,
looks like the attached patch only works if I use
urllib2.ProxyHandler({'https' : proxy}) at the start, but not when I use
request.set_proxy(proxy, 'https') per request. I tested with Python
2.5.1 on Win32.
chrisl commentedon Jun 27, 2008
Hi NL,
Can you please try this patch on top of the original patch?
Thanks
Chris
--- .pc/fix-up/urllib2.py 2008-02-06 01:13:10.000000000 -0800
+++ urllib2.py 2008-06-27 15:45:34.000000000 -0700
@@ -253,7 +253,7 @@ class Request:
return self.__r_host
On Fri, Jun 27, 2008 at 3:22 PM, nfl <report@bugs.python.org> wrote:
nfl commentedon Jun 30, 2008
It makes an infinite loop:
The test program was:
41 remaining items
bitdancer commentedon Jul 17, 2009
Note that this bug is also of concern to Ubuntu. See for example
https://bugs.launchpad.net/python/+bug/94130.
orsenthil commentedon Jul 17, 2009
This is actually fixed in Py2.7 and I am trying to get it Py3.2 before
marking this as fixed.
Based on the comments, should this be backported to Py2.6?
bitdancer commentedon Jul 17, 2009
Personally I would like to see that happen, but I think you should
probably poll python-dev since it can be argued either way as to whether
it is a bug fix or a new feature. FWIW, from the feeder bug reports it
looks to me like the users perceive it as a bug.
jamesbroadhead commentedon Jul 17, 2009
+1 to the above.
(imo, this is a bugfix, not a new feature)
hozn commentedon Jul 17, 2009
Personally, I also think that this is a bugfix, though I realize that in
order to fix the bug an API addition had to be made. So +1 from me on
rolling into 2.6 (though my opinion is simply that of someone who's
eager to see it fixed).
orsenthil commentedon Jul 25, 2009
Fixed for 3.2 in py3k branch revision 74198. Asked python-dev for
objections on back porting changes 2.6.
Shall back port changes to py2.6 and py3.1
gpshead commentedon Jul 25, 2009
*Rereads latest patch*
I'm happy with this being backported as a bugfix to 3.1 and 2.6 so long
as the set_tunnel method is hidden. Call it _set_tunnel() in the
backport (and don't document its existence).
orsenthil commentedon Jul 26, 2009
Agreed to the suggestions made by Greg.
The fix is now backported.
For Py2.6 in the revision 74203.
For Py3k in the revision 74204.
Closing this issue.
l commentedon Aug 8, 2013
Unfortunately this bug was only fixed in urllib2.py but never in urllib.py. This keeps biting people in the butt to this very day. Example: https://chromiumcodereview.appspot.com/10825107/
The attached patch remedies that and also fixes some more issues:
(1) proxy_bypass handling:
URLopener.open() will route the request to one of the open_() methods based on the type of the *request url. However, if a proxy is defined, it instead routes the request based on the type of the *proxy* url. So far so good. But: In open_http(), the code checks if proxy_bypass(realhost) is true and if so it modifies the Host header of the outgoing request. This code only works properly if the request url is by chance of type "http". If the request url type is e.g. "ftp" and the proxy url type is "http" and there's a proxy_bypass defined for realhost, things will go awry since the program will try to speak HTTP with realhost while it should really speak FTP. (In other words, open_ftp() should be used instead of open_http(), the program is stuck in the wrong codepath.) Also, proxy_bypass handling is currently only implemented for the proxy url type "http" (and not, for instance, "https"). The patch solves this by moving the proxy_bypass check to URLopener.open(): If a proxy_bypass is defined for realhost, the request is routed based on the request url type and not based on the proxy url type.
(2) addinfourl construction:
Upon successful retrieval of the URL, open_http() and open_https() will construct an addinfourl object and return that to the caller. The object is constructed with hard coded url type "http" / "https". So if for instance the request url type is "ftp" and the proxy url type is "http", the addinfourl object will contain a url whose type will have magically changed from "ftp" to "http".
(3) Superfluous code:
At the beginning of open_http() and open_https(), the program sets "user_passwd = None". Directly below is an if-else statement. At the beginning of the else block the program again sets "user_passwd = None".
The patch also works with Python 2.6 save for set_tunnel() in httplib.py, which was called _set_tunnel() in 2.6.
l commentedon Sep 12, 2013
*ping*
Anybody, please consider applying the patch I've submitted August 8th so that this issue gets fixed in Python 2.7's urllib.py.
Thanks so much.
orsenthil commentedon Sep 13, 2013
I have a slight fear that this patch could be considered as a feature addition in 2.7 urllib.py, I would like to quell that and ensure that behaviour expectation is consistent when using urllib or urllib2 and latest urllib/request.py modules.
Also, tests + docs can help a lot in speedier reviews.
vadmium commentedon Jul 9, 2015
In the meantime, bpo-24599 has been opened about URLopener(), which I understand is related to the Python 2 “urllib” half of this bug. Since this issue has been closed, perhaps it is best to continue discussion of bpo-1424152-py27-urllib.diff there instead.