blob.download_to_filename fails with google-cloud-storage==1.3.0 #3736
Description
Due to the notable implementation changes in google-cloud-storage 1.3.0, the blob.download_to_filename fails whatever google-cloud from 0.24.0 to 0.27.0 (latest)
Our prod server which using google-cloud==0.24.0 has been broken since last Saturday. I analyzed a bit the error. The problem is come from the upgraded package google-cloud-storage==1.3.0. We are using the google-cloud-storage==1.2.0.
I did several tests which figured out the following problems:
Problem 1: blob.download_to_filename failed when using google-cloud==0.27.0 (included google-cloud-storage==1.3.0 as default, defined in setup.py). Here is a part of stacktrace:
blob.download_to_filename(blob_local_path)
File "/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 482, in download_to_filename
self.download_to_file(file_obj, client=client)
File "/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 464, in download_to_file
self._do_download(transport, file_obj, download_url, headers)
File "/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 418, in _do_download
download.consume(transport)
File "/root/.local/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 101, in consume
self._write_to_stream(result)
File "/root/.local/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 62, in _write_to_stream
with response:
AttributeError: __exit__Problem 2: blob.download_to_filename failed when using google-cloud==0.24.0 (included google-cloud-storage==1.3.0 as default, defined in setup.py). Here is a part of stacktrace:
File "dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 482, in download_to_filename
self.download_to_file(file_obj, client=client)
File "dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 464, in download_to_file
self._do_download(transport, file_obj, download_url, headers)
File "dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 418, in _do_download
download.consume(transport)
File "dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/requests/download.py", line 96, in consume
transport, method, url, **request_kwargs)
File "dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
func, RequestsMixin._get_status_code, retry_strategy)
File "dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
response = func()
File "dev_env/python_venv/local/lib/python2.7/site-packages/google_auth_httplib2.py", line 198, in request
uri, method, body=body, headers=request_headers, **kwargs)
TypeError: request() got an unexpected keyword argument 'data'So, why it happened? I realized that in the REQUIREMENTS of the package google-cloud which is defined in the file setup.py. An example for google-cloud==0.27.0 https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67
- google-cloud==0.24.0 requires google-cloud-storage >= 1.0.0 and < 2.0dev
- google-cloud==0.25.0 requires google-cloud-storage >= 1.1.0 and < 2.0dev => bd9fdfb#diff-2eeaed663bd0d25b7e608891384b7298R66
- google-cloud==0.25.1 requires google-cloud-storage >= 1.1.0 and < 1.2dev => ad4b02c#diff-2eeaed663bd0d25b7e608891384b7298R66
- google-cloud==0.26.0 requires google-cloud-storage >= 1.2.0 and < 2.0dev => 214aba6#diff-2eeaed663bd0d25b7e608891384b7298R67
- google-cloud==0.26.1 requires google-cloud-storage >= 1.2.0 and < 1.3dev => ea5245a#diff-2eeaed663bd0d25b7e608891384b7298R67
- google-cloud==0.27.0 requires google-cloud-storage >= 1.3.0 and < 1.4dev => https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67
We can see that if we only define google-cloud==[0.24.0 to 0.27.0], "pip install" always try to install google-cloud-storage 1.3.0 which requires the dependencies google-cloud-core ~= 0.26. That's why the bug happens.
Solution:
- Downgrade to google-cloud < 0.27.0 and force the requirements google-cloud-storage < 1.3.0
- The problem seems still there with google-cloud==0.27.0 and google-cloud-storage==1.3.0 (to check but at least it has been broken for my case, see Problem 2 above)
Suggestion: better manage the deps for google-cloud in setup.py.
Thanks.