Repository navigation
Exception: OverflowError: string longer than 2147483647 bytes #646
Description
Hi,
i tried to import a large repository (export from a different gitlab instance) which is 2,8GB in size (large LFS objects in there)
importing fails:
i_gl.projects.import_project( open( export_file, 'rb' ), e_project.name, namespace = e_project.namespace[ 'full_path' ], overwrite = True )
File "/usr/local/lib/python2.7/dist-packages/gitlab/v4/objects.py", line 3698, in import_project
files=files, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gitlab/__init__.py", line 600, in http_post
post_data=post_data, files=files, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gitlab/__init__.py", line 481, in http_request
result = self.session.send(prepped, timeout=timeout, **settings)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 423, in send
timeout=timeout
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 595, in urlopen
chunked=chunked)
File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 363, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python2.7/httplib.py", line 1039, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 879, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 855, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/ssl.py", line 701, in sendall
v = self.send(data[count:])
File "/usr/lib/python2.7/ssl.py", line 667, in send
v = self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
any idea how to solve that?
Activity
Flow86 commentedon Nov 27, 2018
I found this: http://docs.python-requests.org/en/master/user/advanced/#streaming-uploads perhaps that helps to fix the problem
gpocentek commentedon Nov 28, 2018
Hi @Flow86
Streaming uploads is probably the best solution indeed. Thanks for reporting the issue.
devrite commentedon Nov 8, 2019
Any progress on it we ran into the same problem. Providing streamed=True via arg add to import_project does not solve the issue.
My best guess is that you probably need to pass stream=streamed when creating the http request?
Any way to work around this issue?
max-wittig commentedon Nov 8, 2019
Haven't looked into it. Not sure. But I would be happy to help and review your MR about this.
zapp42 commentedon Jan 28, 2020
Unfortunately the request multi-part form upload feature does not support streaming. One has to use the MultipartEncoder from the request-toolbelt module for this.
See https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-multipart-data-encoder.
zapp42 commentedon Jan 29, 2020
Check pull request #1004 for a possible solution and please leave some feedback here if you try it.
rcarmonad commentedon Apr 18, 2020
I couldn't import, because the repo was to big.
The workaround i used was to clone a bare repo and then push to the new location.