Skip to content

OkHttp client should not bail out when response body is >2gb - #410

Merged
codefromthecrypt merged 4 commits into
OpenFeign:masterfrom
carterkozak:bugfix/okhttp_handle_large_bodies
Jul 1, 2016
Merged

codefromthecrypt merged 4 commits into
OpenFeign:masterfrom
carterkozak:bugfix/okhttp_handle_large_bodies

Conversation

@carterkozak

@carterkozak carterkozak commented Jun 29, 2016 •

Copy link
Copy Markdown

Instead, we return null length and the response can be processed
similarly to a chunked body.

I'd like to eventually expose the body length as a long, but I'm not sure how we'd prefer to do that.
We could change the return type from Integer -> Long, but that would be api breaking.
Or alternatively add a second method to Body, Long lengthLong(); (or Long getLength();) and deprecate the integer value.

@spencergibb

Copy link
Copy Markdown
Contributor

I guess the question is why are you using feign for requests where the response body is > 2GB?

@carterkozak

Copy link
Copy Markdown
Author

I'm downloading files. This allows me to reuse my connection/retrial/errorDecoder code for the file download, which generally works really well!

@codefromthecrypt

codefromthecrypt commented Jun 30, 2016 •

Copy link
Copy Markdown

the javadoc on this field says the following:

    /**
     * length in bytes, if known. Null if not. <br> <br><br><b>Note</b><br> This is an integer as
     * most implementations cannot do bodies greater than 2GB. Moreover, the scope of this interface
     * doesn't include large bodies.
     */

Feign was made for control APIs, and most of the time they don't attempt to send large content without chunking or breaking it up into pieces. There's some assumptions that both the request and response can fit into a byte array, so just changing here might not be the end of this story. I don't really want to complicate the javadoc saying it might not be set because the body is larger than 2GB, as that makes the feature scope of feign muddy.

It might be ok to make a special repo for big content, with disclaimers that feign wasn't written to be a large content api. For example, a BigBody interface and a trusted implementation like OkHttp could like there. It would make sense to have resume if doing something so huge as restarting from byte 4.9GB would be awful.

@carterkozak

Copy link
Copy Markdown
Author

There's some assumptions that both the request and response can fit into a byte array

I think that is determined by the registered decoder, we use a custom FileDecoder that streams data and writes it out to disk.

I don't really want to complicate the javadoc saying it might not be set because the body is larger than 2GB, as that makes the feature scope of feign muddy.

Currently the ApacheHttpClient implementation will report a potentially negative value if it receives a response length greater than Integer.MAX_VALUE, I think that null length is much cleaner in that case, right?

I'm not especially worried about the body integer length anymore because I can parse the length header externally.
It's funky that different client implementations do different things when the response value is above Integer.MAX_VALUE. The okhttp wrapper will bail out, but others will simply cast it to an int, rather than returning null, which seems more correct in this situation.

I'll push a change to this branch to pull out the long->Integer conversion to a shared static function in Util, and add some testing around that.

@carterkozak
carterkozak force-pushed the bugfix/okhttp_handle_large_bodies branch from 6ab1b89 to 23594e6 Compare June 30, 2016 13:50
@codefromthecrypt

Copy link
Copy Markdown

I don't like toBodyLength.. it doesn't satisfy rule of three to be a util. let's reuse the lines we would have added to prove things work.

Personally, I'd rather throw an exception for all of the clients when they hit >2GB, like we did in OkHttp, but I think the below could be less draconian and help you progress without burdening others.

swap the lines in the toBodyLength commit for below, keeping the actual logic local (in some cases as they were).

  • add a MockWebServer test for no content length -> null
  • add a MockWebServer test for forged very long content-length header -> null
  • clarify the javadoc on Response.Body to say that null means unknown or too long content

sound reasonable?

For super brownie-points on a rainy day... extract a pluggable version of the http client tests :)

@carterkozak

Copy link
Copy Markdown
Author

add a MockWebServer test for forged very long content-length header -> null

Is this possible!? I've tried to do it in the past, but unfortunately the mock server was smart enough to return the actual length of the provided body.

I don't like toBodyLength.. it doesn't satisfy rule of three to be a util. let's reuse the lines we would have added to prove things work.

Makes sense, my rationale for the Utility was to define shared behavior in a shared place, that way if it gets modified in the future, we don't end up with different behavior between client wrapper implementations.
That said, if we document the interface and implementations follow the contract, it shouldn't matter either way :-)

Thanks for the feedback, I'll put together some tests!

Carter Kozak added 2 commits June 30, 2016 14:13
Instead, we return null length and the response can be processed
similarly to a chunked body.
Default, apache, and okhttp client tests extend the Abstract test,
so that we can avoid duplicated code while fully testing each
implementation.
@carterkozak
carterkozak force-pushed the bugfix/okhttp_handle_large_bodies branch from 23594e6 to 39c2293 Compare June 30, 2016 20:34
@carterkozak

Copy link
Copy Markdown
Author

Updated to share Client tests across implementations, no longer shares body length utility, but documents expected functionality in the interface.

I'm not sure how to generate a large response body, MockWebServer only allows a Buffer and String body type, both of which appear to require the full contents in memory.

@codefromthecrypt

codefromthecrypt commented Jul 1, 2016 via email

Copy link
Copy Markdown

server.enqueue(new MockResponse().setStatus("HTTP/1.1 " + 200));

TestInterface api =
Feign.builder().target(TestInterface.class, "http://localhost:" + server.getPort());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.getClient()

@codefromthecrypt

Copy link
Copy Markdown

nicely done. only a couple cleanups. this is notable enough to add to CHANGES I think

@codefromthecrypt
codefromthecrypt merged commit 6717142 into OpenFeign:master Jul 1, 2016
codefromthecrypt pushed a commit that referenced this pull request Jul 2, 2016
OkHttp client should not bail out when response body is >2gb

Instead, we return null length and the response can be processed
similarly to a chunked body.

Also, shares common client tests in AbstractClientTest
velo pushed a commit that referenced this pull request Oct 8, 2024
OkHttp client should not bail out when response body is >2gb

Instead, we return null length and the response can be processed
similarly to a chunked body.

Also, shares common client tests in AbstractClientTest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants