Skip to content

Getbytes encoding fix#131

Merged
NakulSabharwal merged 8 commits intodevfrom
getbytes-encoding-fix
Sep 17, 2018
Merged

Getbytes encoding fix#131
NakulSabharwal merged 8 commits intodevfrom
getbytes-encoding-fix

Conversation

@NakulSabharwal
Copy link
Copy Markdown
Contributor

Fixes #105

Changes proposed in this pull request

  • Set getbytes in Multipart to US-ASCII.
  • Set getbytes in jsonResponse to UTF-8.
  • Added getter and setter for boundary and Content-Type of Multipart.
  • Removed function addPart(String name, String contentType, String content).
  • Renamed function addPart(String name, String contentType, byte[] byteArray) To addFormData(String name, String contentType, byte[] byteArray).
  • Added function addPart(Map<String, String> headers, byte[] content) for user provided headers and byte content.

Other links

-https://tools.ietf.org/html/rfc7578

…Added function addPart in class Multipart for user provided headers and byte content.
@NakulSabharwal NakulSabharwal changed the title getbytes encoding fix in Multipart, OneNoteTests and MockConnection. … Getbytes encoding fix Sep 10, 2018
}

private String createPartHeader(Map<String, String> headers, String name, String contentType, String filename) {
String partContent = addBoundary();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: make partContent a StringBuilder to avoid unnecessary copying

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using String.format(...) now for building String

out.write(returnContent.getBytes(MULTIPART_ENCODING));
}

private String createPartHeader(Map<String, String> headers, String name, String contentType, String filename) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this method static if doesn't use parent object state

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit test coverage of this method.

Copy link
Copy Markdown
Contributor Author

@NakulSabharwal NakulSabharwal Sep 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method cannot be made static because it uses boundary string which is dependent on parent object state. Also, added Unit Test for this method.

partContent += " " + entry.getKey() + "=\"" + entry.getValue() + "\";";
}
}
partContent = partContent.substring(0, partContent.length()-1);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You want to strip the final semicolon. Why's that?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Content-Disposition parameters are prefixed by semi-colon, so removing the trailing one is technically correct.

However, I'm a little puzzled as to this approach where we pretend Content-Disposition parameters are headers and then we lift them out of the headers collection to construct the Content-Disposition header value. The list of headers should just be a list of HTTP headers where the content-disposition value is pre-constructed. If we need a helper to build that value, then let's build a createContentDispositionValue() function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I have now changed the way in developer will use this addPart(Map<String, String> headers, byte[] content) and added Helper function String createContentHeaderValue(String contentValue, Map<String, String> contentDispParameter).
  • contentDispParameter map will contain (key=parameterName, Value=parameterValue). eg. (key="name", value="name-val").
  • createContentHeaderValue helper function will construct the string "contentValue;name="nameval";..." which will be then passed in headers map as (key=Content-Disposition, value=string from helper function), (key=Content-Type, value=string from helper).

*/
public void addHtmlPart(String name, String content) throws IOException {
addPart(name, "text/html", content);
addFormData(name, "text/html", content.getBytes(MULTIPART_ENCODING));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't make assumptions about the encoding of HTML content. If we keep this addHtmlPart method, it needs to accept a byte array and not a string.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed addHtmlPart to accept bytes.

}
else if(filename != null && name != null) {
partContent +=
"Content-Disposition:form-data; name=\"" + name + "\"" + "; filename=\"" + filename + "\"" + RETURN +
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There needs to be a space after the colon in the Content-Disposition header.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added space after the colon in the Content-Disposition header.

Nakul Sabharwal added 2 commits September 11, 2018 17:18
…d addPart for user defined multipart headers, values and parameters.
addPart(name, contentType, fileBytes);
String partContent = addBoundary();
partContent +=
"Content-Disposition:form-data; name=\"" + name + "\"" + "; filename=\"" + file.getName() + "\"" + RETURN +
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please create a reusable method and see if details can be encapsulated in that method.

* Test posting multipart content to a page
*/
@Test
public void testMultipartHeadersMapPost(){
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename to testMultipartPostWithHeadersMap

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed unit test function to testMultipartPostWithHeadersMap.

}

private String createPartHeader(String name, String contentType, String filename) {
String partContent = addBoundary();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you are reworking this use a StringBuilder.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used StringBuilder now for part header's value-parameter.


private String createPartHeader(String name, String contentType, String filename) {
String partContent = addBoundary();
if(filename != null && name != null) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though the if statements may be correct you get more clarity for reading (and stop double checking values) by nesting like so:

if (filename !=null) {
    if (name != null) {
        blah;
    } else {
        blah;
    }
} else if (name!=null) {
   blah;
} else {
   blah;
} 

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restructured the conditional statements.

private String createPartHeader(String name, String contentType, String filename) {
String partContent = addBoundary();
if(filename != null && name != null) {
String partContentWithNameAndFilename = "Content-Disposition: form-data; name=\"%s\"; filename=\"%s\"" + RETURN + "Content-Type:%s" + RETURN + RETURN;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you missing a space again after ContentType:? More occurrences below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected it with space.

* Test posting multipart content to a page
*/
@Test
public void testMultipartPostWithHeadersMap(){
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is useful but what I'd like to see is coverage of the createPartHeader method which has a few conditional branches. If you wrote those simple tests you would see the formatting problems yourself and they would not need review iterations. Because you didn't explicilty add a test for createPartHeader I'm going to run through what I would expect to see for testing of a method like it:

  • make createPartHeader static (including whatever state was used in its parameters)
  • give the method default visibility
  • add the annotation @VisibleForTesting or add comment // VisibleForTesting
  • write numerous small test methods that cover every branch and condition of createPartHeader only (15 minutes of work to knock up)
  • Keep the unit tests low level, only testing createPartHeader

@darrelmiller I know we have discussed briefly how much unit testing to do but this is an obvious easy win and easy to write. I'm happy for testing paths not to be coverered but it should be deliberate and argued as to why it is not being done.

On closer inspection the current test method performs no useful assertions so gives us no coverage at all of the changes made!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Wrote unit tests for createPartHeader.
  • createPartHeader cannot be made static because it uses addBoundary function which uses Boundary string that can be different for different multiparts. Suppose, developer wants to use Multipart at multiple places in application at those places the boundary value will be different so it cannot be made static, in turn then addBoundary and createPartHeader cannot be made static because of usage of non-static boundary member.

.buildRequest(options)
.post(multipart.content());
assertNotNull(page);
} catch (Exception e) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try catch unnecessary, just add throws Exception to the method. At the moment when fails we wouldn't even get a stack trace to help us.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it to throw Exception.

…cting header's value-parameter string and restructured conditions in createPartHeader.
deepak2016
deepak2016 previously approved these changes Sep 14, 2018
@deepak2016
Copy link
Copy Markdown
Contributor

I would like to complete this PR as soon as possible, so that we can refresh SDK on Monday. @davidmoten @darrelmiller kindly review.

.pages()
.buildRequest(options)
.post(multipart.content());
assertNotNull(page);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is only asserting that we aren't going to throw an Exception. The ultimate output of any of these requests and a call to post is that an HTTP POST is made to

  • a url
  • with specific request headers
  • and a chunk of json

The url building process should be tested elsewhere but your work has changed what we see in the second and third aspects. I haven't used the TestBase but it should enable you to assert things about the headers and content being posted. Can you add more assertions here?

Copy link
Copy Markdown
Contributor Author

@NakulSabharwal NakulSabharwal Sep 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Added more assertions now to check url and request header.
  • Content of request cannot be accessed once inside the request because post method calls and carries forward till request is sent on wire. So, I am checking byte content outside the request for being as notNull.


if(contentDispParameter != null) {
for(Map.Entry<String,String> entry : contentDispParameter.entrySet())
contentHeaderValue += ";" + entry.getKey() + "=\"" + entry.getValue() + "\"";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indented now. Although IDE was showing indented earlier.

deepak2016
deepak2016 previously approved these changes Sep 16, 2018
String expectedHeaderValue = "multipart/form-data; boundary=\""+multipart.getBoundary()+"\"";
assertEquals(expectedHeaderValue, headeroption.get(0).getValue().toString());
assertNotNull(multipart.content());
OnenotePage page = pageReq.buildRequest(options)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you have already taken out request object in line 487, why making pageReq.buildRequest(options) call again?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed it to use request.post(...)

* @param contentDispParameter Map containing content paramter's key and value pair
* @return content header value and parameter string
*/
public static String createContentHeaderValue(String contentValue, Map<String, String> contentDispParameter) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just drop this method. There isn't any commonality between Content-Type and Content-Disposition values. We can build helper methods for creating header values that are semi-colon delimited parameter lists at a later point that can be used in other places. This method just creates confusion and we are not using it except for in tests.

Copy link
Copy Markdown
Contributor Author

@NakulSabharwal NakulSabharwal Sep 17, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The content header's value and parameter string building is common in Content-Type and Content-Disposition. eg. Content-Disposition:form-data; name="someName"
Content-Type: text/html; otherparams=paramValue
Also, if some other headers is being provided apart from these then also same helper function can be used.

@NakulSabharwal
Copy link
Copy Markdown
Contributor Author

@davidmoten As all major comments stand resolved, we are going ahead and merging this PR, kindly open a github issue if you see any issue with the code and we will take that as a separate PR/issue.

@NakulSabharwal NakulSabharwal merged commit 918a316 into dev Sep 17, 2018
@deepak2016 deepak2016 deleted the getbytes-encoding-fix branch September 24, 2018 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

getBytes encoding should be explicit

4 participants