1

A scenario, need to consume a rest webservice which provides a huge file as a stream output and vice versa need to handle the stream and directly write to a file rather memory. Service :

@RequestMapping(value = "downloadFile", method = RequestMethod.GET)
public StreamingResponseBody getSteamingFile(HttpServletResponse response) throws IOException {
    response.setContentType("application/octet-stream");
    InputStream inputStream = new FileInputStream(new File("data\\test_big.txt"));
    return outputStream -> {
        int nRead;
        byte[] data = new byte[1024];
        System.out.println("Writing some bytes..");
        while ((nRead = inputStream.read(data, 0, data.length)) != -1) {
            outputStream.write(data, 0, nRead);
        }
        System.out.println("Completed #####");  
        outputStream.flush();
        outputStream.close();
        response.flushBuffer();
    };

}

Consumer Route:

.to("http4://localhost:8080/downloadFile")
        .process(new Processor() {
            public void process(Exchange exchange) throws Exception {
                InputStream is = exchange.getIn().getBody(InputStream.class);
                File ret = File.createTempFile("loadTest", "tmp");
                FileOutputStream fos = new FileOutputStream(ret);
                StreamUtils.copy(is, fos);
                System.out.println("File Name "+ ret.getName());
                is.close();
                fos.flush();
                fos.close();
            }
        });

256 JVM is going out of Memory when processing 300 MB which treats my route is not performing streaming to file.

2 Answers 2

0

.to("http4://localhost:8080/downloadFile?disableStreamCache=true")

disableStreamCache: Determines whether or not the raw input stream from Servlet is cached or not (Camel will read the stream into a in memory/overflow to file, Stream caching) cache. By default Camel will cache the Servlet input stream to support reading it multiple times to ensure it Camel can retrieve all data from the stream. However you can set this option to true when you for example need to access the raw stream, such as streaming it directly to a file or other persistent store

Sign up to request clarification or add additional context in comments.

Comments

-1

You have to enable stream caching read here.

5 Comments

This would only help if you stream large payloads to disk. Which introduces a pretty hefty performance penalty. In principle it should be possible to process everything as streams and with that avoid having to load the payload into memory (or buffer on disk).
If you see consumer code, it is already dumping it to temp file.
I assume this is only to demonstrate the problem. The file has the base name loadTest after all?
Thanks for all your time in giving replies, finally i got solution by enabling a feature in http4. --- http4://localhost:8061?disableStreamCache=true
I Tried with 64 MB RAM and consumed web service and able to write 143 MB file using above disbaleStreamCache property which performed well and potentially there is a crest in CPU as its purley I/O.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.