1

I am facing out of memory error on using the Apache camel Http component to download/stream a file from a public link to an external system (SharePoint). It seems the component internally is reading all of the files into memory at once. Can we do this without loading whole stream to memory? The route is quite simple.

  /**
     * Submit file to upload
     */
    from("direct:fetch-content-data-and-upload")

            .log("** Fetching file content from public link")
            .setHeader(Exchange.HTTP_METHOD, constant("GET"))
            .toD("${header.CamelHttpUri}")

            .process(this.submitFileToUpload)

            .end();

The processor internally uses Graph Client SDK to submit Input Stream to upload in chunks. Error is thrown before it can reach the processor.

Aug  3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onIdleExpired(HttpConnectionOverHTTP.java:160)
Aug  3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.ssl.SslConnection.onIdleExpired(SslConnection.java:391)
Aug  3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:402)
Aug  3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
Aug  3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
Aug  3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
Aug  3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Aug  3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
Aug  3 07:38:37 ip-172-31-9-176 web: ... 3 common frames omitted
Aug  3 07:38:37 ip-172-31-9-176 web: Caused by: java.lang.OutOfMemoryError: Java heap space
Aug  3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.Arrays.copyOf(Arrays.java:3537) ~[na:na]
Aug  3 07:38:37 ip-172-31-9-176 web: at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
Aug  3 07:38:37 ip-172-31-9-176 web: at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:101) ~[camel-support-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:206) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:161) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:156) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:152) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.doExtractResponseBodyAsStream(HttpProducer.java:547) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.extractResponseBody(HttpProducer.java:532) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.populateResponse(HttpProducer.java:332) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:291) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66) ~[camel-support-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:197) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.SendDynamicProcessor$$Lambda$1001/0x000000d8016ec350.doInAsyncProducer(Unknown Source) ~[na:na]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:327) ~[camel-support-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:182) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:477) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:181) ~[camel-base-engine-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.impl.engine.DefaultReactiveExecutor.schedule(DefaultReactiveExecutor.java:54) ~[camel-base-engine-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.lambda$redeliver$2(RedeliveryErrorHandler.java:914) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug  3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask$$Lambda$994/0x000000d8016c1360.done(Unknown Source) ~[na:na] ```
Improve this question
4
  • I think that steam is transform/load total to create body to call "process" method, if you want to process as stream maybe you have to find another component. Commented Aug 4, 2023 at 12:09
  • Is not all the routes/components eventually land in a sort of processor? So not really sure why it would collect all stream data first. I am not even logging any information here. Commented Aug 4, 2023 at 16:05
  • When this.submitFileToUpload is call it create an exchange and a body with the steam maybe use a "to". Commented Aug 7, 2023 at 7:57
  • There seems to be no way of streaming the output of the HTTP component. I tried adding additional "to" route as well. HTTP component always seem to load the whole payload in memory. I had to use a standard Java URL to open the stream directly inside the processor. This does take care of memory issues. Commented Aug 7, 2023 at 20:51
Related questions
1654
What exactly is Apache Camel?
0
Produce messages to IBM MQ using REST API. Apache Camel
1
Apache Camel: How to dynamically set header value in HTTP URI?
Load 3 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.