Progress of download with java.net.http.HttpClient

Question

I use java.net.http.HttpClient to download a large file.

How can I make HttpClient report the progress of the download? For example, can it call a callback every X byte that it has finished downloading?

The download code looks like this:

HttpClient client = HttpClient.newBuilder()
    .followRedirects(Redirect.NORMAL)
    .connectTimeout(Duration.ofSeconds(20))
    .build();

HttpRequest request = HttpRequest.newBuilder().uri(
    new URI("https://example.com/large-file")).build();

// The program blocks here until the download is finished
HttpResponse<Path> response = client.send(request, BodyHandlers.ofFileDownload(
    Path.of("."), StandardOpenOption.WRITE, StandardOpenOption.CREATE));

The file that is downloaded is a couple of hundred MBs. On the user system this takes maybe a minute, during which users are left wondering whether the download is progressing or whether the program is hung.

To help with this I'd like to, while the download is happening, indicate to the user how much of the file that has been downloaded and that the download is proceeding. For example by printing a dot in the terminal when 1 MB of data is completed, or regularly updating a progress bar in a GUI.

How can I do this with HttpClient?

Lii · Accepted Answer · 2023-10-08 16:11:26Z

6

This is rather clunky but the best solution I could find. It uses a custom BodyHandler and BodySubscriber that calls a callback:

Example:

BodyHandler<Path> responceHandler = callbackBodyHandler(
    1_000_000,
    nrBytesReceived -> System.out.print("."),
    BodyHandlers.ofFileDownload(Path.of("."),
        StandardOpenOption.WRITE, StandardOpenOption.CREATE));

HttpResponse<Path> response = client.send(request, responceHandler);

Implementation:

private static <T> BodyHandler<T> callbackBodyHandler(int interval, LongConsumer callback, BodyHandler<T> h) {
    return info -> new BodySubscriber<T>() {
        private BodySubscriber<T> delegateSubscriber = h.apply(info);
        private long receivedBytes = 0;
        private long calledBytes = 0;

        @Override
        public void onSubscribe(Subscription subscription) {
            delegateSubscriber.onSubscribe(subscription);
        }

        @Override
        public void onNext(List<ByteBuffer> item) {
            receivedBytes += item.stream().mapToLong(ByteBuffer::capacity).sum();

            if (receivedBytes - calledBytes > interval) {
                callback.accept(receivedBytes);
                calledBytes = receivedBytes;
            }

            delegateSubscriber.onNext(item);
        }

        @Override
        public void onError(Throwable throwable) {
            delegateSubscriber.onError(throwable);

        }

        @Override
        public void onComplete() {
            delegateSubscriber.onComplete();
        }

        @Override
        public CompletionStage<T> getBody() {
            return delegateSubscriber.getBody();
        }
    };
}

edited Oct 8, 2023 at 16:11

answered Oct 7, 2023 at 14:27

Lii

12.2k9 gold badges69 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

David Newcomb Over a year ago

Thanks for coming back with you solution.

Lii Over a year ago

@DavidNewcomb: Thanks! For some reason people seemed to get upset and downvote the question, I don't understand why... It doesn't matter. :)

David Newcomb · Accepted Answer · 2024-10-18 20:23:34Z

2

You are right @Lil, your solution is a bit clunky but it was very useful for me to find a better solution, although better is subjective. Simpler certainly! So I'm posting mine here:

public void downloadBinaryFileProgressable(
    String address, String localFile, MyModel model)
        throws IOException, InterruptedException {

    URI url = URI.create(address);
    File lf = new File(localFile);
    FileOutputStream fos = new FileOutputStream(lf);

    HttpRequest req = HttpRequest.newBuilder()
        .uri(url).timeout(Duration.ofMillis(10_000)).GET().build();

    BodyHandler<InputStream> handler = HttpResponse.BodyHandlers.ofInputStream();
    HttpResponse<InputStream> resp = client.send(req, handler);
    
    HttpHeaders headers = resp.headers();
    int contentLength = retrieveContentLength(headers);
    
    model.setBytesToDownload(contentLength);
    System.out.println("content-length=" + contentLength);
    
    InputStream is = resp.body();
    
    byte[] buffer = new byte[4096];
    long bytesDownloaded = 0L;
    while(true) {
        int bytesRead = is.read(buffer);
        if (bytesRead == 0) {
            Thread.sleep(Duration.ofMillis(500));
            continue;
        }
        if (bytesRead == -1) {
            break;
        }
        bytesDownloaded += bytesRead;
        model.setBytesDownloaded(bytesDownloaded);
        System.out.println("downloaded=" + bytesDownloaded);
        
        fos.write(buffer, 0, bytesRead);
    }

    fos.close();
    is.close();
}

answered Oct 18, 2024 at 20:23

David Newcomb

11k3 gold badges52 silver badges63 bronze badges

2 Comments

Lii Over a year ago

Nice with an alternative solution. :)

Lii Over a year ago

Maybe it would be an improvement to use a try-with-resource for the InputStream?

Collectives™ on Stack Overflow

Progress of download with java.net.http.HttpClient

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related