What is illegal about using curl with this URL?

Question

I have the below URL:

https://amazon-advertising-api-reports-prod-euamazon.s3.amazonaws.com/apj6jra9ng5v4-avp9363n2w589-report-campaign-45f7c968-d7e8-4d8e-a6ef-2b144268ab36.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210930T164837Z&X-Amz-SignedHeaders=host&X-Amz-Expires=30&X-Amz-Credential=AKIXXXXXXXXXZ6X%2F20210930%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=2a549XXXXXXXXXX0e61f43166c267eaafedf8

It's an original response from Amazon's advertising API with just a few elements replaced by 'X' where there are normal characters you wolud expect in a signature or Credential.

If I copy and paste this url into my browser I can download the file as I would expect. When trying to retrieve this file using any variation of the following curl commands ...

curl -o $REPORT_DATA_FILE_GZ "$REPORT_DOWNLOAD_URL\"

or 

curl -o $REPORT_DATA_FILE_GZ "\"$REPORT_DOWNLOAD_URL\""

or 

curl -o $REPORT_DATA_FILE_GZ $REPORT_DOWNLOAD_URL

... I am getting the error URL using bad/illegal format or missing URL. But I'm certain I am passing the REPORT_DOWNLOAD_URL and REPORT_DATA_FILE_GZ variables with correct values.

I display them with echo just before the call to curl to ensure they are valid.

I also cannot see any characters that look invalid to me in the URL.

All requests to curl are made before my test calls in the browser, so the short expiry period provided by the amazon-advertising-api (30 seconds) would not have expired by that time.

As per the comment I'm running this under Ubuntu 20.04, on WSL2, but most other curl request are working fine.

echo may not show you everything that's in a variable, espeically on a terminal i.e. interactively. Try printf '%q\n' "$REPORT_DOWNLOAD_URL". — dave_thompson_085
– dave_thompson_085, Commented Sep 30, 2021 at 20:57
Thanks @dave_thompson_085 there was a rogue \r at the end variable. That's solved it. — Hans
– Hans, Commented Oct 1, 2021 at 10:51

Hans · Accepted Answer · 2021-10-05 14:22:37Z

0

I finally found the solution based on the output suggested by @dave_thompson_085. I extracted the $REPORT_DOWNLOAD_URL from the header of a different response and it still contained the \r at the end.

answered Oct 5, 2021 at 14:22

Hans

2,9304 gold badges33 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

What is illegal about using curl with this URL?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related