Skip to content

[bug] getting "HTTP/1.1 501 Not Implemented" #10512

Open
Open
[bug] getting "HTTP/1.1 501 Not Implemented"#10512
Labels
bug

Description

@elyran
elyran
opened on May 13, 2024

Describe the bug

hi,
i like the idea of your proxy, but, getting HTTP/1.1 501 Not Implemented for curl -v --proxy http://localhost:8080 -k https://worldtimeapi.org/api/timezone/Asia/Jerusalem.
what am i missing? how can i make it work?

Logs

user1@user1-pc:~/Downloads$ curl -v --proxy http://localhost:8080 -k https://worldtimeapi.org/api/timezone/Asia/Jerusalem
*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to worldtimeapi.org:443
> CONNECT worldtimeapi.org:443 HTTP/1.1
> Host: worldtimeapi.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 501 Not Implemented
< 
* Received HTTP code 501 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 501 from proxy after CONNECT

Additional information

see Reproduction url for code

Reproduction

https://jsfiddle.net/g2fk1vph/

Activity

  1. elyran
    added
    bug
    on May 13, 2024

  2. callumgare commented on May 16, 2024

    @callumgare
    callumgare
    on May 16, 2024

    I believe it's not a bug but rather a feature that this library doesn't (yet?) support. There's different methods to proxy HTTP requests, it can be done using the HTTP method CONNECT (https://en.wikipedia.org/wiki/HTTP_tunnel) or it can be done by supplying the full url instead of just a path in the HTTP request (https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers). It seems this library supports the latter method but not the former.

    # Start the proxy server
node -e 'const {HttpCachingProxy} = require("@loopback/http-caching-proxy"); const proxy = new HttpCachingProxy({cachePath: "/tmp/.proxy-cache", port: 3000}); proxy.start().then(() => console.log(`Proxy started at ${proxy.url}`))'

# Send a request 
curl -v --request-target 'https://worldtimeapi.org/api/timezone/Asia/Jerusalem' -H "Host: worldtimeapi.org" http://localhost:3000

    Personally I'd love to see this library support the HTTP tunnel proxy method (that's what I was looking for when I came across this library) so maybe this ticket can be turned from a bug ticket into a feature request?

  3. callumgare commented on May 16, 2024

    @callumgare
    callumgare
    on May 16, 2024

    Oh, I've just realised that I think the HTTP tunnel method would be largely useless since for HTTPS requests the proxy would just be transparently forwarding encrypted HTTPS traffic and thus would not be able to cache any content.

  4. elyran commented on May 16, 2024

    @elyran
    elyran
    on May 16, 2024
    Author

    I also vote for a feature request. Thank you, appreciate you trying to help out

  5. callumgare commented on May 23, 2024

    @callumgare
    callumgare
    on May 23, 2024

    @elyran I came up with an ugly workaround which may or may not work in your situation. It's detailed in the snippets of code at the end of this post: apify/got-scraping#140 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions