Unify proxy server handling #25618

In my workplace we need to use a proxy server to access the internet. Therefore I use set http_proxy = 'http://ourproxyserver:3128'; in my .duckdbrc to be able to use the ui and install extensions.

I recently started playing with a local s3 storage and realized that the proxy settings make things fail for me, because the local s3 storage needs to be accessed directly and cannot be contacted via our proxy server. At first I looked if there is a no_proxy setting in duckdb but there isn't so I temporarily removed the set http_proxy... statement so that DuckDB would have no proxy configuration at all. To my surprise, when I tried to read a file from my local s3 storage, I still got a HTTP 502 from our proxy server. I then set the HTTP_PROXY option in my s3 secret explicitly to ''and also tested with NULL as shown in below example, but in both cases my s3 calls from duckdb still went to our proxy server.

CREATE OR REPLACE SECRET (
    TYPE s3,
    KEY_ID '*****',
    SECRET '*****',
    ENDPOINT 's3storage.local:8080',
    URL_STYLE 'path',
    REGION 'us-east-1',
    HTTP_PROXY '',
    --HTTP_PROXY NULL, -- does not help either...
    USE_SSL false
);

Only then it came to mind that httpfs maybe does not respect Duckdb's http_proxy option at all and pulls its proxy settings from the os environment and that seems to be the case (I assume the HTTP_PROXY option in the secret can just be used to change the proxy server to something else but not to unset it). I added my s3 storage to my os's NO_PROXY environment var and then was able to successfully pull data from it (I should have done this before anyway, but only had added it to my webbrowser's proxy exception list by then. Oh and yes proxy configuration is quite a mess, in my opionion).

So what I take away from this little journey: It would be great, if duckdb and its (at least core) extensions would share the same proxy configuration (which means either add a no_proxy configuration option to duckdb and use that along with the http_proxy* options, or duckdb itself must respect the defacto-standard of defining os environment vars for proxy configuraiton (HTTP_PROXY, HTTPS_PROXY and NO_PROXY, as well as the vars for username and password).

If this is not possible or at least no priority I would suggest, adding a section to the docs, getting into detail how proxy configuration is pulled into httpfs (at least to me it was not obvious that it is configured completely independent from the duck). I will gladly provide such a docs section if you think that my proposed unification will not take place in the near future :)