where is the documentation ? i have no idea what it does. also the code is full of whitespace issues (mixed tabs and spaces)

but before you spend effort to update manpages and config template (and clean up code), please first describe here detailed with examples what this is supposed to do, what the usecase is, etc.
so in case the PR isn't considered for merge, you don't waste time polishing it.

where is the documentation ? i have no idea what it does.

Apologies, there is not much documentation. It improves the upstream matching algorithm i.e. the otherwise not very useful IP match is now also done on the reverse DNS resolution of the hostname.

It is very unusual to connect to a server by IP and then have tinyproxy match the IP to identify the upstream proxy. This patch will perform a reverse DNS on a hostname and then matches it against to IP/bits or IP/mask site spec of an upstream rule.

Example rule extract:

upstream http proxy1:8888
upstream none "10.0.0.0/255.0.0.0"
upstream none "192.168.0.0/16"

A host accessed with https://myinternalapp.co.uk on IP 10.1.1.1 via tinyproxy would right now go to proxy1 with my patch it would go direct.

also the code is full of whitespace issues (mixed tabs and spaces)

Do you have an astyle ruleset ?

but before you spend effort to update manpages and config template (and clean up code), please first describe here detailed with examples what this is supposed to do, what the usecase is, etc.
so in case the PR isn't considered for merge, you don't waste time polishing it.

Thanks
Markus

A host accessed with https://myinternalapp.co.uk on IP 10.1.1.1 via tinyproxy would right now go to proxy1 with my patch it would go direct.

it would also go "direct" if your rules looked like (i.e., order fixed)

upstream none "10.0.0.0/255.0.0.0"
upstream none "192.168.0.0/16"
upstream http proxy1:8888

assuming that myinternap.whatever has an entry in /etc/hosts pointing to 10.1.1.1

A host accessed with https://myinternalapp.co.uk on IP 10.1.1.1 via tinyproxy would right now go to proxy1 with my patch it would go direct.

it would also go "direct" if your rules looked like (i.e., order fixed)

upstream none "10.0.0.0/255.0.0.0"
upstream none "192.168.0.0/16"
upstream http proxy1:8888

assuming that myinternap.whatever has an entry in /etc/hosts pointing to 10.1.1.1

I did not see anywhere in the code the use of /etc/hosts for matching ( and why would I use hosts instead of DNS ? ). I see hostspec_match in the upstream code which only matches string to string or IP to IP/bits or IP/mask.

So the above would not work.

Markus

I did not see anywhere in the code the use of /etc/hosts

libc getaddrinfo looks in /etc/hosts before doing costly DNS queries

I did not see anywhere in the code the use of /etc/hosts

libc getaddrinfo looks in /etc/hosts before doing costly DNS queries

Yes, but in your code there was no getaddrinfo call to allow a IP based match for upstream proxies when a name string was given.

Regarding costly DNS any further connection to the end host would do the same getaddrinfo lookup i.e. running nscd would be anyway recommended.

BTW don't you do the same getaddrinfo call for your ACL checks ?

Markus

I see hostspec_match in the upstream code which only matches string to string or IP to IP/bits or IP/mask.

i'm not talking about hostspec code but code used for making an actual connection

see also f6d4da5

I see hostspec_match in the upstream code which only matches string to string or IP to IP/bits or IP/mask.

i'm not talking about hostspec code but code used for making an actual connection

see also f6d4da5

So you would prefer to do it once and cache the result in the structure for any follow on use ? Assuming the IP does not change frequent to another IP ?

I would still rely on nscd to perform the caching.

If it is preferred I can set it as disabled and the user can decide to enable it with the knowledge of a possible DNS/nscd dependency. For my use cases it is a required option

Markus

you keep mentioning "reverse DNS", but usually reverse DNS means looking up X.X.X.X.in-addr.arpa where X.X.X.X is the reversed order of the 4 ipv4 octets.

i'm still not sure what you're trying to achieve, is it allowing a hostname in hostspec part of the upstream statement ?

e.g. upstream none "foo.com"

this is already implemented, and i used it in the past; the only gotcha is that you have to put the line
before the default upstream.

you keep mentioning "reverse DNS", but usually reverse DNS means looking up X.X.X.X.in-addr.arpa where X.X.X.X is the reversed order of the 4 ipv4 octets.

i'm still not sure what you're trying to achieve, is it allowing a hostname in hostspec part of the upstream statement ?

e.g. upstream none "foo.com"

this is already implemented, and i used it in the past; the only gotcha is that you have to put the line
before the default upstream.

OOps yes it is a forward DNS to compare the IP. (Embarrassed)

I can't see in the code that this is done. I see that if a site name is given, that you compare it only with a upstream line which has a name or domain and NOT with a upstream line which has an IP.

To compare a name with an IP the name has to be resolved to the IP and that is what I do.

Markus

I can't see in the code that this is done.

is this an indirect affirmation of my question?

what you're trying to achieve, is it allowing a hostname in hostspec part of the upstream statement ?
e.g. upstream none "foo.com"

I can't see in the code that this is done.

well, maybe you just don't have the full picture. i again state that this worked in the past and it still should, unless it has been broken unwittingly. why don't you just try it out instead of repeating over and over that you can't see it in the code ?

I can't see in the code that this is done.

is this an indirect affirmation of my question?

No

what you're trying to achieve, is it allowing a hostname in hostspec part of the upstream statement ?
e.g. upstream none "foo.com"

No

I want that a host abcd.company.com with ip 10.1.1.1 hits the none upstream rule

upstream http proxy:8000
upstream none 10.0.0.0/8

i.e. a https://abcd.company.com hitting tinyproxy goes direct to the destination and not to the upstream proxy

I can't see in the code that this is done.

well, maybe you just don't have the full picture. i again state that this worked in the past and it still should, unless it has been broken unwittingly. why don't you just try it out instead of repeating over and over that you can't see it in the code ?

I did run it with the existing code and it does not work and it seems you have not looked at the code I provided.

Markus

I want that a host abcd.company.com with ip 10.1.1.1 hits the none upstream rule

upstream http proxy:8000
upstream none "10.0.0.0/8"
upstream none "abcd.company.com"

this should make it work, did you try that ?

I want that a host abcd.company.com with ip 10.1.1.1 hits the none upstream rule

upstream http proxy:8000
upstream none "10.0.0.0/8"
upstream none "abcd.company.com"

this should make it work, did you try that ?

Yes, but I do not want to add the thousands of hosts which have a 10.x.x.x IP to the config. I want one fixed rule which covers it automatically.

Yes, but I do not want to add the thousands of hosts which have a 10.x.x.x IP to the config. I want one fixed rule which covers it automatically.

oh, then how about upstream none ".company.com" - which is a wildcard rule for *.company.com ?

Yes, but I do not want to add the thousands of hosts which have a 10.x.x.x IP to the config. I want one fixed rule which covers it automatically.

oh, then how about upstream none ".company.com" - which is a wildcard rule for *.company.com ?

Unfortunately I have .company.com also w/o 10.x.x.x.. I need the IP based check after DNS resolution. So what is the problem here ?

So what is the problem here ?

i'm doing what any reasonable maintainer should do: evaluating whether requested feature additions make sense or can be achieved via already existing functionality - the prerequisite for that is actually understanding what the user is trying to achieve.

So what is the problem here ?

i'm doing what any reasonable maintainer should do: evaluating whether requested feature additions make sense or can be achieved via already existing functionality - the prerequisite for that is actually understanding what the user is trying to achieve.

Ok. What is your evaluation ?

Yes, but I do not want to add the thousands of hosts which have a 10.x.x.x IP to the config. I want one fixed rule which covers it automatically.

oh, then how about upstream none ".company.com" - which is a wildcard rule for *.company.com ?

I have also hosts with *.company.com which are not in 10.x.x.x. I went through all the options and all mean I need to regularly update the tinyproxy config. I really want to avoid unnecessary config adjustments if I can do it with one line.

And it may not even be one domain company.com, but also comp.fr , etc.., i.e. multiple.

well, you managed to produce your own solution, congrats.
i'm currently busy with some other projects but will come back asap to decide about this PR when i have my head free.

well, you managed to produce your own solution, congrats.
i'm currently busy with some other projects but will come back asap to decide about this PR when i have my head free.

Ok. I think others would benefit from it too.

Adjusted names => mm-dns