Making destination IP available in ICAP REQMOD request

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Making destination IP available in ICAP REQMOD request

Moti Berger
Hi

My goal is to obtain the destination IP when sending an HTTP request for my ICAP server so it would be able to decide the kind of adaptation required based on it.

Looking at squid (5.0.4) code I discovered the following:

It seems that "everything" starts at ClientRequestContext.
I've noticed that noteAdaptationAclCheckDone calls startAdaptation which calls more methods, eventually getting to Adaptation::Icap::ModXact::makeRequestHeaders where it iterates over headers defined by the adaptation_meta configurations in squid.conf.
For each, it calls the 'match' method where it tries to format (and assemble) it. There it seems that the value is taken from an AccessLogEntry:
   case LFT_SERVER_IP_ADDRESS:
            if (al->hier.tcpServer)
                out = al->hier.tcpServer->remote.toStr(tmp, sizeof(tmp));
            break;

So the AccessLogEntry object seems to be the key.
At REQMOD time, I don't get the value of the destination IP.
Looking further I found that the DNS resolving happens when it's decided that the request should be forwarded to the destination server.

So I tracked the flow and it seems to start from FwdState::Start method which gets an AccessLogEntryPointer.
Then it calls methods that eventually do the DNS resolving (Dns::nbgethostbyname) and ending in (FwdState::connectStart) which have the IP to connect to.
So it seems that this flow will populate the AccessLogEntry.
This seems right since during RESPMOD, the same code above (in Adaptation::Icap::ModXact::makeRequestHeaders) is running and this time the `match` method eventually gets the destination IP.
I added logs that prints the AccessLogEntryPointer and in the FwdState.cc the log says address 0x5592ab521e30*12 and in the Notes.cc the log says address 0x5592ab521e30*25.

Two things that I haven't found yet:
1. The place where the AccessLogEntry is populated
2. Where after the adaptation, the forwarding to the destination server occured (assuming it should be forwarded)

I couldn't figure out a way to start the DNS resolving just before the startAdaptation starts as it requires all sorts of objects that seem to be unavailable there.
I wonder if you can help me to find a way to do it.

Thanks,
Moti


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Making destination IP available in ICAP REQMOD request

Amos Jeffries
Administrator
As you have found. There is no destination IP at REQMOD time. Even if squid were to do a lookup it does not know the outcome of the routing decision in order to select which IP to send REQMOD. Especially if REQMOD is the source of that decision.

A normal (forward) proxy has only a server host name (aka URI domain) to work with until the server connection is about to be opened. The REQMOD can be passed that name, but needs to account for *any* of the IPs it finds for that (or none) might be used.

Interception proxy try to connect to the same server IP as the client was using. But may not need a server connection at all if cache storage or collapsed forwarding catches the transaction.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Making destination IP available in ICAP REQMOD request

Eliezer Croitoru-3
In reply to this post by Moti Berger
Hey Moti,

It is a good assumption that the same caching dns server (not 8.8.8.8 or 1.1.1.1) that the client use will return the relevant destination ip for the domain.
Its possible to do such a query in the icap service with low timeout(2-3) seconds.
can this be good enough for your use case?

Eliezer

On Mon, Jan 18, 2021, 00:28 Moti Berger <[hidden email]> wrote:
Hi

My goal is to obtain the destination IP when sending an HTTP request for my ICAP server so it would be able to decide the kind of adaptation required based on it.

Looking at squid (5.0.4) code I discovered the following:

It seems that "everything" starts at ClientRequestContext.
I've noticed that noteAdaptationAclCheckDone calls startAdaptation which calls more methods, eventually getting to Adaptation::Icap::ModXact::makeRequestHeaders where it iterates over headers defined by the adaptation_meta configurations in squid.conf.
For each, it calls the 'match' method where it tries to format (and assemble) it. There it seems that the value is taken from an AccessLogEntry:
   case LFT_SERVER_IP_ADDRESS:
            if (al->hier.tcpServer)
                out = al->hier.tcpServer->remote.toStr(tmp, sizeof(tmp));
            break;

So the AccessLogEntry object seems to be the key.
At REQMOD time, I don't get the value of the destination IP.
Looking further I found that the DNS resolving happens when it's decided that the request should be forwarded to the destination server.

So I tracked the flow and it seems to start from FwdState::Start method which gets an AccessLogEntryPointer.
Then it calls methods that eventually do the DNS resolving (Dns::nbgethostbyname) and ending in (FwdState::connectStart) which have the IP to connect to.
So it seems that this flow will populate the AccessLogEntry.
This seems right since during RESPMOD, the same code above (in Adaptation::Icap::ModXact::makeRequestHeaders) is running and this time the `match` method eventually gets the destination IP.
I added logs that prints the AccessLogEntryPointer and in the FwdState.cc the log says address 0x5592ab521e30*12 and in the Notes.cc the log says address 0x5592ab521e30*25.

Two things that I haven't found yet:
1. The place where the AccessLogEntry is populated
2. Where after the adaptation, the forwarding to the destination server occured (assuming it should be forwarded)

I couldn't figure out a way to start the DNS resolving just before the startAdaptation starts as it requires all sorts of objects that seem to be unavailable there.
I wonder if you can help me to find a way to do it.

Thanks,
Moti

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Making destination IP available in ICAP REQMOD request

Alex Rousskov
In reply to this post by Moti Berger
On 1/17/21 5:28 PM, Moti Berger wrote:
> I couldn't figure out a way to start the DNS resolving just before
> the startAdaptation starts as it requires all sorts of objects that seem
> to be unavailable there.

Please ask development questions on squid-dev:
http://www.squid-cache.org/Support/mailing-lists.html#squid-dev

Triggering a DNS lookup before REQMOD is easy, but waiting for its
asynchronous result requires a bit of development work outside this
mailing list scope.

More importantly, a DNS lookup alone is probably not enough. If your
REQMOD service relies on the destination address that Squid will use
when/if forwarding the request, then Squid should be modified to
(optionally) decide on the forwarding destination first, _before_
adapting the request and then, to the extent possible, stick with that
original decision after REQMOD. This may be equivalent to implementing
post-cache REQMOD!

It is possible to implement all that, and there are legitimate use cases
that would benefit from such functionality, but a quality implementation
will require serious development work. If you are sure that you need
this functionality, and you want to implement it yourself, then I
recommend starting with an RFC on squid-dev:
https://wiki.squid-cache.org/MergeProcedure#Before_you_start_coding


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users