Forwarded-for functionality(squid)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Forwarded-for functionality(squid)

piyush.gaba

Hi Team,

 

I would like to ask for your inputs/resolution for one of the issues that I am facing while using forwarded_for functionality with new version of squid i.e. v4.13

 

UseCase:

I have a client, a webserver and squid hosted in 3 different VMs. I have set up interfaces in such a way that Client cannot access webserver directly, but only through squid proxy.(Works fine)

 

Now, When I append forwarded_for off to my squid.conf and then curl the webserver from client [ curl {Webserver.mgmtIP} ], the expected logs on webserver should have, “unknown” at the end of line.

And if there is nothing appended or forwarded_for on is appended, it should have “Client_IP” at the end of the line in log file.

But I am not getting expected output.

 

PFA squid.conf ,output at httpd , httpd.conf.

 

After several tries, I am looking forward for your help/advice.

 

Thank You.  

 

Bien cordialement,

Piyush Gaba

Software Engineer

TGI/OLN- INDIA

Tower B, 8th Floor, DLF Infinity Towers, 

DLF Cyber City Phase - II
Gurgaon - 122002, Haryana, INDIA

( Mobile: +91-9818498198

[Email] [hidden email]

 

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

httpd.conf (16K) Download Attachment
httpdlogsaftercurl.PNG (67K) Download Attachment
squid.conf (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

Matus UHLAR - fantomas
On 08.09.20 09:01, [hidden email] wrote:

>I would like to ask for your inputs/resolution for one of the issues that I am facing while using forwarded_for functionality with new version of squid i.e. v4.13
>
>UseCase:
>I have a client, a webserver and squid hosted in 3 different VMs.  I have
> set up interfaces in such a way that Client cannot access webserver
> directly, but only through squid proxy.(Works fine)
>
>Now, When I append forwarded_for off to my squid.conf and then curl the
> webserver from client [ curl {Webserver.mgmtIP} ], the expected logs on
> webserver should have, "unknown" at the end of line.

You can configure webserver to log contents of X-Forwarded-For: line.
Note that that line can contain anything clients can send.
So, unless you really want to see that content, don't log it.

>And if there is nothing appended or forwarded_for on is appended, it should have "Client_IP" at the end of the line in log file.
>But I am not getting expected output.


--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
He who laughs last thinks slowest.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

Amos Jeffries
Administrator
On 8/09/20 9:52 pm, Matus UHLAR - fantomas wrote:

> On 08.09.20 09:01, piyush.gaba wrote:
>> I would like to ask for your inputs/resolution for one of the issues
>> that I am facing while using forwarded_for functionality with new
>> version of squid i.e. v4.13
>>
>> UseCase:
>> I have a client, a webserver and squid hosted in 3 different VMs.  I have
>> set up interfaces in such a way that Client cannot access webserver
>> directly, but only through squid proxy.(Works fine)
>>
>> Now, When I append forwarded_for off to my squid.conf and then curl the
>> webserver from client [ curl {Webserver.mgmtIP} ], the expected logs on
>> webserver should have, "unknown" at the end of line.
>
> You can configure webserver to log contents of X-Forwarded-For: line.
> Note that that line can contain anything clients can send.
> So, unless you really want to see that content, don't log it.
>
>> And if there is nothing appended or forwarded_for on is appended, it
>> should have "Client_IP" at the end of the line in log file.
>> But I am not getting expected output.
>

In squid.conf add this to see the HTTP messages going through Squid and
understand what is going on:

  debug_options 11,2


HTH
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

piyush.gaba
In reply to this post by Matus UHLAR - fantomas
Hi Matus,

Thank you for your reply.

I think my whole struggle is to get the desired output as:
<ip>- - [17/Aug/2018:08:43:29 +0200] "GET /index1.html HTTP/1.1" 200 36 "-" "curl/7.29.0" unknown

Which has "unknown" at the end because forwarded_for was set to "off".

But now when I am working with squid 4.13 I am not getting the desired output, I am getting the output as,
<ip> - - [08/Sep/2020:15:07:19 +0200] "GET /index1.html HTTP/1.1" 200 8 "-" "curl/7.29.0"

Which does not have anything at the end, while the forwarded_for is set to "off".

Please let me know if you have any advice to give for this logging problem.

I am using below log format in my httpd file,
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Bien cordialement,
Piyush Gaba
Software Engineer
TGI/OLN- INDIA
Tower B, 8th Floor, DLF Infinity Towers,
DLF Cyber City Phase - II
Gurgaon - 122002, Haryana, INDIA
 [Email] [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Matus UHLAR - fantomas
Sent: Tuesday, September 8, 2020 15:23
To: [hidden email]
Subject: Re: [squid-users] Forwarded-for functionality(squid)

On 08.09.20 09:01, [hidden email] wrote:

>I would like to ask for your inputs/resolution for one of the issues that I am facing while using forwarded_for functionality with new version of squid i.e. v4.13
>
>UseCase:
>I have a client, a webserver and squid hosted in 3 different VMs.  I have
> set up interfaces in such a way that Client cannot access webserver
> directly, but only through squid proxy.(Works fine)
>
>Now, When I append forwarded_for off to my squid.conf and then curl the
> webserver from client [ curl {Webserver.mgmtIP} ], the expected logs on
> webserver should have, "unknown" at the end of line.

You can configure webserver to log contents of X-Forwarded-For: line.
Note that that line can contain anything clients can send.
So, unless you really want to see that content, don't log it.

>And if there is nothing appended or forwarded_for on is appended, it should have "Client_IP" at the end of the line in log file.
>But I am not getting expected output.


--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
He who laughs last thinks slowest.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

Amos Jeffries
Administrator
On 9/09/20 1:15 am, piyush.gaba wrote:

> Hi Matus,
>
> Thank you for your reply.
>
> I think my whole struggle is to get the desired output as:
> <ip>- - [17/Aug/2018:08:43:29 +0200] "GET /index1.html HTTP/1.1" 200 36 "-" "curl/7.29.0" unknown
>
> Which has "unknown" at the end because forwarded_for was set to "off".
>
> But now when I am working with squid 4.13 I am not getting the desired output, I am getting the output as,
> <ip> - - [08/Sep/2020:15:07:19 +0200] "GET /index1.html HTTP/1.1" 200 8 "-" "curl/7.29.0"
>
> Which does not have anything at the end, while the forwarded_for is set to "off".
>
> Please let me know if you have any advice to give for this logging problem.
>
> I am using below log format in my httpd file,
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
>

That LogFormat tells us the first column of the log file is the contents
of the X-Forwarded-For header. The first column of the log lines you
showed is "<ip>" - the value of XFF header is supposed to be one or more
IPs, so that looks like it is working.

Use the debug_options setting I gave you to *actually* see what is
happening. The Apache log is only showing you what values are _after_
the httpd process and all modules have done their modifications to the
HTTP input ... including replacing the transaction client-IP with any
value from XFF header.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

piyush.gaba
Hi Amos,

Thank you so much for your reply.

I did debug and I found below output in the squid logs,

1599582079.051      3 <Client_Ip> TCP_MISS/200 430 GET http:// <webserverIp>/index1.html - HIER_DIRECT/<webserverIp> text/html

I would like to share a URL for squid directive and that is the output I am expecting at the end of the line of my output and as I have given in the trail.
http://www.squid-cache.org/Doc/config/forwarded_for/ 

When set to off, it should show "unknown". Right ?

Bien cordialement,
Piyush Gaba
Software Engineer
TGI/OLN- INDIA
Tower B, 8th Floor, DLF Infinity Towers,
DLF Cyber City Phase - II
Gurgaon - 122002, Haryana, INDIA
[Email] [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
Sent: Tuesday, September 8, 2020 20:42
To: [hidden email]
Subject: Re: [squid-users] Forwarded-for functionality(squid)

On 9/09/20 1:15 am, piyush.gaba wrote:

> Hi Matus,
>
> Thank you for your reply.
>
> I think my whole struggle is to get the desired output as:
> <ip>- - [17/Aug/2018:08:43:29 +0200] "GET /index1.html HTTP/1.1" 200 36 "-" "curl/7.29.0" unknown
>
> Which has "unknown" at the end because forwarded_for was set to "off".
>
> But now when I am working with squid 4.13 I am not getting the desired output, I am getting the output as,
> <ip> - - [08/Sep/2020:15:07:19 +0200] "GET /index1.html HTTP/1.1" 200 8 "-" "curl/7.29.0"
>
> Which does not have anything at the end, while the forwarded_for is set to "off".
>
> Please let me know if you have any advice to give for this logging problem.
>
> I am using below log format in my httpd file,
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
>

That LogFormat tells us the first column of the log file is the contents
of the X-Forwarded-For header. The first column of the log lines you
showed is "<ip>" - the value of XFF header is supposed to be one or more
IPs, so that looks like it is working.

Use the debug_options setting I gave you to *actually* see what is
happening. The Apache log is only showing you what values are _after_
the httpd process and all modules have done their modifications to the
HTTP input ... including replacing the transaction client-IP with any
value from XFF header.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

piyush.gaba
In reply to this post by Amos Jeffries
By the way guys. It just worked. Thanks alot Amos and Matus.

Thank You.

Bien cordialement,
Piyush Gaba
Software Engineer
TGI/OLN- INDIA
Tower B, 8th Floor, DLF Infinity Towers, 
DLF Cyber City Phase - II
Gurgaon - 122002, Haryana, INDIA
G Mobile: +91-9818498198
[Email] [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
Sent: Tuesday, September 8, 2020 20:42
To: [hidden email]
Subject: Re: [squid-users] Forwarded-for functionality(squid)

On 9/09/20 1:15 am, piyush.gaba wrote:

> Hi Matus,
>
> Thank you for your reply.
>
> I think my whole struggle is to get the desired output as:
> <ip>- - [17/Aug/2018:08:43:29 +0200] "GET /index1.html HTTP/1.1" 200 36 "-" "curl/7.29.0" unknown
>
> Which has "unknown" at the end because forwarded_for was set to "off".
>
> But now when I am working with squid 4.13 I am not getting the desired output, I am getting the output as,
> <ip> - - [08/Sep/2020:15:07:19 +0200] "GET /index1.html HTTP/1.1" 200 8 "-" "curl/7.29.0"
>
> Which does not have anything at the end, while the forwarded_for is set to "off".
>
> Please let me know if you have any advice to give for this logging problem.
>
> I am using below log format in my httpd file,
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
>

That LogFormat tells us the first column of the log file is the contents
of the X-Forwarded-For header. The first column of the log lines you
showed is "<ip>" - the value of XFF header is supposed to be one or more
IPs, so that looks like it is working.

Use the debug_options setting I gave you to *actually* see what is
happening. The Apache log is only showing you what values are _after_
the httpd process and all modules have done their modifications to the
HTTP input ... including replacing the transaction client-IP with any
value from XFF header.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

Matus UHLAR - fantomas
On 08.09.20 16:42, [hidden email] wrote:
>By the way guys. It just worked. Thanks alot Amos and Matus.

just to make sure - are you aware that anything in the X-Forwarded-For:
header can be fake and you should only use trusted IPs?

The follow_x_forwarded_for describes how:
http://www.squid-cache.org/Doc/config/follow_x_forwarded_for/


>On 9/09/20 1:15 am, piyush.gaba wrote:
>> I think my whole struggle is to get the desired output as:
>> <ip>- - [17/Aug/2018:08:43:29 +0200] "GET /index1.html HTTP/1.1" 200 36 "-" "curl/7.29.0" unknown
>>
>> Which has "unknown" at the end because forwarded_for was set to "off".
>>
>> But now when I am working with squid 4.13 I am not getting the desired output, I am getting the output as,
>> <ip> - - [08/Sep/2020:15:07:19 +0200] "GET /index1.html HTTP/1.1" 200 8 "-" "curl/7.29.0"
>>
>> Which does not have anything at the end, while the forwarded_for is set to "off".
>>
>> Please let me know if you have any advice to give for this logging problem.
>>
>> I am using below log format in my httpd file,
>> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

>-----Original Message-----
>From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
>Sent: Tuesday, September 8, 2020 20:42
>To: [hidden email]
>Subject: Re: [squid-users] Forwarded-for functionality(squid)

>That LogFormat tells us the first column of the log file is the contents
>of the X-Forwarded-For header. The first column of the log lines you
>showed is "<ip>" - the value of XFF header is supposed to be one or more
>IPs, so that looks like it is working.
>
>Use the debug_options setting I gave you to *actually* see what is
>happening. The Apache log is only showing you what values are _after_
>the httpd process and all modules have done their modifications to the
>HTTP input ... including replacing the transaction client-IP with any
>value from XFF header.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Saving Private Ryan...
Private Ryan exists. Overwrite? (Y/N)
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Forwarded-for functionality(squid)

piyush.gaba
Yes Matus, I understand. It's just for temporary purpose since I am just testing the capabilities of squid v4.13.

Bien cordialement,
Piyush Gaba

-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Matus UHLAR - fantomas
Sent: Wednesday, September 9, 2020 14:13
To: [hidden email]
Subject: Re: [squid-users] Forwarded-for functionality(squid)

On 08.09.20 16:42, [hidden email] wrote:
>By the way guys. It just worked. Thanks alot Amos and Matus.

just to make sure - are you aware that anything in the X-Forwarded-For:
header can be fake and you should only use trusted IPs?

The follow_x_forwarded_for describes how:
http://www.squid-cache.org/Doc/config/follow_x_forwarded_for/


>On 9/09/20 1:15 am, piyush.gaba wrote:
>> I think my whole struggle is to get the desired output as:
>> <ip>- - [17/Aug/2018:08:43:29 +0200] "GET /index1.html HTTP/1.1" 200 36 "-" "curl/7.29.0" unknown
>>
>> Which has "unknown" at the end because forwarded_for was set to "off".
>>
>> But now when I am working with squid 4.13 I am not getting the desired output, I am getting the output as,
>> <ip> - - [08/Sep/2020:15:07:19 +0200] "GET /index1.html HTTP/1.1" 200 8 "-" "curl/7.29.0"
>>
>> Which does not have anything at the end, while the forwarded_for is set to "off".
>>
>> Please let me know if you have any advice to give for this logging problem.
>>
>> I am using below log format in my httpd file,
>> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

>-----Original Message-----
>From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
>Sent: Tuesday, September 8, 2020 20:42
>To: [hidden email]
>Subject: Re: [squid-users] Forwarded-for functionality(squid)

>That LogFormat tells us the first column of the log file is the contents
>of the X-Forwarded-For header. The first column of the log lines you
>showed is "<ip>" - the value of XFF header is supposed to be one or more
>IPs, so that looks like it is working.
>
>Use the debug_options setting I gave you to *actually* see what is
>happening. The Apache log is only showing you what values are _after_
>the httpd process and all modules have done their modifications to the
>HTTP input ... including replacing the transaction client-IP with any
>value from XFF header.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Saving Private Ryan...
Private Ryan exists. Overwrite? (Y/N)
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users