Re: logformat for requests using PROXY protocol

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: logformat for requests using PROXY protocol

Amos Jeffries
Administrator
On 15/11/19 2:56 pm, chammidhan wrote:

> I have configured a Squid ECS cluster behind a network load balancer in AWS.
> To reflect the original client IP, I needed to enable PROXY Protocol V2 on
> the load balancer. The service itself is working fine and I can create rules
> based on the original client IP and these are applied as expected. However,
> it doesn't seem that logformat format codes are working as expected. No
> matter how I format the logs, I'm always seeing the logs in the same format.
> Which looks like below.
>
> 1573771498.693 240116 10.181.3.10 TCP_TUNNEL/200 1742 CONNECT
> id.google.com:443 - HIER_DIRECT/172.217.167.67 -
>
> My logformat directive is the default
> logformat squid %{%Y/%m/%d-%H:%M:%S}tl %>A/%>a %un %rm/%rv %ru %mt
> %{User-Agent}>h %>st/%<st %tr %>Hs %Ss %Sh/%<A
>
> Appreciate any insight to what I may be doing wrong. Things were working
> fine before enabling PROXY protocol on the NLB
>

Please run "squid -k parse" on your config and fix the errors and
warnings it produces.

"
2019/11/15 18:11:50| Processing: logformat squid %{%Y/%m/%d-...
2019/11/15 18:11:50| ERROR: logformat squid is already defined. Ignoring.
"

To use a custom log format you need a custom name.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: logformat for requests using PROXY protocol

chammidhan
I have my logformat as follows.
logformat jsonformat {"Client Hostname":"%>A","Source IP":"%>a","HTTP Method":"%rm","HTTP Protocol version":"%rv","Request Domain":"%>rd","Port":"%>rP","User Agent":"%{User-Agent}>h","Request Size":"%>st","Reply Size":"%<st","Response Time(ms)":"%tr","Status Code":"%>Hs","Request Status":"%Ss","Server FQDN":"%<A"} 

The proxy is sitting behind a load balancer in AWS and Proxy Protocol V2 is enabled on both the LB and Squid. Everything seems to work fine. I can create rules based on source IP of the client. However. I want to be able to  create rules based on the hostname of the original client. But it doesn't seem that Squid sees the original client's hostname. Rather it takes the hostname of the LB as seen by below log.

{ "Client Hostname": "ip-10-181-3-213.ap-southeast-2.compute.internal", "Source IP": "10.181.3.10", "HTTP Method": "CONNECT", "HTTP Protocol version": "1.1", "Request Domain": "clientservices.googleapis.com", "Port": "443", "User Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36", "Request Size": "253", "Reply Size": "4138", "Response Time(ms)": "0", "Status Code": "403", "Request Status": "TCP_DENIED", "Server FQDN": "-" } 

On Fri, Nov 15, 2019 at 3:15 PM Amos Jeffries <[hidden email]> wrote:
On 15/11/19 2:56 pm, chammidhan wrote:
> I have configured a Squid ECS cluster behind a network load balancer in AWS.
> To reflect the original client IP, I needed to enable PROXY Protocol V2 on
> the load balancer. The service itself is working fine and I can create rules
> based on the original client IP and these are applied as expected. However,
> it doesn't seem that logformat format codes are working as expected. No
> matter how I format the logs, I'm always seeing the logs in the same format.
> Which looks like below.
>
> 1573771498.693 240116 10.181.3.10 TCP_TUNNEL/200 1742 CONNECT
> id.google.com:443 - HIER_DIRECT/172.217.167.67 -
>
> My logformat directive is the default
> logformat squid %{%Y/%m/%d-%H:%M:%S}tl %>A/%>a %un %rm/%rv %ru %mt
> %{User-Agent}>h %>st/%<st %tr %>Hs %Ss %Sh/%<A
>
> Appreciate any insight to what I may be doing wrong. Things were working
> fine before enabling PROXY protocol on the NLB
>

Please run "squid -k parse" on your config and fix the errors and
warnings it produces.

"
2019/11/15 18:11:50| Processing: logformat squid %{%Y/%m/%d-...
2019/11/15 18:11:50| ERROR: logformat squid is already defined. Ignoring.
"

To use a custom log format you need a custom name.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: logformat for requests using PROXY protocol

Amos Jeffries
Administrator
On 22/11/19 4:05 pm, Chammi Kumarapathirage wrote:

> I have my logformat as follows.
> logformat jsonformat {"Client Hostname":"%>A","Source IP":"%>a","HTTP
> Method":"%rm","HTTP Protocol version":"%rv","Request
> Domain":"%>rd","Port":"%>rP","User Agent":"%{User-Agent}>h","Request
> Size":"%>st","Reply Size":"%<st","Response Time(ms)":"%tr","Status
> Code":"%>Hs","Request Status":"%Ss","Server FQDN":"%<A"} 
>
> The proxy is sitting behind a load balancer in AWS and Proxy Protocol V2
> is enabled on both the LB and Squid. Everything seems to work fine. I
> can create rules based on source IP of the client. However. I want to be
> able to  create rules based on the hostname of the original client. But
> it doesn't seem that Squid sees the original client's hostname. Rather
> it takes the hostname of the LB as seen by below log.

The %>A log code is still tied to old logging state instead of the IP
values updated by PROXY protocol.

The only way I can see to log that value without patching Squid is with
something complicated like an external_acl_type helper to do the lookup
and supply it as a tag or note to Squid.

If you are happy to patch I can make a PR for you to try.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: logformat for requests using PROXY protocol

chammidhan
Thanks for the response Amos. This is an AWS Fargate instance and I'm not exactly sure how patching works in that space. I'm rather new to both serverless concept and Squid. I will research this and get back to you. Thanks!

On Friday, November 22, 2019, Amos Jeffries <[hidden email]> wrote:
On 22/11/19 4:05 pm, Chammi Kumarapathirage wrote:
> I have my logformat as follows.
> logformat jsonformat {"Client Hostname":"%>A","Source IP":"%>a","HTTP
> Method":"%rm","HTTP Protocol version":"%rv","Request
> Domain":"%>rd","Port":"%>rP","User Agent":"%{User-Agent}>h","Request
> Size":"%>st","Reply Size":"%<st","Response Time(ms)":"%tr","Status
> Code":"%>Hs","Request Status":"%Ss","Server FQDN":"%<A"} 
>
> The proxy is sitting behind a load balancer in AWS and Proxy Protocol V2
> is enabled on both the LB and Squid. Everything seems to work fine. I
> can create rules based on source IP of the client. However. I want to be
> able to  create rules based on the hostname of the original client. But
> it doesn't seem that Squid sees the original client's hostname. Rather
> it takes the hostname of the LB as seen by below log.

The %>A log code is still tied to old logging state instead of the IP
values updated by PROXY protocol.

The only way I can see to log that value without patching Squid is with
something complicated like an external_acl_type helper to do the lookup
and supply it as a tag or note to Squid.

If you are happy to patch I can make a PR for you to try.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users