SQUID with cache_peer config + E2guardian - too many connections

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

SQUID with cache_peer config + E2guardian - too many connections

Contato - KONNTROL
Hello Everyone,
Greetings.

Background:
OS - FreeBSD 12.1
SQUID ver 4.10
OpenSSL 1.0.2u

I am trying to use SQUID in front of E2Guardian (content filter) with the
following configuration at the SQUID side.

###
cache_peer 127.0.0.1 parent 8080 0 login=*:password
client_persistent_connections on
always_direct deny all
never_direct allow all
###

It works fine till the point  SQUID exhausts all E2Guardian threads/workers,
no matter the amount you set. If 1000, SQUID is opening 1000 connections. If
10.000, squid also opens 10.000 connections.
I tried the directive "client_persistent_connections on and off" with no
success.
Even using a single browser for testing purposes, for some reason SQUID
opens thousands of connections against the E2guardian.
I did a wireshark capture to "see" what is  happening and it seems like a
lot of ACK/SYN with no payload.

Any idea? Maybe I am using a wrong configuration.

By the way, I am using SQUID in front of E2Guardian cause I use Kerberos
authentication (not supported by E2guardian) with FORWARDX option enable.

Thanks In advance,

Cordially.
Fabricio.


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Amos Jeffries
Administrator
On 29/07/20 1:07 pm, Contato - KONNTROL wrote:

> Hello Everyone,
> Greetings.
>
> Background:
> OS - FreeBSD 12.1
> SQUID ver 4.10
> OpenSSL 1.0.2u
>
> I am trying to use SQUID in front of E2Guardian (content filter) with the
> following configuration at the SQUID side.
>
> ###
> cache_peer 127.0.0.1 parent 8080 0 login=*:password
> client_persistent_connections on
> always_direct deny all
> never_direct allow all
> ###
>
> It works fine till the point  SQUID exhausts all E2Guardian threads/workers,
> no matter the amount you set. If 1000, SQUID is opening 1000 connections. If
> 10.000, squid also opens 10.000 connections.
> I tried the directive "client_persistent_connections on and off" with no
> success.
> Even using a single browser for testing purposes, for some reason SQUID
> opens thousands of connections against the E2guardian.
> I did a wireshark capture to "see" what is  happening and it seems like a
> lot of ACK/SYN with no payload.
>
> Any idea? Maybe I am using a wrong configuration.
>

You are. BUT, I think you have a forwarding loop happening so the
correct config for limiting connections will not help.

You should be able to test for loops by enabling the Via header. If your
squid.conf contains "via off" remove that line. Assuming e2g is not
removing that header Squid will reject loops with an error message.


Check that the traffic leaving e2g is not going back into Squid. With
the setup described e2g should be connecting directly to
upstream/Internet servers - it should have no settings about Squid
except those for processing the X-Forwarded-For header.

If you are intercepting traffic to deliver it to Squid make sure the
connections leaving e2g are not being caught by those firewall rules.


If you are certain there is no loop the cache_peer max-conn=N is the way
to limit the connections made to a peer. This will only help if the
problem is high traffic flow. It will not help if there is a forwarding
loop happening.


> By the way, I am using SQUID in front of E2Guardian cause I use Kerberos
> authentication (not supported by E2guardian) with FORWARDX option enable.
>

Sure. You may want to look at the features of e2g you are using and see
whether Squid can do them instead. The idea there being to make deny
decisions early as possible to minimize the total amount of processing
work those transactions consume.
 You may find you can get rid of e2g entirely, which will improve
overall performance and reduce management headaches from layers of proxy.


Cheers
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Klaus Brandl
On Wednesday 29 July 2020 14:50:11 Amos Jeffries wrote:

> On 29/07/20 1:07 pm, Contato - KONNTROL wrote:
> > Hello Everyone,
> > Greetings.
> >
> > Background:
> > OS - FreeBSD 12.1
> > SQUID ver 4.10
> > OpenSSL 1.0.2u
> >
> > I am trying to use SQUID in front of E2Guardian (content filter) with the
> > following configuration at the SQUID side.
> >
> > ###
> > cache_peer 127.0.0.1 parent 8080 0 login=*:password
> > client_persistent_connections on
> > always_direct deny all
> > never_direct allow all
> > ###
> >
> > It works fine till the point  SQUID exhausts all E2Guardian
> > threads/workers, no matter the amount you set. If 1000, SQUID is opening
> > 1000 connections. If 10.000, squid also opens 10.000 connections.
> > I tried the directive "client_persistent_connections on and off" with no
> > success.
> > Even using a single browser for testing purposes, for some reason SQUID
> > opens thousands of connections against the E2guardian.
> > I did a wireshark capture to "see" what is  happening and it seems like a
> > lot of ACK/SYN with no payload.
> >
> > Any idea? Maybe I am using a wrong configuration.
>
> You are. BUT, I think you have a forwarding loop happening so the
> correct config for limiting connections will not help.
>
> You should be able to test for loops by enabling the Via header. If your
> squid.conf contains "via off" remove that line. Assuming e2g is not
> removing that header Squid will reject loops with an error message.

Setting an other "visible_hostname" may also help.

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Amos Jeffries
Administrator
On 29/07/20 8:29 pm, Klaus Brandl wrote:
>
> Setting an other "visible_hostname" may also help.
>

Why do you think the hostname has any relation to the problem?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Klaus Brandl
On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
> On 29/07/20 8:29 pm, Klaus Brandl wrote:
> > Setting an other "visible_hostname" may also help.
>
> Why do you think the hostname has any relation to the problem?

because we had also a forwarding loop by connection 2 squids on the same host
together via a parent statement. Then in the via header there was the same
hostname 2 times, and this issued squid to detect a forwarding loop.
Setting an other visible_hostname on one of the squids solved this problem.

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Matus UHLAR - fantomas
>> On 29/07/20 8:29 pm, Klaus Brandl wrote:
>> > Setting an other "visible_hostname" may also help.

>On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
>> Why do you think the hostname has any relation to the problem?

On 29.07.20 13:58, Klaus Brandl wrote:
>because we had also a forwarding loop by connection 2 squids on the same host
>together via a parent statement. Then in the via header there was the same
>hostname 2 times, and this issued squid to detect a forwarding loop.
>Setting an other visible_hostname on one of the squids solved this problem.

there's a specific setting "unique_hostname" for these cases.

#  TAG: unique_hostname
#       If you want to have multiple machines with the same
#       'visible_hostname' you must give each machine a different
#       'unique_hostname' so forwarding loops can be detected.
#Default:
# Copy the value from visible_hostname


I just wonder why is this not set to $HOSTNAME by default, so setting same
visible_hostname on different servers would not require setting different
unique_hostname

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
The only substitute for good manners is fast reflexes.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Amos Jeffries
Administrator
In reply to this post by Klaus Brandl
On 29/07/20 11:58 pm, Klaus Brandl wrote:

> On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
>> On 29/07/20 8:29 pm, Klaus Brandl wrote:
>>> Setting an other "visible_hostname" may also help.
>>
>> Why do you think the hostname has any relation to the problem?
>
> because we had also a forwarding loop by connection 2 squids on the same host
> together via a parent statement. Then in the via header there was the same
> hostname 2 times, and this issued squid to detect a forwarding loop.
> Setting an other visible_hostname on one of the squids solved this problem.>

In your situation the parent proxy was wrongly reporting loops since it
saw its own name coming out of the child proxy. That is not a real loop,
just a misconfiguration on your part to begin with.

NP: Klaus, unique_hostname is probably a better solution to your
problem. That lets loop detection work properly but both proxies send
URLs containing the shared visible_hostname to clients when they need to
reference proxy resources.


For this thread the OP has only one Squid and it is first in the proxy
chain.

For a loop to happen the peer must already be accepting traffic from
Squid with its current visible/public hostname. Only after that the
traffic might loop back to Squid to begin another circle. So setting the
Squid hostname to a different value will not stop any real loops, only
alter the string placed in the Via header each cycle.

Right now in the troubleshooting we are trying to get loops to show up
to see whether that is the hidden problem.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Klaus Brandl
On Thursday 30 July 2020 00:24:52 Amos Jeffries wrote:

> On 29/07/20 11:58 pm, Klaus Brandl wrote:
> > On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
> >> On 29/07/20 8:29 pm, Klaus Brandl wrote:
> >>> Setting an other "visible_hostname" may also help.
> >>
> >> Why do you think the hostname has any relation to the problem?
> >
> > because we had also a forwarding loop by connection 2 squids on the same
> > host together via a parent statement. Then in the via header there was
> > the same hostname 2 times, and this issued squid to detect a forwarding
> > loop. Setting an other visible_hostname on one of the squids solved this
> > problem.>
> In your situation the parent proxy was wrongly reporting loops since it
> saw its own name coming out of the child proxy. That is not a real loop,
> just a misconfiguration on your part to begin with.
>
> NP: Klaus, unique_hostname is probably a better solution to your
> problem. That lets loop detection work properly but both proxies send
> URLs containing the shared visible_hostname to clients when they need to
> reference proxy resources.

ok, thank you, i will use this next time we have this problem.

>
>
> For this thread the OP has only one Squid and it is first in the proxy
> chain.

i saw the "cache_peer 127.0.0.1 parent 8080..." entry and it remembered me to
our problem, sorry.

>
> For a loop to happen the peer must already be accepting traffic from
> Squid with its current visible/public hostname. Only after that the
> traffic might loop back to Squid to begin another circle. So setting the
> Squid hostname to a different value will not stop any real loops, only
> alter the string placed in the Via header each cycle.
>
> Right now in the troubleshooting we are trying to get loops to show up
> to see whether that is the hidden problem.
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Contato - KONNTROL
Thanks Amos, Klaus and Fantomas.
I already send another message with a lot of info and an attachment, that is now under moderator's review. If blocked, I will delete the attachment and send it again.

Thanks
Fabricio.


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Klaus Brandl
Sent: Wednesday, July 29, 2020 9:37 AM
To: [hidden email]
Subject: Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

On Thursday 30 July 2020 00:24:52 Amos Jeffries wrote:

> On 29/07/20 11:58 pm, Klaus Brandl wrote:
> > On Wednesday 29 July 2020 23:03:43 Amos Jeffries wrote:
> >> On 29/07/20 8:29 pm, Klaus Brandl wrote:
> >>> Setting an other "visible_hostname" may also help.
> >>
> >> Why do you think the hostname has any relation to the problem?
> >
> > because we had also a forwarding loop by connection 2 squids on the
> > same host together via a parent statement. Then in the via header
> > there was the same hostname 2 times, and this issued squid to detect
> > a forwarding loop. Setting an other visible_hostname on one of the
> > squids solved this problem.>
> In your situation the parent proxy was wrongly reporting loops since
> it saw its own name coming out of the child proxy. That is not a real
> loop, just a misconfiguration on your part to begin with.
>
> NP: Klaus, unique_hostname is probably a better solution to your
> problem. That lets loop detection work properly but both proxies send
> URLs containing the shared visible_hostname to clients when they need
> to reference proxy resources.

ok, thank you, i will use this next time we have this problem.

>
>
> For this thread the OP has only one Squid and it is first in the proxy
> chain.

i saw the "cache_peer 127.0.0.1 parent 8080..." entry and it remembered me to our problem, sorry.

>
> For a loop to happen the peer must already be accepting traffic from
> Squid with its current visible/public hostname. Only after that the
> traffic might loop back to Squid to begin another circle. So setting
> the Squid hostname to a different value will not stop any real loops,
> only alter the string placed in the Via header each cycle.
>
> Right now in the troubleshooting we are trying to get loops to show up
> to see whether that is the hidden problem.
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch Amtsgericht Muenchen HRB 98238 genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Contato - KONNTROL
In reply to this post by Amos Jeffries
Hello Amos/Klaus/Fantomas,
Thanks for your help.

I have searched for the "via off"  and "via on" - The directive was not present on the config file but I am assuming "via on" it's the default option. Anyway I added it to the config file. No Success.
I also checked the visible_hostname, as suggested by Klaus (Thanks Klaus for the help!!) . It was set to localhost, then I changed to something different. No success as well.

So, I decided to make another test using Wireshark. For that, I put the squid down before starting the capture, then started Squid after capturing.
What I can see is squid sending thousands of requests (like a machine gun) against the E2G (loopback interface on port 8080) with thousands of "408 Request Time Out" entries.
I also see the following HTTP GET: "http://127.0.0.1:8080/squid-internal-dynamic/netdb"  by the way,  8080 is E2G port. Not sure what it is. The 408 above are probably because of this calling.

Attached you can see the capture file, just in case you have wireshark or any other software able to read .cap file. Don't worry, there is nothing confidential on the file. That's a LAB environment.
That is really confusing me.

Thank You very Much!

Regards
Fabricio.



-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Amos Jeffries
Sent: Tuesday, July 28, 2020 11:50 PM
To: [hidden email]
Subject: Re: [squid-users] SQUID with cache_peer config + E2guardian - too many connections

On 29/07/20 1:07 pm, Contato - KONNTROL wrote:

> Hello Everyone,
> Greetings.
>
> Background:
> OS - FreeBSD 12.1
> SQUID ver 4.10
> OpenSSL 1.0.2u
>
> I am trying to use SQUID in front of E2Guardian (content filter) with
> the following configuration at the SQUID side.
>
> ###
> cache_peer 127.0.0.1 parent 8080 0 login=*:password
> client_persistent_connections on always_direct deny all never_direct
> allow all ###
>
> It works fine till the point  SQUID exhausts all E2Guardian
> threads/workers, no matter the amount you set. If 1000, SQUID is
> opening 1000 connections. If 10.000, squid also opens 10.000 connections.
> I tried the directive "client_persistent_connections on and off" with
> no success.
> Even using a single browser for testing purposes, for some reason
> SQUID opens thousands of connections against the E2guardian.
> I did a wireshark capture to "see" what is  happening and it seems
> like a lot of ACK/SYN with no payload.
>
> Any idea? Maybe I am using a wrong configuration.
>
You are. BUT, I think you have a forwarding loop happening so the correct config for limiting connections will not help.

You should be able to test for loops by enabling the Via header. If your squid.conf contains "via off" remove that line. Assuming e2g is not removing that header Squid will reject loops with an error message.


Check that the traffic leaving e2g is not going back into Squid. With the setup described e2g should be connecting directly to upstream/Internet servers - it should have no settings about Squid except those for processing the X-Forwarded-For header.

If you are intercepting traffic to deliver it to Squid make sure the connections leaving e2g are not being caught by those firewall rules.


If you are certain there is no loop the cache_peer max-conn=N is the way to limit the connections made to a peer. This will only help if the problem is high traffic flow. It will not help if there is a forwarding loop happening.


> By the way, I am using SQUID in front of E2Guardian cause I use
> Kerberos authentication (not supported by E2guardian) with FORWARDX option enable.
>

Sure. You may want to look at the features of e2g you are using and see whether Squid can do them instead. The idea there being to make deny decisions early as possible to minimize the total amount of processing work those transactions consume.
 You may find you can get rid of e2g entirely, which will improve overall performance and reduce management headaches from layers of proxy.


Cheers
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Squid_E2G_Loopback_Capture.zip (62K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SQUID with cache_peer config + E2guardian - too many connections

Klaus Brandl
tried to add the option "no-netdb-exchange" on your cache_peer line?

On Wednesday 29 July 2020 11:50:20 Contato - KONNTROL wrote:

> Hello Amos/Klaus/Fantomas,
> Thanks for your help.
>
> I have searched for the "via off"  and "via on" - The directive was not
> present on the config file but I am assuming "via on" it's the default
> option. Anyway I added it to the config file. No Success. I also checked
> the visible_hostname, as suggested by Klaus (Thanks Klaus for the help!!) .
> It was set to localhost, then I changed to something different. No success
> as well.
>
> So, I decided to make another test using Wireshark. For that, I put the
> squid down before starting the capture, then started Squid after capturing.
> What I can see is squid sending thousands of requests (like a machine gun)
> against the E2G (loopback interface on port 8080) with thousands of "408
> Request Time Out" entries. I also see the following HTTP GET:
> "http://127.0.0.1:8080/squid-internal-dynamic/netdb"  by the way,  8080 is
> E2G port. Not sure what it is. The 408 above are probably because of this
> calling.
>
> Attached you can see the capture file, just in case you have wireshark or
> any other software able to read .cap file. Don't worry, there is nothing
> confidential on the file. That's a LAB environment. That is really
> confusing me.
>
> Thank You very Much!
>
> Regards
> Fabricio.
>
>

Klaus

---

genua GmbH
Domagkstrasse 7, 85551 Kirchheim bei Muenchen
tel +49 89 991950-0, fax -999, www.genua.de

Geschaeftsfuehrer: Matthias Ochs, Marc Tesch
Amtsgericht Muenchen HRB 98238
genua ist ein Unternehmen der Bundesdruckerei-Gruppe.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users