Squid slow down after awhile

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid slow down after awhile

masoud mazarei
I setup squid as traspsrent proxy with wccp configuration(layer 2).
All things work propery but after awhile it slows down. 
I try to findout whats going to happen using tcpdump 
And i know packets have delay to go out from squid machine. And I guess it may happened cause of tproxy config of linux kernel then i check linux kernel tproxy with 
And there is not delay to send packet out. All things work correctly with no delay. but when I stsrt to use squid after awhile I have delay in packet sending.
Whats your opinion? 


--
Yours Sincerely
Masoud Mazarei



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid slow down after awhile

Amos Jeffries
Administrator
On 13/10/17 22:09, masoud mazarei wrote:

> I setup squid as traspsrent proxy with wccp configuration(layer 2).
> All things work propery but after awhile it slows down.
> I try to findout whats going to happen using tcpdump
> And i know packets have delay to go out from squid machine. And I guess
> it may happened cause of tproxy config of linux kernel then i check
> linux kernel tproxy with
> https://github.com/LiamHaworth/go-tproxy
> And there is not delay to send packet out. All things work correctly
> with no delay. but when I stsrt to use squid after awhile I have delay
> in packet sending.
> Whats your opinion?
>

Insufficient data:

* Check you WCCP TCAM table size.

* Check your iptables/netfilter memory usage and table capacities.

* Check your available TCP ports.

* Check your Squid machines per-process FD limitations. As viewed by the
OS and what Squid thinks are available.

* Check for forwarding loops (set 'via on' in squid.conf).

* Check your logs for any issues that may be reported. Especially squid
cache.log and OS system log.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid slow down after awhile

masoud mazarei
TCP ESTABLISHED=54
TCP SYNC/WAIT/FIN=0

FD limitation=64000
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_window_scaling=1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 50576   64768   98152
net.core.netdev_max_backlog = 2500
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.ipv4.ip_local_port_range = 18000    65535

CISCO TCAM 10 mac learned(cache server mac,gateway mac ...)

no forward loop
no error in cache.log and syslog
iptables totaly have 10 rows in all chains. and no mem usage which affects os and machine
and i have only 1 user behind cache server.
also i have checked with many users behind cache but no differ in result.
i have same problem with squid 3.5 and squid 3.1




On Sat, Oct 14, 2017 at 12:36 PM, Amos Jeffries <[hidden email]> wrote:
On 13/10/17 22:09, masoud mazarei wrote:
I setup squid as traspsrent proxy with wccp configuration(layer 2).
All things work propery but after awhile it slows down.
I try to findout whats going to happen using tcpdump
And i know packets have delay to go out from squid machine. And I guess it may happened cause of tproxy config of linux kernel then i check linux kernel tproxy with
https://github.com/LiamHaworth/go-tproxy
And there is not delay to send packet out. All things work correctly with no delay. but when I stsrt to use squid after awhile I have delay in packet sending.
Whats your opinion?


Insufficient data:

* Check you WCCP TCAM table size.

* Check your iptables/netfilter memory usage and table capacities.

* Check your available TCP ports.

* Check your Squid machines per-process FD limitations. As viewed by the OS and what Squid thinks are available.

* Check for forwarding loops (set 'via on' in squid.conf).

* Check your logs for any issues that may be reported. Especially squid cache.log and OS system log.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users



--
Yours Sincerely
Masoud Mazarei


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid slow down after awhile

Amos Jeffries
Administrator
In reply to this post by Amos Jeffries
On 15/10/17 05:20, masoud mazarei wrote:
> i sent you an pcap file which shows the problem.
> my client ip is 172.22.127.1 and target host is  94.182.227.21.
> squid machine mac is e4:11:5b:ea:30:c2.
> filter pcap file in wireshark by filter "ip.host==94.182.227.21" you
> will see that the first SYN packet arrived in No.304 and relative time
> 6.637173 but first packet which goes out from cache machine as client by
> squid happend in No.371 and relative time 45.013691

That kind of indicates the problem is either in how long the client
takes to deliver the HTTP request to Squid, or DNS lookups to find the
destination(s).

> what is happend in (6.637173 - 45.013691) duration.?

For a transparent proxy these things have happened between SYN on
client<->Squid and SYN on Squid<->server:


* NAT/TPROXY record lookups for client connection state

* wait for the client to send its HTTP request.
  - with happy eyeballs there may be a large wait between the SYN and
first data sent by client for ~50% of connections.

* parsing of that HTTP request message.

* DNS lookup(s) for Host header verification

* http_access checks

* Adaptation hooks (ICAP / eCAP), if any

* URL re-writer lookups, if any

* HTTP 'cache' directive ACL checks

* HTTP cache lookup

* DNS lookups to find destination, if any
  - this should be very fast since the Host verify results should be
cached. But if any of the above took longer than DNS TTL new lookups may
be required - naturally increasing the delay further.

* Destination selection

* TCP server connection(s) setup
  - if you are only looking at IPv4 packets you may be missing multiple
SYN packets for IPv6 servers before the first IPv4 SYN packet appears.


The points above with sub-notes are the ones most likely to be delayed
for seconds. You may be seeing one particular source of the problem, or
multiple adding together. 45 sec seems an unusual number. Most of the
timeouts in Squid and networking are multiples of 30 sec.



> which debug level will help me to know what is happend in background?
> i enabled
> "debug_options 5,3 6,3 46,3 11,3 19,3 55,3 58,3"
> BUT there is no valuable data to solve this problem.
>

You may need an ALL,6 trace then to see if there are any clues in odd
places. As verbose as it is the debugging in Squid is far from complete
so for some of these delay issues there no specific lines to look for
mention and we have to go by relative timing of things.

The durations between actions on the list of points above should narrow
down a bit better what to look at.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users