Performance

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance

Snyder, Brian

Hello All,

 

I apologise for asking another squid performance question, but I have been banging my head against the wall for the better part of three months. Squid is installed and working. However, over time it slows down significantly. I have tried everything from turning off caching to trying to load balance several squid machines. Would someone mind giving my config a quick look.

 

Hardware:

Centos 7

32GB Ram

Xeon E5 4 cores

4x 140G SAS 15k (Cache)

2x 64G SSD (OS mirror)

10G Network connection

 

Users:

about 10k devices random web traffic

 

squid.conf scrubbed

# RADIUS Config

auth_param basic program /usr/lib64/squid/basic_radius_auth -h 10.10. -w password

auth_param basic children 2

auth_param basic realm BCS External Proxy

auth_param basic credentialsttl 24 hour

 

# ACL Lists

acl ACL_All src all

acl ACL_Password proxy_auth REQUIRED

acl ACL_SSL_Ports port 80 443 1443 7446 8443

acl ACL_Safe_Ports port "/etc/squid/ports.conf"

acl ACL_Connect method CONNECT

acl ACL_Purge method PURGE

acl ACL_Do_Not_Cache dstdomain "/etc/squid/lists.conf"

acl ACL_Query urlpath_regex cgi-bin \?

acl ACL_Deny_Url dstdomain "/etc/squid/deny.conf"

acl ACL_Allow_Url dstdomain "/etc/squid/allow.conf"

acl ACL_Web_Filter dst 10.10.18.1/32

acl ACL_Beavercreek_Clients src 10.10.0.0/16 172.16.0.0/16

acl ACL_Beavercreek_Networks dst 10.10.0.0/16 172.16.0.0/16

acl ACL_MVECA_Networks dst 10.3.0.0/16

acl ACL_Manager proto cache_object

acl ACL_Apple dstdomain .appldnld.apple.com .gspe19.ls.apple.com .init-p01md.apple.com .init-p01st.push.apple.com .init.ess.apple.com .iosapps.itunes.apple.com .mesu.apple.com .pancake.apple.com .phobos.apple.com .ocsp.apple.com

acl ACL_Apple_Dest dst 10.10.18.31/32 10.10.18.32/32 10.10.18.33/32

acl ACL_School_Hours time MTWHF 07:30-16:00

acl ACL_Block_Apps urlpath_regex -i \.ipa(\?.*)?$ \.mobileconfig(\?.*)?$ \.plist(\?.*)?$

#acl ACL_Block_IOS urlpath_regex -i

acl ACL_Full_Speed src 10.10.0.0/16 172.16.160.0/20

 

# Delay Pools

delay_pools 2

delay_class 1 2

delay_parameters 1 -1/-1 -1/-1

delay_access 1 allow ACL_Full_Speed

delay_access 1 deny ACL_All

 

delay_class 2 2

delay_parameters 2 -1/-1 200000/200000

delay_access 2 allow ACL_Apple ACL_School_Hours

delay_access 2 allow ACL_Apple_Dest ACL_School_Hours

delay_access 2 deny ACL_All

 

delay_initial_bucket_level 50

 

# Access Rules

http_access allow ACL_Web_Filter

#http_access deny ACL_Block_IOS

http_access allow ACL_Apple_Dest

http_access allow ACL_Allow_Url

http_access allow ACL_Apple

http_access deny ACL_Deny_Url

http_access deny !ACL_Safe_Ports

http_access deny ACL_Block_Apps

http_access allow ACL_Connect ACL_SSL_Ports

http_access deny ACL_Connect !ACL_SSL_Ports

http_access allow ACL_Beavercreek_Clients

http_access allow ACL_Purge ACL_Beavercreek_Clients

http_access allow ACL_Beavercreek_Clients ACL_Manager

http_access allow ACL_Password

http_access deny !ACL_Password

http_access deny ACL_All

 

#Forward

forwarded_for truncate

via on

 

#Do not cache rules

#cache deny all

always_direct allow ACL_Do_Not_Cache ACL_Apple_Dest ACL_Apple ACL_Beavercreek_Networks ACL_MVECA_Networks

cache deny ACL_Do_Not_Cache ACL_Query ACL_Apple ACL_Apple_Dest ACL_Beavercreek_Networks ACL_MVECA_Networks

 

# Network Info

http_port 10.10.:8888

http_port 10.10.:3128

 

#Worker info

workers 4

cpu_affinity_map process_numbers=1,2,3,4 cores=1,3,5,7

 

# DNS Config

dns_v4_first on

dns_nameservers 10.10. 10.10.

append_domain xxx

 

# Cache config

cache_mem 16 GB

cache_effective_user squid

cache_effective_group squid

memory_cache_mode always

memory_replacement_policy heap GDSF

cache_replacement_policy heap LFUDA

max_open_disk_fds 0

maximum_object_size_in_memory 1 MB

maximum_object_size 100 MB

 

cache_swap_high 95

cache_swap_low 90

 

max_filedesc 16384

fqdncache_size 8192

ipcache_size 8192

 

# MISC Settings

visible_hostname xxx

cache_mgr xxx

logfile_rotate 1

half_closed_clients off

dead_peer_timeout 30 second

dns_timeout 5 second

connect_timeout 30 second

shutdown_lifetime 10 second

server_persistent_connections off

authenticate_ttl 1 hour

authenticate_ip_ttl 1 hour

#ignore_expect_100 on

reply_header_max_size 128 KB

 

# Uncomment and adjust the following to add a disk cache directory.

 

cache_dir rock /var/spool/squid 20480 min-size=1 max-size=31000 max-swap-rate=100 swap-timeout=1000

 

if ${process_number} = 1

cache_dir diskd /squid/data1/aufs 51200 32 256 min-size=31001 max-size=104857600

endif

if ${process_number} = 2

cache_dir diskd /squid/data2/aufs 51200 32 256 min-size=31001 max-size=104857600

endif

if ${process_number} = 3

cache_dir diskd /squid/data3/aufs 51200 32 256 min-size=31001 max-size=104857600

endif

if ${process_number} = 4

cache_dir diskd /squid/data4/aufs 51200 32 256 min-size=31001 max-size=104857600

endif

 

logformat squid %tl.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %<a %mt

access_log stdio:/var/log/squid/access.log squid

 

# Leave coredumps in the first cache dir

coredump_dir /var/spool/squid

 

# We recommend you to use at least the following line.

#hierarchy_stoplist cgi-bin ?

 

# Add any of your own refresh_pattern entries above these.

refresh_pattern ^ftp:           1440    20%     10080

refresh_pattern ^gopher:        1440    0%      1440

refresh_pattern -i (/cgi-bin/|\?) 0     0%      0

refresh_pattern .               0       20%     4320

 

#Custom error URL

error_directory /etc/squid/custom

 

 

Info:

HTTP/1.1 200 OK

Server: squid/3.5.20

Mime-Version: 1.0

Date: Thu, 11 Jan 2018 17:04:50 GMT

Content-Type: text/plain

Expires: Thu, 11 Jan 2018 17:04:50 GMT

Last-Modified: Thu, 11 Jan 2018 17:04:50 GMT

Connection: close

 

Squid Object Cache: Version 3.5.20

Build Info:

Service Name: squid

Start Time:     Thu, 11 Jan 2018 03:26:58 GMT

Current Time:   Thu, 11 Jan 2018 17:04:50 GMT

Connection information for squid:

        Number of clients accessing cache:      35529

        Number of HTTP requests received:       4231669

        Number of ICP messages received:        0

        Number of ICP messages sent:    0

        Number of queued ICP replies:   0

        Number of HTCP messages received:       0

        Number of HTCP messages sent:   0

        Request failure ratio:   0.00

        Average HTTP requests per minute since start:   5174.0

        Average ICP messages per minute since start:    0.0

        Select loop called: 299015711 times, 12.376 ms avg

Cache information for squid:

        Hits as % of all requests:      5min: 5.4%, 60min: 3.6%

        Hits as % of bytes sent:        5min: 2.2%, 60min: 2.6%

        Memory hits as % of hit requests:       5min: 21.2%, 60min: 15.4%

        Disk hits as % of hit requests: 5min: 47.1%, 60min: 40.2%

        Storage Swap size:      60801452 KB

        Storage Swap capacity:  26.4% used, 73.6% free

        Storage Mem size:       2005632 KB

        Storage Mem capacity:   12.0% used, 88.0% free

        Mean Object Size:       119.80 KB

        Requests given to unlinkd:      0

Median Service Times (seconds)  5 min    60 min:

        HTTP Requests (All):   0.90173  0.47329   I have seen these go as high as 20 seconds

        Cache Misses:          0.04781  0.06103

        Cache Hits:            0.00000  0.00000

        Near Hits:             0.02599  0.04127

        Not-Modified Replies:  0.00000  0.00000

        DNS Lookups:           0.00019  0.00056

        ICP Queries:           0.00000  0.00000

Resource usage for squid:

        UP Time:        49072.590 seconds

        CPU Time:       4228.270 seconds

        CPU Usage:      8.62%

        CPU Usage, 5 minute avg:        10.12%

        CPU Usage, 60 minute avg:       15.49%

        Maximum Resident Size: 25868624 KB

        Page faults with physical i/o: 3

Memory accounted for:

        Total accounted:       212706 KB

        memPoolAlloc calls: 647869201

        memPoolFree calls:  656372502

File descriptor usage for squid:

        Maximum number of file descriptors:   81920

        Largest file desc currently in use:   10848

        Number of file desc currently in use: 9169

        Files queued for open:                   0

        Available number of file descriptors: 72751

        Reserved number of file descriptors:   500

        Store Disk files open:                   1

Internal Data Structures:

        162462 StoreEntries

           291 StoreEntries with MemObjects

         62657 Hot Object Cache Items

        507526 on-disk objects

 

60Min:

HTTP/1.1 200 OK

Server: squid/3.5.20

Mime-Version: 1.0

Date: Thu, 11 Jan 2018 17:06:23 GMT

Content-Type: text/plain

Expires: Thu, 11 Jan 2018 17:06:23 GMT

Last-Modified: Thu, 11 Jan 2018 17:06:23 GMT

Connection: close

 

sample_start_time = 1515686758.822127 (Thu, 11 Jan 2018 16:05:58 GMT)

sample_end_time = 1515690358.901608 (Thu, 11 Jan 2018 17:05:58 GMT)

client_http.requests = 184.169353/sec

client_http.hits = 7.716049/sec

client_http.errors = 13.524058/sec

client_http.kbytes_in = 549.705029/sec

client_http.kbytes_out = 13344.171474/sec

client_http.all_median_svc_time = 0.499589 seconds

client_http.miss_median_svc_time = 0.061028 seconds

client_http.nm_median_svc_time = 0.000000 seconds

client_http.nh_median_svc_time = 0.041120 seconds

client_http.hit_median_svc_time = 0.000000 seconds

server.all.requests = 164.990063/sec

server.all.errors = 0.000000/sec

server.all.kbytes_in = 13006.455302/sec

server.all.kbytes_out = 510.372012/sec

server.http.requests = 50.421262/sec

server.http.errors = 0.000000/sec

server.http.kbytes_in = 2179.449438/sec

server.http.kbytes_out = 69.110279/sec

server.ftp.requests = 0.000000/sec

server.ftp.errors = 0.000000/sec

server.ftp.kbytes_in = 0.000000/sec

server.ftp.kbytes_out = 0.000000/sec

server.other.requests = 114.568802/sec

server.other.errors = 0.000000/sec

server.other.kbytes_in = 10827.005308/sec

server.other.kbytes_out = 441.261456/sec

icp.pkts_sent = 0.000000/sec

icp.pkts_recv = 0.000000/sec

icp.queries_sent = 0.000000/sec

icp.replies_sent = 0.000000/sec

icp.queries_recv = 0.000000/sec

icp.replies_recv = 0.000000/sec

icp.replies_queued = 0.000000/sec

icp.query_timeouts = 0.000000/sec

icp.kbytes_sent = 0.000000/sec

icp.kbytes_recv = 0.000000/sec

icp.q_kbytes_sent = 0.000000/sec

icp.r_kbytes_sent = 0.000000/sec

icp.q_kbytes_recv = 0.000000/sec

icp.r_kbytes_recv = 0.000000/sec

icp.query_median_svc_time = 0.000000 seconds

icp.reply_median_svc_time = 0.000000 seconds

dns.median_svc_time = 0.000557 seconds

unlink.requests = 0.000000/sec

page_faults = 0.000833/sec

select_loops = 12880.416435/sec

select_fds = 9811.879014/sec

average_select_fd_period = 0.000000/fd

median_select_fds = 0.000000

swap.outs = 8.875762/sec

swap.ins = 8.034935/sec

swap.files_cleaned = 0.000000/sec

aborted_requests = 1.342211/sec

syscalls.disk.opens = 3.490250/sec

syscalls.disk.closes = 3.489972/sec

syscalls.disk.reads = 76.404351/sec

syscalls.disk.writes = 104.594693/sec

syscalls.disk.seeks = 0.000000/sec

syscalls.disk.unlinks = 0.146943/sec

syscalls.sock.accepts = 569.969450/sec

syscalls.sock.sockets = 197.239195/sec

syscalls.sock.connects = 164.998952/sec

syscalls.sock.binds = 0.000000/sec

syscalls.sock.closes = 363.784801/sec

syscalls.sock.reads = 4398.237408/sec

syscalls.sock.writes = 4622.031167/sec

syscalls.sock.recvfroms = 118.198756/sec

syscalls.sock.sendtos = 61.557552/sec

cpu_time = 558.603203 seconds

wall_time = 18000.159289 seconds

cpu_usage = 3.103324%

 

tail end of cache.log

[root@proxy ~]# tail -f /var/log/squid/cache.log

2018/01/11 12:05:39 kid3| urlParse: URL too large (12594 bytes)

2018/01/11 12:05:39 kid3| urlParse: URL too large (12602 bytes)

2018/01/11 12:05:47 kid1| fqdncacheParse: No PTR record for '91.212.150.79'

2018/01/11 12:06:10 kid3| urlParse: URL too large (12720 bytes)

2018/01/11 12:06:20 kid4| urlParse: URL too large (13122 bytes)

2018/01/11 12:06:20 kid3| urlParse: URL too large (13166 bytes)

2018/01/11 12:06:32 kid3| urlParse: URL too large (12599 bytes)

2018/01/11 12:06:34 kid3| urlParse: URL too large (12598 bytes)

2018/01/11 12:06:52 kid3| urlParse: URL too large (12724 bytes)

2018/01/11 12:06:53 kid3| urlParse: URL too large (12636 bytes)

 

I should also mention this proxy is behind a web content filter which I believe may be running proxy services. Any help would be appreciated.

 

Thanks,

Brian Snyder

 

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Performance

Alex Rousskov
On 01/11/2018 10:14 AM, Snyder, Brian wrote:

> I apologise for asking another squid performance question,

There is nothing wrong with that! It is often very difficult to solve
performance problems on the mailing list, but that does not imply folks
should not ask performance questions.


> over time it slows down significantly.

In case nobody looks for or finds problems in your configuration: What
is the bottleneck? CPU? RAM? Disk I/O? NIC interrupts?

A tool like atop may be able to answer that question for you, especially
if you let it collect stats from before Squid start to the time when the
Squid becomes very slow.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Performance

Snyder, Brian
Thank you for your reply. I have not found an issue the hardware. Atop shows everything in normal ranges. I do know the squid is about 50% faster with our filter set up as a parent vs routing through it normally. Not sure why that would be. However, when I set it up as a parent I notice quite a few connect to parent ip failed. There does not seem to be a network issue.

Thanks,
Brian

On Jan 11, 2018, at 2:23 PM, Alex Rousskov <[hidden email]<mailto:[hidden email]>> wrote:

On 01/11/2018 10:14 AM, Snyder, Brian wrote:

I apologise for asking another squid performance question,

There is nothing wrong with that! It is often very difficult to solve
performance problems on the mailing list, but that does not imply folks
should not ask performance questions.


over time it slows down significantly.

In case nobody looks for or finds problems in your configuration: What
is the bottleneck? CPU? RAM? Disk I/O? NIC interrupts?

A tool like atop may be able to answer that question for you, especially
if you let it collect stats from before Squid start to the time when the
Squid becomes very slow.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Performance

Alex Rousskov
On 01/15/2018 11:56 AM, Snyder, Brian wrote:
> I have not found an issue the hardware.

To avoid misunderstanding, I did not imply that there are
hardware-related issues. My question was about bottlenecks (i.e.,
resources that are being overused, including hardware resources like CPU
or RAM and soft resources like the number of file descriptors or
conntrack buckets).


> Atop shows everything in normal ranges.

If all resource usage is normal when the problem is apparent, then the
source of the "over time it slows down significantly" problem most
likely lies outside of Squid. An external agent (e.g., a parent proxy,
an adaptation service, a DNS server, etc.) must be delaying or dropping
messages. You can narrow down the list of suspects by investigating
Squid-reported errors and identifying Squid transaction stage(s) that
incur delays. Squid logs various transaction response times and error
details that may help in this analysis.


HTH,

Alex.


> On Jan 11, 2018, at 2:23 PM, Alex Rousskov <[hidden email]<mailto:[hidden email]>> wrote:
>
> On 01/11/2018 10:14 AM, Snyder, Brian wrote:
>
> I apologise for asking another squid performance question,
>
> There is nothing wrong with that! It is often very difficult to solve
> performance problems on the mailing list, but that does not imply folks
> should not ask performance questions.
>
>
> over time it slows down significantly.
>
> In case nobody looks for or finds problems in your configuration: What
> is the bottleneck? CPU? RAM? Disk I/O? NIC interrupts?
>
> A tool like atop may be able to answer that question for you, especially
> if you let it collect stats from before Squid start to the time when the
> Squid becomes very slow.
>
> Alex.
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users