squid restart

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

squid restart

Vieri
Hi,

Around every hour or so, the Squid proxy client experience comes to a crawl.
It takes a very long time to load a simple web page.

This is a snapshot taken when this happens:

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Sat, 31 Oct 2020 10:43:21 GMT
Content-Type: text/plain;charset=utf-8
Expires: Sat, 31 Oct 2020 10:43:21 GMT
Last-Modified: Sat, 31 Oct 2020 10:43:21 GMT
X-Cache: MISS from inf-fw1
X-Cache-Lookup: MISS from inf-fw1:3128
Connection: close

Squid Object Cache: Version 5.0.4-20201013-r6b13b73d3
Build Info:
Service Name: squid
Start Time:     Sat, 31 Oct 2020 04:50:45 GMT
Current Time:   Sat, 31 Oct 2020 10:43:21 GMT
Connection information for squid:
        Number of clients accessing cache:      320
        Number of HTTP requests received:       519828
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   1474.3
        Average ICP messages per minute since start:    0.0
        Select loop called: 9044075 times, 2.339 ms avg
Cache information for squid:
        Hits as % of all requests:      5min: 2.1%, 60min: 2.5%
        Hits as % of bytes sent:        5min: -68.9%, 60min: -402.5%
        Memory hits as % of hit requests:       5min: 78.3%, 60min: 62.3%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 1.7%
        Storage Swap size:      29040 KB
        Storage Swap capacity:  88.6% used, 11.4% free
        Storage Mem size:       29212 KB
        Storage Mem capacity:   89.1% used, 10.9% free
        Mean Object Size:       17.31 KB
        Requests given to unlinkd:      11815
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.04519  0.04776
        Cache Misses:          0.06286  0.06286
        Cache Hits:            0.00000  0.00000
        Near Hits:             0.04277  0.02317
        Not-Modified Replies:  0.00000  0.00000
        DNS Lookups:           0.00000  0.00000
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        21155.513 seconds
        CPU Time:       1334.166 seconds
        CPU Usage:      6.31%
        CPU Usage, 5 minute avg:        8.60%
        CPU Usage, 60 minute avg:       9.88%
        Maximum Resident Size: 4287872 KB
        Page faults with physical i/o: 0
Memory accounted for:
        Total accounted:       744703 KB
        memPoolAlloc calls: 136343652
        memPoolFree calls:  140190831
File descriptor usage for squid:
        Maximum number of file descriptors:   4096
        Largest file desc currently in use:   4009
        Number of file desc currently in use: 3997
        Files queued for open:                   0
        Available number of file descriptors:   99
        Reserved number of file descriptors:   100
        Store Disk files open:                   0
Internal Data Structures:
          1852 StoreEntries
          1849 StoreEntries with MemObjects
          1754 Hot Object Cache Items
          1678 on-disk objects

If I issue the '-k reconfigure' command then the user experience is "great again".

A data snapshot taken right after the latter command shows this:

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Sat, 31 Oct 2020 10:48:40 GMT
Content-Type: text/plain;charset=utf-8
Expires: Sat, 31 Oct 2020 10:48:40 GMT
Last-Modified: Sat, 31 Oct 2020 10:48:40 GMT
X-Cache: MISS from inf-fw1
X-Cache-Lookup: MISS from inf-fw1:3128
Connection: close

Squid Object Cache: Version 5.0.4-20201013-r6b13b73d3
Build Info:
Service Name: squid
Start Time:     Sat, 31 Oct 2020 10:46:51 GMT
Current Time:   Sat, 31 Oct 2020 10:48:40 GMT
Connection information for squid:
        Number of clients accessing cache:      179
        Number of HTTP requests received:       4663
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   2575.3
        Average ICP messages per minute since start:    0.0
        Select loop called: 51220 times, 2.121 ms avg
Cache information for squid:
        Hits as % of all requests:      5min: 0.3%, 60min: 0.3%
        Hits as % of bytes sent:        5min: 3.6%, 60min: 3.6%
        Memory hits as % of hit requests:       5min: 60.0%, 60min: 60.0%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
        Storage Swap size:      10292 KB
        Storage Swap capacity:  31.4% used, 68.6% free
        Storage Mem size:       10456 KB
        Storage Mem capacity:   31.9% used, 68.1% free
        Mean Object Size:       15.07 KB
        Requests given to unlinkd:      4657
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.05046  0.05046
        Cache Misses:          0.06286  0.06286
        Cache Hits:            0.00000  0.00000
        Near Hits:             0.15048  0.15048
        Not-Modified Replies:  0.00000  0.00000
        DNS Lookups:           0.00000  0.00000
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        108.639 seconds
        CPU Time:       10.588 seconds
        CPU Usage:      9.75%
        CPU Usage, 5 minute avg:        12.90%
        CPU Usage, 60 minute avg:       12.90%
        Maximum Resident Size: 462736 KB
        Page faults with physical i/o: 0
Memory accounted for:
        Total accounted:        37879 KB
        memPoolAlloc calls:   1256976
        memPoolFree calls:    1307898
File descriptor usage for squid:
        Maximum number of file descriptors:   4096
        Largest file desc currently in use:    567
        Number of file desc currently in use:  559
        Files queued for open:                   0
        Available number of file descriptors: 3537
        Reserved number of file descriptors:   100
        Store Disk files open:                   0
Internal Data Structures:
           997 StoreEntries
           997 StoreEntries with MemObjects
           683 Hot Object Cache Items
           683 on-disk objects

This did not happen with Squid 4, or maybe it wasn't as obvious.


I guess the reason could be for this:

        Maximum number of file descriptors:   4096
        Largest file desc currently in use:   4009
        Number of file desc currently in use: 3997

However, I set the following directive in squid.conf:

max_filedescriptors 65536

It doesn't seem to be honored here unless I stop and restart the squid service again (/etc/init.d/squid restart from command line):

File descriptor usage for squid:
        Maximum number of file descriptors:   65535

It seems that if I run the same command (/etc/init.d/squid restart) from crontab, that ulimit is not honored. I guess that's the root cause of my issue because I am asking cron to restart Squid once daily. I'll try not to, but I was hoping to see if there was a reliable way to fully restart the Squid process.

Vieri



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid restart

Amos Jeffries
Administrator
On 1/11/20 12:02 am, Vieri wrote:
> Hi,
>
> Around every hour or so, the Squid proxy client experience comes to a crawl.
> It takes a very long time to load a simple web page.

...

>
> I guess the reason could be for this:
>
>          Maximum number of file descriptors:   4096
>          Largest file desc currently in use:   4009
>          Number of file desc currently in use: 3997
>

Yes, that looks like it.


> However, I set the following directive in squid.conf:
>
> max_filedescriptors 65536
>

Are you using systemd, SysV or another init ?

systemd default limits has this as a known issue for initial startup.


> It doesn't seem to be honored here unless I stop and restart the squid service again (/etc/init.d/squid restart from command line):
>
> File descriptor usage for squid:
>          Maximum number of file descriptors:   65535
>
> It seems that if I run the same command (/etc/init.d/squid restart) from crontab, that ulimit is not honored. I guess that's the root cause of my issue because I am asking cron to restart Squid once daily. I'll try not to, but I was hoping to see if there was a reliable way to fully restart the Squid process.
>
> Vieri
>

The init system restart command is the preferred one - it handles any
system details that need updating. Alternatively, "squid -k restart" can
be used.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid restart

Vieri


On Saturday, October 31, 2020, 4:08:23 PM GMT+1, Amos Jeffries <[hidden email]> wrote:

>> However, I set the following directive in squid.conf:
>>
>> max_filedescriptors 65536
>>
> Are you using systemd, SysV or another init ?

I'm using SysV on Gentoo Linux.

> It doesn't seem to be honored here unless I stop and restart the squid service again (/etc/init.d/squid restart from command line):
>
> File descriptor usage for squid:
>          Maximum number of file descriptors:   65535
>
> It seems that if I run the same command (/etc/init.d/squid restart) from crontab, that ulimit is not honored. I guess that's the root cause of my issue because I am asking cron to restart Squid once daily. I'll try not to, but I was hoping to see if there was a reliable way to fully restart the Squid process.
>
> Vieri

>

The init system restart command is the preferred one - it handles any
system details that need updating. Alternatively, "squid -k restart" can
be used.

The SysV init script works fine when run from command line or at boot time (and probably from a custom inittab script -- cannot confirm it yet). The problem shows up when running it from cron (I have cronie-1.5.4).
I'll take a look at the '-k restart' alternative.

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid restart

Eliezer Croitoru-3
In reply to this post by Vieri
Hey Vieri,

FD and other limits are a tricky thing in gentoo and couple other systems.
In ubuntu for example to apply a ulimit for squid what they did is add a ulimit command at the beginning of the init script.
Ie 'ulimit -hn 65535;ulimit -sn 65535'

In other init or startup systems like systemd which are not started from a restricted shell like bash you can define these settings.

P.S. You should not restart squid because of a FD limit from a crontab job....
If a specific user abuses the service you should try create some firewall rules to limit clients requests bursts.

All The Best,
Elieser

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]

-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Vieri
Sent: Saturday, October 31, 2020 1:03 PM
To: Squid Users <[hidden email]>
Subject: [squid-users] squid restart

Hi,

Around every hour or so, the Squid proxy client experience comes to a crawl.
It takes a very long time to load a simple web page.

This is a snapshot taken when this happens:

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Sat, 31 Oct 2020 10:43:21 GMT
Content-Type: text/plain;charset=utf-8
Expires: Sat, 31 Oct 2020 10:43:21 GMT
Last-Modified: Sat, 31 Oct 2020 10:43:21 GMT
X-Cache: MISS from inf-fw1
X-Cache-Lookup: MISS from inf-fw1:3128
Connection: close

Squid Object Cache: Version 5.0.4-20201013-r6b13b73d3
Build Info:
Service Name: squid
Start Time:     Sat, 31 Oct 2020 04:50:45 GMT
Current Time:   Sat, 31 Oct 2020 10:43:21 GMT
Connection information for squid:
        Number of clients accessing cache:      320
        Number of HTTP requests received:       519828
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   1474.3
        Average ICP messages per minute since start:    0.0
        Select loop called: 9044075 times, 2.339 ms avg
Cache information for squid:
        Hits as % of all requests:      5min: 2.1%, 60min: 2.5%
        Hits as % of bytes sent:        5min: -68.9%, 60min: -402.5%
        Memory hits as % of hit requests:       5min: 78.3%, 60min: 62.3%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 1.7%
        Storage Swap size:      29040 KB
        Storage Swap capacity:  88.6% used, 11.4% free
        Storage Mem size:       29212 KB
        Storage Mem capacity:   89.1% used, 10.9% free
        Mean Object Size:       17.31 KB
        Requests given to unlinkd:      11815
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.04519  0.04776
        Cache Misses:          0.06286  0.06286
        Cache Hits:            0.00000  0.00000
        Near Hits:             0.04277  0.02317
        Not-Modified Replies:  0.00000  0.00000
        DNS Lookups:           0.00000  0.00000
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        21155.513 seconds
        CPU Time:       1334.166 seconds
        CPU Usage:      6.31%
        CPU Usage, 5 minute avg:        8.60%
        CPU Usage, 60 minute avg:       9.88%
        Maximum Resident Size: 4287872 KB
        Page faults with physical i/o: 0
Memory accounted for:
        Total accounted:       744703 KB
        memPoolAlloc calls: 136343652
        memPoolFree calls:  140190831
File descriptor usage for squid:
        Maximum number of file descriptors:   4096
        Largest file desc currently in use:   4009
        Number of file desc currently in use: 3997
        Files queued for open:                   0
        Available number of file descriptors:   99
        Reserved number of file descriptors:   100
        Store Disk files open:                   0
Internal Data Structures:
          1852 StoreEntries
          1849 StoreEntries with MemObjects
          1754 Hot Object Cache Items
          1678 on-disk objects

If I issue the '-k reconfigure' command then the user experience is "great again".

A data snapshot taken right after the latter command shows this:

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Sat, 31 Oct 2020 10:48:40 GMT
Content-Type: text/plain;charset=utf-8
Expires: Sat, 31 Oct 2020 10:48:40 GMT
Last-Modified: Sat, 31 Oct 2020 10:48:40 GMT
X-Cache: MISS from inf-fw1
X-Cache-Lookup: MISS from inf-fw1:3128
Connection: close

Squid Object Cache: Version 5.0.4-20201013-r6b13b73d3
Build Info:
Service Name: squid
Start Time:     Sat, 31 Oct 2020 10:46:51 GMT
Current Time:   Sat, 31 Oct 2020 10:48:40 GMT
Connection information for squid:
        Number of clients accessing cache:      179
        Number of HTTP requests received:       4663
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   2575.3
        Average ICP messages per minute since start:    0.0
        Select loop called: 51220 times, 2.121 ms avg
Cache information for squid:
        Hits as % of all requests:      5min: 0.3%, 60min: 0.3%
        Hits as % of bytes sent:        5min: 3.6%, 60min: 3.6%
        Memory hits as % of hit requests:       5min: 60.0%, 60min: 60.0%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
        Storage Swap size:      10292 KB
        Storage Swap capacity:  31.4% used, 68.6% free
        Storage Mem size:       10456 KB
        Storage Mem capacity:   31.9% used, 68.1% free
        Mean Object Size:       15.07 KB
        Requests given to unlinkd:      4657
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.05046  0.05046
        Cache Misses:          0.06286  0.06286
        Cache Hits:            0.00000  0.00000
        Near Hits:             0.15048  0.15048
        Not-Modified Replies:  0.00000  0.00000
        DNS Lookups:           0.00000  0.00000
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        108.639 seconds
        CPU Time:       10.588 seconds
        CPU Usage:      9.75%
        CPU Usage, 5 minute avg:        12.90%
        CPU Usage, 60 minute avg:       12.90%
        Maximum Resident Size: 462736 KB
        Page faults with physical i/o: 0
Memory accounted for:
        Total accounted:        37879 KB
        memPoolAlloc calls:   1256976
        memPoolFree calls:    1307898
File descriptor usage for squid:
        Maximum number of file descriptors:   4096
        Largest file desc currently in use:    567
        Number of file desc currently in use:  559
        Files queued for open:                   0
        Available number of file descriptors: 3537
        Reserved number of file descriptors:   100
        Store Disk files open:                   0
Internal Data Structures:
           997 StoreEntries
           997 StoreEntries with MemObjects
           683 Hot Object Cache Items
           683 on-disk objects

This did not happen with Squid 4, or maybe it wasn't as obvious.


I guess the reason could be for this:

        Maximum number of file descriptors:   4096
        Largest file desc currently in use:   4009
        Number of file desc currently in use: 3997

However, I set the following directive in squid.conf:

max_filedescriptors 65536

It doesn't seem to be honored here unless I stop and restart the squid service again (/etc/init.d/squid restart from command line):

File descriptor usage for squid:
        Maximum number of file descriptors:   65535

It seems that if I run the same command (/etc/init.d/squid restart) from crontab, that ulimit is not honored. I guess that's the root cause of my issue because I am asking cron to restart Squid once daily. I'll try not to, but I was hoping to see if there was a reliable way to fully restart the Squid process.

Vieri



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users