High memory usage under load with caching disabled, memory is not being freed even with no load

Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

High memory usage under load with caching disabled, memory is not being freed even with no load

Ivan Bulatovic
Hi all,

I am trying to configure squid to run as a forward proxy with no
caching (cache deny all) with an option to choose the outgoing IP
address based on the username. So all squid has to do is to use a
certain outgoing IP address for a certain user, return the data from
the server to that user and cache nothing.

For that I created a special authentication helper and used the ACLs
and tcp_outgoing_address to create a lot of users and outgoing IP
addresses (about 260 at the moment). Example (not the real IP I use,
of course):

acl use_IP1 proxy_auth user1
tcp_outgoing_address 1.2.3.4   use_IP1

I also configured the squid to use 4 workers, but this happens even
when I use only one worker (default)

And this works. However, under heavy load, Squid eats all of the RAM
and then starts going to swap. And the memory usage does not drop when
I remove all the load from squid (I shut down all clients).

I left it to see if the memory will be freed but even after leaving it
for an hour the info page reports this:
Cache information for squid:
        Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
        Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
        Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
        Storage Swap size:      0 KB
        Storage Swap capacity:   0.0% used, 100.0% free
        Storage Mem size:       0 KB
        Storage Mem capacity:    0.0% used, 100.0% free
        Mean Object Size:       0.00 KB
        Requests given to unlinkd:      0

Resource usage for squid:
        UP Time:        255334.875 seconds
        CPU Time:       7122.436 seconds
        CPU Usage:      2.79%
        CPU Usage, 5 minute avg:        0.05%
        CPU Usage, 60 minute avg:       37.66%
        Maximum Resident Size: 41500720 KB
        Page faults with physical i/o: 1003410

And here is the listing of free and top commands (with no load on the server):

# free -h
              total        used        free      shared  buff/cache   available
Mem:            11G         10G        791M        676K        491M        1.0G
Swap:           11G        5.5G        6.5G

# top
top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 91.2/12251688
[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
        ]
KiB Swap: 45.8/12582904
[||||||||||||||||||||||||||||||||||||||||||||||
                              ]

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
  7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
  7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
  7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
  1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
  1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
   474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
systemd-journal
  7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
  1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
 43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
 43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
 43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
 43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
 48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
 48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
     1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
...
... lines ommited
...

In the attachment you can find the printout from squidclient mgr:info
and squidclient mgr:mem. These are both taken at the moment when there
is no more load on the proxy. I also included my squid.conf file
(minus the two files where acls are defined and outgoing IP addresses,
these two contain only acl and tcp_outgoing_address lines as in the
example above).

Machine info:
OS: Ubuntu 18.04 with latest updates
Squid version 4.12 (from diladele repository)
Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM

I can not understand what is eating all of the memory, if I disabled the cache.

Maybe I configured something wrong but I can not find what.

Thank you for any help you can provide.

Best regards,
Ivan

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

mem.txt (66K) Download Attachment
squid.conf (5K) Download Attachment
info.txt (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Ivan Bulatovic
Hi all,

I tried with stock squid from Ubuntu 18.04 (version 4.10) and basic
config, but still no luck. Here is the config I tried:
-----------------------------------------
acl localnet src 10.20.0.0/16           # My network
acl localnet src fc00::/7               # RFC 4193 local private network range
acl localnet src fe80::/10              # RFC 4291 link-local
(directly plugged) machines

acl SSL_ports port 443

acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http

acl CONNECT method CONNECT

cache deny all

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

http_access allow localhost manager
http_access allow localnet manager
http_access deny manager
http_access deny to_localhost
http_access allow localnet
http_access allow localhost
http_access deny all

http_port localhost:3128
http_port 10.20.6.8:50505

coredump_dir /var/spool/squid

shutdown_lifetime 2 seconds
max_filedescriptors 262143

server_persistent_connections off
client_persistent_connections off

request_header_access Via deny all
request_header_access X-Forwarded-For deny all
dns_v4_first on
------------------------------------

Looks like squid has some serious memory issues when under heavy load
(90 servers that crawl Internet sites). It just eats up memory, and
does not free it up even days after it is being used (with no load on
the proxy for days). So I guess I have to look for another solution.

Best regards,
Ivan Bulatovic


On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic
<[hidden email]> wrote:

>
> Hi all,
>
> I am trying to configure squid to run as a forward proxy with no
> caching (cache deny all) with an option to choose the outgoing IP
> address based on the username. So all squid has to do is to use a
> certain outgoing IP address for a certain user, return the data from
> the server to that user and cache nothing.
>
> For that I created a special authentication helper and used the ACLs
> and tcp_outgoing_address to create a lot of users and outgoing IP
> addresses (about 260 at the moment). Example (not the real IP I use,
> of course):
>
> acl use_IP1 proxy_auth user1
> tcp_outgoing_address 1.2.3.4   use_IP1
>
> I also configured the squid to use 4 workers, but this happens even
> when I use only one worker (default)
>
> And this works. However, under heavy load, Squid eats all of the RAM
> and then starts going to swap. And the memory usage does not drop when
> I remove all the load from squid (I shut down all clients).
>
> I left it to see if the memory will be freed but even after leaving it
> for an hour the info page reports this:
> Cache information for squid:
>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
>         Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
>         Storage Swap size:      0 KB
>         Storage Swap capacity:   0.0% used, 100.0% free
>         Storage Mem size:       0 KB
>         Storage Mem capacity:    0.0% used, 100.0% free
>         Mean Object Size:       0.00 KB
>         Requests given to unlinkd:      0
>
> Resource usage for squid:
>         UP Time:        255334.875 seconds
>         CPU Time:       7122.436 seconds
>         CPU Usage:      2.79%
>         CPU Usage, 5 minute avg:        0.05%
>         CPU Usage, 60 minute avg:       37.66%
>         Maximum Resident Size: 41500720 KB
>         Page faults with physical i/o: 1003410
>
> And here is the listing of free and top commands (with no load on the server):
>
> # free -h
>               total        used        free      shared  buff/cache   available
> Mem:            11G         10G        791M        676K        491M        1.0G
> Swap:           11G        5.5G        6.5G
>
> # top
> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> KiB Mem : 91.2/12251688
> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>         ]
> KiB Swap: 45.8/12582904
> [||||||||||||||||||||||||||||||||||||||||||||||
>                               ]
>
>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
>    474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
> systemd-journal
>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
>  48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
>      1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
> ...
> ... lines ommited
> ...
>
> In the attachment you can find the printout from squidclient mgr:info
> and squidclient mgr:mem. These are both taken at the moment when there
> is no more load on the proxy. I also included my squid.conf file
> (minus the two files where acls are defined and outgoing IP addresses,
> these two contain only acl and tcp_outgoing_address lines as in the
> example above).
>
> Machine info:
> OS: Ubuntu 18.04 with latest updates
> Squid version 4.12 (from diladele repository)
> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>
> I can not understand what is eating all of the memory, if I disabled the cache.
>
> Maybe I configured something wrong but I can not find what.
>
> Thank you for any help you can provide.
>
> Best regards,
> Ivan
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Alex Rousskov
On 8/3/20 9:11 AM, Ivan Bulatovic wrote:

> Looks like squid has some serious memory issues when under heavy load
> (90 servers that crawl Internet sites).

>         Maximum Resident Size: 41500720 KB

If the above (unreliable) report matches your observations using system
tools like "top", then it is indeed likely that your Squid is suffering
from a memory leak -- 41GB is usually too much for most non-caching
Squid instances.

Identifying the leak may take some time, and I am not volunteering to do
the necessary legwork personally, but the Squid Project does fix
virtually all runtime leaks that we know about. If you want to speed up
the process, one of the best things you can do is to run Squid under
valgrind with a good suppression file. This requires building Squid with
a special ./configure option. Several testing iterations may be
necessary. If you are willing to do this, please file a bug report and
somebody will guide you through the steps.


> It just eats up memory, and
> does not free it up even days after it is being used (with no load on
> the proxy for days).

Some memory retention is expected by default. See
http://www.squid-cache.org/Doc/config/memory_pools/

Unfortunately, AFAICT, your mgr:mem output does not show any obvious
leaks -- all numbers are very small. If something is leaking a lot, then
it is probably not pooled by Squid.


HTH,

Alex.


> On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
>>
>> Hi all,
>>
>> I am trying to configure squid to run as a forward proxy with no
>> caching (cache deny all) with an option to choose the outgoing IP
>> address based on the username. So all squid has to do is to use a
>> certain outgoing IP address for a certain user, return the data from
>> the server to that user and cache nothing.
>>
>> For that I created a special authentication helper and used the ACLs
>> and tcp_outgoing_address to create a lot of users and outgoing IP
>> addresses (about 260 at the moment). Example (not the real IP I use,
>> of course):
>>
>> acl use_IP1 proxy_auth user1
>> tcp_outgoing_address 1.2.3.4   use_IP1
>>
>> I also configured the squid to use 4 workers, but this happens even
>> when I use only one worker (default)
>>
>> And this works. However, under heavy load, Squid eats all of the RAM
>> and then starts going to swap. And the memory usage does not drop when
>> I remove all the load from squid (I shut down all clients).
>>
>> I left it to see if the memory will be freed but even after leaving it
>> for an hour the info page reports this:
>> Cache information for squid:
>>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
>>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
>>         Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
>>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
>>         Storage Swap size:      0 KB
>>         Storage Swap capacity:   0.0% used, 100.0% free
>>         Storage Mem size:       0 KB
>>         Storage Mem capacity:    0.0% used, 100.0% free
>>         Mean Object Size:       0.00 KB
>>         Requests given to unlinkd:      0
>>
>> Resource usage for squid:
>>         UP Time:        255334.875 seconds
>>         CPU Time:       7122.436 seconds
>>         CPU Usage:      2.79%
>>         CPU Usage, 5 minute avg:        0.05%
>>         CPU Usage, 60 minute avg:       37.66%
>>         Maximum Resident Size: 41500720 KB
>>         Page faults with physical i/o: 1003410
>>
>> And here is the listing of free and top commands (with no load on the server):
>>
>> # free -h
>>               total        used        free      shared  buff/cache   available
>> Mem:            11G         10G        791M        676K        491M        1.0G
>> Swap:           11G        5.5G        6.5G
>>
>> # top
>> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
>> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
>> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> KiB Mem : 91.2/12251688
>> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>>         ]
>> KiB Swap: 45.8/12582904
>> [||||||||||||||||||||||||||||||||||||||||||||||
>>                               ]
>>
>>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
>>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
>>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
>>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
>>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
>>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
>>    474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
>> systemd-journal
>>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
>>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
>>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
>>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
>>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
>>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
>>  48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
>>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
>>      1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
>> ...
>> ... lines ommited
>> ...
>>
>> In the attachment you can find the printout from squidclient mgr:info
>> and squidclient mgr:mem. These are both taken at the moment when there
>> is no more load on the proxy. I also included my squid.conf file
>> (minus the two files where acls are defined and outgoing IP addresses,
>> these two contain only acl and tcp_outgoing_address lines as in the
>> example above).
>>
>> Machine info:
>> OS: Ubuntu 18.04 with latest updates
>> Squid version 4.12 (from diladele repository)
>> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>>
>> I can not understand what is eating all of the memory, if I disabled the cache.
>>
>> Maybe I configured something wrong but I can not find what.
>>
>> Thank you for any help you can provide.
>>
>> Best regards,
>> Ivan
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Ivan Bulatovic
Hi Alex,

Thank you very much for your help.

I opened a bug on bugs.squid-cache.org
(https://bugs.squid-cache.org/show_bug.cgi?id=5071).

Best regards,
Ivan

On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
<[hidden email]> wrote:

>
> On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
>
> > Looks like squid has some serious memory issues when under heavy load
> > (90 servers that crawl Internet sites).
>
> >         Maximum Resident Size: 41500720 KB
>
> If the above (unreliable) report matches your observations using system
> tools like "top", then it is indeed likely that your Squid is suffering
> from a memory leak -- 41GB is usually too much for most non-caching
> Squid instances.
>
> Identifying the leak may take some time, and I am not volunteering to do
> the necessary legwork personally, but the Squid Project does fix
> virtually all runtime leaks that we know about. If you want to speed up
> the process, one of the best things you can do is to run Squid under
> valgrind with a good suppression file. This requires building Squid with
> a special ./configure option. Several testing iterations may be
> necessary. If you are willing to do this, please file a bug report and
> somebody will guide you through the steps.
>
>
> > It just eats up memory, and
> > does not free it up even days after it is being used (with no load on
> > the proxy for days).
>
> Some memory retention is expected by default. See
> http://www.squid-cache.org/Doc/config/memory_pools/
>
> Unfortunately, AFAICT, your mgr:mem output does not show any obvious
> leaks -- all numbers are very small. If something is leaking a lot, then
> it is probably not pooled by Squid.
>
>
> HTH,
>
> Alex.
>
>
> > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
> >>
> >> Hi all,
> >>
> >> I am trying to configure squid to run as a forward proxy with no
> >> caching (cache deny all) with an option to choose the outgoing IP
> >> address based on the username. So all squid has to do is to use a
> >> certain outgoing IP address for a certain user, return the data from
> >> the server to that user and cache nothing.
> >>
> >> For that I created a special authentication helper and used the ACLs
> >> and tcp_outgoing_address to create a lot of users and outgoing IP
> >> addresses (about 260 at the moment). Example (not the real IP I use,
> >> of course):
> >>
> >> acl use_IP1 proxy_auth user1
> >> tcp_outgoing_address 1.2.3.4   use_IP1
> >>
> >> I also configured the squid to use 4 workers, but this happens even
> >> when I use only one worker (default)
> >>
> >> And this works. However, under heavy load, Squid eats all of the RAM
> >> and then starts going to swap. And the memory usage does not drop when
> >> I remove all the load from squid (I shut down all clients).
> >>
> >> I left it to see if the memory will be freed but even after leaving it
> >> for an hour the info page reports this:
> >> Cache information for squid:
> >>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
> >>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
> >>         Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
> >>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
> >>         Storage Swap size:      0 KB
> >>         Storage Swap capacity:   0.0% used, 100.0% free
> >>         Storage Mem size:       0 KB
> >>         Storage Mem capacity:    0.0% used, 100.0% free
> >>         Mean Object Size:       0.00 KB
> >>         Requests given to unlinkd:      0
> >>
> >> Resource usage for squid:
> >>         UP Time:        255334.875 seconds
> >>         CPU Time:       7122.436 seconds
> >>         CPU Usage:      2.79%
> >>         CPU Usage, 5 minute avg:        0.05%
> >>         CPU Usage, 60 minute avg:       37.66%
> >>         Maximum Resident Size: 41500720 KB
> >>         Page faults with physical i/o: 1003410
> >>
> >> And here is the listing of free and top commands (with no load on the server):
> >>
> >> # free -h
> >>               total        used        free      shared  buff/cache   available
> >> Mem:            11G         10G        791M        676K        491M        1.0G
> >> Swap:           11G        5.5G        6.5G
> >>
> >> # top
> >> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
> >> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
> >> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> KiB Mem : 91.2/12251688
> >> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> >>         ]
> >> KiB Swap: 45.8/12582904
> >> [||||||||||||||||||||||||||||||||||||||||||||||
> >>                               ]
> >>
> >>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> >>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
> >>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
> >>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
> >>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
> >>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
> >>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
> >>    474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
> >> systemd-journal
> >>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
> >>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
> >>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
> >>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
> >>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
> >>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
> >>  48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
> >>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
> >>      1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
> >> ...
> >> ... lines ommited
> >> ...
> >>
> >> In the attachment you can find the printout from squidclient mgr:info
> >> and squidclient mgr:mem. These are both taken at the moment when there
> >> is no more load on the proxy. I also included my squid.conf file
> >> (minus the two files where acls are defined and outgoing IP addresses,
> >> these two contain only acl and tcp_outgoing_address lines as in the
> >> example above).
> >>
> >> Machine info:
> >> OS: Ubuntu 18.04 with latest updates
> >> Squid version 4.12 (from diladele repository)
> >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
> >>
> >> I can not understand what is eating all of the memory, if I disabled the cache.
> >>
> >> Maybe I configured something wrong but I can not find what.
> >>
> >> Thank you for any help you can provide.
> >>
> >> Best regards,
> >> Ivan
> > _______________________________________________
> > squid-users mailing list
> > [hidden email]
> > http://lists.squid-cache.org/listinfo/squid-users
> >
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Eliezer Croitoru-3
I think that the mgr:info or another page there contains the amount of requests per second etc.
also netstat or ss -ntp might give some basic understanding about this server size.

are you using dynamic memory on the hyper-v hypervisor?

Eliezer

On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <[hidden email]> wrote:
Hi Alex,

Thank you very much for your help.

I opened a bug on bugs.squid-cache.org
(https://bugs.squid-cache.org/show_bug.cgi?id=5071).

Best regards,
Ivan

On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
<[hidden email]> wrote:
>
> On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
>
> > Looks like squid has some serious memory issues when under heavy load
> > (90 servers that crawl Internet sites).
>
> >         Maximum Resident Size: 41500720 KB
>
> If the above (unreliable) report matches your observations using system
> tools like "top", then it is indeed likely that your Squid is suffering
> from a memory leak -- 41GB is usually too much for most non-caching
> Squid instances.
>
> Identifying the leak may take some time, and I am not volunteering to do
> the necessary legwork personally, but the Squid Project does fix
> virtually all runtime leaks that we know about. If you want to speed up
> the process, one of the best things you can do is to run Squid under
> valgrind with a good suppression file. This requires building Squid with
> a special ./configure option. Several testing iterations may be
> necessary. If you are willing to do this, please file a bug report and
> somebody will guide you through the steps.
>
>
> > It just eats up memory, and
> > does not free it up even days after it is being used (with no load on
> > the proxy for days).
>
> Some memory retention is expected by default. See
> http://www.squid-cache.org/Doc/config/memory_pools/
>
> Unfortunately, AFAICT, your mgr:mem output does not show any obvious
> leaks -- all numbers are very small. If something is leaking a lot, then
> it is probably not pooled by Squid.
>
>
> HTH,
>
> Alex.
>
>
> > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
> >>
> >> Hi all,
> >>
> >> I am trying to configure squid to run as a forward proxy with no
> >> caching (cache deny all) with an option to choose the outgoing IP
> >> address based on the username. So all squid has to do is to use a
> >> certain outgoing IP address for a certain user, return the data from
> >> the server to that user and cache nothing.
> >>
> >> For that I created a special authentication helper and used the ACLs
> >> and tcp_outgoing_address to create a lot of users and outgoing IP
> >> addresses (about 260 at the moment). Example (not the real IP I use,
> >> of course):
> >>
> >> acl use_IP1 proxy_auth user1
> >> tcp_outgoing_address 1.2.3.4   use_IP1
> >>
> >> I also configured the squid to use 4 workers, but this happens even
> >> when I use only one worker (default)
> >>
> >> And this works. However, under heavy load, Squid eats all of the RAM
> >> and then starts going to swap. And the memory usage does not drop when
> >> I remove all the load from squid (I shut down all clients).
> >>
> >> I left it to see if the memory will be freed but even after leaving it
> >> for an hour the info page reports this:
> >> Cache information for squid:
> >>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
> >>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
> >>         Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
> >>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
> >>         Storage Swap size:      0 KB
> >>         Storage Swap capacity:   0.0% used, 100.0% free
> >>         Storage Mem size:       0 KB
> >>         Storage Mem capacity:    0.0% used, 100.0% free
> >>         Mean Object Size:       0.00 KB
> >>         Requests given to unlinkd:      0
> >>
> >> Resource usage for squid:
> >>         UP Time:        255334.875 seconds
> >>         CPU Time:       7122.436 seconds
> >>         CPU Usage:      2.79%
> >>         CPU Usage, 5 minute avg:        0.05%
> >>         CPU Usage, 60 minute avg:       37.66%
> >>         Maximum Resident Size: 41500720 KB
> >>         Page faults with physical i/o: 1003410
> >>
> >> And here is the listing of free and top commands (with no load on the server):
> >>
> >> # free -h
> >>               total        used        free      shared  buff/cache   available
> >> Mem:            11G         10G        791M        676K        491M        1.0G
> >> Swap:           11G        5.5G        6.5G
> >>
> >> # top
> >> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
> >> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
> >> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >> KiB Mem : 91.2/12251688
> >> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> >>         ]
> >> KiB Swap: 45.8/12582904
> >> [||||||||||||||||||||||||||||||||||||||||||||||
> >>                               ]
> >>
> >>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> >>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
> >>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
> >>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
> >>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
> >>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
> >>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
> >>    474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
> >> systemd-journal
> >>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
> >>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
> >>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
> >>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
> >>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
> >>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
> >>  48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
> >>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
> >>      1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
> >> ...
> >> ... lines ommited
> >> ...
> >>
> >> In the attachment you can find the printout from squidclient mgr:info
> >> and squidclient mgr:mem. These are both taken at the moment when there
> >> is no more load on the proxy. I also included my squid.conf file
> >> (minus the two files where acls are defined and outgoing IP addresses,
> >> these two contain only acl and tcp_outgoing_address lines as in the
> >> example above).
> >>
> >> Machine info:
> >> OS: Ubuntu 18.04 with latest updates
> >> Squid version 4.12 (from diladele repository)
> >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
> >>
> >> I can not understand what is eating all of the memory, if I disabled the cache.
> >>
> >> Maybe I configured something wrong but I can not find what.
> >>
> >> Thank you for any help you can provide.
> >>
> >> Best regards,
> >> Ivan
> > _______________________________________________
> > squid-users mailing list
> > [hidden email]
> > http://lists.squid-cache.org/listinfo/squid-users
> >
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Ivan Bulatovic
Hi Eliezer,

In the original message I sent to the squid-users mail list, I
attached listings from mgr:info and mgr:mem. The server is definitely
using a lot of connections (close to 200K connections), which is why I
increased the open files limits in linux as well as in squid.conf. And
there are a lot of requests per second, when the server is running. I
could understand that it needs a lot of memory for in-transit cache,
but that memory should be later released  back to OS, once the
requests load goes down. However, that is not the case, even days
after there is no load on the server, it stays at 11GB of RAM and 5.5
GB of swap used. The second I restart the squid process, everything
goes to normal, memory is released. That is why I suspect there is
some memory leak somewhere.

The server is a Ubuntu 18.04 LTS VM (running on Hyper-V 2019 server),
with 8 virtual processors and 12GB of RAM (although I can increase
that if that is the problem, but I thought that without caching this
would be more than enough).

I am not using dynamic memory on Hyper-V (it is turned off for this VM).

Best regards,
Ivan

On Wed, Aug 5, 2020 at 7:14 PM NgTech LTD <[hidden email]> wrote:

>
> I think that the mgr:info or another page there contains the amount of requests per second etc.
> also netstat or ss -ntp might give some basic understanding about this server size.
>
> are you using dynamic memory on the hyper-v hypervisor?
>
> Eliezer
>
> On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <[hidden email]> wrote:
>>
>> Hi Alex,
>>
>> Thank you very much for your help.
>>
>> I opened a bug on bugs.squid-cache.org
>> (https://bugs.squid-cache.org/show_bug.cgi?id=5071).
>>
>> Best regards,
>> Ivan
>>
>> On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
>> <[hidden email]> wrote:
>> >
>> > On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
>> >
>> > > Looks like squid has some serious memory issues when under heavy load
>> > > (90 servers that crawl Internet sites).
>> >
>> > >         Maximum Resident Size: 41500720 KB
>> >
>> > If the above (unreliable) report matches your observations using system
>> > tools like "top", then it is indeed likely that your Squid is suffering
>> > from a memory leak -- 41GB is usually too much for most non-caching
>> > Squid instances.
>> >
>> > Identifying the leak may take some time, and I am not volunteering to do
>> > the necessary legwork personally, but the Squid Project does fix
>> > virtually all runtime leaks that we know about. If you want to speed up
>> > the process, one of the best things you can do is to run Squid under
>> > valgrind with a good suppression file. This requires building Squid with
>> > a special ./configure option. Several testing iterations may be
>> > necessary. If you are willing to do this, please file a bug report and
>> > somebody will guide you through the steps.
>> >
>> >
>> > > It just eats up memory, and
>> > > does not free it up even days after it is being used (with no load on
>> > > the proxy for days).
>> >
>> > Some memory retention is expected by default. See
>> > http://www.squid-cache.org/Doc/config/memory_pools/
>> >
>> > Unfortunately, AFAICT, your mgr:mem output does not show any obvious
>> > leaks -- all numbers are very small. If something is leaking a lot, then
>> > it is probably not pooled by Squid.
>> >
>> >
>> > HTH,
>> >
>> > Alex.
>> >
>> >
>> > > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
>> > >>
>> > >> Hi all,
>> > >>
>> > >> I am trying to configure squid to run as a forward proxy with no
>> > >> caching (cache deny all) with an option to choose the outgoing IP
>> > >> address based on the username. So all squid has to do is to use a
>> > >> certain outgoing IP address for a certain user, return the data from
>> > >> the server to that user and cache nothing.
>> > >>
>> > >> For that I created a special authentication helper and used the ACLs
>> > >> and tcp_outgoing_address to create a lot of users and outgoing IP
>> > >> addresses (about 260 at the moment). Example (not the real IP I use,
>> > >> of course):
>> > >>
>> > >> acl use_IP1 proxy_auth user1
>> > >> tcp_outgoing_address 1.2.3.4   use_IP1
>> > >>
>> > >> I also configured the squid to use 4 workers, but this happens even
>> > >> when I use only one worker (default)
>> > >>
>> > >> And this works. However, under heavy load, Squid eats all of the RAM
>> > >> and then starts going to swap. And the memory usage does not drop when
>> > >> I remove all the load from squid (I shut down all clients).
>> > >>
>> > >> I left it to see if the memory will be freed but even after leaving it
>> > >> for an hour the info page reports this:
>> > >> Cache information for squid:
>> > >>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
>> > >>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
>> > >>         Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
>> > >>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
>> > >>         Storage Swap size:      0 KB
>> > >>         Storage Swap capacity:   0.0% used, 100.0% free
>> > >>         Storage Mem size:       0 KB
>> > >>         Storage Mem capacity:    0.0% used, 100.0% free
>> > >>         Mean Object Size:       0.00 KB
>> > >>         Requests given to unlinkd:      0
>> > >>
>> > >> Resource usage for squid:
>> > >>         UP Time:        255334.875 seconds
>> > >>         CPU Time:       7122.436 seconds
>> > >>         CPU Usage:      2.79%
>> > >>         CPU Usage, 5 minute avg:        0.05%
>> > >>         CPU Usage, 60 minute avg:       37.66%
>> > >>         Maximum Resident Size: 41500720 KB
>> > >>         Page faults with physical i/o: 1003410
>> > >>
>> > >> And here is the listing of free and top commands (with no load on the server):
>> > >>
>> > >> # free -h
>> > >>               total        used        free      shared  buff/cache   available
>> > >> Mem:            11G         10G        791M        676K        491M        1.0G
>> > >> Swap:           11G        5.5G        6.5G
>> > >>
>> > >> # top
>> > >> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
>> > >> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
>> > >> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> KiB Mem : 91.2/12251688
>> > >> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > >>         ]
>> > >> KiB Swap: 45.8/12582904
>> > >> [||||||||||||||||||||||||||||||||||||||||||||||
>> > >>                               ]
>> > >>
>> > >>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>> > >>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
>> > >>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
>> > >>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
>> > >>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
>> > >>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
>> > >>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
>> > >>    474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
>> > >> systemd-journal
>> > >>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
>> > >>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
>> > >>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
>> > >>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
>> > >>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
>> > >>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
>> > >>  48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
>> > >>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
>> > >>      1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
>> > >> ...
>> > >> ... lines ommited
>> > >> ...
>> > >>
>> > >> In the attachment you can find the printout from squidclient mgr:info
>> > >> and squidclient mgr:mem. These are both taken at the moment when there
>> > >> is no more load on the proxy. I also included my squid.conf file
>> > >> (minus the two files where acls are defined and outgoing IP addresses,
>> > >> these two contain only acl and tcp_outgoing_address lines as in the
>> > >> example above).
>> > >>
>> > >> Machine info:
>> > >> OS: Ubuntu 18.04 with latest updates
>> > >> Squid version 4.12 (from diladele repository)
>> > >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>> > >>
>> > >> I can not understand what is eating all of the memory, if I disabled the cache.
>> > >>
>> > >> Maybe I configured something wrong but I can not find what.
>> > >>
>> > >> Thank you for any help you can provide.
>> > >>
>> > >> Best regards,
>> > >> Ivan
>> > > _______________________________________________
>> > > squid-users mailing list
>> > > [hidden email]
>> > > http://lists.squid-cache.org/listinfo/squid-users
>> > >
>> >
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Eliezer Croitoru-3
Hey Ivan,

From what i remember there is a calculation for how much k per conn should squid use.
another thing is that squid is not returning memory once ot took it.
Amos knows about this and might be able to respond.

Eliezer

On Wed, Aug 5, 2020, 20:34 Ivan Bulatovic <[hidden email]> wrote:
Hi Eliezer,

In the original message I sent to the squid-users mail list, I
attached listings from mgr:info and mgr:mem. The server is definitely
using a lot of connections (close to 200K connections), which is why I
increased the open files limits in linux as well as in squid.conf. And
there are a lot of requests per second, when the server is running. I
could understand that it needs a lot of memory for in-transit cache,
but that memory should be later released  back to OS, once the
requests load goes down. However, that is not the case, even days
after there is no load on the server, it stays at 11GB of RAM and 5.5
GB of swap used. The second I restart the squid process, everything
goes to normal, memory is released. That is why I suspect there is
some memory leak somewhere.

The server is a Ubuntu 18.04 LTS VM (running on Hyper-V 2019 server),
with 8 virtual processors and 12GB of RAM (although I can increase
that if that is the problem, but I thought that without caching this
would be more than enough).

I am not using dynamic memory on Hyper-V (it is turned off for this VM).

Best regards,
Ivan

On Wed, Aug 5, 2020 at 7:14 PM NgTech LTD <[hidden email]> wrote:
>
> I think that the mgr:info or another page there contains the amount of requests per second etc.
> also netstat or ss -ntp might give some basic understanding about this server size.
>
> are you using dynamic memory on the hyper-v hypervisor?
>
> Eliezer
>
> On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <[hidden email]> wrote:
>>
>> Hi Alex,
>>
>> Thank you very much for your help.
>>
>> I opened a bug on bugs.squid-cache.org
>> (https://bugs.squid-cache.org/show_bug.cgi?id=5071).
>>
>> Best regards,
>> Ivan
>>
>> On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
>> <[hidden email]> wrote:
>> >
>> > On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
>> >
>> > > Looks like squid has some serious memory issues when under heavy load
>> > > (90 servers that crawl Internet sites).
>> >
>> > >         Maximum Resident Size: 41500720 KB
>> >
>> > If the above (unreliable) report matches your observations using system
>> > tools like "top", then it is indeed likely that your Squid is suffering
>> > from a memory leak -- 41GB is usually too much for most non-caching
>> > Squid instances.
>> >
>> > Identifying the leak may take some time, and I am not volunteering to do
>> > the necessary legwork personally, but the Squid Project does fix
>> > virtually all runtime leaks that we know about. If you want to speed up
>> > the process, one of the best things you can do is to run Squid under
>> > valgrind with a good suppression file. This requires building Squid with
>> > a special ./configure option. Several testing iterations may be
>> > necessary. If you are willing to do this, please file a bug report and
>> > somebody will guide you through the steps.
>> >
>> >
>> > > It just eats up memory, and
>> > > does not free it up even days after it is being used (with no load on
>> > > the proxy for days).
>> >
>> > Some memory retention is expected by default. See
>> > http://www.squid-cache.org/Doc/config/memory_pools/
>> >
>> > Unfortunately, AFAICT, your mgr:mem output does not show any obvious
>> > leaks -- all numbers are very small. If something is leaking a lot, then
>> > it is probably not pooled by Squid.
>> >
>> >
>> > HTH,
>> >
>> > Alex.
>> >
>> >
>> > > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
>> > >>
>> > >> Hi all,
>> > >>
>> > >> I am trying to configure squid to run as a forward proxy with no
>> > >> caching (cache deny all) with an option to choose the outgoing IP
>> > >> address based on the username. So all squid has to do is to use a
>> > >> certain outgoing IP address for a certain user, return the data from
>> > >> the server to that user and cache nothing.
>> > >>
>> > >> For that I created a special authentication helper and used the ACLs
>> > >> and tcp_outgoing_address to create a lot of users and outgoing IP
>> > >> addresses (about 260 at the moment). Example (not the real IP I use,
>> > >> of course):
>> > >>
>> > >> acl use_IP1 proxy_auth user1
>> > >> tcp_outgoing_address 1.2.3.4   use_IP1
>> > >>
>> > >> I also configured the squid to use 4 workers, but this happens even
>> > >> when I use only one worker (default)
>> > >>
>> > >> And this works. However, under heavy load, Squid eats all of the RAM
>> > >> and then starts going to swap. And the memory usage does not drop when
>> > >> I remove all the load from squid (I shut down all clients).
>> > >>
>> > >> I left it to see if the memory will be freed but even after leaving it
>> > >> for an hour the info page reports this:
>> > >> Cache information for squid:
>> > >>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
>> > >>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
>> > >>         Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
>> > >>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
>> > >>         Storage Swap size:      0 KB
>> > >>         Storage Swap capacity:   0.0% used, 100.0% free
>> > >>         Storage Mem size:       0 KB
>> > >>         Storage Mem capacity:    0.0% used, 100.0% free
>> > >>         Mean Object Size:       0.00 KB
>> > >>         Requests given to unlinkd:      0
>> > >>
>> > >> Resource usage for squid:
>> > >>         UP Time:        255334.875 seconds
>> > >>         CPU Time:       7122.436 seconds
>> > >>         CPU Usage:      2.79%
>> > >>         CPU Usage, 5 minute avg:        0.05%
>> > >>         CPU Usage, 60 minute avg:       37.66%
>> > >>         Maximum Resident Size: 41500720 KB
>> > >>         Page faults with physical i/o: 1003410
>> > >>
>> > >> And here is the listing of free and top commands (with no load on the server):
>> > >>
>> > >> # free -h
>> > >>               total        used        free      shared  buff/cache   available
>> > >> Mem:            11G         10G        791M        676K        491M        1.0G
>> > >> Swap:           11G        5.5G        6.5G
>> > >>
>> > >> # top
>> > >> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00, 0.00, 0.00
>> > >> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
>> > >> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
>> > >> KiB Mem : 91.2/12251688
>> > >> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > >>         ]
>> > >> KiB Swap: 45.8/12582904
>> > >> [||||||||||||||||||||||||||||||||||||||||||||||
>> > >>                               ]
>> > >>
>> > >>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>> > >>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5  29:43.74 squid
>> > >>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1  29:58.17 squid
>> > >>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0  29:47.56 squid
>> > >>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8  29:15.60 squid
>> > >>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4 136:18.57 metricbeat
>> > >>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4  14:32.10 filebeat
>> > >>    474 root      19  -1  127796  17576  17144 S   0.0  0.1   0:27.01
>> > >> systemd-journal
>> > >>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1   0:20.87 squid
>> > >>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1   0:31.83 snapd
>> > >>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1   0:00.14 python3
>> > >>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1   0:00.10 python3
>> > >>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1   0:00.12 python3
>> > >>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1   0:00.16 python3
>> > >>  48848 root      20   0  105688   6960   5968 S   0.0  0.1   0:00.02 sshd
>> > >>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0   0:00.00 sshd
>> > >>      1 root      20   0   86360   4364   2488 S   0.0  0.0  32:46.22 systemd
>> > >> ...
>> > >> ... lines ommited
>> > >> ...
>> > >>
>> > >> In the attachment you can find the printout from squidclient mgr:info
>> > >> and squidclient mgr:mem. These are both taken at the moment when there
>> > >> is no more load on the proxy. I also included my squid.conf file
>> > >> (minus the two files where acls are defined and outgoing IP addresses,
>> > >> these two contain only acl and tcp_outgoing_address lines as in the
>> > >> example above).
>> > >>
>> > >> Machine info:
>> > >> OS: Ubuntu 18.04 with latest updates
>> > >> Squid version 4.12 (from diladele repository)
>> > >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>> > >>
>> > >> I can not understand what is eating all of the memory, if I disabled the cache.
>> > >>
>> > >> Maybe I configured something wrong but I can not find what.
>> > >>
>> > >> Thank you for any help you can provide.
>> > >>
>> > >> Best regards,
>> > >> Ivan
>> > > _______________________________________________
>> > > squid-users mailing list
>> > > [hidden email]
>> > > http://lists.squid-cache.org/listinfo/squid-users
>> > >
>> >
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Amos Jeffries
Administrator
On 6/08/20 11:06 am, NgTech LTD wrote:
> Hey Ivan,
>
> From what i remember there is a calculation for how much k per conn
> should squid use.

Aye;
 256KB * number of currently open FD
 + read_ahead_gap
 + received size of current in-transit response (if cacheable MISS)


> another thing is that squid is not returning memory once ot took it.

The calculation for this is _minimum_ 5MB per type of memory allocating
object is retained by Squid for quick re-use. The mgr:mem report lists
details of those allocations.



Alex didn't mention this earlier but what I am seeing in your "top" tool
output is that there are 5x 'squid' processes running. It looks like 4
of them are SMP worker or disker processes each using 2.5GB of RAM.

The "free" tool is confirming this with its report of "used: 10G" (4x
2.5GB) of memory actually being used on the machine.

Most kernels fork() implementation is terrible with virtual memory
calculations. Most of that number will never actually be used. So they
can be ignored so long as the per-process number does not exceed the
actual physical RAM installed (beyond that kernel refuses to spawn with
fork()).
 The numbers your tools are reporting are kind of reasonable - maximum
about 7GB *per process* allocated.


The 41GB "resident size" is from old memory allocation APIs in the
kernel which suffer from 32-bit issues. When this value has odd numbers
and/or conflicts with the system tools - believe the tools instead.



So to summarize; what I am seeing there is that during *Peak* load times
your proxy workers (combined) are *maybe* using up to 41GB of memory. At
the off-peak time you are doing your analysis reports they have dropped
down to 10GB.
 With one data point there is no sign of a memory leak happening. Just a
normal machine handling far more peak traffic than its available amount
of memory can cope with.

 That is not to rule out a leak entirely. More measurements over time
may show a pattern of increasing off-peak memory allocated. But just one
comparison of peak vs off-peak is not going to reveal that type of pattern.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: High memory usage under load with caching disabled, memory is not being freed even with no load

Ivan Bulatovic
Hi Amos,

> > From what i remember there is a calculation for how much k per conn
> > should squid use.
>
> Aye;
>  256KB * number of currently open FD
>  + read_ahead_gap
>  + received size of current in-transit response (if cacheable MISS)

I tried to reduce the number of in-memory objects using following (I
read somewhere that 4KB is the minimum block of memory that squid can
allocate):
  cache_mem 8 MB
  maximum_object_size 4 KB
  maximum_object_size_in_memory 4 KB
  read_ahead_gap 4 KB

But the above settings did not help much.

At the moment I am running a much lighter load on the squid VM to see
how it behaves.

So, right now, the machine has about 110.000 open TCP connections (I
guess half are from clients and the other half is to the Internet,
which my firewall also confirms). It has been running like this for
the last 4 hours or so.

Here is the situation (in the attachment you will find full command
print-outs and config file):

- Running squid 4.12 from diladele repository on Ubuntu 18.04 LTS

- RAM used: around 9 GB out of 16 GB (no of swap is used)

- I am running 2 squid workers at the moment (see attached squid.conf)

- Top reports this (removed other processes, they practically have no
impact on the memory listing):

top - 10:22:03 up 18:00,  1 user,  load average: 1.88, 1.58, 1.43
Tasks: 169 total,   1 running,  94 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.6 us,  9.6 sy,  0.0 ni, 72.7 id,  0.1 wa,  0.0 hi,  3.9 si,  0.0 st
KiB Mem : 16380456 total,  5011560 free,  9205660 used,  2163236 buff/cache
KiB Swap: 12582904 total, 12582904 free,        0 used.  7121124 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  1515 proxy     20   0 5713808 4.396g  14188 S  36.5 28.1 130:03.34 squid
  1514 proxy     20   0 4360348 3.329g  14380 S  28.9 21.3 104:15.21 squid

 - mgr:info show some weird stats:
    Number of clients accessing cache:      156   (which is exactly
twice the number of actual clients, but this is probably due to the
number of workers)
    Maximum Resident Size: 32467680 KB   (which is 32GB. At no time
during these 4 hours has this value of RAM consumption ever been
reached. The memory is steadily, but slowly increasing to where it is
now - at 9 GB. I have no idea what this value is.)

I have no idea if the rest of the stats from mgr:info are OK or not, I
really have no way of checking that.

I added to the configuration memory pools option, we will see if it
helps (I think I already tried this, but I can not be sure, I ran a
lot of tests trying to fix this myself before I reached out to you):
memory_pools off

If there is anything else I can do to help with debugging this, please
let me know.

Thank you for your time and help,
Ivan


On Fri, Aug 7, 2020 at 2:23 AM Amos Jeffries <[hidden email]> wrote:

>
> On 6/08/20 11:06 am, NgTech LTD wrote:
> > Hey Ivan,
> >
> > From what i remember there is a calculation for how much k per conn
> > should squid use.
>
> Aye;
>  256KB * number of currently open FD
>  + read_ahead_gap
>  + received size of current in-transit response (if cacheable MISS)
>
>
> > another thing is that squid is not returning memory once ot took it.
>
> The calculation for this is _minimum_ 5MB per type of memory allocating
> object is retained by Squid for quick re-use. The mgr:mem report lists
> details of those allocations.
>
>
>
> Alex didn't mention this earlier but what I am seeing in your "top" tool
> output is that there are 5x 'squid' processes running. It looks like 4
> of them are SMP worker or disker processes each using 2.5GB of RAM.
>
> The "free" tool is confirming this with its report of "used: 10G" (4x
> 2.5GB) of memory actually being used on the machine.
>
> Most kernels fork() implementation is terrible with virtual memory
> calculations. Most of that number will never actually be used. So they
> can be ignored so long as the per-process number does not exceed the
> actual physical RAM installed (beyond that kernel refuses to spawn with
> fork()).
>  The numbers your tools are reporting are kind of reasonable - maximum
> about 7GB *per process* allocated.
>
>
> The 41GB "resident size" is from old memory allocation APIs in the
> kernel which suffer from 32-bit issues. When this value has odd numbers
> and/or conflicts with the system tools - believe the tools instead.
>
>
>
> So to summarize; what I am seeing there is that during *Peak* load times
> your proxy workers (combined) are *maybe* using up to 41GB of memory. At
> the off-peak time you are doing your analysis reports they have dropped
> down to 10GB.
>  With one data point there is no sign of a memory leak happening. Just a
> normal machine handling far more peak traffic than its available amount
> of memory can cope with.
>
>  That is not to rule out a leak entirely. More measurements over time
> may show a pattern of increasing off-peak memory allocated. But just one
> comparison of peak vs off-peak is not going to reveal that type of pattern.
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

Proxy01 test 2020-08-07.zip (11K) Download Attachment