Why `Storage Mem capacity` has a value larger than 100%.

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Why `Storage Mem capacity` has a value larger than 100%.

kmiku7
Hello
My squid consume too much memory at startup time grow.
From manager cgi /squid-internal-mgr/info, I saw the `Storage Mem capacity`
in Cache information for squid is 2429.7%, larger than 100%. In my opinion,
this value should be less or equal to 100%. I have searched but find nothing
about it.
Can someone tell me why? And how to limit squid memory usage, to avoid
memory grow as startup time grow and handle more and more request.
Thanks~.
The value of `cache_mem` option is 256M.

The following is outputs:

curl http://127.0.0.1:155/squid-internal-mgr/info
Squid Object Cache: Version 4.4
Build Info:
Service Name: squidHot
Start Time: Thu, 13 Feb 2019 06:37:48 GMT
Current Time: Wed, 17 Jul 2019 10:46:26 GMT
Connection information for squid:
        Number of clients accessing cache: (client_db off)
        Number of HTTP requests received: 562562
        Number of ICP messages received: 11385
        Number of ICP messages sent: 11421
        Number of queued ICP replies: 0
        Number of HTCP messages received: 0
        Number of HTCP messages sent: 0
        Request failure ratio: 0.00
        Average HTTP requests per minute since start: 2.6
        Average ICP messages per minute since start: 0.1
        Select loop called: 2360855132 times, 5.604 ms avg
Cache information for squid:
        Hits as % of all requests: 5min: 100.0%, 60min: 98.9%
        Hits as % of bytes sent: 5min: -0.0%, 60min: 90.6%
        Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
        Disk hits as % of hit requests: 5min: 100.0%, 60min: 83.3%
        Storage Swap size: 62529876 KB
        Storage Swap capacity: 1.5% used, 98.5% free
* Storage Mem size: 6369400 KB
        Storage Mem capacity: 2429.7% used, -2329.7% free*
        Mean Object Size: 68338.66 KB
        Requests given to unlinkd: 3



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Why `Storage Mem capacity` has a value larger than 100%.

Amos Jeffries
Administrator
On 29/07/19 2:46 pm, kmiku7 wrote:
> Hello
> My squid consume too much memory at startup time grow.
> From manager cgi /squid-internal-mgr/info, I saw the `Storage Mem capacity`
> in Cache information for squid is 2429.7%, larger than 100%. In my opinion,
> this value should be less or equal to 100%. I have searched but find nothing
> about it.
> Can someone tell me why? And how to limit squid memory usage, to avoid
> memory grow as startup time grow and handle more and more request.

The numbers in these reports can get a bit weird if you are running a
32-bit build of Squid on a 64-bit system, or just with very large sizes
for some things like cache.

When the value seems absurd like this use an external tool such as 'top'
to see if Squid is actually using as much memory as claimed.

If you have SMP workers configured then shared memory not working
properly could result in excessive memory usage across all of them.


Also, your proxy is apparently trying to fit objects with an *average*
size exceeding 70MB into that 256MB of cache. The bit of the report you
elided shows how many it is trying to fit in there.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Why `Storage Mem capacity` has a value larger than 100%.

kmiku7
Amos Jeffries wrote

> On 29/07/19 2:46 pm, kmiku7 wrote:
>> Hello
>> My squid consume too much memory at startup time grow.
>> From manager cgi /squid-internal-mgr/info, I saw the `Storage Mem
>> capacity`
>> in Cache information for squid is 2429.7%, larger than 100%. In my
>> opinion,
>> this value should be less or equal to 100%. I have searched but find
>> nothing
>> about it.
>> Can someone tell me why? And how to limit squid memory usage, to avoid
>> memory grow as startup time grow and handle more and more request.
>
> The numbers in these reports can get a bit weird if you are running a
> 32-bit build of Squid on a 64-bit system, or just with very large sizes
> for some things like cache.
>
> When the value seems absurd like this use an external tool such as 'top'
> to see if Squid is actually using as much memory as claimed.
>
> If you have SMP workers configured then shared memory not working
> properly could result in excessive memory usage across all of them.
>
>
> Also, your proxy is apparently trying to fit objects with an *average*
> size exceeding 70MB into that 256MB of cache. The bit of the report you
> elided shows how many it is trying to fit in there.
>
>
> Amos
> _______________________________________________
> squid-users mailing list

> squid-users@.squid-cache

> http://lists.squid-cache.org/listinfo/squid-users

Thanks your reply.
I am running 64bit build of squid on 64bit system. The output of top/ps
shows that squid is using as much memory as claimed in report.
I configure cache directory with size of 4T: cache_dir ufs PATH 4194304 128
256.

There are many child-process start, following is output of `ps`:
USER    20401  0.0  0.0  71020  2728 ?        Ss   Feb14   0:00
/PATH/TO/squid -f /PATH/TO/CONFIG/FILE -n squidHot
USER    20405  0.5 11.8 8258832 7771408 ?     S    Feb14 1298:30 (squid-1)
--kid squid-1 -f /PATH/TO/CONFIG/FILE -n squidHot
USER    20440  0.0  0.0  29468  1444 ?        S    Feb14   0:16
(logfile-daemon) /PATH/TO/access.log
USER    20441  0.0  0.0  29460  1256 ?        S    Feb14   0:00 (unlinkd)
USER    20444  0.0  0.0  29468  1252 ?        S    Feb14   0:00
(logfile-daemon) /PATH/TO/store.log

Process 20405 cost maximum memory.

Other part of report also make me puzzled:
        Internal Data Structures:
                  1185 StoreEntries
                  1184 StoreEntries with MemObjects
                     8 Hot Object Cache Items
                     9 on-disk objects
`9 on-disk objects` means only 9 entries of 1185 are stored on disk, and
others are stored in memory?



Amos Jeffries wrote
> Also, your proxy is apparently trying to fit objects with an *average*
> size exceeding 70MB into that 256MB of cache. The bit of the report you
> elided shows how many it is trying to fit in there.

Yes, we have many file larger than 256MB. But what problem will this lead
to? And why?



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Why `Storage Mem capacity` has a value larger than 100%.

Amos Jeffries
Administrator
On 31/07/19 3:23 pm, kmiku7 wrote:
>
> Thanks your reply.
> I am running 64bit build of squid on 64bit system. The output of top/ps
> shows that squid is using as much memory as claimed in report.

Okay. That means the negative values are just an artifact of 32-bit
types used in the report display, not actually overflow bugs in the
store code.

> I configure cache directory with size of 4T: cache_dir ufs PATH 4194304 128
> 256.

FYI: On 64-bit systems each 1GB of disk storage needs approximately 15MB
of RAM for the index and metadata.

Does your machine actually have 60GB of RAM available for the proxy to
use for this large cache_dir index?


Those numbers are relative to the avg object size. So I advise tuning
the min-size= parameter so only the many-MB objects get stored there.
That should cut the index RAM requirement by a few orders of magnitude.


>
> There are many child-process start, following is output of `ps`:
> USER    20401  0.0  0.0  71020  2728 ?        Ss   Feb14   0:00
> /PATH/TO/squid -f /PATH/TO/CONFIG/FILE -n squidHot
> USER    20405  0.5 11.8 8258832 7771408 ?     S    Feb14 1298:30 (squid-1)
> --kid squid-1 -f /PATH/TO/CONFIG/FILE -n squidHot
> USER    20440  0.0  0.0  29468  1444 ?        S    Feb14   0:16
> (logfile-daemon) /PATH/TO/access.log
> USER    20441  0.0  0.0  29460  1256 ?        S    Feb14   0:00 (unlinkd)
> USER    20444  0.0  0.0  29468  1252 ?        S    Feb14   0:00
> (logfile-daemon) /PATH/TO/store.log
>
> Process 20405 cost maximum memory.
>
> Other part of report also make me puzzled:
> Internal Data Structures:
>  1185 StoreEntries

ie. Total number of objects being cached by this proxy.


>  1184 StoreEntries with MemObjects

ie. Total number of objects which have at least some portion stored in
RAM for fast access.

This includes:
 * all objects in cache_mem
 * all cacheable objects currently being received from a server
 * all cacheable objects currently being delivered to a client
(though objects may match multiple of those criteria, each is only
counted once).

The difference between this and total objects (1185 - 1184 = 1) is the
number of objects *only* stored in a cache_dir.


>     8 Hot Object Cache Items

ie. Total count of items in cache_mem area of RAM.

>     9 on-disk objects

ie. Total count of objects stored in all configured cache_dir.


> `9 on-disk objects` means only 9 entries of 1185 are stored on disk, and
> others are stored in memory?
>

Essentially, yes.


>
>
> Amos Jeffries wrote
>> Also, your proxy is apparently trying to fit objects with an *average*
>> size exceeding 70MB into that 256MB of cache. The bit of the report you
>> elided shows how many it is trying to fit in there.
>
> Yes, we have many file larger than 256MB. But what problem will this lead
> to? And why?
>

That will lead to disk I/O capacity being a major limiting factor in
delivery speed for all those objects. Since their data has to be saved
to or read from disk in order to be used.

Modulo bugs, the store is only supposed to be keeping a small portion of
them in memory awaiting delivery (on sending them) or waiting for
swapout to disk (on receiving). Though maximum_object_size_in_memory
place a role there, objects under that limit *may* be loaded fully into
cache_mem.

Check your object size limits:
<http://www.squid-cache.org/Doc/config/minimum_object_size/>
<http://www.squid-cache.org/Doc/config/maximum_object_size/>
<http://www.squid-cache.org/Doc/config/maximum_object_size_in_memory/>


PS. I also advise upgrading to the latest v4 release to avoid the
security issues and a memory leak that have been fixed since v4.4.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users