WARNING: Disk space over limit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

WARNING: Disk space over limit

erdosain9
Hi to all.
Im having this warning

2017/07/11 11:37:44 kid1| WARNING: Disk space over limit: 17241692.00 KB > 15360000 KB
2017/07/11 11:37:56 kid1| WARNING: Disk space over limit: 16800692.00 KB > 15360000 KB
2017/07/11 11:38:07 kid1| WARNING: Disk space over limit: 16466860.00 KB > 15360000 KB
2017/07/11 11:38:19 kid1| WARNING: Disk space over limit: 16152960.00 KB > 15360000 KB
2017/07/11 11:38:30 kid1| WARNING: Disk space over limit: 15905772.00 KB > 15360000 KB
2017/07/11 11:38:41 kid1| WARNING: Disk space over limit: 15664684.00 KB > 15360000 KB
2017/07/11 11:38:52 kid1| WARNING: Disk space over limit: 15499404.00 KB > 15360000 KB


Why is this happening?? why is over limit??

[root@squid ~]# df -h
S.ficheros              Tamaño Usados  Disp Uso% Montado en
/dev/mapper/centos-root    48G    17G   31G  36% /
devtmpfs                  1,9G      0  1,9G   0% /dev
tmpfs                     1,9G   2,1M  1,9G   1% /dev/shm
tmpfs                     1,9G   8,5M  1,9G   1% /run
tmpfs                     1,9G      0  1,9G   0% /sys/fs/cgroup
/dev/sda1                 497M   143M  355M  29% /boot
tmpfs                     380M      0  380M   0% /run/user/0

SQUID.CONF

cache_dir diskd /var/spool/squid 15000 16 256
cache_mem 1000 MB
maximum_object_size_in_memory 1 MB

cache_swap_low 90
cache_swap_high 95

Yes, it is over cache_dir... but why?? and how i correct this?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Eliezer Croitoru
Hey,

If you are using a single process ie not SMP(default) use aufs instead of diskd cache_dir.
It's much more stable and efficient then diskd(to my knowledge).

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of erdosain9
Sent: Tuesday, July 11, 2017 17:25
To: [hidden email]
Subject: [squid-users] WARNING: Disk space over limit

Hi to all.
Im having this warning

2017/07/11 11:37:44 kid1| WARNING: Disk space over limit: 17241692.00 KB >
15360000 KB
2017/07/11 11:37:56 kid1| WARNING: Disk space over limit: 16800692.00 KB >
15360000 KB
2017/07/11 11:38:07 kid1| WARNING: Disk space over limit: 16466860.00 KB >
15360000 KB
2017/07/11 11:38:19 kid1| WARNING: Disk space over limit: 16152960.00 KB >
15360000 KB
2017/07/11 11:38:30 kid1| WARNING: Disk space over limit: 15905772.00 KB >
15360000 KB
2017/07/11 11:38:41 kid1| WARNING: Disk space over limit: 15664684.00 KB >
15360000 KB
2017/07/11 11:38:52 kid1| WARNING: Disk space over limit: 15499404.00 KB >
15360000 KB


Why is this happening?? why is over limit??

[root@squid ~]# df -h
S.ficheros              Tamaño Usados  Disp Uso% Montado en
/dev/mapper/centos-root    48G    17G   31G  36% /
devtmpfs                  1,9G      0  1,9G   0% /dev
tmpfs                     1,9G   2,1M  1,9G   1% /dev/shm
tmpfs                     1,9G   8,5M  1,9G   1% /run
tmpfs                     1,9G      0  1,9G   0% /sys/fs/cgroup
/dev/sda1                 497M   143M  355M  29% /boot
tmpfs                     380M      0  380M   0% /run/user/0

SQUID.CONF

cache_dir diskd /var/spool/squid 15000 16 256
cache_mem 1000 MB
maximum_object_size_in_memory 1 MB

cache_swap_low 90
cache_swap_high 95

Yes, it is over cache_dir... but why?? and how i correct this?



--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/WARNING-Disk-space-over-limit-tp4683055.html
Sent from the Squid - Users mailing list archive at Nabble.com.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Yuri Voinov


12.07.2017 3:25, Eliezer Croitoru пишет:
> Hey,
>
> If you are using a single process ie not SMP(default) use aufs instead of diskd cache_dir.
> It's much more stable and efficient then diskd(to my knowledge).
Not sure about you knowledge. Diskd is simple designed for another OS
than Linux. On appropriate platform it is not only stable, but quite
efficient than aufs. :)

>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of erdosain9
> Sent: Tuesday, July 11, 2017 17:25
> To: [hidden email]
> Subject: [squid-users] WARNING: Disk space over limit
>
> Hi to all.
> Im having this warning
>
> 2017/07/11 11:37:44 kid1| WARNING: Disk space over limit: 17241692.00 KB >
> 15360000 KB
> 2017/07/11 11:37:56 kid1| WARNING: Disk space over limit: 16800692.00 KB >
> 15360000 KB
> 2017/07/11 11:38:07 kid1| WARNING: Disk space over limit: 16466860.00 KB >
> 15360000 KB
> 2017/07/11 11:38:19 kid1| WARNING: Disk space over limit: 16152960.00 KB >
> 15360000 KB
> 2017/07/11 11:38:30 kid1| WARNING: Disk space over limit: 15905772.00 KB >
> 15360000 KB
> 2017/07/11 11:38:41 kid1| WARNING: Disk space over limit: 15664684.00 KB >
> 15360000 KB
> 2017/07/11 11:38:52 kid1| WARNING: Disk space over limit: 15499404.00 KB >
> 15360000 KB
>
>
> Why is this happening?? why is over limit??
>
> [root@squid ~]# df -h
> S.ficheros              Tamaño Usados  Disp Uso% Montado en
> /dev/mapper/centos-root    48G    17G   31G  36% /
> devtmpfs                  1,9G      0  1,9G   0% /dev
> tmpfs                     1,9G   2,1M  1,9G   1% /dev/shm
> tmpfs                     1,9G   8,5M  1,9G   1% /run
> tmpfs                     1,9G      0  1,9G   0% /sys/fs/cgroup
> /dev/sda1                 497M   143M  355M  29% /boot
> tmpfs                     380M      0  380M   0% /run/user/0
Where's the /var mountpoint?

>
> SQUID.CONF
>
> cache_dir diskd /var/spool/squid 15000 16 256
You specified 15000 megabytes for cache. So, reasonable

WARNING: Disk space over limit: 17241692.00 KB >
15360000 KB

Squid tries to save more than you specified.
> cache_mem 1000 MB
> maximum_object_size_in_memory 1 MB
>
> cache_swap_low 90
> cache_swap_high 95
and if you take a look onto your swap_hi/low values you can see it's not
do replacement to the end of specified store limit.
>
> Yes, it is over cache_dir... but why?? and how i correct this?
Try to do two things. First - increase cache_dir formal size. Second -
return

cache_swap_low

to it's default value.

Because of "Premature optimization is root of all evils".

>
>
>
> --
> View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/WARNING-Disk-space-over-limit-tp4683055.html
> Sent from the Squid - Users mailing list archive at Nabble.com.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Yuri Voinov


12.07.2017 4:34, Yuri пишет:

>
> 12.07.2017 3:25, Eliezer Croitoru пишет:
>> Hey,
>>
>> If you are using a single process ie not SMP(default) use aufs instead of diskd cache_dir.
>> It's much more stable and efficient then diskd(to my knowledge).
> Not sure about you knowledge. Diskd is simple designed for another OS
> than Linux. On appropriate platform it is not only stable, but quite
> efficient than aufs. :)
>> Eliezer
>>
>> ----
>> Eliezer Croitoru
>> Linux System Administrator
>> Mobile: +972-5-28704261
>> Email: [hidden email]
>>
>>
>>
>> -----Original Message-----
>> From: squid-users [mailto:[hidden email]] On Behalf Of erdosain9
>> Sent: Tuesday, July 11, 2017 17:25
>> To: [hidden email]
>> Subject: [squid-users] WARNING: Disk space over limit
>>
>> Hi to all.
>> Im having this warning
>>
>> 2017/07/11 11:37:44 kid1| WARNING: Disk space over limit: 17241692.00 KB >
>> 15360000 KB
>> 2017/07/11 11:37:56 kid1| WARNING: Disk space over limit: 16800692.00 KB >
>> 15360000 KB
>> 2017/07/11 11:38:07 kid1| WARNING: Disk space over limit: 16466860.00 KB >
>> 15360000 KB
>> 2017/07/11 11:38:19 kid1| WARNING: Disk space over limit: 16152960.00 KB >
>> 15360000 KB
>> 2017/07/11 11:38:30 kid1| WARNING: Disk space over limit: 15905772.00 KB >
>> 15360000 KB
>> 2017/07/11 11:38:41 kid1| WARNING: Disk space over limit: 15664684.00 KB >
>> 15360000 KB
>> 2017/07/11 11:38:52 kid1| WARNING: Disk space over limit: 15499404.00 KB >
>> 15360000 KB
>>
>>
>> Why is this happening?? why is over limit??
>>
>> [root@squid ~]# df -h
>> S.ficheros              Tamaño Usados  Disp Uso% Montado en
>> /dev/mapper/centos-root    48G    17G   31G  36% /
>> devtmpfs                  1,9G      0  1,9G   0% /dev
>> tmpfs                     1,9G   2,1M  1,9G   1% /dev/shm
>> tmpfs                     1,9G   8,5M  1,9G   1% /run
>> tmpfs                     1,9G      0  1,9G   0% /sys/fs/cgroup
>> /dev/sda1                 497M   143M  355M  29% /boot
>> tmpfs                     380M      0  380M   0% /run/user/0
> Where's the /var mountpoint?
>
>> SQUID.CONF
>>
>> cache_dir diskd /var/spool/squid 15000 16 256
> You specified 15000 megabytes for cache. So, reasonable
>
> WARNING: Disk space over limit: 17241692.00 KB >
> 15360000 KB
>
> Squid tries to save more than you specified.
>> cache_mem 1000 MB
>> maximum_object_size_in_memory 1 MB
>>
>> cache_swap_low 90
>> cache_swap_high 95
> and if you take a look onto your swap_hi/low values you can see it's not
> do replacement to the end of specified store limit.
>> Yes, it is over cache_dir... but why?? and how i correct this?
> Try to do two things. First - increase cache_dir formal size. Second -
> return
>
> cache_swap_low
>
> to it's default value.
>
> Because of "Premature optimization is root of all evils".
Ooooops, it default value is 90 :) My bad, too much nightly work :)

Well. Let's read squid.conf.documented:

#  TAG: cache_swap_low    (percent, 0-100)
#    The low-water mark for AUFS/UFS/diskd cache object eviction by
#    the cache_replacement_policy algorithm.
#
#    Removal begins when the swap (disk) usage of a cache_dir is
#    above this low-water mark and attempts to maintain utilization
#    near the low-water mark.
#
#    As swap utilization increases towards the high-water mark set
#    by cache_swap_high object eviction becomes more agressive.
#
#    The value difference in percentages between low- and high-water
#    marks represent an eviction rate of 300 objects per second and
#    the rate continues to scale in agressiveness by multiples of
#    this above the high-water mark.
#
#    Defaults are 90% and 95%. If you have a large cache, 5% could be
#    hundreds of MB. If this is the case you may wish to set these
#    numbers closer together.
#
#    See also cache_swap_high and cache_replacement_policy

Eviction starts when you cache achieves cache_swap_low. So, may be you
traffic is more than squid can be replaced between 90 and 95%. So, you
can try to decrease cache_swap_low. Or, review your caching policy to be
less aggressive. Or, increase cache_dir limit. Or all together.


>
>>
>>
>> --
>> View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/WARNING-Disk-space-over-limit-tp4683055.html
>> Sent from the Squid - Users mailing list archive at Nabble.com.
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Amos Jeffries
Administrator
In reply to this post by Yuri Voinov
On 12/07/17 10:34, Yuri wrote:
>
>
> 12.07.2017 3:25, Eliezer Croitoru пишет:
>> Hey,
>>
>> If you are using a single process ie not SMP(default) use aufs instead of diskd cache_dir.
>> It's much more stable and efficient then diskd(to my knowledge).
 >
> Not sure about you knowledge. Diskd is simple designed for another OS
> than Linux. On appropriate platform it is not only stable, but quite
> efficient than aufs. :)
>>

Not quite. AUFS was supposed to succeed diskd since it allows far
greater parallelism, however it often does not work very well on
BSD-like systems and nobody managed to fix it yet.

The latest BSDs threading does handle AUFS better than older ones do
though so YMMV.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Amos Jeffries
Administrator
In reply to this post by Yuri Voinov
On 12/07/17 10:47, Yuri wrote:

>
> Well. Let's read squid.conf.documented:
>
> #  TAG: cache_swap_low    (percent, 0-100)
> #    The low-water mark for AUFS/UFS/diskd cache object eviction by
> #    the cache_replacement_policy algorithm.
> #
> #    Removal begins when the swap (disk) usage of a cache_dir is
> #    above this low-water mark and attempts to maintain utilization
> #    near the low-water mark.
> #
> #    As swap utilization increases towards the high-water mark set
> #    by cache_swap_high object eviction becomes more agressive.
> #
> #    The value difference in percentages between low- and high-water
> #    marks represent an eviction rate of 300 objects per second and
> #    the rate continues to scale in agressiveness by multiples of
> #    this above the high-water mark.
> #
> #    Defaults are 90% and 95%. If you have a large cache, 5% could be
> #    hundreds of MB. If this is the case you may wish to set these
> #    numbers closer together.
> #
> #    See also cache_swap_high and cache_replacement_policy
>
> Eviction starts when you cache achieves cache_swap_low. So, may be you
> traffic is more than squid can be replaced between 90 and 95%. So, you
> can try to decrease cache_swap_low. Or, review your caching policy to be
> less aggressive. Or, increase cache_dir limit. Or all together.
>

Or adjust the maximum object size that cache can store.

Since the warnings are clearly decreasing in overage value things seem
to be recovering properly. The warning is to alert you that it happened
in the first place, so traffic may need re-evaluation. Your proxy seems
to be able to evict ~400 MB/cycle from disk.


Additional to traffic rate, it is rare but can happen that one or more
very large objects are fetched very close together. Squid then discovers
it has to throw away GB of objects to make room for them, even though
they may not have fully arrived yet. This may exceed the eviction rate
for a while, depending on your disk I/O speed.




The math if you are interested:

At your total cache size, the 5% high-low gap is only 750 MB, and the
warning occurs at 100% - just another 750 MB above that gap. Giving a
total burst size of 1.5 GB of object(s) necessary to trigger it.

NP: that 1.5GB needs to be above the regular eviction rate per garbage
cycle. If you are having trouble the fix is to a) get faster RPM disks,
b) spread the load over more proxies,


I notice that the overage value in the trace is only 2.2 GB which is
within realm for DVD images, HD movies, Windows 10 updates, etc.
If you are intending to cache those you need; multiple cache disks,
lower high/low percentages.

  You can also ensure the physical disk has a sufficient buffer of space
to not worry about occasional of these warnings. That is effectivively
the same as lowering the low watermark but prohibits Squid from using
that space when traffic is not being such a problem.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Eliezer Croitoru
In reply to this post by Amos Jeffries
Just wondering what or who is required to make this fix happen?
A donation? A programmer? A Tester?

Thanks,
Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Amos Jeffries
Sent: Wednesday, July 12, 2017 09:58
To: [hidden email]
Subject: Re: [squid-users] WARNING: Disk space over limit

On 12/07/17 10:34, Yuri wrote:
>
>
> 12.07.2017 3:25, Eliezer Croitoru пишет:
>> Hey,
>>
>> If you are using a single process ie not SMP(default) use aufs instead of diskd cache_dir.
>> It's much more stable and efficient then diskd(to my knowledge).
 >
> Not sure about you knowledge. Diskd is simple designed for another OS
> than Linux. On appropriate platform it is not only stable, but quite
> efficient than aufs. :)
>>

Not quite. AUFS was supposed to succeed diskd since it allows far
greater parallelism, however it often does not work very well on
BSD-like systems and nobody managed to fix it yet.

The latest BSDs threading does handle AUFS better than older ones do
though so YMMV.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: WARNING: Disk space over limit

Amos Jeffries
Administrator
On 12/07/17 22:55, Eliezer Croitoru wrote:
> Just wondering what or who is required to make this fix happen?
> A donation? A programmer? A Tester?

AFAIK, someone with knowledge of multi-threaded process design and both
Linux and BSD kernel disk I/O. Adrian was that person for Squid-2 and
IIRC had a theory that it was related to the thread blocking behaviour
of the kernel somehow.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...