4.1 exception PageStack.cc(106)

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

4.1 exception PageStack.cc(106)

Heiler Bemerguy
2018/08/04 04:42:59 kid1| FATAL: Dying from an exception handling
failure; exception: check failed: false
     exception location: mem/PageStack.cc(106) push

Then it tries to run again and again every kid but they always die:

2018/08/04 04:43:05 kid1| FATAL: kid1 registration timed out
2018/08/04 04:43:05 kid1| Squid Cache (Version 4.1-20180725-r86244dd):
Terminated abnormally.
CPU Usage: 0.036 seconds = 0.016 user + 0.020 sys
Maximum Resident Size: 155984 KB
Page faults with physical i/o: 0


Squid Cache: Version 4.1-20180725-r86244dd
Service Name: squid

This binary uses OpenSSL 1.0.1t  3 May 2016. For legal restrictions on
distribution see https://www.openssl.org/source/license.html

configure options:  '--enable-delay-pools' '--with-openssl'
'--enable-htcp' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--disable-wccp' '--disable-snmp'
'--enable-inline' '--enable-async-io=32' '--enable-storeio=aufs,rock'
'--enable-underscores' '--enable-removal-policies=lru,heap'
'--enable-http-violations' '--disable-ident-lookups'
'--with-large-files' '--enable-ssl' '--enable-ltdl-convenience'
'--prefix=/usr' '--localstatedir=/var' '--libexecdir=/lib/squid'
'--srcdir=.' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid'
'--with-default-user=proxy' '--with-logdir=/var/log'
'--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=16384'
'--with-aufs-threads=32' '--disable-translation'

--
Atenciosamente,

Heiler Bensimon Bemerguy - CINBESA
Analista de Redes, Wi-Fi,
Virtualização e Serviços Internet
(55) 91 98151-4894

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Amos Jeffries
Administrator
On 05/08/18 14:41, Heiler Bemerguy wrote:
> 2018/08/04 04:42:59 kid1| FATAL: Dying from an exception handling
> failure; exception: check failed: false
>     exception location: mem/PageStack.cc(106) push
>

What are your cache_mem and cache_dir settings?

And what *-bit type (eg 16, 32, 64, 128) is the machine displaying these
errors?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Heiler Bemerguy
Em 05/08/2018 05:47, Amos Jeffries escreveu:
> On 05/08/18 14:41, Heiler Bemerguy wrote:
>> 2018/08/04 04:42:59 kid1| FATAL: Dying from an exception handling
>> failure; exception: check failed: false
>>      exception location: mem/PageStack.cc(106) push
>>
> What are your cache_mem and cache_dir settings?

workers 5
cache_mem 5500 MB
cache_dir rock /cache2 130000 min-size=1 max-size=196608
cache_dir rock /cache3 130000 min-size=196609 max-size=624288
cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
swap-timeout=500

> And what *-bit type (eg 16, 32, 64, 128) is the machine displaying these
> errors?
How can I get that?!


--
Atenciosamente,

Heiler Bensimon Bemerguy - CINBESA
Analista de Redes, Wi-Fi,
Virtualização e Serviços Internet
(55) 91 98151-4894

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Amos Jeffries
Administrator
On 07/08/18 03:32, Heiler Bemerguy wrote:

> Em 05/08/2018 05:47, Amos Jeffries escreveu:
>> On 05/08/18 14:41, Heiler Bemerguy wrote:
>>> 2018/08/04 04:42:59 kid1| FATAL: Dying from an exception handling
>>> failure; exception: check failed: false
>>>      exception location: mem/PageStack.cc(106) push
>>>
>> What are your cache_mem and cache_dir settings?
>
> workers 5
> cache_mem 5500 MB
> cache_dir rock /cache2 130000 min-size=1 max-size=196608
> cache_dir rock /cache3 130000 min-size=196609 max-size=624288
> cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
> swap-timeout=500
>
>> And what *-bit type (eg 16, 32, 64, 128) is the machine displaying these
>> errors?
> How can I get that?!

That depends on your OS.  On Linux "uname -a" provides the relevant type
info. I'm not sure for other OS which lack that tool.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Heiler Bemerguy
Em 06/08/2018 13:39, Amos Jeffries escreveu:

>
>> workers 5
>> cache_mem 5500 MB
>> cache_dir rock /cache2 130000 min-size=1 max-size=196608
>> cache_dir rock /cache3 130000 min-size=196609 max-size=624288
>> cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
>> swap-timeout=500
>>
>>> And what *-bit type (eg 16, 32, 64, 128) is the machine displaying these
>>> errors?
>> How can I get that?!
> That depends on your OS.  On Linux "uname -a" provides the relevant type
> info. I'm not sure for other OS which lack that tool.

Linux proxy 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u1 (2017-06-18)
x86_64 GNU/Linux

--
Atenciosamente,

Heiler Bensimon Bemerguy - CINBESA
Analista de Redes, Wi-Fi,
Virtualização e Serviços Internet
(55) 91 98151-4894

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Matus UHLAR - fantomas
>Em 06/08/2018 13:39, Amos Jeffries escreveu:
>>
>>>workers 5
>>>cache_mem 5500 MB
>>>cache_dir rock /cache2 130000 min-size=1 max-size=196608
>>>cache_dir rock /cache3 130000 min-size=196609 max-size=624288
>>>cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
>>>swap-timeout=500
>>>
>>>>And what *-bit type (eg 16, 32, 64, 128) is the machine displaying these
>>>>errors?
>>>How can I get that?!
>>That depends on your OS.  On Linux "uname -a" provides the relevant type
>>info. I'm not sure for other OS which lack that tool.

On 06.08.18 13:53, Heiler Bemerguy wrote:
>Linux proxy 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u1 (2017-06-18)
>x86_64 GNU/Linux

on debian, also:

dpkg --print-architecture

and

dpkg -s squid squid3

are important.

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Honk if you love peace and quiet.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Amos Jeffries
Administrator
In reply to this post by Heiler Bemerguy
On 07/08/18 04:53, Heiler Bemerguy wrote:
> Em 06/08/2018 13:39, Amos Jeffries escreveu:
>>
>>> workers 5
>>> cache_mem 5500 MB
>>> cache_dir rock /cache2 130000 min-size=1 max-size=196608
>>> cache_dir rock /cache3 130000 min-size=196609 max-size=624288
>>> cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
>>> swap-timeout=500

The number of slots these cache_dir require is 0-800 million, 800-1600
million, and 2.4+ billion. The latter is outside of the range of values
which the 32-bit slot IDs can be assigned. I think this is where your
initial exception is coming from.

You need to increase the slot sizes to be larger than the default 32KB
on at least that large caches.


>>>
>>>> And what *-bit type (eg 16, 32, 64, 128) is the machine displaying
>>>> these
>>>> errors?
>>> How can I get that?!
>> That depends on your OS.  On Linux "uname -a" provides the relevant type
>> info. I'm not sure for other OS which lack that tool.
>
> Linux proxy 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u1 (2017-06-18)
> x86_64 GNU/Linux
>


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Alex Rousskov
On 08/07/2018 08:01 AM, Amos Jeffries wrote:
> On 07/08/18 04:53, Heiler Bemerguy wrote:
>> Em 06/08/2018 13:39, Amos Jeffries escreveu:
>>>> cache_mem 5500 MB
>>>> cache_dir rock /cache2 130000 min-size=1 max-size=196608
>>>> cache_dir rock /cache3 130000 min-size=196609 max-size=624288
>>>> cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
>>>> swap-timeout=500


> The number of slots these cache_dir require is 0-800 million, 800-1600
> million, and 2.4+ billion.


What makes you think that? AFAICT, all three cache_dirs have the same
130'000 MB size and, hence, each requires 8'320'000 default 16KB slots:

    130000 * 1024 / 16 = 8320000

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Amos Jeffries
Administrator
On 08/08/18 04:30, Alex Rousskov wrote:

> On 08/07/2018 08:01 AM, Amos Jeffries wrote:
>> On 07/08/18 04:53, Heiler Bemerguy wrote:
>>> Em 06/08/2018 13:39, Amos Jeffries escreveu:
>>>>> cache_mem 5500 MB
>>>>> cache_dir rock /cache2 130000 min-size=1 max-size=196608
>>>>> cache_dir rock /cache3 130000 min-size=196609 max-size=624288
>>>>> cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
>>>>> swap-timeout=500
>
>
>> The number of slots these cache_dir require is 0-800 million, 800-1600
>> million, and 2.4+ billion.
>
>
> What makes you think that? AFAICT, all three cache_dirs have the same
> 130'000 MB size and, hence, each requires 8'320'000 default 16KB slots:
>
>     130000 * 1024 / 16 = 8320000
>


I was working from the number slots needed to store the files Squid is
going to try to put in there (the 2^27 sfileno).

You are right, its the other way around.

Do you have any idea then why the cunt of used slots should be over 2^32
? the exception being encountered is thrown when used slot count (an
int) exceeds the value of largest available slot index (an uint32_t).

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Heiler Bemerguy
In reply to this post by Amos Jeffries
Em 07/08/2018 11:01, Amos Jeffries escreveu:

> On 07/08/18 04:53, Heiler Bemerguy wrote:
>> Em 06/08/2018 13:39, Amos Jeffries escreveu:
>>>> workers 5
>>>> cache_mem 5500 MB
>>>> cache_dir rock /cache2 130000 min-size=1 max-size=196608
>>>> cache_dir rock /cache3 130000 min-size=196609 max-size=624288
>>>> cache_dir rock /cache4 130000 min-size=624289 max-swap-rate=500
>>>> swap-timeout=500
> The number of slots these cache_dir require is 0-800 million, 800-1600
> million, and 2.4+ billion. The latter is outside of the range of values
> which the 32-bit slot IDs can be assigned. I think this is where your
> initial exception is coming from.
>
> You need to increase the slot sizes to be larger than the default 32KB
> on at least that large caches.
>
>

This was changed from 3.5.27 ?? It used to work flawlessy just before
the upgrade to 4.1. (now 4.2). And shouldn't this crash only the disker,
not the worker?

Thanks


--
Atenciosamente,

Heiler Bensimon Bemerguy - CINBESA
Analista de Redes, Wi-Fi,
Virtualização e Serviços Internet
(55) 91 98151-4894

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Alex Rousskov
In reply to this post by Amos Jeffries
On 08/07/2018 11:04 AM, Amos Jeffries wrote:

> Do you have any idea then why the cunt of used slots should be over
> 2^32?

I suspect this is a Squid bug. We have seen the same symptoms while
working on not-directly-related bug 4823. I do not know whether OP
suffers from the same underlying problem. Unfortunately, I do not
remember the exact explanation for the bug we saw.

FWIW, there is a work-in-progress Factory PR that may eventually fix
this bug as a side effect of other upcoming Rock improvements[1]. That
PR, including its description, is not ready for the official review, and
may change a lot until it is ready.

Alex.
[1] https://github.com/measurement-factory/squid/pull/10
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: 4.1 exception PageStack.cc(106)

Heiler Bemerguy
Em 07/08/2018 14:55, Alex Rousskov escreveu:
On 08/07/2018 11:04 AM, Amos Jeffries wrote:

Do you have any idea then why the cunt of used slots should be over
2^32?
I suspect this is a Squid bug. We have seen the same symptoms while
working on not-directly-related bug 4823. I do not know whether OP
suffers from the same underlying problem. Unfortunately, I do not
remember the exact explanation for the bug we saw.

FWIW, there is a work-in-progress Factory PR that may eventually fix
this bug as a side effect of other upcoming Rock improvements[1]. That
PR, including its description, is not ready for the official review, and
may change a lot until it is ready.


proxy     1307  0.0  0.0 6501296 26612 ?       S    08:28   0:01 (squid-coord-9) --kid squid-coord-9 -s
proxy     1308  0.7  4.0 7417212 2693168 ?     S    08:28   2:53 (squid-disk-8) --kid squid-disk-8 -s
proxy     1309  0.4  2.9 7417212 1940332 ?     S    08:28   1:44 (squid-disk-7) --kid squid-disk-7 -s
proxy     1310  1.0  5.7 7417212 3785400 ?     S    08:28   4:04 (squid-disk-6) --kid squid-disk-6 -s
proxy     4326  2.0  1.3 9393888 913204 ?      S    13:46   1:39 (squid-3) --kid squid-3 -s
proxy     4440  2.8  1.4 9508812 958080 ?      S    14:00   1:52 (squid-5) --kid squid-5 -s
proxy     4646  6.7  2.2 9537240 1490604 ?     S    14:06   4:06 (squid-1) --kid squid-1 -s
proxy     4925  9.1  1.3 9529772 861360 ?      S    14:47   1:49 (squid-2) --kid squid-2 -s
proxy     4956 14.4  1.5 9605584 991504 ?      S    14:50   2:26 (squid-4) --kid squid-4 -s

Our 5 "workers" are dying frequently. And the same .conf and rock files were used with 3.5 without any problems.. this 4.2 is compiled with O2 (the default Debian-amd64 CFLAGS)
If you need any more info, just ask :)

Some other log entries:

2018/08/07 00:45:41 kid6| WARNING: Ignoring store entry because store entry metadata is too big
    exception location: StoreMetaUnpacker.cc(52) checkBuffer

2018/08/07 00:45:42 kid6| WARNING: Ignoring store entry because store entry metadata is too big
    exception location: StoreMetaUnpacker.cc(52) checkBuffer

2018/08/07 00:45:42 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 1505!=1550
2018/08/07 00:45:43 kid6| Store rebuilding is 57.20% complete
2018/08/07 00:45:43 kid7| WARNING: Ignoring store entry because store entry metadata is corrupted
    exception location: StoreMetaUnpacker.cc(43) checkBuffer

2018/08/07 00:45:45 kid6| WARNING: Ignoring store entry because store entry metadata is too big
    exception location: StoreMetaUnpacker.cc(52) checkBuffer

2018/08/07 00:45:45 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 2006!=2005
2018/08/07 00:45:45 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 6492!=6491
2018/08/07 00:45:46 kid7| WARNING: Ignoring store entry because store entry metadata is corrupted
    exception location: StoreMetaUnpacker.cc(43) checkBuffer

2018/08/07 00:45:46 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 681!=691
2018/08/07 00:45:46 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 794!=825
2018/08/07 00:45:46 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 3848!=3954
2018/08/07 00:45:47 kid7| Store rebuilding is 96.91% complete
2018/08/07 00:45:48 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 6550!=6547
2018/08/07 00:45:48 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 6942!=6943
2018/08/07 00:45:50 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 2123!=2127
2018/08/07 00:45:51 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 4071!=4069
2018/08/07 00:45:51 kid6| WARNING: Ignoring cache entry due to a SIZE MISMATCH 1140!=1141
2018/08/07 00:45:51 kid7| Finished rebuilding storage from disk.
2018/08/07 00:45:51 kid7|   8086747 Entries scanned
2018/08/07 00:45:51 kid7|    148604 Invalid entries.
2018/08/07 00:45:51 kid7|         0 With invalid flags.
2018/08/07 00:45:51 kid7|        15 Objects loaded.
2018/08/07 00:45:51 kid7|         0 Objects expired.
2018/08/07 00:45:51 kid7|         0 Objects cancelled.
2018/08/07 00:45:51 kid7|        92 Duplicate URLs purged.
2018/08/07 00:45:51 kid7|      1063 Swapfile clashes avoided.
2018/08/07 00:45:51 kid7|   Took 1234.47 seconds (  0.01 objects/sec).
2018/08/07 00:45:51 kid7| Beginning Validation Procedure
2018/08/07 00:45:51 kid7|   Completed Validation Procedure
2018/08/07 00:45:51 kid7|   Validated 0 Entries
2018/08/07 00:45:51 kid7|   store_swap_size = 16.00 KB
-- 
Atenciosamente,

Heiler Bensimon Bemerguy - CINBESA
Analista de Redes, Wi-Fi,
Virtualização e Serviços Internet
(55) 91 98151-4894

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users