Squid 5 service stops after assertion failure

[Vieri]

Hi,

My Squid web proxy crashed as shown in this log:

2021/01/24 13:18:13 kid1| helperHandleRead: unexpected reply on channel 0 from bllookup #Hlpr21 '43 ERR message=[...]
    current master transaction: master65
2021/01/24 13:18:13 kid1| assertion failed: helper.cc:1066: "skip == 0 && eom == NULL"
    current master transaction: master65
2021/01/24 13:18:13 kid1| Set Current Directory to /var/cache/squid
2021/01/24 13:18:13 kid1| Starting Squid Cache version 5.0.4-20201125-r5fadc09ee for x86_64-pc-linux-gnu...
2021/01/24 13:18:13 kid1| Service Name: squid
[...]
REPEATS (assertion failure & squid restart)
[...]
2021/01/24 13:18:27 kid1| helperHandleRead: unexpected reply on channel 0 from bllookup #Hlpr21 '2 ERR message=[...]
    current master transaction: master76
2021/01/24 13:18:27 kid1| assertion failed: helper.cc:1066: "skip == 0 && eom == NULL"
    current master transaction: master76
2021/01/24 13:18:27| Removing PID file (/run/squid.pid)
2021/01/24 13:18:34| Pinger exiting.
2021/01/24 13:18:37| Pinger exiting.

After the assertion failure Squid tries to restart a few times (assertion failures seen again) and finally exits.
A manual restart works, but I don't know for how long.

The external script "bllookup" is probably responsible for bad output, but maybe Squid could handle it without crashing.

Regards,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

[Amos Jeffries]

On 25/01/21 10:42 am, Vieri wrote:
>
> After the assertion failure Squid tries to restart a few times (assertion failures seen again) and finally exits.
> A manual restart works, but I don't know for how long.
>
> The external script "bllookup" is probably responsible for bad output,

That is a certainty.


> but maybe Squid could handle it without crashing.
>

As you noticed, Squid halts service only after the helper fails 10
multiple times in a row. Before that Squid is restarting the helper to
see if it was a temporary issue.


Would you prefer Squid sucks up all the TCP/IP resources on the machine
for clients waiting on this helper to start work?

Or Squid to start mixing up the results so each client A lookup is
determining the output of some other client B's transaction?


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

[Alex Rousskov]

On 1/24/21 4:42 PM, Vieri wrote:

> 2021/01/24 13:18:13 kid1| helperHandleRead: unexpected reply on channel 0 from bllookup #Hlpr21 '43 ERR message=[...]
>     current master transaction: master65
> 2021/01/24 13:18:13 kid1| assertion failed: helper.cc:1066: "skip == 0 && eom == NULL"
>     current master transaction: master65

> The external script "bllookup" is probably responsible for bad
> output, but maybe Squid could handle it without crashing.

You are right. The corresponding code needs some work, hopefully driven
by some well-established error handling principles rather than
semi-random ad hoc decisions. Filing a bug report with Squid Bugzilla
may increase chances of this problem getting fixed.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

[Alex Rousskov]

On 1/24/21 5:00 PM, Amos Jeffries wrote:
> On 25/01/21 10:42 am, Vieri wrote:
>>
>> After the assertion failure Squid tries to restart a few times
>> (assertion failures seen again) and finally exits.
>> A manual restart works, but I don't know for how long.
>>
>> The external script "bllookup" is probably responsible for bad output,

> That is a certainty.

>> but maybe Squid could handle it without crashing.


> As you noticed, Squid halts service only after the helper fails 10
> multiple times in a row. Before that Squid is restarting the helper to
> see if it was a temporary issue.

AFAICT, this Squid worker (rather than the helper) crashes/asserts. The
master process restarts the worker 10 times (because the worker keeps
crashing/asserting) and then gives up and kills the entire instance.


> Would you prefer Squid sucks up all the TCP/IP resources on the machine
> for clients waiting on this helper to start work?

> Or Squid to start mixing up the results so each client A lookup is
> determining the output of some other client B's transaction?


FWIW, I would prefer if Squid worker did not assert. Whether the worker
should quit when its helper fails is debatable and perhaps should be
configurable. In most environments/cases, killing the affected Squid
_transaction_ is much better than killing the Squid worker (and then
killing the entire Squid instance), but there may be exception worth
supporting (via configuration).


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

[Vieri]

On Sunday, January 24, 2021, 11:03:19 PM GMT+1, Amos Jeffries <[hidden email]> wrote:

>> The external script "bllookup" is probably responsible for bad output,
>
> That is a certainty.
>
>> but maybe Squid could handle it without crashing.
>
> As you noticed, Squid halts service only after the helper fails 10
> multiple times in a row. Before that Squid is restarting the helper to
> see if it was a temporary issue.

OK, the external script is definitely guilty. However, it is buggy and triggers the Squid assertion failure only in specific circumstances. So it's trasaction-specific. In my use case I would definitely prefer that only a few transactions were "killed", and that the whole of the proxy service would keep working.
Of course, I would still need to identify these cases and fix them, but in the meantime I would not get a general crash.
On the other hand a general failure forces me to look into this issue with greater celerity. ;-)

Thanks,

Vieri

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

[Vieri]

On Sunday, January 24, 2021, 11:08:49 PM GMT+1, Alex Rousskov <[hidden email]> wrote:

> Filing a bug report with Squid Bugzilla may increase chances of this problem getting fixed.

Done here:

https://bugs.squid-cache.org/show_bug.cgi?id=5100

Thanks,

Vieri
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users