Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Francisco Amaro
Hello all,

We are running stock CentOS 6 and CentOS 7 Squid builds, patched to the latest version, 3.1.23 and 3.5.20.
We have several machines, running on separate locations, with an ACL file unique to all, that get's pushed to all machines using an homemade script. So local configuration (interfaces, cache locations, etc) is one file, and the bulk of the ACL's are included on
a separate file, that is the same on all servers.

For some time now, but getting more frequent on the last couple of weeks, we have been getting errors on our cache.log about bogus ACL's, forcing a squid restart. This happens on configuration reloads, which we do very frequently,  maybe 10-20 times a day.

On CentOS 6, the most frequent error is :

FATAL: Bungled (null) line 203: icap_retry deny all

We do not use icap_retry, it is missing from the config file, but this seems to be fixed on 3.2.4:

Changes to squid-3.2.4 (03 Dec 2012):
- Remove 'Bungled' warning on missing component directives

On CentOS 7, the most frequent error is :

FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

Now this one even has a bug entry, and was fixed on 3.3.5:

Changes to squid-3.3.5 (20 May 2013):
- Bug 3744: squid terminated: FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

But we are still seeing it on 3.5.20...

Now what is even more weird is that besides that, we are getting what seem complete random bogus entries, like this ones:

FATAL: Bungled /etc/squid/squid_acls line 11796: acl xyz_allow_url
FATAL: Bungled /etc/squid/squid_acls line 10905: acl abc_url108 dstdomai

FATAL: Bungled squid_acls line 10033: http_access allow def_
FATAL: Bungled squid_acls line 9605: http_access allow ms_src_teste ms_t
FATAL: Bungled squid_acls line 11025: http_access allow xpto

The problem is that these lines are 100% correct , they are not "bungled" at all,  it seems the parser as some issue with it.

Each time we get an error like this, Squid does a full restart, and that on CentOS 6 is time consuming, maybe 2-3 minutes, time enough for our users reaching for the Helpdesk tickets... On CentOS 7, since the restart is very fast, we do not have many complaints...

Our ACL file is currently 7500 lines, mostly comprising of dstdomain and url_regex entries. We support a lot of costumers (about 100) and each one has a specific configuration. Mostly have while lists, and there are a  few blacklists in the middle... so, not very big in terms of line numbers, but somewhat  complex regarding site/VLAN entries...

For security/privacy reasons I cannot share my complete configuration, namely the ACL's part, since they contain client information, but I am able to get detailed logging and/or redacted information...

So... anyone has any clue what is happenning ?


-- 


Francisco Amaro
Email: [hidden email]

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Amos Jeffries
Administrator
On 13/10/17 05:24, Francisco Amaro wrote:

> Hello all,
>
> We are running stock CentOS 6 and CentOS 7 Squid builds, patched to the
> latest version, 3.1.23 and 3.5.20.
> We have several machines, running on separate locations, with an ACL
> file unique to all, that get's pushed to all machines using an homemade
> script. So local configuration (interfaces, cache locations, etc) is one
> file, and the bulk of the ACL's are included on
> a separate file, that is the same on all servers.
>
> For some time now, but getting more frequent on the last couple of
> weeks, we have been getting errors on our cache.log about bogus ACL's,
> forcing a squid restart. This happens on configuration reloads, which we
> do very frequently,  maybe 10-20 times a day.
>
> On CentOS 6, the most frequent error is :
>
> FATAL: Bungled (null) line 203: icap_retry deny all
>
> We do not use icap_retry, it is missing from the config file, but this
> seems to be fixed on 3.2.4:
>
> Changes to squid-3.2.4 (03 Dec 2012):
> - Remove 'Bungled' warning on missing component directives
>
> On CentOS 7, the most frequent error is :
>
> FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all
>
> Now this one even has a bug entry, and was fixed on 3.3.5:
>
> Changes to squid-3.3.5 (20 May 2013):
> - Bug 3744: squid terminated: FATAL: Bungled (null) line 3:
> sslproxy_cert_sign signTrusted all
>
> But we are still seeing it on 3.5.20...
>
> Now what is even more weird is that besides that, we are getting what
> seem complete random bogus entries, like this ones:
>
> FATAL: Bungled /etc/squid/squid_acls line 11796: acl xyz_allow_url
> FATAL: Bungled /etc/squid/squid_acls line 10905: acl abc_url108 dstdomai
>
> FATAL: Bungled squid_acls line 10033: http_access allow def_
> FATAL: Bungled squid_acls line 9605: http_access allow ms_src_teste ms_t
> FATAL: Bungled squid_acls line 11025: http_access allow xpto
>
> The problem is that these lines are 100% correct , they are not
> "bungled" at all,  it seems the parser as some issue with it.
>
> Each time we get an error like this, Squid does a full restart, and that
> on CentOS 6 is time consuming, maybe 2-3 minutes, time enough for our
> users reaching for the Helpdesk tickets... On CentOS 7, since the
> restart is very fast, we do not have many complaints.....

Eliezer provides some more up to date packages for CentOS
<https://wiki.squid-cache.org/KnowledgeBase/CentOS#Squid-3.5>


> So... anyone has any clue what is happenning ?
>

Two things come to mind:

* non-ASCII characters in your config file. That used to be a regular
problem with cut-n-paste on some systems.



* the wrong binary running somehow.

Try shutting Squid down completely then restarting it.

If that does not work by itself you can repeat with a re-install after
the shutdown.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Eliezer Croitoru
In reply to this post by Francisco Amaro
Are you validating your config files before committing them??
If you have about 100 clients this is the first thing to do...
Either run a asciii validation script for the whole file(I can write it if you really need it) or maintain two configuration testing node (CentOS 6+7).
You don't even need to apply the configurations into the a running squid, you just need to run a command like "squid -kparse -f /etc/squid/squid.conf" and it will tell you what's wrong.

Let me know if you need more help.

Eliezer

* CC me since I'm not watching the list daily these days.

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On Behalf Of Francisco Amaro
Sent: Thursday, October 12, 2017 19:25
To: [hidden email]
Subject: [squid-users] Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Hello all,

We are running stock CentOS 6 and CentOS 7 Squid builds, patched to the latest version, 3.1.23 and 3.5.20.
We have several machines, running on separate locations, with an ACL file unique to all, that get's pushed to all machines using an homemade script. So local configuration (interfaces, cache locations, etc) is one file, and the bulk of the ACL's are included on
a separate file, that is the same on all servers.

For some time now, but getting more frequent on the last couple of weeks, we have been getting errors on our cache.log about bogus ACL's, forcing a squid restart. This happens on configuration reloads, which we do very frequently,  maybe 10-20 times a day.

On CentOS 6, the most frequent error is :

FATAL: Bungled (null) line 203: icap_retry deny all

We do not use icap_retry, it is missing from the config file, but this seems to be fixed on 3.2.4:

Changes to squid-3.2.4 (03 Dec 2012):
- Remove 'Bungled' warning on missing component directives

On CentOS 7, the most frequent error is :

FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

Now this one even has a bug entry, and was fixed on 3.3.5:

Changes to squid-3.3.5 (20 May 2013):
- Bug 3744: squid terminated: FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

But we are still seeing it on 3.5.20...

Now what is even more weird is that besides that, we are getting what seem complete random bogus entries, like this ones:

FATAL: Bungled /etc/squid/squid_acls line 11796: acl xyz_allow_url
FATAL: Bungled /etc/squid/squid_acls line 10905: acl abc_url108 dstdomai

FATAL: Bungled squid_acls line 10033: http_access allow def_
FATAL: Bungled squid_acls line 9605: http_access allow ms_src_teste ms_t
FATAL: Bungled squid_acls line 11025: http_access allow xpto

The problem is that these lines are 100% correct , they are not "bungled" at all,  it seems the parser as some issue with it.

Each time we get an error like this, Squid does a full restart, and that on CentOS 6 is time consuming, maybe 2-3 minutes, time enough for our users reaching for the Helpdesk tickets... On CentOS 7, since the restart is very fast, we do not have many complaints...

Our ACL file is currently 7500 lines, mostly comprising of dstdomain and url_regex entries. We support a lot of costumers (about 100) and each one has a specific configuration. Mostly have while lists, and there are a  few blacklists in the middle... so, not very big in terms of line numbers, but somewhat  complex regarding site/VLAN entries...

For security/privacy reasons I cannot share my complete configuration, namely the ACL's part, since they contain client information, but I am able to get detailed logging and/or redacted information...

So... anyone has any clue what is happenning ?



--


Francisco Amaro
Email: mailto:[hidden email]

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Francisco Amaro
Hi all,

So, we've been (slowly) investigating this issue these past days, and have reached some conclusions.

a) First, and sorry for not being clear on that, this only happens on "squid reload", not on full stop/start cycles.
A full start cycle does not give this error, that's why we are pretty sure the issue is not a bungled ACL, per se.

b) Besides that, it only happens when the proxy has some load on it, we have been unable to reproduce this on
our test machines, since they do not have a significant load. We've been postponing our ACL's changes for
early morning/lunch-hour these past days, and we didn't have had a FATAL error since last week.

c) Although we don't have non US-ASCII on the ACL's themselves, we have a lot of them on comments, so
I've stripped down all the comments of the ACL's files and tried that on our test bed, it didn't generare any
kind of errors, but since they do not have some load on it, we are not sure if it was the comments stripping 
or the lack of load on the server...

So, we are now searching for an "easy" way of generating traffic on a squid server, so we can, at least, be
able to reproduce the error, so we can try the different solutions....








2017-10-18 1:06 GMT+01:00 Eliezer Croitoru <[hidden email]>:
Are you validating your config files before committing them??
If you have about 100 clients this is the first thing to do...
Either run a asciii validation script for the whole file(I can write it if you really need it) or maintain two configuration testing node (CentOS 6+7).
You don't even need to apply the configurations into the a running squid, you just need to run a command like "squid -kparse -f /etc/squid/squid.conf" and it will tell you what's wrong.

Let me know if you need more help.

Eliezer

* CC me since I'm not watching the list daily these days.

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: <a href="tel:%2B972-5-28704261" value="+972528704261">+972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On Behalf Of Francisco Amaro
Sent: Thursday, October 12, 2017 19:25
To: [hidden email]
Subject: [squid-users] Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Hello all,

We are running stock CentOS 6 and CentOS 7 Squid builds, patched to the latest version, 3.1.23 and 3.5.20.
We have several machines, running on separate locations, with an ACL file unique to all, that get's pushed to all machines using an homemade script. So local configuration (interfaces, cache locations, etc) is one file, and the bulk of the ACL's are included on
a separate file, that is the same on all servers.

For some time now, but getting more frequent on the last couple of weeks, we have been getting errors on our cache.log about bogus ACL's, forcing a squid restart. This happens on configuration reloads, which we do very frequently,  maybe 10-20 times a day.

On CentOS 6, the most frequent error is :

FATAL: Bungled (null) line 203: icap_retry deny all

We do not use icap_retry, it is missing from the config file, but this seems to be fixed on 3.2.4:

Changes to squid-3.2.4 (03 Dec 2012):
- Remove 'Bungled' warning on missing component directives

On CentOS 7, the most frequent error is :

FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

Now this one even has a bug entry, and was fixed on 3.3.5:

Changes to squid-3.3.5 (20 May 2013):
- Bug 3744: squid terminated: FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all

But we are still seeing it on 3.5.20...

Now what is even more weird is that besides that, we are getting what seem complete random bogus entries, like this ones:

FATAL: Bungled /etc/squid/squid_acls line 11796: acl xyz_allow_url
FATAL: Bungled /etc/squid/squid_acls line 10905: acl abc_url108 dstdomai

FATAL: Bungled squid_acls line 10033: http_access allow def_
FATAL: Bungled squid_acls line 9605: http_access allow ms_src_teste ms_t
FATAL: Bungled squid_acls line 11025: http_access allow xpto

The problem is that these lines are 100% correct , they are not "bungled" at all,  it seems the parser as some issue with it.

Each time we get an error like this, Squid does a full restart, and that on CentOS 6 is time consuming, maybe 2-3 minutes, time enough for our users reaching for the Helpdesk tickets... On CentOS 7, since the restart is very fast, we do not have many complaints...

Our ACL file is currently 7500 lines, mostly comprising of dstdomain and url_regex entries. We support a lot of costumers (about 100) and each one has a specific configuration. Mostly have while lists, and there are a  few blacklists in the middle... so, not very big in terms of line numbers, but somewhat  complex regarding site/VLAN entries...

For security/privacy reasons I cannot share my complete configuration, namely the ACL's part, since they contain client information, but I am able to get detailed logging and/or redacted information...

So... anyone has any clue what is happenning ?



--


Francisco Amaro
Email: mailto:[hidden email]




--
Francisco Amaro
Email: [hidden email]

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Amos Jeffries
Administrator
On 19/10/17 03:33, Francisco Amaro wrote:
> Hi all,
>
> So, we've been (slowly) investigating this issue these past days, and
> have reached some conclusions.
>
> a) First, and sorry for not being clear on that, this only happens on
> "squid reload", not on full stop/start cycles.
> A full start cycle does not give this error, that's why we are pretty
> sure the issue is not a bungled ACL, per se.

That is unfortunately not a guarantee of success. squid.conf and files
it refers to can be changed while Squid is running and have no effect
until next reload happens.

Also what system infrastructure is running this "squid reload" command?

The correct commands to reload squid.conf are "squid -k reconfigure"
[just reload] or "squid -k check" [validate then reload *if* it passes].
Any other system scripts or processes (eg systemd or openrc, or upstart)
trying to manage Squid may be adding bugs of its own.


>
> b) Besides that, it only happens when the proxy has some load on it, we
> have been unable to reproduce this on
> our test machines, since they do not have a significant load. We've been
> postponing our ACL's changes for
> early morning/lunch-hour these past days, and we didn't have had a FATAL
> error since last week.
>
> c) Although we don't have non US-ASCII on the ACL's themselves, we have
> a lot of them on comments, so
> I've stripped down all the comments of the ACL's files and tried that on
> our test bed, it didn't generare any
> kind of errors, but since they do not have some load on it, we are not
> sure if it was the comments stripping
> or the lack of load on the server...

Filtering the config down to 7-bit / US-ASCII characters is a very good
idea. We have done a bunch of work making Squid handle UTF-8, but the
config file is one are where that is far from complete so the behaviour
is unpredictable.

I do know that a Unicode character at the end of a line or Windows /
MacOS line terminators have been known to cause admin a lot of headaches.


>
> So, we are now searching for an "easy" way of generating traffic on a
> squid server, so we can, at least, be
> able to reproduce the error, so we can try the different solutions....
>


The Squid dev team tend to use Apache Bench ("ab" tool) to generate lots
of simple transactions very fast if the same URL repeatedly is
sufficient for the test or caching is disabled.
   Or a tool called Polygraph if the URLs have to be variable and
stressing all sorts of cache and post-cache logic.

I believe there are also tools that can take a feed of web traffic and
replay it. But I have not had to use any of those yet on Squid.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5.20 and 3.1.23 getting re-started with bogus "FATAL: Bungled" acls

Alex Rousskov
On 10/19/2017 08:11 AM, Amos Jeffries wrote:

> The Squid dev team tend to use Apache Bench ("ab" tool) to generate lots
> of simple transactions very fast if the same URL repeatedly is
> sufficient for the test or caching is disabled.
>   Or a tool called Polygraph if the URLs have to be variable and
> stressing all sorts of cache and post-cache logic.
>
> I believe there are also tools that can take a feed of web traffic and
> replay it. But I have not had to use any of those yet on Squid.

Web Polygraph can play URL lists (but not full HTTP request captures).

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users