New Squid 3.5 reconfigure causes service down

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

New Squid 3.5 reconfigure causes service down

Nicola Ferrari (#554252)
Hi List!

We're experiencing problems with a just-upgraded squid install (from
Debian 8 to Debian 9, using packages in repos). Here are the details
from squid -version:

Squid Cache: Version 3.5.23
Service Name: squid
Debian linux

We use "negotiate kerberos" authenticators to offer Active Directory SSO.

We're also running squidguard.

Lines in config file are:

[...]
# NEGOTIATE KERBEROS AUTH
auth_param negotiate program /usr/lib/squid/negotiate_wrapper_auth
--ntlm /usr/$
auth_param negotiate children 60
auth_param negotiate keep_alive off
[...]
url_rewrite_program /usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf


The problem is that, issuing the "squid -k reconfigure" command (i.e. to
adjust acls in conf file) the result is not just a configuration reload,
but authenticators processes are restarting, causing an "out-of-service"
for all users, for a courple of minutes.

Basically the same issue as in this thread:
https://serverfault.com/questions/247835/squid-3-reloading-makes-it-stop-serving-requests

I'm in doubt if reducing helpers number would be a good idea, since we
need to serve ca. 300 simultaneous users.

Before the recent upgrade, with the previous Debian8, reload took some
seconds only..

Is there any best-practice to get an "Hot-Configurable" system?
Do you have any suggestion?

Thanks!
Best regards,


PS: English isn't my first language, so please excuse any mistakes..

--
+---------------------+
| Linux User  #554252 |
+---------------------+

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: New Squid 3.5 reconfigure causes service down

Alex Rousskov
On 10/05/2017 03:20 AM, Nicola Ferrari (#554252) wrote:

> issuing the "squid -k reconfigure" command (i.e. to
> adjust acls in conf file) the result is not just a configuration reload,
> but authenticators processes are restarting,

As you have discovered already, running heavy unnecessary actions is a
known problem with Squid hot reconfiguration support. An upgrade may
have an effect on certain aspects of that problem, but the problem
itself is as old as Squid.


> causing an "out-of-service" for all users, for a courple of minutes.

The "couple of minutes" part might be related to your upgrade and, if
so, you may be able to avoid such delays. For list readers not familiar
with Debian releases, which _Squid_ version are you upgrading from?


> Basically the same issue as in this thread:
> https://serverfault.com/questions/247835/squid-3-reloading-makes-it-stop-serving-requests

The symptoms are the same but the underlying cause may be different
(unless you have already checked but did not tell us).


> I'm in doubt if reducing helpers number would be a good idea, since we
> need to serve ca. 300 simultaneous users.
>
> Before the recent upgrade, with the previous Debian8, reload took some
> seconds only..
>
> Is there any best-practice to get an "Hot-Configurable" system?
> Do you have any suggestion?


I suggest to start by figuring our what Squid is doing during those
"couple of minutes" if you have not already. The mailing list thread
linked from the above serverfault answer shows how to do that and has
several potentially useful comments. Compare the new logs with those of
your older Squid. What has changed related to the startup delays?


FWIW, there is now a low-priority project to support fast ACL-only
reconfiguration. We have the initial high-level design and some code,
but it will take a while (possibly a year or more) to complete at its
current priority.

http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: New Squid 3.5 reconfigure causes service down

Nicola Ferrari (#554252)
On 05/10/2017 18:25, Alex Rousskov wrote:
> The "couple of minutes" part might be related to your upgrade and, if
> so, you may be able to avoid such delays. For list readers not familiar
> with Debian releases, which _Squid_ version are you upgrading from?
>

I was running squid 3.4 on top of Debian 8 (jessie)
I upgraded to squid 3.5 on top of Debian 9 (stretch)

> I suggest to start by figuring our what Squid is doing during those
> "couple of minutes" if you have not already.

What I notice by checking cache.log is that it stops for a while on

helperOpenServers: Starting 1/60 'ntlm_auth' processes
2017/10/05 11:36:06 kid1| Starting new ntlmauthenticator helpers...

This was not a usual behaviour on Squid 3.4;

At the moment of the upgrade, I had to adjust various path from
"/squid3" to "/squid" ..

I checked authenticators path and other occurrences in conf file,
everything seems to be ok.

Just for testing purposes, I would try my config on a new clean install,
just to be sure this is not related to the upgrade in some way, and let
you know!

Nick



--
+---------------------+
| Linux User  #554252 |
+---------------------+

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: New Squid 3.5 reconfigure causes service down

Alex Rousskov
On 10/05/2017 10:44 AM, Nicola Ferrari (#554252) wrote:
> On 05/10/2017 18:25, Alex Rousskov wrote:
>> The "couple of minutes" part might be related to your upgrade and, if
>> so, you may be able to avoid such delays. For list readers not familiar
>> with Debian releases, which _Squid_ version are you upgrading from?

> I was running squid 3.4 on top of Debian 8 (jessie)
> I upgraded to squid 3.5 on top of Debian 9 (stretch)

>> I suggest to start by figuring our what Squid is doing during those
>> "couple of minutes" if you have not already.
>
> What I notice by checking cache.log is that it stops for a while on
>
> helperOpenServers: Starting 1/60 'ntlm_auth' processes
> 2017/10/05 11:36:06 kid1| Starting new ntlmauthenticator helpers...
>
> This was not a usual behaviour on Squid 3.4;

The next task is to figure out what changed related to that line (i.e.,
to starting ntlmauthenticator helpers). Here are a few things you may
want to check: Do you start the same number of helpers as before? Does
starting a single helper take longer in v3.5 than in v3.4? Does Squid
v3.5 consume a lot more RAM before it tries to start that helper than
Squid v3.4 consumed? Does Squid v3.5 helper itself consume a lot more
RAM than Squid v3.4 helper?

Something must have changed. If you can pinpoint that change, it is
likely that you can reverse or work around it. Since we probably do not
know what that change is, have no access to your server, and no free
time to investigate, you have to narrow the suspects down yourself.


> Just for testing purposes, I would try my config on a new clean install

That is a good initial test as well. And remember that you are not
looking for something that broke or does not work. You are looking for
something that works differently. And your initial focus should be on
things that affect helper startup (i.e., fork() and exec() system
calls): process size, number of processes, etc. Commands like strace can
help you measure delays down to a single system call level if needed.


Good luck,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: New Squid 3.5 reconfigure causes service down

Amos Jeffries
Administrator
In reply to this post by Nicola Ferrari (#554252)
On 06/10/17 05:44, Nicola Ferrari (#554252) wrote:

> On 05/10/2017 18:25, Alex Rousskov wrote:
>> The "couple of minutes" part might be related to your upgrade and, if
>> so, you may be able to avoid such delays. For list readers not familiar
>> with Debian releases, which _Squid_ version are you upgrading from?
>>
>
> I was running squid 3.4 on top of Debian 8 (jessie)
> I upgraded to squid 3.5 on top of Debian 9 (stretch)
>
>> I suggest to start by figuring our what Squid is doing during those
>> "couple of minutes" if you have not already.
>
> What I notice by checking cache.log is that it stops for a while on
>
> helperOpenServers: Starting 1/60 'ntlm_auth' processes
> 2017/10/05 11:36:06 kid1| Starting new ntlmauthenticator helpers...
>
> This was not a usual behaviour on Squid 3.4;

The behaviour of starting helpers has been present since forever -
though it may not have been logged correctly. The "Starting N/N
'helper_name' processes" log entry was added with dynamic helper in
Squid-3.2, so should have been visible in Jesse.

The 1/60 indicates that the number of ntlm_auth helpers running was 1
less than your startup=N configuration value. The N defaults to the max
value (60) if not configured explicitly.


>
> At the moment of the upgrade, I had to adjust various path from
> "/squid3" to "/squid" ..
>
> I checked authenticators path and other occurrences in conf file,
> everything seems to be ok.

Did Squid start properly and at least seem to work okay after the
upgrade and before you manually ran the "-k reconfigure" ?

FYI: Stretch brings somewhat deeper SELinux integration in the
background. The packaged init script updates the SELinux permissions for
cache_dir. But if you have any custom directories for other things you
may need to run /sbin/restorecon on them manually after any changes to
the path or OS permissions - or do it anyway just in case SELinux is
being confused.

>
> Just for testing purposes, I would try my config on a new clean install,
> just to be sure this is not related to the upgrade in some way, and let
> you know!
>

Please also try a full clean restart of Squid:

  Shutdown completely using the init script. If any 'squid' or
'(squid-N)' process remains after that use kill -9 to halt that process,
and manually delete the squid.pid / squid3.pid file if any still exists.

  Starting Squid using the Stretch package init script should then
ensure that the expected paths have the right permissions, and runs the
'-k parse' checks for you.

Since this is the Samba NTLM helper you should also check that the
Samba, winbind etc components still have it enabled. Behaviour is
undefined if the OS components are only partially functioning.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users