Question about shutdown_lifetime behavior.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about shutdown_lifetime behavior.

Cody Herzog
Hello.

I have a question about shutdown_lifetime:

http://www.squid-cache.org/Doc/config/shutdown_lifetime/

Does Squid always wait the full amount of time before shutting down, even after all active connections have closed?

Based on my testing, it seems like it does.

However, I found some documentation which indicates that Squid should close as soon as all active connections have closed:

https://www.safaribooksonline.com/library/view/squid-the-definitive/0596001622/re91.html

"Squid finally exits when all client connections have been closed or when this timeout occurs."

Is that the expected behavior, or will Squid always stay open for the full timeout, as I'm observing in my testing?

I searched the FAQ and the Internet at large, but couldn't find a definitive answer.

I'm running Squid 3.5.27 on Ubuntu 16.04.

If Squid does not have an internal mechanism to complete the shutdown when all active connections have closed, then I may have to create my own based on polling with 'netstat'.

Thanks.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question about shutdown_lifetime behavior.

Amos Jeffries
Administrator
On 01/05/18 18:04, Cody Herzog wrote:
> Hello.
>
> I have a question about shutdown_lifetime:
>
> http://www.squid-cache.org/Doc/config/shutdown_lifetime/
>
> Does Squid always wait the full amount of time before shutting down, even after all active connections have closed?

For now yes. Long-term the plan is to have it exit if no clients are
connected, but we have not yet finalized how that is to work internally.


>
> I'm running Squid 3.5.27 on Ubuntu 16.04.
>
> If Squid does not have an internal mechanism to complete the shutdown when all active connections have closed, then I may have to create my own based on polling with 'netstat'.
>

You can try. Squid maintains a number of helpers though which may still
be doing things with sockets after clients are gone. Besides dev time
detecting when all that is completed is the largest blocker at present
to the long-term project.

If you really need a fast shutdown feel free to set the lifetime config
value to a shorter time, or simply call "squid -k shutdown" (or
equivalent init script command).

The "-k shutdown" behaviour is; on first use to begin the shutdown
period, on second call to trigger the end of shutdown as if
shutdown_lifetime was reached.


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question about shutdown_lifetime behavior.

Cody Herzog
In reply to this post by Cody Herzog

Thanks very much for the quick response, Amos.

 

For my use case, I would like Squid to exit when all client connections have been closed or when the timeout occurs, whichever comes first.

 

My instances of Squid may be handling several persistent WebSocket connections, and I don't want to disrupt those. I will occasionally need to perform maintenance, so I want a safe way to stop Squid without disrupting user activity.

 

I am using a fairly simple Squid configuration, with no caching, so I suspect that I can simply monitor the number of active Squid TCP connections using 'netstat', and then execute the second shutdown command when I detect that all those connections are closed.

 

I've been using the following command to count the number of active Squid TCP connections of port 443, which is the only port I use:

 

netstat -nat | grep ".*:443.*:" | grep ESTABLISHED | wc -l

 

That seems to give me what I want.

 

Is it possible that bad things could happen by stopping Squid when I see that all the TCP connections have closed?

 

Thanks.

 

-Cody


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question about shutdown_lifetime behavior.

Amos Jeffries
Administrator
On 02/05/18 05:17, Cody Herzog wrote:
> Thanks very much for the quick response, Amos.
>
>  
>
> For my use case, I would like Squid to exit when all client connections
> have been closed or when the timeout occurs, whichever comes first.
>

Then Squids behaviour already matches your requirements. The "or when
timeout occurs" is shutdown_lifetime and you do not have to do anything.

>
> My instances of Squid may be handling several persistent WebSocket
> connections, and I don't want to disrupt those. I will occasionally need
> to perform maintenance, so I want a safe way to stop Squid without
> disrupting user activity.
>

Nod. It will also likely be handling CONNECT tunnels for other things.
Be aware that these connections can last indefinitely - some have been
known to last on a timescale of several weeks.

If your maintenance is with squid.conf or things loaded by it use "squid
-k reconfigure" instead of a restart cycle. Squid can reload its config
fine with just a pause for any active clients - your netstat approach
could be useful to pick a time with minimal connections to reload the
config.

>
> I am using a fairly simple Squid configuration, with no caching, so I
> suspect that I can simply monitor the number of active Squid TCP
> connections using 'netstat', and then execute the second shutdown
> command when I detect that all those connections are closed.
>

Since WebSockets is part of your situation netstat will almost certainly
not work as well as you suspect. These connections can be very
surprising in their lifetimes.


>
> I've been using the following command to count the number of active
> Squid TCP connections of port 443, which is the only port I use:
>
>  
>
> netstat -nat | grep ".*:443.*:" | grep ESTABLISHED | wc -l
>
>  
>
> That seems to give me what I want.
>

Hmm, you might be able to get more useful info from the Squid
filedescriptors report.
  squidclient mgr:filedescriptors


>  
>
> Is it possible that bad things could happen by stopping Squid when I see
> that all the TCP connections have closed?

Should not be any bad things if you just use -k shutdown (twice). Squid
will take the time it needs for a clean (but immediate) shutdown so long
as you only do it twice, not more and do not use "kill" command.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question about shutdown_lifetime behavior.

Cody Herzog
In reply to this post by Cody Herzog
Thanks again, Amos.

>Then Squids behaviour already matches your requirements. The "or when timeout occurs" is shutdown_lifetime and you do not have to do anything.

I'm confused by this. After issuing the first shutdown command, my desired behavior is for Squid to shut itself down fully as soon as it detects that there is no more client activity. My understanding from your first response is that Squid will always wait the full timeout, regardless of whether activity seems to have stopped.

Ultimately, I ended up having to implement something custom anyway.

My clients have multiple persistent WebSocket connections to different services. Some of those services are critical, and some are not. Shutdown must be postponed until there are no more active connections to critical services. Connections to the non-critical services can last a very long time, and I don't want to postpone shutdown because of those connections.
 
To get my desired behavior, I ended up polling 'cache_object://cache.host.name/active_requests' to check if any critical requests are active, and if not, then I issue the shutdown command.

The tricky thing is that 'cache_object' cannot be queried after the first shutdown command has been issued, because Squid does not accept any new connections. Therefore, I had to find a way to prevent new client connections, while still allowing 'cache_object' to keep working. I was able to accomplish this by modifying squid.conf and issuing a 'reconfigure'.  Thankfully, the 'reconfigure' which prevents new client connections does not seem to break any of the active WebSocket connections.

So, here is my shutdown sequence:

1.) Modify config file to prevent new client connections and 'reconfigure'.
2.) Poll active requests until there are no connections to critical services.
3.) Issue the shutdown command with a small value for shutdown_lifetime.

Does that sounds reasonable?
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question about shutdown_lifetime behavior.

Alex Rousskov
On 05/02/2018 03:07 PM, Cody Herzog wrote:

> So, here is my shutdown sequence:
>
> 1.) Modify config file to prevent new client connections and 'reconfigure'.
> 2.) Poll active requests until there are no connections to critical services.
> 3.) Issue the shutdown command with a small value for shutdown_lifetime.
>
> Does that sounds reasonable?

It sounds like a reasonable (and clever!) workaround to me.

Ideally, a single "squid -k shutdown" should result in everything you
need done by Squid automatically, with a new ACL-driven directive to
identify "connections to critical services".

Please keep in mind that Squid reconfiguration is still a disruptive
action (unfortunately). If it does not usually affect your critical
services, great, but I am fairly sure it is possible to come up with
specific cases where reconfiguration kills in-progress transactions.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Question about shutdown_lifetime behavior.

Cody Herzog
In reply to this post by Cody Herzog
Thanks, Alex.

I do have some concerns that 'reconfigure' may cause disruptions in certain situations, but I haven't seen it yet.

Perhaps it is most likely to cause problems when connections are first being established, or when they are changing states.

It seems to do a good job of not disrupting established WebSocket connections.

One other idea I had was to use 'iptables' to prevent new TCP connections to port 443 as the first phase of shutdown. I think that would probably work, and would not have the potential bad behavior of 'reconfigure'.

For my 'reconfigure', I'm simply commenting out the https_port line which causes Squid to listen on 443.

Thanks again.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users