squid+apache+php-fpm How squid could mark DEAD a peer where Apache is alive and php-fpm dead?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

squid+apache+php-fpm How squid could mark DEAD a peer where Apache is alive and php-fpm dead?

Patrick Chemla

Hi,


Thank you all for the great work on Squid.


I got last week a bad "issue" where a squid Version 4.0.23 set to send requests to 4 VMs didn't detect a dead peer.


On each VM are running Apache as the frontal, and php-fpm as php scripts execution.


Squid is using sourcehash load balancing algorithm.


But, if the php-fpm is down or busy, Apache will return a 504 Gateway timeout to Squid and browser, and squid will continue to send requests to same Apache, even if the answer will always be HTTP code 504, because Apache is ALIVE.


Apache is there, respawning, but the user is not getting the right answer.


And all IPs that are connecting to this VM, will fail, so 25% of all users will suffer unavailability.


I have setup some watchdogs scripts that will now, from the VM intern itself, check if there is a 504 code returned by Apache, and in that case, will restart php-fpm.


But how to tell squid to mark dead a peer that returns 504 code ?


Thanks

Patrick



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid+apache+php-fpm How squid could mark DEAD a peer where Apache is alive and php-fpm dead?

Alex Rousskov
On 6/28/20 10:56 AM, Patrick Chemla wrote:

> I got last week a bad "issue" where a squid Version 4.0.23

Please upgrade to the latest Squid v4 (or v5). While the problem you are
writing about is unlikely to disappear, there are other problems in your
Squid version. My answer is based on Squid v5, but I hope it applies to
the latest v4 as well.


> But how to tell squid to mark dead a peer that returns 504 code ?

I did not check closely, but I believe that Squid's peer viability
decisions are based primarily on transport layer problems -- valid HTTP
responses will not mark a peer dead, regardless of the HTTP status code.

Modern Squids should reforward HTTP 504 responses to other peers (where
possible). See the last bullet in reforwarding _exceptions_ list at
https://wiki.squid-cache.org/SquidFaq/InnerWorkings#When_does_Squid_re-forward_a_client_request.3F

Please note that your configuration may not have any peers eligible for
reforwarding the request to. For example, if you have no parent caches,
then a sourcehash peer will be the only candidate, with no reforwarding
candidates at all AFAICT. The algorithm that computes the list of
peering candidates (including candidates for reforwarding) is documented
at
https://wiki.squid-cache.org/Features/LoadBalance#Overall_peer_selection_logic

If reforwarding is not good enough in your use case, you should be able
to ban a peer using cache_peer_access ACLs. The trigger for a ban would
probably be an annotate_transaction ACL in http_reply_access. Turning
the ban off would probably require an external ACL (or adding a
counter-like ACL support to Squid itself).

Alternatively, one could enhance Squid with a new ACL-driven
cache_peer_failure directive that would allow the admin to determine
which valid HTTP responses should qualify for marking a cache peer as
dead. Adding this feature would be straightforward AFAICT. It would be a
generally useful option. IIRC, there were requests for a similar feature
in the past. Quality pull requests or development sponsorship should be
welcomed IMO.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users