Originserver load balancing and health checks in Squid reverse proxy mode

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Originserver load balancing and health checks in Squid reverse proxy mode

Chris-4
Hi all,

I'm trying to figure out the best way to use squid (version 3.5.27) in
reverse proxy mode in regard to originserver health checks and load
balancing.

So far I had been using the round-robin originserver cache peer
selection algorithm while using weight to favor originservers with
closer proximity/lower latency.

The problem: if one cache_peer is dead it takes ages for squid to choose
the second originserver. It does look as if (e.g. if one originserver
has a weight of 32, the other of 2) squid tries the dead server several
times before accessing the other one.

Now instead of using round-robin plus weight it would be best to use
weighted-round-robin. But as I understand it, this wouldn't work with
originserver if (as it's normally the case) the originserver won't
handle icp or htcp requests. Did I miss sth. here? Would background-ping
work?

I tried weighted-round-robin and background-ping on originservers but
got only an evenly distributed request handling even if ones
originservers rtt would be less than half of the others. But then again,
those originservers won't handle icp requests.

So what's the best solution to a) choose the originserver with the
lowest rtt and b) still have a fast switch if one of the originservers
switches into dead state?

Would I have to span another proxy (like e.g. HAProxy) between Squid and
originserver or better install Squid on those originservers as well
(only for serving icp requests from the squid fellows)?

Is there a better way to update the dead state of an originserver?

How do you handle this?

Thanks a lot,

Chris

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Eliezer Croitoru-3
Hey Chris,

The main question is " what do you need squid for?"
If you need squid for caching it one thing.
RFC compliance is another thing..
Anyway Haproxy is better in Load Balancing and traffic control/management.
If you need load balancer use haproxy.
If you need caching for very specific known use cases then use it.
For general purpose these days it might not work as you might expect.

Take into account that browsers cache lots of things, even these who shouldn't so the gain/profit
should be tested first.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Chris
Sent: Monday, February 8, 2021 4:41 PM
To: [hidden email]
Subject: [squid-users] Originserver load balancing and health checks in Squid reverse proxy mode

Hi all,

I'm trying to figure out the best way to use squid (version 3.5.27) in
reverse proxy mode in regard to originserver health checks and load
balancing.

So far I had been using the round-robin originserver cache peer
selection algorithm while using weight to favor originservers with
closer proximity/lower latency.

The problem: if one cache_peer is dead it takes ages for squid to choose
the second originserver. It does look as if (e.g. if one originserver
has a weight of 32, the other of 2) squid tries the dead server several
times before accessing the other one.

Now instead of using round-robin plus weight it would be best to use
weighted-round-robin. But as I understand it, this wouldn't work with
originserver if (as it's normally the case) the originserver won't
handle icp or htcp requests. Did I miss sth. here? Would background-ping
work?

I tried weighted-round-robin and background-ping on originservers but
got only an evenly distributed request handling even if ones
originservers rtt would be less than half of the others. But then again,
those originservers won't handle icp requests.

So what's the best solution to a) choose the originserver with the
lowest rtt and b) still have a fast switch if one of the originservers
switches into dead state?

Would I have to span another proxy (like e.g. HAProxy) between Squid and
originserver or better install Squid on those originservers as well
(only for serving icp requests from the squid fellows)?

Is there a better way to update the dead state of an originserver?

How do you handle this?

Thanks a lot,

Chris

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Amos Jeffries
Administrator
In reply to this post by Chris-4
On 9/02/21 3:40 am, Chris wrote:

> Hi all,
>
> I'm trying to figure out the best way to use squid (version 3.5.27) in
> reverse proxy mode in regard to originserver health checks and load
> balancing.
>
> So far I had been using the round-robin originserver cache peer
> selection algorithm while using weight to favor originservers with
> closer proximity/lower latency.
>

Ok.


> The problem: if one cache_peer is dead it takes ages for squid to choose
> the second originserver. It does look as if (e.g. if one originserver
> has a weight of 32, the other of 2) squid tries the dead server several
> times before accessing the other one.
>

The DEAD check by default requires 10 failures in a row to trigger. This
is configurable with the connect-fail-limit=N option.


> Now instead of using round-robin plus weight it would be best to use
> weighted-round-robin. But as I understand it, this wouldn't work with
> originserver if (as it's normally the case) the originserver won't
> handle icp or htcp requests. Did I miss sth. here? Would background-ping
> work?

Well, kind of.

ICP/HTCP is just a protocol. Most origin servers do not support them,
but some do. Especially if the server is not a true origin but a
reverse-proxy.


>
> I tried weighted-round-robin and background-ping on originservers but
> got only an evenly distributed request handling even if ones
> originservers rtt would be less than half of the others. But then again,
> those originservers won't handle icp requests.

RTT is retrieved from ICMP data primarily. Check your Squid is built
with --enable-icmp, the pinger helper is operational, and that ICMP Echo
traffic is working on all possible network routes between your Squid and
the peer server(s).


>
> So what's the best solution to a) choose the originserver with the
> lowest rtt and b) still have a fast switch if one of the originservers
> switches into dead state?


Check whether the RTT is actually being measured properly by Squid
(debug_options ALL,1 44,3 15,8). If the peers are fast enough responding
or close enough in the network RTT could come out as a 0 value or some N
value equal for both peer. ie. neither being "closer".


Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Chris-4
Hi,

thank you Amos, this is bringing me into the right direction.

Now I know what I'll have to debug: the pinger.

Cache.log shows:

2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
2021/02/09 14:49:27| pinger: ICMP socket opened.
2021/02/09 14:49:27| pinger: ICMPv6 socket opened
2021/02/09 14:49:27| Pinger exiting.

and that last line "pinger exiting" looks like a problem here.

Squid is used as a package from ubuntu bionic, it's configured with
"--enable-icmp" as stated by squid -v.

Now I explicitly wrote a "pinger_enable on" and the pinger_program path
(in this case: "/usr/lib/squid/pinger" ) into the squid.conf  (as well
as icmp_query on) and reconfigured but the cache.log still shows:

"Pinger exiting"

So I don't understand why the pinger is exiting. The pinger_program is
owned by root and has 0755 execution rights. Normal ping commands do
work and show the one originserver at ttl=53 and time=50 while the other
is at ttl=56 and time=155 - so a RTT comparison for weighted-round-robin
should work here.

Any hints on how I can find out why the pinger is exiting? Right now I'm
debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
the pinger exits.

The Originservers are defined by (with icp/htcp disabled):

cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
no-netdb-exchange weighted-round-robin originserver name=srv1
forceddomain=[domainname]

cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
no-netdb-exchange weighted-round-robin originserver name=srv2
forceddomain=[domainname]


Thank you for your help,

Chris





On 09.02.21 04:23, Amos Jeffries wrote:

> On 9/02/21 3:40 am, Chris wrote:
>> Hi all,
>>
>> I'm trying to figure out the best way to use squid (version 3.5.27)
>> in reverse proxy mode in regard to originserver health checks and
>> load balancing.
>>
>> So far I had been using the round-robin originserver cache peer
>> selection algorithm while using weight to favor originservers with
>> closer proximity/lower latency.
>>
>
> Ok.
>
>
>> The problem: if one cache_peer is dead it takes ages for squid to
>> choose the second originserver. It does look as if (e.g. if one
>> originserver has a weight of 32, the other of 2) squid tries the dead
>> server several times before accessing the other one.
>>
>
> The DEAD check by default requires 10 failures in a row to trigger.
> This is configurable with the connect-fail-limit=N option.
>
>
>> Now instead of using round-robin plus weight it would be best to use
>> weighted-round-robin. But as I understand it, this wouldn't work with
>> originserver if (as it's normally the case) the originserver won't
>> handle icp or htcp requests. Did I miss sth. here? Would
>> background-ping work?
>
> Well, kind of.
>
> ICP/HTCP is just a protocol. Most origin servers do not support them,
> but some do. Especially if the server is not a true origin but a
> reverse-proxy.
>
>
>>
>> I tried weighted-round-robin and background-ping on originservers but
>> got only an evenly distributed request handling even if ones
>> originservers rtt would be less than half of the others. But then
>> again, those originservers won't handle icp requests.
>
> RTT is retrieved from ICMP data primarily. Check your Squid is built
> with --enable-icmp, the pinger helper is operational, and that ICMP
> Echo traffic is working on all possible network routes between your
> Squid and the peer server(s).
>
>
>>
>> So what's the best solution to a) choose the originserver with the
>> lowest rtt and b) still have a fast switch if one of the
>> originservers switches into dead state?
>
>
> Check whether the RTT is actually being measured properly by Squid
> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
> responding or close enough in the network RTT could come out as a 0
> value or some N value equal for both peer. ie. neither being "closer".
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Chris-4
Oh, that lib won't help, sorry, forget about my pinger_program path

So do I have to recompile squid myself and than install the pinger as
described here:
https://wiki.squid-cache.org/SquidFaq/OperatingSquid#Using_ICMP_to_Measure_the_Network 
?

On 09.02.21 16:03, Chris wrote:

> Hi,
>
> thank you Amos, this is bringing me into the right direction.
>
> Now I know what I'll have to debug: the pinger.
>
> Cache.log shows:
>
> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
> 2021/02/09 14:49:27| pinger: ICMP socket opened.
> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
> 2021/02/09 14:49:27| Pinger exiting.
>
> and that last line "pinger exiting" looks like a problem here.
>
> Squid is used as a package from ubuntu bionic, it's configured with
> "--enable-icmp" as stated by squid -v.
>
> Now I explicitly wrote a "pinger_enable on" and the pinger_program
> path (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as
> well as icmp_query on) and reconfigured but the cache.log still shows:
>
> "Pinger exiting"
>
> So I don't understand why the pinger is exiting. The pinger_program is
> owned by root and has 0755 execution rights. Normal ping commands do
> work and show the one originserver at ttl=53 and time=50 while the
> other is at ttl=56 and time=155 - so a RTT comparison for
> weighted-round-robin should work here.
>
> Any hints on how I can find out why the pinger is exiting? Right now
> I'm debuging with debug_options ALL,1 44,3 15,8 but don't see a reason
> why the pinger exits.
>
> The Originservers are defined by (with icp/htcp disabled):
>
> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
> no-netdb-exchange weighted-round-robin originserver name=srv1
> forceddomain=[domainname]
>
> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
> no-netdb-exchange weighted-round-robin originserver name=srv2
> forceddomain=[domainname]
>
>
> Thank you for your help,
>
> Chris
>
>
>
>
>
> On 09.02.21 04:23, Amos Jeffries wrote:
>> On 9/02/21 3:40 am, Chris wrote:
>>> Hi all,
>>>
>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>> in reverse proxy mode in regard to originserver health checks and
>>> load balancing.
>>>
>>> So far I had been using the round-robin originserver cache peer
>>> selection algorithm while using weight to favor originservers with
>>> closer proximity/lower latency.
>>>
>>
>> Ok.
>>
>>
>>> The problem: if one cache_peer is dead it takes ages for squid to
>>> choose the second originserver. It does look as if (e.g. if one
>>> originserver has a weight of 32, the other of 2) squid tries the
>>> dead server several times before accessing the other one.
>>>
>>
>> The DEAD check by default requires 10 failures in a row to trigger.
>> This is configurable with the connect-fail-limit=N option.
>>
>>
>>> Now instead of using round-robin plus weight it would be best to use
>>> weighted-round-robin. But as I understand it, this wouldn't work
>>> with originserver if (as it's normally the case) the originserver
>>> won't handle icp or htcp requests. Did I miss sth. here? Would
>>> background-ping work?
>>
>> Well, kind of.
>>
>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>> but some do. Especially if the server is not a true origin but a
>> reverse-proxy.
>>
>>
>>>
>>> I tried weighted-round-robin and background-ping on originservers
>>> but got only an evenly distributed request handling even if ones
>>> originservers rtt would be less than half of the others. But then
>>> again, those originservers won't handle icp requests.
>>
>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>> with --enable-icmp, the pinger helper is operational, and that ICMP
>> Echo traffic is working on all possible network routes between your
>> Squid and the peer server(s).
>>
>>
>>>
>>> So what's the best solution to a) choose the originserver with the
>>> lowest rtt and b) still have a fast switch if one of the
>>> originservers switches into dead state?
>>
>>
>> Check whether the RTT is actually being measured properly by Squid
>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>> responding or close enough in the network RTT could come out as a 0
>> value or some N value equal for both peer. ie. neither being "closer".
>>
>>
>> Amos
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Eliezer Croitoru-3
In reply to this post by Chris-4
Maybe its apparmor.
pinger needs to have a setuid permission as root.
its a pinger and needs root privleges as far as i remember.

Eliezer


On Tue, Feb 9, 2021, 17:03 Chris <[hidden email]> wrote:
Hi,

thank you Amos, this is bringing me into the right direction.

Now I know what I'll have to debug: the pinger.

Cache.log shows:

2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
2021/02/09 14:49:27| pinger: ICMP socket opened.
2021/02/09 14:49:27| pinger: ICMPv6 socket opened
2021/02/09 14:49:27| Pinger exiting.

and that last line "pinger exiting" looks like a problem here.

Squid is used as a package from ubuntu bionic, it's configured with
"--enable-icmp" as stated by squid -v.

Now I explicitly wrote a "pinger_enable on" and the pinger_program path
(in this case: "/usr/lib/squid/pinger" ) into the squid.conf  (as well
as icmp_query on) and reconfigured but the cache.log still shows:

"Pinger exiting"

So I don't understand why the pinger is exiting. The pinger_program is
owned by root and has 0755 execution rights. Normal ping commands do
work and show the one originserver at ttl=53 and time=50 while the other
is at ttl=56 and time=155 - so a RTT comparison for weighted-round-robin
should work here.

Any hints on how I can find out why the pinger is exiting? Right now I'm
debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
the pinger exits.

The Originservers are defined by (with icp/htcp disabled):

cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
no-netdb-exchange weighted-round-robin originserver name=srv1
forceddomain=[domainname]

cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
no-netdb-exchange weighted-round-robin originserver name=srv2
forceddomain=[domainname]


Thank you for your help,

Chris





On 09.02.21 04:23, Amos Jeffries wrote:
> On 9/02/21 3:40 am, Chris wrote:
>> Hi all,
>>
>> I'm trying to figure out the best way to use squid (version 3.5.27)
>> in reverse proxy mode in regard to originserver health checks and
>> load balancing.
>>
>> So far I had been using the round-robin originserver cache peer
>> selection algorithm while using weight to favor originservers with
>> closer proximity/lower latency.
>>
>
> Ok.
>
>
>> The problem: if one cache_peer is dead it takes ages for squid to
>> choose the second originserver. It does look as if (e.g. if one
>> originserver has a weight of 32, the other of 2) squid tries the dead
>> server several times before accessing the other one.
>>
>
> The DEAD check by default requires 10 failures in a row to trigger.
> This is configurable with the connect-fail-limit=N option.
>
>
>> Now instead of using round-robin plus weight it would be best to use
>> weighted-round-robin. But as I understand it, this wouldn't work with
>> originserver if (as it's normally the case) the originserver won't
>> handle icp or htcp requests. Did I miss sth. here? Would
>> background-ping work?
>
> Well, kind of.
>
> ICP/HTCP is just a protocol. Most origin servers do not support them,
> but some do. Especially if the server is not a true origin but a
> reverse-proxy.
>
>
>>
>> I tried weighted-round-robin and background-ping on originservers but
>> got only an evenly distributed request handling even if ones
>> originservers rtt would be less than half of the others. But then
>> again, those originservers won't handle icp requests.
>
> RTT is retrieved from ICMP data primarily. Check your Squid is built
> with --enable-icmp, the pinger helper is operational, and that ICMP
> Echo traffic is working on all possible network routes between your
> Squid and the peer server(s).
>
>
>>
>> So what's the best solution to a) choose the originserver with the
>> lowest rtt and b) still have a fast switch if one of the
>> originservers switches into dead state?
>
>
> Check whether the RTT is actually being measured properly by Squid
> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
> responding or close enough in the network RTT could come out as a 0
> value or some N value equal for both peer. ie. neither being "closer".
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Chris-4
Hi Elizer, this helped, it seems as if I got the pinger working.

It's now owned by root in the same group as the squid user and the
setuid set.

So I used chown root:squidusergroup and chmod u+s on the pinger (and in
ubuntu it is actually found under /usr/lib/squid/pinger ).

Now with debug 42,3 I get some values as:

Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2]
                                 0 Echo Reply      155ms 7 hops

and

Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1]
                                 0 Echo Reply      11ms 9 hops

but squid is still allocating the requests evenly and not using those
ping times in weighted-round-robin.

Does the weighted-round-robin need some time to use those rtt values?

Best Regards,

Chris


On 09.02.21 16:19, NgTech LTD wrote:

> Maybe its apparmor.
> pinger needs to have a setuid permission as root.
> its a pinger and needs root privleges as far as i remember.
>
> Eliezer
>
>
> On Tue, Feb 9, 2021, 17:03 Chris <[hidden email]> wrote:
>
>> Hi,
>>
>> thank you Amos, this is bringing me into the right direction.
>>
>> Now I know what I'll have to debug: the pinger.
>>
>> Cache.log shows:
>>
>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>> 2021/02/09 14:49:27| Pinger exiting.
>>
>> and that last line "pinger exiting" looks like a problem here.
>>
>> Squid is used as a package from ubuntu bionic, it's configured with
>> "--enable-icmp" as stated by squid -v.
>>
>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf  (as well
>> as icmp_query on) and reconfigured but the cache.log still shows:
>>
>> "Pinger exiting"
>>
>> So I don't understand why the pinger is exiting. The pinger_program is
>> owned by root and has 0755 execution rights. Normal ping commands do
>> work and show the one originserver at ttl=53 and time=50 while the other
>> is at ttl=56 and time=155 - so a RTT comparison for weighted-round-robin
>> should work here.
>>
>> Any hints on how I can find out why the pinger is exiting? Right now I'm
>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>> the pinger exits.
>>
>> The Originservers are defined by (with icp/htcp disabled):
>>
>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>> no-netdb-exchange weighted-round-robin originserver name=srv1
>> forceddomain=[domainname]
>>
>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>> no-netdb-exchange weighted-round-robin originserver name=srv2
>> forceddomain=[domainname]
>>
>>
>> Thank you for your help,
>>
>> Chris
>>
>>
>>
>>
>>
>> On 09.02.21 04:23, Amos Jeffries wrote:
>>> On 9/02/21 3:40 am, Chris wrote:
>>>> Hi all,
>>>>
>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>> in reverse proxy mode in regard to originserver health checks and
>>>> load balancing.
>>>>
>>>> So far I had been using the round-robin originserver cache peer
>>>> selection algorithm while using weight to favor originservers with
>>>> closer proximity/lower latency.
>>>>
>>> Ok.
>>>
>>>
>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>> choose the second originserver. It does look as if (e.g. if one
>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>> server several times before accessing the other one.
>>>>
>>> The DEAD check by default requires 10 failures in a row to trigger.
>>> This is configurable with the connect-fail-limit=N option.
>>>
>>>
>>>> Now instead of using round-robin plus weight it would be best to use
>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>> originserver if (as it's normally the case) the originserver won't
>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>> background-ping work?
>>> Well, kind of.
>>>
>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>> but some do. Especially if the server is not a true origin but a
>>> reverse-proxy.
>>>
>>>
>>>> I tried weighted-round-robin and background-ping on originservers but
>>>> got only an evenly distributed request handling even if ones
>>>> originservers rtt would be less than half of the others. But then
>>>> again, those originservers won't handle icp requests.
>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>> Echo traffic is working on all possible network routes between your
>>> Squid and the peer server(s).
>>>
>>>
>>>> So what's the best solution to a) choose the originserver with the
>>>> lowest rtt and b) still have a fast switch if one of the
>>>> originservers switches into dead state?
>>>
>>> Check whether the RTT is actually being measured properly by Squid
>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>> responding or close enough in the network RTT could come out as a 0
>>> value or some N value equal for both peer. ie. neither being "closer".
>>>
>>>
>>> Amos
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Chris-4
This is what I'm seeing in peer_select in cache_log with 44,3 debug options:

2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(258)
peerSelectDnsPaths: Find IP destination for: '[the_request]' via
[ip_cache_peer_srv1]
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(280)
peerSelectDnsPaths: Found sources for '[the_request]'
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(281)
peerSelectDnsPaths:   always_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(282)
peerSelectDnsPaths:    never_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv2]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv3]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(295)
peerSelectDnsPaths:        timedout = 0
2021/02/09 16:25:11.588 kid1| 44,3| peer_select.cc(79) ~ps_state:
[the_request]

and than in access.log I have:

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv1]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv2]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv3]

evenly distributed.

So it's not using the weighted-round-robin that should have srv1 at
11ms, while srv2 and srv3 are at about 150ms in regard to pinger.

What did I miss in configuring weighted-round-robin?

Best Regards,

Chris







On 09.02.21 17:09, Chris wrote:

> Hi Elizer, this helped, it seems as if I got the pinger working.
>
> It's now owned by root in the same group as the squid user and the
> setuid set.
>
> So I used chown root:squidusergroup and chmod u+s on the pinger (and
> in ubuntu it is actually found under /usr/lib/squid/pinger ).
>
> Now with debug 42,3 I get some values as:
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2]
>                                 0 Echo Reply      155ms 7 hops
>
> and
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1]
>                                 0 Echo Reply      11ms 9 hops
>
> but squid is still allocating the requests evenly and not using those
> ping times in weighted-round-robin.
>
> Does the weighted-round-robin need some time to use those rtt values?
>
> Best Regards,
>
> Chris
>
>
> On 09.02.21 16:19, NgTech LTD wrote:
>> Maybe its apparmor.
>> pinger needs to have a setuid permission as root.
>> its a pinger and needs root privleges as far as i remember.
>>
>> Eliezer
>>
>>
>> On Tue, Feb 9, 2021, 17:03 Chris <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> thank you Amos, this is bringing me into the right direction.
>>>
>>> Now I know what I'll have to debug: the pinger.
>>>
>>> Cache.log shows:
>>>
>>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>>> 2021/02/09 14:49:27| Pinger exiting.
>>>
>>> and that last line "pinger exiting" looks like a problem here.
>>>
>>> Squid is used as a package from ubuntu bionic, it's configured with
>>> "--enable-icmp" as stated by squid -v.
>>>
>>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as well
>>> as icmp_query on) and reconfigured but the cache.log still shows:
>>>
>>> "Pinger exiting"
>>>
>>> So I don't understand why the pinger is exiting. The pinger_program is
>>> owned by root and has 0755 execution rights. Normal ping commands do
>>> work and show the one originserver at ttl=53 and time=50 while the
>>> other
>>> is at ttl=56 and time=155 - so a RTT comparison for
>>> weighted-round-robin
>>> should work here.
>>>
>>> Any hints on how I can find out why the pinger is exiting? Right now
>>> I'm
>>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>>> the pinger exits.
>>>
>>> The Originservers are defined by (with icp/htcp disabled):
>>>
>>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv1
>>> forceddomain=[domainname]
>>>
>>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv2
>>> forceddomain=[domainname]
>>>
>>>
>>> Thank you for your help,
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>> On 09.02.21 04:23, Amos Jeffries wrote:
>>>> On 9/02/21 3:40 am, Chris wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>>> in reverse proxy mode in regard to originserver health checks and
>>>>> load balancing.
>>>>>
>>>>> So far I had been using the round-robin originserver cache peer
>>>>> selection algorithm while using weight to favor originservers with
>>>>> closer proximity/lower latency.
>>>>>
>>>> Ok.
>>>>
>>>>
>>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>>> choose the second originserver. It does look as if (e.g. if one
>>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>>> server several times before accessing the other one.
>>>>>
>>>> The DEAD check by default requires 10 failures in a row to trigger.
>>>> This is configurable with the connect-fail-limit=N option.
>>>>
>>>>
>>>>> Now instead of using round-robin plus weight it would be best to use
>>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>>> originserver if (as it's normally the case) the originserver won't
>>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>>> background-ping work?
>>>> Well, kind of.
>>>>
>>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>>> but some do. Especially if the server is not a true origin but a
>>>> reverse-proxy.
>>>>
>>>>
>>>>> I tried weighted-round-robin and background-ping on originservers but
>>>>> got only an evenly distributed request handling even if ones
>>>>> originservers rtt would be less than half of the others. But then
>>>>> again, those originservers won't handle icp requests.
>>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>>> Echo traffic is working on all possible network routes between your
>>>> Squid and the peer server(s).
>>>>
>>>>
>>>>> So what's the best solution to a) choose the originserver with the
>>>>> lowest rtt and b) still have a fast switch if one of the
>>>>> originservers switches into dead state?
>>>>
>>>> Check whether the RTT is actually being measured properly by Squid
>>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>>> responding or close enough in the network RTT could come out as a 0
>>>> value or some N value equal for both peer. ie. neither being "closer".
>>>>
>>>>
>>>> Amos
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Eliezer Croitoru-3
This is more of Amos and Alex area.
In general I think that haproxy does load balancing much more efficiently then squid.
It is being used in production for years so I'm not sure why you should use Squid for LB.
If you want to resolve this issue then be my guest I can only offer so QA and advice here and there.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: [hidden email]
Zoom: Coming soon


-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Chris
Sent: Tuesday, February 9, 2021 6:36 PM
To: [hidden email]
Subject: Re: [squid-users] Originserver load balancing and health checks in Squid reverse proxy mode

This is what I'm seeing in peer_select in cache_log with 44,3 debug options:

2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(258)
peerSelectDnsPaths: Find IP destination for: '[the_request]' via
[ip_cache_peer_srv1]
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(280)
peerSelectDnsPaths: Found sources for '[the_request]'
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(281)
peerSelectDnsPaths:   always_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(282)
peerSelectDnsPaths:    never_direct = DENIED
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv2]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv3]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
peerSelectDnsPaths:      cache_peer = local=0.0.0.0
remote=[ip_cache_peer_srv1]:[port] flags=1
2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(295)
peerSelectDnsPaths:        timedout = 0
2021/02/09 16:25:11.588 kid1| 44,3| peer_select.cc(79) ~ps_state:
[the_request]

and than in access.log I have:

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv1]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv2]

TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv3]

evenly distributed.

So it's not using the weighted-round-robin that should have srv1 at
11ms, while srv2 and srv3 are at about 150ms in regard to pinger.

What did I miss in configuring weighted-round-robin?

Best Regards,

Chris







On 09.02.21 17:09, Chris wrote:

> Hi Elizer, this helped, it seems as if I got the pinger working.
>
> It's now owned by root in the same group as the squid user and the
> setuid set.
>
> So I used chown root:squidusergroup and chmod u+s on the pinger (and
> in ubuntu it is actually found under /usr/lib/squid/pinger ).
>
> Now with debug 42,3 I get some values as:
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2]
>                                 0 Echo Reply      155ms 7 hops
>
> and
>
> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1]
>                                 0 Echo Reply      11ms 9 hops
>
> but squid is still allocating the requests evenly and not using those
> ping times in weighted-round-robin.
>
> Does the weighted-round-robin need some time to use those rtt values?
>
> Best Regards,
>
> Chris
>
>
> On 09.02.21 16:19, NgTech LTD wrote:
>> Maybe its apparmor.
>> pinger needs to have a setuid permission as root.
>> its a pinger and needs root privleges as far as i remember.
>>
>> Eliezer
>>
>>
>> On Tue, Feb 9, 2021, 17:03 Chris <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> thank you Amos, this is bringing me into the right direction.
>>>
>>> Now I know what I'll have to debug: the pinger.
>>>
>>> Cache.log shows:
>>>
>>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>>> 2021/02/09 14:49:27| Pinger exiting.
>>>
>>> and that last line "pinger exiting" looks like a problem here.
>>>
>>> Squid is used as a package from ubuntu bionic, it's configured with
>>> "--enable-icmp" as stated by squid -v.
>>>
>>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as well
>>> as icmp_query on) and reconfigured but the cache.log still shows:
>>>
>>> "Pinger exiting"
>>>
>>> So I don't understand why the pinger is exiting. The pinger_program is
>>> owned by root and has 0755 execution rights. Normal ping commands do
>>> work and show the one originserver at ttl=53 and time=50 while the
>>> other
>>> is at ttl=56 and time=155 - so a RTT comparison for
>>> weighted-round-robin
>>> should work here.
>>>
>>> Any hints on how I can find out why the pinger is exiting? Right now
>>> I'm
>>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>>> the pinger exits.
>>>
>>> The Originservers are defined by (with icp/htcp disabled):
>>>
>>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv1
>>> forceddomain=[domainname]
>>>
>>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>>> no-netdb-exchange weighted-round-robin originserver name=srv2
>>> forceddomain=[domainname]
>>>
>>>
>>> Thank you for your help,
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>> On 09.02.21 04:23, Amos Jeffries wrote:
>>>> On 9/02/21 3:40 am, Chris wrote:
>>>>> Hi all,
>>>>>
>>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>>> in reverse proxy mode in regard to originserver health checks and
>>>>> load balancing.
>>>>>
>>>>> So far I had been using the round-robin originserver cache peer
>>>>> selection algorithm while using weight to favor originservers with
>>>>> closer proximity/lower latency.
>>>>>
>>>> Ok.
>>>>
>>>>
>>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>>> choose the second originserver. It does look as if (e.g. if one
>>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>>> server several times before accessing the other one.
>>>>>
>>>> The DEAD check by default requires 10 failures in a row to trigger.
>>>> This is configurable with the connect-fail-limit=N option.
>>>>
>>>>
>>>>> Now instead of using round-robin plus weight it would be best to use
>>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>>> originserver if (as it's normally the case) the originserver won't
>>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>>> background-ping work?
>>>> Well, kind of.
>>>>
>>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>>> but some do. Especially if the server is not a true origin but a
>>>> reverse-proxy.
>>>>
>>>>
>>>>> I tried weighted-round-robin and background-ping on originservers but
>>>>> got only an evenly distributed request handling even if ones
>>>>> originservers rtt would be less than half of the others. But then
>>>>> again, those originservers won't handle icp requests.
>>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>>> Echo traffic is working on all possible network routes between your
>>>> Squid and the peer server(s).
>>>>
>>>>
>>>>> So what's the best solution to a) choose the originserver with the
>>>>> lowest rtt and b) still have a fast switch if one of the
>>>>> originservers switches into dead state?
>>>>
>>>> Check whether the RTT is actually being measured properly by Squid
>>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>>> responding or close enough in the network RTT could come out as a 0
>>>> value or some N value equal for both peer. ie. neither being "closer".
>>>>
>>>>
>>>> Amos
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>>>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Alex Rousskov
In reply to this post by Chris-4
On 2/9/21 11:35 AM, Chris wrote:
> This is what I'm seeing in peer_select in cache_log with 44,3 debug
> options:

Add (at least) "15,3" to your debug_options and then look for
getWeightedRoundRobinParent lines. Looking at mgr:server_list Cache
Manager page may also be useful.


> Does the weighted-round-robin need some time to use those rtt values?

I am not 100% sure, but I think the answer to that question is "no".

If you want to see the details of that peer selection algorithm, look
for the Squid function with that name. I bet it has _some_ undocumented
surprises, but I do not know whether they are relevant to your specific
use case.


HTH,

Alex.


> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(258)
> peerSelectDnsPaths: Find IP destination for: '[the_request]' via
> [ip_cache_peer_srv1]
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(280)
> peerSelectDnsPaths: Found sources for '[the_request]'
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(281)
> peerSelectDnsPaths:   always_direct = DENIED
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(282)
> peerSelectDnsPaths:    never_direct = DENIED
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv1]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv2]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv3]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(292)
> peerSelectDnsPaths:      cache_peer = local=0.0.0.0
> remote=[ip_cache_peer_srv1]:[port] flags=1
> 2021/02/09 16:25:11.588 kid1| 44,2| peer_select.cc(295)
> peerSelectDnsPaths:        timedout = 0
> 2021/02/09 16:25:11.588 kid1| 44,3| peer_select.cc(79) ~ps_state:
> [the_request]
>
> and than in access.log I have:
>
> TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv1]
>
> TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv2]
>
> TCP_MISS/200 [the_request] ROUND_ROBIN_PARENT/[ip_cache_peer_srv3]
>
> evenly distributed.
>
> So it's not using the weighted-round-robin that should have srv1 at
> 11ms, while srv2 and srv3 are at about 150ms in regard to pinger.
>
> What did I miss in configuring weighted-round-robin?
>
> Best Regards,
>
> Chris
>
>
>
>
>
>
>
> On 09.02.21 17:09, Chris wrote:
>> Hi Elizer, this helped, it seems as if I got the pinger working.
>>
>> It's now owned by root in the same group as the squid user and the
>> setuid set.
>>
>> So I used chown root:squidusergroup and chmod u+s on the pinger (and
>> in ubuntu it is actually found under /usr/lib/squid/pinger ).
>>
>> Now with debug 42,3 I get some values as:
>>
>> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv2]
>>                                 0 Echo Reply      155ms 7 hops
>>
>> and
>>
>> Icmp.cc(95) Log: pingerLog: [timestamp] [ip_srv1]
>>                                 0 Echo Reply      11ms 9 hops
>>
>> but squid is still allocating the requests evenly and not using those
>> ping times in weighted-round-robin.
>>
>> Does the weighted-round-robin need some time to use those rtt values?
>>
>> Best Regards,
>>
>> Chris
>>
>>
>> On 09.02.21 16:19, NgTech LTD wrote:
>>> Maybe its apparmor.
>>> pinger needs to have a setuid permission as root.
>>> its a pinger and needs root privleges as far as i remember.
>>>
>>> Eliezer
>>>
>>>
>>> On Tue, Feb 9, 2021, 17:03 Chris <[hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> thank you Amos, this is bringing me into the right direction.
>>>>
>>>> Now I know what I'll have to debug: the pinger.
>>>>
>>>> Cache.log shows:
>>>>
>>>> 2021/02/09 14:49:27| pinger: Initialising ICMP pinger ...
>>>> 2021/02/09 14:49:27| pinger: ICMP socket opened.
>>>> 2021/02/09 14:49:27| pinger: ICMPv6 socket opened
>>>> 2021/02/09 14:49:27| Pinger exiting.
>>>>
>>>> and that last line "pinger exiting" looks like a problem here.
>>>>
>>>> Squid is used as a package from ubuntu bionic, it's configured with
>>>> "--enable-icmp" as stated by squid -v.
>>>>
>>>> Now I explicitly wrote a "pinger_enable on" and the pinger_program path
>>>> (in this case: "/usr/lib/squid/pinger" ) into the squid.conf (as well
>>>> as icmp_query on) and reconfigured but the cache.log still shows:
>>>>
>>>> "Pinger exiting"
>>>>
>>>> So I don't understand why the pinger is exiting. The pinger_program is
>>>> owned by root and has 0755 execution rights. Normal ping commands do
>>>> work and show the one originserver at ttl=53 and time=50 while the
>>>> other
>>>> is at ttl=56 and time=155 - so a RTT comparison for
>>>> weighted-round-robin
>>>> should work here.
>>>>
>>>> Any hints on how I can find out why the pinger is exiting? Right now
>>>> I'm
>>>> debuging with debug_options ALL,1 44,3 15,8 but don't see a reason why
>>>> the pinger exits.
>>>>
>>>> The Originservers are defined by (with icp/htcp disabled):
>>>>
>>>> cache_peer [ipv4_address_srv1] parent [http_port] 0 no-digest
>>>> no-netdb-exchange weighted-round-robin originserver name=srv1
>>>> forceddomain=[domainname]
>>>>
>>>> cache_peer [ipv4_address_srv2] parent [http_port] 0 no-digest
>>>> no-netdb-exchange weighted-round-robin originserver name=srv2
>>>> forceddomain=[domainname]
>>>>
>>>>
>>>> Thank you for your help,
>>>>
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 09.02.21 04:23, Amos Jeffries wrote:
>>>>> On 9/02/21 3:40 am, Chris wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'm trying to figure out the best way to use squid (version 3.5.27)
>>>>>> in reverse proxy mode in regard to originserver health checks and
>>>>>> load balancing.
>>>>>>
>>>>>> So far I had been using the round-robin originserver cache peer
>>>>>> selection algorithm while using weight to favor originservers with
>>>>>> closer proximity/lower latency.
>>>>>>
>>>>> Ok.
>>>>>
>>>>>
>>>>>> The problem: if one cache_peer is dead it takes ages for squid to
>>>>>> choose the second originserver. It does look as if (e.g. if one
>>>>>> originserver has a weight of 32, the other of 2) squid tries the dead
>>>>>> server several times before accessing the other one.
>>>>>>
>>>>> The DEAD check by default requires 10 failures in a row to trigger.
>>>>> This is configurable with the connect-fail-limit=N option.
>>>>>
>>>>>
>>>>>> Now instead of using round-robin plus weight it would be best to use
>>>>>> weighted-round-robin. But as I understand it, this wouldn't work with
>>>>>> originserver if (as it's normally the case) the originserver won't
>>>>>> handle icp or htcp requests. Did I miss sth. here? Would
>>>>>> background-ping work?
>>>>> Well, kind of.
>>>>>
>>>>> ICP/HTCP is just a protocol. Most origin servers do not support them,
>>>>> but some do. Especially if the server is not a true origin but a
>>>>> reverse-proxy.
>>>>>
>>>>>
>>>>>> I tried weighted-round-robin and background-ping on originservers but
>>>>>> got only an evenly distributed request handling even if ones
>>>>>> originservers rtt would be less than half of the others. But then
>>>>>> again, those originservers won't handle icp requests.
>>>>> RTT is retrieved from ICMP data primarily. Check your Squid is built
>>>>> with --enable-icmp, the pinger helper is operational, and that ICMP
>>>>> Echo traffic is working on all possible network routes between your
>>>>> Squid and the peer server(s).
>>>>>
>>>>>
>>>>>> So what's the best solution to a) choose the originserver with the
>>>>>> lowest rtt and b) still have a fast switch if one of the
>>>>>> originservers switches into dead state?
>>>>>
>>>>> Check whether the RTT is actually being measured properly by Squid
>>>>> (debug_options ALL,1 44,3 15,8). If the peers are fast enough
>>>>> responding or close enough in the network RTT could come out as a 0
>>>>> value or some N value equal for both peer. ie. neither being "closer".
>>>>>
>>>>>
>>>>> Amos
>>>>> _______________________________________________
>>>>> squid-users mailing list
>>>>> [hidden email]
>>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Originserver load balancing and health checks in Squid reverse proxy mode

Amos Jeffries
Administrator
On 10/02/21 9:59 am, Alex Rousskov wrote:

> On 2/9/21 11:35 AM, Chris wrote:
>> This is what I'm seeing in peer_select in cache_log with 44,3 debug
>> options:
>
> Add (at least) "15,3" to your debug_options and then look for
> getWeightedRoundRobinParent lines. Looking at mgr:server_list Cache
> Manager page may also be useful.
>
>
>> Does the weighted-round-robin need some time to use those rtt values?
>
> I am not 100% sure, but I think the answer to that question is "no".
>

Well, only the time it takes to find out what the RTT actually is. As
soon as RTT is known it is used.

This algorithm works like round-robin in that each selection cycle the
peer with fewest "uses" gets selected. However, where normal round-robin
adds "(1 * weight) uses" to a peer each time it gets selected, this
algorithm adds "(RTT * weight) uses".

The debug lines Alex mentioned will tell you what weighted_rtt your
proxy is using as its RTT count per use. If that shows "1", then the RTT
is too small to calculate speed difference (math code improvements
needed to cope with modern fast networks).

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users