Quantcast

No failover when default parent proxy fails (Squid 3.5.12)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

No failover when default parent proxy fails (Squid 3.5.12)

Jens Offenbach
Hi,
I have two parent proxies configured, but Squid seems to stick to the default proxy even when the proxy cannot be reached:
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:13 kid1| Detected DEAD Parent: proxy.mycompany.de
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed

No failover takes place... I must miss someting in my config. Can someone please help me. I am on Ubuntu 16.04.2:
$ squid -v
Squid Cache: Version 3.5.12
Service Name: squid
Ubuntu linux
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' 'BUILDCXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-build-info=Ubuntu linux' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'

This is my squid.conf
# ACCESS CONTROLS
# -----------------------------------------------------------------------------
  # OpenStack Networks
  acl localnet src 10.116.0.0/20
  acl localnet src 10.30.200.0/21
  acl localnet src 10.30.216.0/22

  # mycompany Networks
  acl to_matnet dst 139.2.0.0/16
  acl to_matnet dst 193.96.112.0/21
  acl to_matnet dst 192.109.216.0/24
  acl to_matnet dst 100.1.4.0/22
  acl to_matnet dst 10.0.0.0/8
  acl to_matnet dst 172.16.0.0/12
  acl to_matnet dst 192.168.0.0/16

  # SSL-Ports
  acl SSL_ports port 443 # https
  acl SSL_ports port 563 # snews
  acl SSL_ports port 873 # rsync

  # Safe-Ports
  acl Safe_ports port 80  # http
  acl Safe_ports port 21  # ftp
  acl Safe_ports port 443 # https
  acl Safe_ports port 70  # gopher
  acl Safe_ports port 210 # wais
  acl Safe_ports port 1025-65535 # unregistered ports
  acl Safe_ports port 280 # http-mgmt
  acl Safe_ports port 488 # gss-http
  acl Safe_ports port 591 # filemaker
  acl Safe_ports port 777 # multiling http
  acl Safe_ports port 631 # cups
  acl Safe_ports port 873 # rsync
  acl Safe_ports port 901 # SWAT

  # HTTPS
  acl CONNECT method CONNECT

  http_access deny  !Safe_ports
  http_access deny  CONNECT !SSL_ports
  http_access allow manager localhost
  http_access deny  manager
  http_access allow localnet
  http_access allow localhost
  http_access deny all

# NETWORK OPTIONS
# -----------------------------------------------------------------------------
  http_port 10.30.202.99:3128

# OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
# -----------------------------------------------------------------------------
  cache_peer proxy.mycompany.de parent 8080 0 no-query no-digest default
  cache_peer  roxy.mycompany.de parent 8080 0 no-query no-digest

# MEMORY CACHE OPTIONS
# -----------------------------------------------------------------------------
  maximum_object_size_in_memory 8 MB
  memory_replacement_policy heap LFUDA
  cache_mem 256 MB

# DISK CACHE OPTIONS
# -----------------------------------------------------------------------------
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir ufs /var/cache/squid 88894 16 256 max-size=10737418240

# LOGFILE OPTIONS
# -----------------------------------------------------------------------------
  access_log daemon:/var/log/squid/access.log squid

# OPTIONS FOR TROUBLESHOOTING
# -----------------------------------------------------------------------------
  cache_log /var/log/squid/cache.log
  coredump_dir /var/log/squid

# OPTIONS FOR TUNING THE CACHE
# -----------------------------------------------------------------------------
  max_stale 6 days
  shutdown_lifetime 5 seconds

# ADMINISTRATIVE PARAMETERS
# -----------------------------------------------------------------------------
  visible_hostname mos-proxy.mycompany.com

# OPTIONS INFLUENCING REQUEST FORWARDING
# -----------------------------------------------------------------------------
  always_direct allow to_matnet
  never_direct  allow all

# DNS OPTIONS
# -----------------------------------------------------------------------------
  dns_nameservers 139.2.34.171
  dns_nameservers 139.2.34.37

# MISCELLANEOUS
# -----------------------------------------------------------------------------
  memory_pools off
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Amos Jeffries
Administrator
On 15/03/2017 7:06 p.m., Jens Offenbach wrote:

> Hi,
> I have two parent proxies configured, but Squid seems to stick to the default proxy even when the proxy cannot be reached:
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| Detected DEAD Parent: proxy.mycompany.de
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed


Appearances can be deceiving at times. This shows that several "a
connections were attempted but does not specify if that was the orider
they were initiated or not.
 It also does not indicate whether that was HTTP or probe for
ressurected peer.

You have disabled the probes though (no-query no-digest) so it is
unlikely Squid will ever detect a DEAD peer as coming alive again. This
makes me suspect that the "roxy" peer was also detected dead some time
earlier. If that has happened then the "FIRST_AVAILABLE" peer selection
algorithm will produce no possible routes and Squid falls back to the
DEFAULT peer ... which happens to be that one you see in the log and
will try that peer until it works again.

If you want to see the exact route selection results add this to your
squid.conf:
 debug_options 44,2


Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Jens Offenbach
@Amos:
Thanks a lot for your help.

I have modified my squid.conf:
  cache_peer proxy.mycompany.de parent 8080 0 connect-timeout=5 connect-fail-limit=3 default
  cache_peer  roxy.mycompany.de parent 8080 0 connect-timeout=5 connect-fail-limit=3
 
Do you think this will work? It it currently a little bit difficult for my to test the new configuration.

Regards,
Jens


Gesendet: Mittwoch, 15. März 2017 um 15:38 Uhr
Von: "Amos Jeffries" <[hidden email]>
An: [hidden email]
Betreff: Re: [squid-users] No failover when default parent proxy fails (Squid 3.5.12)
On 15/03/2017 7:06 p.m., Jens Offenbach wrote:

> Hi,
> I have two parent proxies configured, but Squid seems to stick to the default proxy even when the proxy cannot be reached:
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| Detected DEAD Parent: proxy.mycompany.de
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed


Appearances can be deceiving at times. This shows that several "a
connections were attempted but does not specify if that was the orider
they were initiated or not.
It also does not indicate whether that was HTTP or probe for
ressurected peer.

You have disabled the probes though (no-query no-digest) so it is
unlikely Squid will ever detect a DEAD peer as coming alive again. This
makes me suspect that the "roxy" peer was also detected dead some time
earlier. If that has happened then the "FIRST_AVAILABLE" peer selection
algorithm will produce no possible routes and Squid falls back to the
DEFAULT peer ... which happens to be that one you see in the log and
will try that peer until it works again.

If you want to see the exact route selection results add this to your
squid.conf:
debug_options 44,2


Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Jens Offenbach
In reply to this post by Amos Jeffries
Failover does not seem to work properly in case of HTTPS. When the primary parent proxy fails, it takes minutes until the download starts and in some cases it never starts.

Is there anything that must be configured specifially in case of HTTPS and timeouts for failover?

This is my squid.conf:
# ACCESS CONTROLS
# -----------------------------------------------------------------------------
  # Local Networks
  acl localnet src 139.2.0.0/16
  acl localnet src 193.96.112.0/21
  acl localnet src 192.109.216.0/24
  acl localnet src 100.1.4.0/22
  acl localnet src 10.0.0.0/8
  acl localnet src 172.16.0.0/12
  acl localnet src 192.168.0.0/16

  # mycompany Networks
  acl to_matnet dst 139.2.0.0/16
  acl to_matnet dst 193.96.112.0/21
  acl to_matnet dst 192.109.216.0/24
  acl to_matnet dst 100.1.4.0/22
  acl to_matnet dst 10.0.0.0/8
  acl to_matnet dst 172.16.0.0/12
  acl to_matnet dst 192.168.0.0/16

  # SSL-Ports
  acl SSL_ports port 443 # https
  acl SSL_ports port 563 # snews
  acl SSL_ports port 873 # rsync

  # Safe-Ports
  acl Safe_ports port 80  # http
  acl Safe_ports port 21  # ftp
  acl Safe_ports port 443 # https
  acl Safe_ports port 70  # gopher
  acl Safe_ports port 210 # wais
  acl Safe_ports port 1025-65535 # unregistered ports
  acl Safe_ports port 280 # http-mgmt
  acl Safe_ports port 488 # gss-http
  acl Safe_ports port 591 # filemaker
  acl Safe_ports port 777 # multiling http
  acl Safe_ports port 631 # cups
  acl Safe_ports port 873 # rsync
  acl Safe_ports port 901 # SWAT

  # HTTPS
  acl CONNECT method CONNECT

  http_access deny !Safe_ports
  http_access deny CONNECT !SSL_ports
  http_access allow manager localhost
  http_access deny  manager
  http_access allow localnet
  http_access allow localhost
  http_access deny all

# NETWORK OPTIONS
# -----------------------------------------------------------------------------
  http_port 3128

# OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
# -----------------------------------------------------------------------------
  cache_peer proxy.mycompany.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=3 default
  cache_peer  roxy.mycompany.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=3

# MEMORY CACHE OPTIONS
# -----------------------------------------------------------------------------
  maximum_object_size_in_memory 8 MB
  memory_replacement_policy heap LFUDA
  cache_mem 256 MB

# DISK CACHE OPTIONS
# -----------------------------------------------------------------------------
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir ufs /var/cache/squid 88894 16 256 max-size=10737418240

# LOGFILE OPTIONS
# -----------------------------------------------------------------------------
  access_log daemon:/var/log/squid/access.log squid

# OPTIONS FOR TROUBLESHOOTING
# -----------------------------------------------------------------------------
  cache_log /var/log/squid/cache.log
  coredump_dir /var/log/squid

# OPTIONS FOR TUNING THE CACHE
# -----------------------------------------------------------------------------
  max_stale 6 days
  shutdown_lifetime 5 seconds

# ADMINISTRATIVE PARAMETERS
# -----------------------------------------------------------------------------
  visible_hostname proxy.mycompany.com

# OPTIONS INFLUENCING REQUEST FORWARDING
# -----------------------------------------------------------------------------
  always_direct allow to_matnet
  never_direct  allow all

# DNS OPTIONS
# -----------------------------------------------------------------------------
  dns_nameservers 139.2.34.171
  dns_nameservers 139.2.34.37

# MISCELLANEOUS
# -----------------------------------------------------------------------------
  memory_pools off

Regards,
Jens
 

Gesendet: Mittwoch, 15. März 2017 um 15:38 Uhr
Von: "Amos Jeffries" <[hidden email]>
An: [hidden email]
Betreff: Re: [squid-users] No failover when default parent proxy fails (Squid 3.5.12)
On 15/03/2017 7:06 p.m., Jens Offenbach wrote:

> Hi,
> I have two parent proxies configured, but Squid seems to stick to the default proxy even when the proxy cannot be reached:
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| Detected DEAD Parent: proxy.mycompany.de
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:13 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed
> 2017/03/15 06:40:43 kid1| TCP connection to proxy.mycompany.de/8080 failed


Appearances can be deceiving at times. This shows that several "a
connections were attempted but does not specify if that was the orider
they were initiated or not.
It also does not indicate whether that was HTTP or probe for
ressurected peer.

You have disabled the probes though (no-query no-digest) so it is
unlikely Squid will ever detect a DEAD peer as coming alive again. This
makes me suspect that the "roxy" peer was also detected dead some time
earlier. If that has happened then the "FIRST_AVAILABLE" peer selection
algorithm will produce no possible routes and Squid falls back to the
DEFAULT peer ... which happens to be that one you see in the log and
will try that peer until it works again.

If you want to see the exact route selection results add this to your
squid.conf:
debug_options 44,2


Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Alex Rousskov
On 03/15/2017 11:23 PM, Jens Offenbach wrote:
> Failover does not seem to work properly in case of HTTPS.

We have recently discovered and fixed two bugs in failure recovery code
code for tunneled (not bumped) HTTPS connections:
http://lists.squid-cache.org/pipermail/squid-dev/2017-March/008243.html

I do not know whether our (not yet officially reviewed) fixes will help
in your use case and whether you can try v4 or v5 to test them.


HTH,

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Jens Offenbach
Thanks for your quick response...

I have also configured, but the value seems not to be honored:
connect_timeout 30 seconds

The primary peer is down, but Squid does not print any "Dead parent" in the logs. Every HTTPS request is forwarded to the primary peer and it takes 1 minute until the secondary peer gets used, even with "connect_timeout 30 seconds". I think, I am facing the first issue that has been fixed by your patch.

Are there any plans to backport this fix to Xenial APT repositories or to create a new Debian package for Squid4/5?

Regards,
Jens 


Gesendet: Donnerstag, 16. März 2017 um 06:37 Uhr
Von: "Alex Rousskov" <[hidden email]>
An: [hidden email]
Cc: "Jens Offenbach" <[hidden email]>
Betreff: Re: [squid-users] No failover when default parent proxy fails (Squid 3.5.12)
On 03/15/2017 11:23 PM, Jens Offenbach wrote:
> Failover does not seem to work properly in case of HTTPS.

We have recently discovered and fixed two bugs in failure recovery code
code for tunneled (not bumped) HTTPS connections:
http://lists.squid-cache.org/pipermail/squid-dev/2017-March/008243.html

I do not know whether our (not yet officially reviewed) fixes will help
in your use case and whether you can try v4 or v5 to test them.


HTH,

Alex.
 
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Amos Jeffries
Administrator
On 16/03/2017 7:05 p.m., Jens Offenbach wrote:

> Thanks for your quick response...
>
> I have also configured, but the value seems not to be honored:
> connect_timeout 30 seconds
>
> The primary peer is down, but Squid does not print any "Dead parent"
> in the logs. Every HTTPS request is forwarded to the primary peer and
> it takes 1 minute until the secondary peer gets used, even with
> "connect_timeout 30 seconds". I think, I am facing the first issue
> that has been fixed by your patch.
>

The global config options being ignored completely is correct because
your peer have individual connect-timeout=5 settings.

So, those 5sec timeouts should be used instead now as before.

Though note that they apply only to how long a TCP connection (SYN,
SYN-ACK) is waited for. There is also a dns_timeout and peer selection
timeout that apply separately to the act of connecting. And a
forward_timeout global limit that all those operations have to fit
within, including retries.


Did you have a chance to try the debug setting I suggested at the
beginning? That will give you an immediate view about what Squid is
detecting as usable paths for each and every request and at what times
relative to the DEAD/LIVE notice.



> Are there any plans to backport this fix to Xenial APT repositories
> or to create a new Debian package for Squid4/5?

That is up to the Ubuntu server team, but I think it Unlikely. Zesty is
the current stable and things like this generally dont have enough
widespread impact to qualify for LTS backports.

Debian is now frozen to stabilize for the "Buster" release, that will
contain Squid-3.5.23 plus some few critical patches which are already
set. A Squid-4 package is ready and waiting for the release freeze to
end before it goes public in the Debian Unstable/Testing repos.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Jens Offenbach
This is the sceanrio;

Squid 3.5.12 is installed on "squid-proxy.mycompany.com". The two parent proxies are:
- Primary: proxy.mycompany.de:8080 (139.2.1.3)
- Fallback: roxy.mycompany.de:8080 (139.2.1.4)

I have misunderstood the "default" option in "cache_peer". When I got it right, it has the meaning of a fallback, so I switched it to "roxy.mycompany.de". "proxy.mycompany.de" should always be used and "roxy.mycompany.de" only when "proxy.mycompany.de" fails.

squid.conf:

# ACCESS CONTROLS
# -----------------------------------------------------------------------------
  # Local Networks
  acl localnet src 139.2.0.0/16
  acl localnet src 193.96.112.0/21
  acl localnet src 192.109.216.0/24
  acl localnet src 100.1.4.0/22
  acl localnet src 10.0.0.0/8
  acl localnet src 172.16.0.0/12
  acl localnet src 192.168.0.0/16

  # Materna Networks
  acl to_matnet dst 139.2.0.0/16
  acl to_matnet dst 193.96.112.0/21
  acl to_matnet dst 192.109.216.0/24
  acl to_matnet dst 100.1.4.0/22
  acl to_matnet dst 10.0.0.0/8
  acl to_matnet dst 172.16.0.0/12
  acl to_matnet dst 192.168.0.0/16

  # SSL-Ports
  acl SSL_ports port 443 # https
  acl SSL_ports port 563 # snews
  acl SSL_ports port 873 # rsync

  # Safe-Ports
  acl Safe_ports port 80  # http
  acl Safe_ports port 21  # ftp
  acl Safe_ports port 443 # https
  acl Safe_ports port 70  # gopher
  acl Safe_ports port 210 # wais
  acl Safe_ports port 1025-65535 # unregistered ports
  acl Safe_ports port 280 # http-mgmt
  acl Safe_ports port 488 # gss-http
  acl Safe_ports port 591 # filemaker
  acl Safe_ports port 777 # multiling http
  acl Safe_ports port 631 # cups
  acl Safe_ports port 873 # rsync
  acl Safe_ports port 901 # SWAT

  # HTTPS
  acl CONNECT method CONNECT

  http_access deny !Safe_ports
  http_access deny CONNECT !SSL_ports
  http_access allow manager localhost
  http_access deny  manager
  http_access allow localnet
  http_access allow localhost
  http_access deny all

# NETWORK OPTIONS
# -----------------------------------------------------------------------------
  http_port 3128
  http_port 3129 intercept

# OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
# -----------------------------------------------------------------------------
  cache_peer proxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2
  cache_peer  roxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2 default

# MEMORY CACHE OPTIONS
# -----------------------------------------------------------------------------
  maximum_object_size_in_memory 8 MB
  memory_replacement_policy heap LFUDA
  cache_mem 256 MB

# DISK CACHE OPTIONS
# -----------------------------------------------------------------------------
  maximum_object_size 10 GB
  cache_replacement_policy heap GDSF
  cache_dir ufs /var/cache/squid 88894 16 256 max-size=10737418240

# LOGFILE OPTIONS
# -----------------------------------------------------------------------------
  access_log daemon:/var/log/squid/access.log squid

# OPTIONS FOR TROUBLESHOOTING
# -----------------------------------------------------------------------------
  cache_log /var/log/squid/cache.log
  coredump_dir /var/log/squid
  debug_options 44,2

# OPTIONS FOR TUNING THE CACHE
# -----------------------------------------------------------------------------
  max_stale 6 days
  shutdown_lifetime 5 seconds

# ADMINISTRATIVE PARAMETERS
# -----------------------------------------------------------------------------
  visible_hostname proxy.materna.com

# OPTIONS INFLUENCING REQUEST FORWARDING
# -----------------------------------------------------------------------------
  always_direct allow to_matnet
  never_direct  allow all

# DNS OPTIONS
# -----------------------------------------------------------------------------
  dns_nameservers 139.2.34.171
  dns_nameservers 139.2.34.37

# MISCELLANEOUS
# -----------------------------------------------------------------------------
  memory_pools off

Now, I block traffic on "squid-proxy.mycompany.com" to the primary proxy "proxy.mycompany.de" (139.2.1.3) using IPTables:
$ iptables -A OUTPUT -p icmp -d 139.2.1.3 -j DROP
$ iptables -A OUTPUT -p tcp -d 139.2.1.3 -j DROP
$ iptables -A OUTPUT -p udp -d 139.2.1.3 -j DROP

On the test machine, I use:
$ export http_proxy=http://squid-proxy.mycompany.com:3128/
$ export https_proxy=http://squid-proxy.mycompany.com:3128/
$ export HTTP_PROXY=http://squid-proxy.mycompany.com:3128/
$ export HTTPS_PROXY=http://squid-proxy.mycompany.com:3128/

Trying to download a resource:
$ wget https://repository.apache.org/content/groups/snapshots/org/apache/karaf/apache-karaf/4.1.1-SNAPSHOT/apache-karaf-4.1.1-20170315.084054-35.tar.gz

The download hangs for 2 minutes until it gets started. A retry shows the same results, the download starts after 2 minutes showing:
--2017-03-16 09:31:26--  https://repository.apache.org/content/groups/snapshots/org/apache/karaf/apache-karaf/4.1.1-SNAPSHOT/apache-karaf-4.1.1-20170314.154157-34.tar.gz
Resolving squid-proxy.mycompany.com (squid-proxy.mycompany.com)... 10.152.132.41
Connecting to squid-proxy.mycompany.com (squid-proxy.mycompany.com)|10.152.132.41|:3128... connected.

cache.log:

2017/03/16 10:17:47 kid1| Shutdown: NTLM authentication.
2017/03/16 10:17:47 kid1| Shutdown: Negotiate authentication.
2017/03/16 10:17:47 kid1| Shutdown: Digest authentication.
2017/03/16 10:17:47 kid1| Shutdown: Basic authentication.
CPU Usage: 0.084 seconds = 0.052 user + 0.032 sys
Maximum Resident Size: 113840 KB
Page faults with physical i/o: 0
2017/03/16 10:17:48 kid1| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
2017/03/16 10:17:48 kid1| Service Name: squid
2017/03/16 10:17:48| pinger: Initialising ICMP pinger ...
2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: http://proxy.materna.de:8080/squid-internal-dynamic/netdb' via proxy.materna.de
2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://proxy.materna.de:8080/squid-internal-dynamic/netdb'
2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:   always_direct = ALLOWED
2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:    never_direct = DUNNO
2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:          DIRECT = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths:        timedout = 0
2017/03/16 10:18:12.279 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: http://roxy.materna.de:8080/squid-internal-dynamic/netdb' via roxy.materna.de
2017/03/16 10:18:12.279 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://roxy.materna.de:8080/squid-internal-dynamic/netdb'
2017/03/16 10:18:12.279 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:   always_direct = ALLOWED
2017/03/16 10:18:12.279 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:    never_direct = DUNNO
2017/03/16 10:18:12.279 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths:          DIRECT = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
2017/03/16 10:18:12.279 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths:        timedout = 0
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via proxy.materna.de
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via proxy.materna.de
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via roxy.materna.de
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via roxy.materna.de
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'repository.apache.org:443'
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:   always_direct = DENIED
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:    never_direct = ALLOWED
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths:        timedout = 0

access.log

1489656077.628 159679 10.30.216.160 TCP_TUNNEL/200 26328966 CONNECT repository.apache.org:443 - ANY_OLD_PARENT/139.2.1.4 -


Any hints?

Jens
 

Gesendet: Donnerstag, 16. März 2017 um 09:27 Uhr
Von: "Amos Jeffries" <[hidden email]>
An: [hidden email]
Betreff: Re: [squid-users] No failover when default parent proxy fails (Squid 3.5.12)
On 16/03/2017 7:05 p.m., Jens Offenbach wrote:

> Thanks for your quick response...
>
> I have also configured, but the value seems not to be honored:
> connect_timeout 30 seconds
>
> The primary peer is down, but Squid does not print any "Dead parent"
> in the logs. Every HTTPS request is forwarded to the primary peer and
> it takes 1 minute until the secondary peer gets used, even with
> "connect_timeout 30 seconds". I think, I am facing the first issue
> that has been fixed by your patch.
>

The global config options being ignored completely is correct because
your peer have individual connect-timeout=5 settings.

So, those 5sec timeouts should be used instead now as before.

Though note that they apply only to how long a TCP connection (SYN,
SYN-ACK) is waited for. There is also a dns_timeout and peer selection
timeout that apply separately to the act of connecting. And a
forward_timeout global limit that all those operations have to fit
within, including retries.


Did you have a chance to try the debug setting I suggested at the
beginning? That will give you an immediate view about what Squid is
detecting as usable paths for each and every request and at what times
relative to the DEAD/LIVE notice.



> Are there any plans to backport this fix to Xenial APT repositories
> or to create a new Debian package for Squid4/5?

That is up to the Ubuntu server team, but I think it Unlikely. Zesty is
the current stable and things like this generally dont have enough
widespread impact to qualify for LTS backports.

Debian is now frozen to stabilize for the "Buster" release, that will
contain Squid-3.5.23 plus some few critical patches which are already
set. A Squid-4 package is ready and waiting for the release freeze to
end before it goes public in the Debian Unstable/Testing repos.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Amos Jeffries
Administrator
On 16/03/2017 10:39 p.m., Jens Offenbach wrote:
> This is the sceanrio;
>
> Squid 3.5.12 is installed on "squid-proxy.mycompany.com". The two parent proxies are:
> - Primary: proxy.mycompany.de:8080 (139.2.1.3)
> - Fallback: roxy.mycompany.de:8080 (139.2.1.4)
>
> I have misunderstood the "default" option in "cache_peer". When I got it right, it has the meaning of a fallback, so I switched it to "roxy.mycompany.de". "proxy.mycompany.de" should always be used and "roxy.mycompany.de" only when "proxy.mycompany.de" fails.
>

Well, kind of. Unless that peer is selected by one of the other
algorithms (for that it has to be 'alive') it will be appended as the
last-resort peer to be used regardless of DEAD/alive status.


> squid.conf:
>
...
>
> # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
> # -----------------------------------------------------------------------------
>   cache_peer proxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2
>   cache_peer  roxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2 default
>
...

> # OPTIONS INFLUENCING REQUEST FORWARDING
> # -----------------------------------------------------------------------------
>   always_direct allow to_matnet
>   never_direct  allow all
>
> # DNS OPTIONS
> # -----------------------------------------------------------------------------
>   dns_nameservers 139.2.34.171
>   dns_nameservers 139.2.34.37
>
...
>
> Now, I block traffic on "squid-proxy.mycompany.com" to the primary proxy "proxy.mycompany.de" (139.2.1.3) using IPTables:
> $ iptables -A OUTPUT -p icmp -d 139.2.1.3 -j DROP
> $ iptables -A OUTPUT -p tcp -d 139.2.1.3 -j DROP
> $ iptables -A OUTPUT -p udp -d 139.2.1.3 -j DROP
>

Are you trying to test connection timeout issues or a host going offline?
These iptables rules will force a timeout but not emulate a host
disconnection. Particularly when ICMP is also dropped.

When a host disconnects Squid will receive active signals (maybe via
ICMP) that the TCP SYN packet cannot get through. That speeds failure
recovery things up enormously. If the peer software simply
crashes/exits, different signals happen but with the same super fast
effects.

REJECT rules would be a better emulation of a machine disconnecting, or
an only-TCP REJECT rule emulates a peer software crash, etc. That way
the ICMP signalling still happens similar to those types of failure.



> On the test machine, I use:
> $ export http_proxy=http://squid-proxy.mycompany.com:3128/
> $ export https_proxy=http://squid-proxy.mycompany.com:3128/
> $ export HTTP_PROXY=http://squid-proxy.mycompany.com:3128/
> $ export HTTPS_PROXY=http://squid-proxy.mycompany.com:3128/
>
> Trying to download a resource:
> $ wget https://repository.apache.org/content/groups/snapshots/org/apache/karaf/apache-karaf/4.1.1-SNAPSHOT/apache-karaf-4.1.1-20170315.084054-35.tar.gz
>
> The download hangs for 2 minutes until it gets started. A retry shows the same results, the download starts after 2 minutes showing:
> --2017-03-16 09:31:26--  https://repository.apache.org/content/groups/snapshots/org/apache/karaf/apache-karaf/4.1.1-SNAPSHOT/apache-karaf-4.1.1-20170314.154157-34.tar.gz
> Resolving squid-proxy.mycompany.com (squid-proxy.mycompany.com)... 10.152.132.41
> Connecting to squid-proxy.mycompany.com (squid-proxy.mycompany.com)|10.152.132.41|:3128... connected.
>
> cache.log:
>
...
> 2017/03/16 10:17:48 kid1| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
> 2017/03/16 10:17:48 kid1| Service Name: squid
> 2017/03/16 10:17:48| pinger: Initialising ICMP pinger ...
> 2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: http://proxy.materna.de:8080/squid-internal-dynamic/netdb' via proxy.materna.de
> 2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://proxy.materna.de:8080/squid-internal-dynamic/netdb'

These can be avoided by adding no-netdb-exchange option to the
cache_peer config lines. But it is probably a good idea to keep them for
production use as they will be the way of detecting a peer recovery to
live status.

...

> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via proxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via proxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via roxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via roxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'repository.apache.org:443'
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths:   always_direct = DENIED
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths:    never_direct = ALLOWED
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths:      cache_peer = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths:        timedout = 0
>

Hmm. Something is going wrong with our logic to ensure unique IP:port
entries in the list of selected paths. It should not be affecting your
issue much though.

> access.log
>
> 1489656077.628 159679 10.30.216.160 TCP_TUNNEL/200 26328966 CONNECT repository.apache.org:443 - ANY_OLD_PARENT/139.2.1.4 -
>

Uhm. One thing to be very wary of is that transactions are not logged
until they are completed. So things like their full duration and bytes
can be recorded.

When CONNECT are involved some people who are not fully aware of the
meanings of that request method can be surprised by lack of log entries.
It is a tunnel and whole *weeks* worth of various traffic can happen
inside before it reaches that complete state for logging.
 You might see nothing actually happening except CONNECT lines being
logged with zero sizes, or huge amounts of https:// URLs being fetched
without a single access.log line occuring ... or any mix of behaviour in
between.

This connection had 26MB transferred over it. The 'connect' stage (TCP
SYN / SYN-ACK exchange) may have been successful within the first 11
seconds (5sec timeout on first two cache_peer in that cache.log list,
then immediate success on the third) and just nothing visibly happening
on it at the HTTP level for a bit while the TLS crypto did things.

If things are breaking or going slowly at the TLS layer or higher, then
there is nothing you can do in this Squid. As far as this Squid is
concerned the TCP tunnel was setup fine and working. What is inside it
is opaque.

I have just done a test of those two peers from here to see how the
setup goes, and there is an over 2min 10-12sec delay before my ISPs NAT
system cuts the connection. Something is very broken with those
particular peers or the network they reside in. That whole process
should have taken under 350ms and been terminated by their end.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Jens Offenbach
Thanks for your detailed explaination...

Well, the "default" option does not seem to be the right choice to achieve the expected behaviour. Hopefully, the following change will make all the traffic passing through the primary proxy when it is reachable:

# OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
# -----------------------------------------------------------------------------
  cache_peer proxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2 weight=2
  cache_peer roxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2

I am sorry for the incorrect test setup... I just added:
$ iptables -A OUTPUT -p tcp -d 139.2.1.3 -j REJECT

and the failover takes place very fast:
# iptables -D OUTPUT -p tcp -d 139.2.1.3 -j REJECT (Simulate primary proxy is online)
1489663755.767   4063 10.30.216.160 TCP_TUNNEL/200 26329103 CONNECT repository.apache.org:443 - FIRSTUP_PARENT/139.2.1.3 -
# iptables -A OUTPUT -p tcp -d 139.2.1.3 -j REJECT (Simulate primary proxy offline)
1489663818.913  33845 10.30.216.160 TCP_TUNNEL/200 20569674 CONNECT repository.apache.org:443 - ANY_OLD_PARENT/139.2.1.4 -
# iptables -D OUTPUT -p tcp -d 139.2.1.3 -j REJECT (Simulate primary proxy back online)
1489663850.148   4521 10.30.216.160 TCP_TUNNEL/200 20809195 CONNECT repository.apache.org:443 - FIRSTUP_PARENT/139.2.1.3 -

Those two parent proxies are not under my control. I will talk with my IT guys what is wrong here.

Next Wednesday is the next planned downtime of the primary proxy. I will test the failover at this time again.

You really helped me a lot. Thank you every much!

Regards,
Jens

Gesendet: Donnerstag, 16. März 2017 um 11:51 Uhr
Von: "Amos Jeffries" <[hidden email]>
An: [hidden email]
Betreff: Re: [squid-users] No failover when default parent proxy fails (Squid 3.5.12)
On 16/03/2017 10:39 p.m., Jens Offenbach wrote:
> This is the sceanrio;
>
> Squid 3.5.12 is installed on "squid-proxy.mycompany.com". The two parent proxies are:
> - Primary: proxy.mycompany.de:8080 (139.2.1.3)
> - Fallback: roxy.mycompany.de:8080 (139.2.1.4)
>
> I have misunderstood the "default" option in "cache_peer". When I got it right, it has the meaning of a fallback, so I switched it to "roxy.mycompany.de". "proxy.mycompany.de" should always be used and "roxy.mycompany.de" only when "proxy.mycompany.de" fails.
>

Well, kind of. Unless that peer is selected by one of the other
algorithms (for that it has to be 'alive') it will be appended as the
last-resort peer to be used regardless of DEAD/alive status.


> squid.conf:
>
...
>
> # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
> # -----------------------------------------------------------------------------
> cache_peer proxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2
> cache_peer roxy.materna.de parent 8080 0 no-digest no-query connect-timeout=5 connect-fail-limit=2 default
>
...

> # OPTIONS INFLUENCING REQUEST FORWARDING
> # -----------------------------------------------------------------------------
> always_direct allow to_matnet
> never_direct allow all
>
> # DNS OPTIONS
> # -----------------------------------------------------------------------------
> dns_nameservers 139.2.34.171
> dns_nameservers 139.2.34.37
>
...
>
> Now, I block traffic on "squid-proxy.mycompany.com" to the primary proxy "proxy.mycompany.de" (139.2.1.3) using IPTables:
> $ iptables -A OUTPUT -p icmp -d 139.2.1.3 -j DROP
> $ iptables -A OUTPUT -p tcp -d 139.2.1.3 -j DROP
> $ iptables -A OUTPUT -p udp -d 139.2.1.3 -j DROP
>

Are you trying to test connection timeout issues or a host going offline?
These iptables rules will force a timeout but not emulate a host
disconnection. Particularly when ICMP is also dropped.

When a host disconnects Squid will receive active signals (maybe via
ICMP) that the TCP SYN packet cannot get through. That speeds failure
recovery things up enormously. If the peer software simply
crashes/exits, different signals happen but with the same super fast
effects.

REJECT rules would be a better emulation of a machine disconnecting, or
an only-TCP REJECT rule emulates a peer software crash, etc. That way
the ICMP signalling still happens similar to those types of failure.


...
> 2017/03/16 10:17:48 kid1| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
> 2017/03/16 10:17:48 kid1| Service Name: squid
> 2017/03/16 10:17:48| pinger: Initialising ICMP pinger ...
> 2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: http://proxy.materna.de:8080/squid-internal-dynamic/netdb'[http://proxy.materna.de:8080/squid-internal-dynamic/netdb'] via proxy.materna.de
> 2017/03/16 10:18:09.579 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'http://proxy.materna.de:8080/squid-internal-dynamic/netdb'[http://proxy.materna.de:8080/squid-internal-dynamic/netdb']

These can be avoided by adding no-netdb-exchange option to the
cache_peer config lines. But it is probably a good idea to keep them for
production use as they will be the way of detecting a peer recovery to
live status.

...

> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via proxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via proxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via roxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: repository.apache.org:443' via roxy.materna.de
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for 'repository.apache.org:443'
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths: never_direct = ALLOWED
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths: cache_peer = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths: cache_peer = local=0.0.0.0 remote=139.2.1.3:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths: cache_peer = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(292) peerSelectDnsPaths: cache_peer = local=0.0.0.0 remote=139.2.1.4:8080 flags=1
> 2017/03/16 10:18:37.951 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths: timedout = 0
>

Hmm. Something is going wrong with our logic to ensure unique IP:port
entries in the list of selected paths. It should not be affecting your
issue much though.

> access.log
>
> 1489656077.628 159679 10.30.216.160 TCP_TUNNEL/200 26328966 CONNECT repository.apache.org:443 - ANY_OLD_PARENT/139.2.1.4 -
>

Uhm. One thing to be very wary of is that transactions are not logged
until they are completed. So things like their full duration and bytes
can be recorded.

When CONNECT are involved some people who are not fully aware of the
meanings of that request method can be surprised by lack of log entries.
It is a tunnel and whole *weeks* worth of various traffic can happen
inside before it reaches that complete state for logging.
You might see nothing actually happening except CONNECT lines being
logged with zero sizes, or huge amounts of https:// URLs being fetched
without a single access.log line occuring ... or any mix of behaviour in
between.

This connection had 26MB transferred over it. The 'connect' stage (TCP
SYN / SYN-ACK exchange) may have been successful within the first 11
seconds (5sec timeout on first two cache_peer in that cache.log list,
then immediate success on the third) and just nothing visibly happening
on it at the HTTP level for a bit while the TLS crypto did things.

If things are breaking or going slowly at the TLS layer or higher, then
there is nothing you can do in this Squid. As far as this Squid is
concerned the TCP tunnel was setup fine and working. What is inside it
is opaque.

I have just done a test of those two peers from here to see how the
setup goes, and there is an over 2min 10-12sec delay before my ISPs NAT
system cuts the connection. Something is very broken with those
particular peers or the network they reside in. That whole process
should have taken under 350ms and been terminated by their end.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users]
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: No failover when default parent proxy fails (Squid 3.5.12)

Alex Rousskov
In reply to this post by Amos Jeffries
On 03/16/2017 02:27 AM, Amos Jeffries wrote:

> The global config options being ignored completely is correct because
> your peer have individual connect-timeout=5 settings.

Please note that [individual] peer connect-timeout settings are ignored
on the tunneling path in unpatched Squid v4 and v5 (possibly v3 too):
http://lists.squid-cache.org/pipermail/squid-dev/2017-March/008243.html

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...