Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

John Sweet-Escott
Hi All

We are trying to run Squid 4.8, compiled with OpenSSL 1.1.1 (see [1]) on Ubuntu 18.04 as a transparent proxy for the purpose of egress filtering of HTTPS traffic using SNI (see config in [2]). It it works correctly when contacting some addresses (e.g. https://www.ubuntu.com) but not others (e.g. https://www.google.com). When we contact https://www.google.com using TLS1.2 we get the error in the logs:
2019/09/15 10:33:09 kid1| ERROR: negotiating TLS on FD 19: error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)
and the page returned to the client contains ERR_SECURE_CONNECT_FAIL. When TLS1.3 is used, the connections are made correctly, however my application is constrained to java8 & tomcat8 which does not support TLS1.3.

Connections are made using curl or openssl s_client. For example:
openssl s_client -tls1_2  -CAfile squid.crt -connect www.google.com:443 -tlsextdebug
[237/1854]CONNECTED(00000005)
TLS server extension "renegotiation info" (id=65281), len=1
0000 - 00                                                .
TLS server extension "EC point formats" (id=11), len=4
0000 - 03 00 01 02                                       ....
TLS server extension "session ticket" (id=35), len=0
TLS server extension "extended master secret" (id=23), len=0
depth=1 C = GB, ST = London, L = squid, O = squid, CN = squid
verify return:1
depth=0 CN = www.google.com
verify return:1
---
Certificate chain
 0 s:CN = www.google.com
   i:C = GB, ST = London, L = squid, O = squid, CN = squid
 1 s:C = GB, ST = London, L = squid, O = squid, CN = squid
   i:C = GB, ST = London, L = squid, O = squid, CN = squid
 2 s:C = GB, ST = London, L = squid, O = squid, CN = squid
   i:C = GB, ST = London, L = squid, O = squid, CN = squid
etc

Attached are pcap files showing first a bad connection to google and then a working connection to ubuntu. Looking at the pcap files the difference in the google and ubuntu server hello lies in the extensions and the cypher:
Google:
Handshake Protocol: Server Hello
    Handshake Type: Server Hello (2)
    Length: 59
    Version: TLS 1.2 (0x0303)
    Random: 5d7e05552e1fdea260f67e0bdf413f6a9837fbaffdebeb35…
    Session ID Length: 0
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
    Compression Method: null (0)
    Extensions Length: 19
    Extension: extended_master_secret (len=0)
    Extension: renegotiation_info (len=1)
    Extension: ec_point_formats (len=2)
    Extension: session_ticket (len=0)
Ubuntu:
Handshake Protocol: Server Hello
    Handshake Type: Server Hello (2)
    Length: 61
    Version: TLS 1.2 (0x0303)
    Random: 7ec2c3a2554bac610e0290ac1f160c3ed185bdd1159e377c…
    Session ID Length: 0
    Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
    Compression Method: null (0)
    Extensions Length: 21
    Extension: server_name (len=0)
    Extension: renegotiation_info (len=1)
    Extension: ec_point_formats (len=4)
    Extension: session_ticket (len=0)
Differences are that Google supplies extended_master_secretand Ubuntu supplies server_name extensions. The cyphers chosen by the server is also different. Enabling debug using squid -k debug indicates that this is the likely problematic area:

2019/09/15 11:21:02.486 kid1| 83,5| PeerConnector.cc(712) checkForMissingCertificates: SSL server sent 2 certificates
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCall.cc(26) AsyncCall: The AsyncCall Security::PeerConnector::negotiate constructed, this=0x560d937c94b0 [call515701]2019/09/15 11:21:02.486 kid1| 83,7| AsyncCall.cc(93) ScheduleCall: PeerConnector.cc(391) will call Security::PeerConnector::negotiate() [call515701]
2019/09/15 11:21:02.486 kid1| 83,7| AsyncJob.cc(154) callEnd: Ssl::PeekingPeerConnector status out: [ FD 19 job24663]
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCallQueue.cc(57) fireNext: leaving Security::PeerConnector::negotiate()
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCallQueue.cc(55) fireNext: entering Security::PeerConnector::negotiate()
2019/09/15 11:21:02.486 kid1| 83,7| AsyncCall.cc(38) make: make call Security::PeerConnector::negotiate [call515701]
2019/09/15 11:21:02.486 kid1| 83,7| AsyncJob.cc(123) callStart: Ssl::PeekingPeerConnector status in: [ FD 19 job24663]
2019/09/15 11:21:02.486 kid1| 83,5| PeerConnector.cc(188) negotiate: SSL_connect session=0x560d93835950
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(356) giveBuffered: 5<=5 bytes to OpenSSL
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(356) giveBuffered: 63<=63 bytes to OpenSSL
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(164) stateChanged: FD 19 now: 0x1001 TWCH (SSLv3/TLS write client hello)
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(471) write: postpone writing 7 bytes to SSL FD 19
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(164) stateChanged: FD 19 now: 0x4008 SSLERR (error)
2019/09/15 11:21:02.486 kid1| 83,7| bio.cc(164) stateChanged: FD 19 now: 0x1002 SSLERR (error)
2019/09/15 11:21:02.486 kid1| 83,5| NegotiationHistory.cc(83) retrieveNegotiatedInfo: SSL connection info on FD 19 SSL version NONE/0.0 negotiated cipher
2019/09/15 11:21:02.486 kid1| ERROR: negotiating TLS on FD 19: error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)
2019/09/15 11:21:02.486 kid1| 83,5| PeerConnector.cc(570) callBack: TLS setup ended for local=10.20.251.235:37882 remote=216.58.210.196:443 FD 19 flags=1

I am not sure how to resolve this issue. I am guessing that there is something in the response from Google that is causing OpenSSL to reject the connections. From what I can see, there is no evidence of a downgrade being attempted, which is what the inappropriate fallback error message might indicate.

Any advice/guidance greatfully recieved.
John

[1] OpenSSL and Squid versions
openssl version
OpenSSL 1.1.1  11 Sep 2018

squid -v
Squid Cache: Version 4.8
Service Name: squid
Ubuntu linux
This binary uses OpenSSL 1.1.1  11 Sep 2018. For legal restrictions on distribution see https://www.openssl.org/source/license.html
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--$
ysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silen$
-rules' 'BUILDCXXFLAGS=-g -O2 -fdebug-prefix-map=/home/builder/diladele/squid-ubuntu/src/ubuntu18/scripts.squid4/build/squid/squid-4.8=. -fstack-protector-strong -$
format -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -latomic' 'BUILDCXX=x86_64-linux-gn$
-g++' '--with-build-environment=default' '--enable-build-info=Ubuntu linux' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid' '-$
mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' $
--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-ssl' '--enable-ssl-crtd' '--with-openssl' '--enable-follow-x-forwarded-for' '--enab$
e-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntl
m=fake,SMB_LM' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-security-
cert-validators=fake' '--enable-storeid-rewrite-helpers=file' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '
--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=
65536' '--with-large-files' '--with-default-user=proxy' '--with-gnutls' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CC=x86_64-linux-gnu-gcc' 'CFLAGS=
-g -O2 -fdebug-prefix-map=/home/builder/diladele/squid-ubuntu/src/ubuntu18/scripts.squid4/build/squid/squid-4.8=. -fstack-protector-strong -Wformat -Werror=format-s
ecurity -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -latomic' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXX=x86_64-linux-gnu-
g++' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/home/builder/diladele/squid-ubuntu/src/ubuntu18/scripts.squid4/build/squid/squid-4.8=. -fstack-protector-strong -Wformat -
Werror=format-security'

[2] Configuration squid.conf
visible_hostname squid
max_filedesc 4096

http_port 3128
#Handling HTTP requests
http_port 3129 intercept
include /etc/squid/http_sites.conf
http_access allow allowed_http_sites

#Handling HTTPS requests
https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept
acl SSL_port port 443http_access allow SSL_port
# Allows access to www.google.com and www.ubuntu.com
include /etc/squid/https_sites.conf

# TLS peek-and-splice configuration
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump peek step2 allowed_https_sites
ssl_bump splice step3 allowed_https_sitesssl_bump terminate step2 all
http_access deny all

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

badgoodonly.pcap (21K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

Amos Jeffries
Administrator

On 15/09/19 10:41 pm, John Sweet-Escott wrote:

> Hi All
>
> We are trying to run Squid 4.8, compiled with OpenSSL 1.1.1 (see [1]) on
> Ubuntu 18.04 as a transparent proxy for the purpose of egress filtering
> of HTTPS traffic using SNI (see config in [2]). It it works correctly
> when contacting some addresses (e.g. https://www.ubuntu.com) but not
> others (e.g. https://www.google.com). When we contact
> https://www.google.com using TLS1.2 we get the error in the logs:
> 2019/09/15 10:33:09 kid1| ERROR: negotiating TLS on FD 19:
> error:1425F175:SSL routines:ssl_choose_client_version:inappropriate
> fallback (1/-1/0)
...
>     Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)

I suspect it might have something to do with these ECDSA keys.

You do not have Elliptic-Curves enabled on the https_port client-facing
connection. So the TLS extensions associated are likely not to be
compatible between the client and the server connections Squid is
attempting to bridge between.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

John Sweet-Escott
Hi Amos

Thank you for your help.

On Tue, 17 Sep 2019 at 07:26, Amos Jeffries <[hidden email]> wrote:

> ...
> >     Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
>
> I suspect it might have something to do with these ECDSA keys.
>
> You do not have Elliptic-Curves enabled on the https_port client-facing
> connection. So the TLS extensions associated are likely not to be
> compatible between the client and the server connections Squid is
> attempting to bridge between.
>
I generated a dhparams file using the command:
openssl dhparam -out dhparams.pem 2048
and then I configured the port with the following options:
https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept
tls-dh=prime256v1:/etc/squid/dhparams.pem
options=SINGLE_ECDH_USE,SINGLE_DH_USE

But this still gives this in the log when I connect:
2019/09/18 08:19:44 kid1| ERROR: negotiating TLS on FD 17:
error:1425F175:SSL routines:ssl_choose_client_version:inappropriate
fallback (1/-1/0)

I have also tried restricting the cipher to the same cipher that works
for the ubuntu connection and I get the same error:
openssl s_client -tls1_2  -CAfile squid.crt -cipher
ECDHE-RSA-AES128-GCM-SHA256  -connect www.google.com:443

With this restriction, the client hello to squid is:
Handshake Protocol: Client Hello
    Handshake Type: Client Hello (1)
    Length: 156
    Version: TLS 1.2 (0x0303)
    Random: e52eb8a54705dc32774c5832694dd4567cd9b0f34556ebf3…
    Session ID Length: 0
    Cipher Suites Length: 4
    Cipher Suites (2 suites)
        Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
        Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
    Compression Methods Length: 1
    Compression Methods (1 method)
    Extensions Length: 111
    Extension: server_name (len=19)
    Extension: ec_point_formats (len=4)
    Extension: supported_groups (len=12)
    Extension: session_ticket (len=0)
    Extension: encrypt_then_mac (len=0)
    Extension: extended_master_secret (len=0)
    Extension: signature_algorithms (len=48)
The proxied hello to google is identical to the above.
The server hello from google is:
Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Server Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 63
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 59
            Version: TLS 1.2 (0x0303)
            Random: 5d81da909e779d7e67f2663d6563236721b0906d09dacf02…
            Session ID Length: 0
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
            Compression Method: null (0)
            Extensions Length: 19
            Extension: extended_master_secret (len=0)
            Extension: renegotiation_info (len=1)
            Extension: ec_point_formats (len=2)
            Extension: session_ticket (len=0)
    TLSv1.2 Record Layer: Handshake Protocol: Certificate
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 2537
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 2533
            Certificates Length: 2530
            Certificates (2530 bytes)
                Certificate Length: 1422
                Certificate:
3082058a30820472a0030201020210556630a312faeab908…
(id-at-commonName=www.google.com,id-at-organizationName=Google
LLC,id-at-localityName=Mountain
View,id-at-stateOrProvinceName=California,id-at-countryName=US)
                Certificate Length: 1102
                Certificate:
3082044a30820332a003020102020d01e3b49aa18d8aa981…
(id-at-commonName=GTS CA 1O1,id-at-organizationName=Google Trust
Services,id-at-countryName=US)
    TLSv1.2 Record Layer: Handshake Protocol: Server Key Exchange
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 300
        Handshake Protocol: Server Key Exchange
            Handshake Type: Server Key Exchange (12)
            Length: 296
            EC Diffie-Hellman Server Params
    TLSv1.2 Record Layer: Handshake Protocol: Server Hello Done
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 4
        Handshake Protocol: Server Hello Done

If you have any further suggestions as to how/where I should debug I
would be extremely grateful.

John

On Tue, 17 Sep 2019 at 07:26, Amos Jeffries <[hidden email]> wrote:

>
>
> On 15/09/19 10:41 pm, John Sweet-Escott wrote:
> > Hi All
> >
> > We are trying to run Squid 4.8, compiled with OpenSSL 1.1.1 (see [1]) on
> > Ubuntu 18.04 as a transparent proxy for the purpose of egress filtering
> > of HTTPS traffic using SNI (see config in [2]). It it works correctly
> > when contacting some addresses (e.g. https://www.ubuntu.com) but not
> > others (e.g. https://www.google.com). When we contact
> > https://www.google.com using TLS1.2 we get the error in the logs:
> > 2019/09/15 10:33:09 kid1| ERROR: negotiating TLS on FD 19:
> > error:1425F175:SSL routines:ssl_choose_client_version:inappropriate
> > fallback (1/-1/0)
> ...
> >     Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
>
> I suspect it might have something to do with these ECDSA keys.
>
> You do not have Elliptic-Curves enabled on the https_port client-facing
> connection. So the TLS extensions associated are likely not to be
> compatible between the client and the server connections Squid is
> attempting to bridge between.
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

tannmann
Hey John,

It looks like we have a very similar setup and configuration as you, and we
are experiencing the same problem. Have you been able to figure out a way to
get connections to google to work with Squid 4.8 as a transparent proxy?

Thanks,

Tanner



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Problem with ssl_choose_client_version:inappropriate fallback on some sites when using TLS1.2

John Sweet-Escott
Hi Tanner

Unfortunately not. We have tried everything we can think of, plus suggested items from this list, with no success. If you figure it out let me know.

Many thanks
John

Sent from my iPhone

> On 20 Nov 2019, at 21:34, tannmann <[hidden email]> wrote:
>
> Hey John,
>
> It looks like we have a very similar setup and configuration as you, and we
> are experiencing the same problem. Have you been able to figure out a way to
> get connections to google to work with Squid 4.8 as a transparent proxy?
>
> Thanks,
>
> Tanner
>
>
>
> --
> Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users