Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

classic Classic list List threaded Threaded
10 messages Options
dc
Reply | Threaded
Open this post in threaded view
|

Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

dc
I have a transparent squid 4.8 proxy peek-and-splice setup acting as a
TLS domain filtering proxy. The setup worked well, until more and more
servers started adopting TLS 1.3. In this case, depending on the client
TLS version, errors started to appear:

If server, squid and client use TLS 1.3: Everything works as expected.
If server and squid use TLS 1.3, but client only supports TLS 1.2: The
client terminates the connection due to certificate verification errors.

I have had a look at what happens at TLS protocol level using wireshark,
and it seems that in the latter case, squid - for some reason - performs
(something similar to) bumping instead of splicing! That is, squid sends
back certificates to the client which are completely different than the
ones received from the server, and appear to be generated. Any
ClientKeyExchange received from the client also wouldn't be forwarded to
the server.

The following is the relevant part of my squid config:

https_port 3443 intercept ssl-bump cert=/etc/squid/dummy.pem.crt
key=/etc/squid/dummy.pem.key
ssl_bump peek step1 all
ssl_bump peek step2 allowed_https_connections
ssl_bump terminate step2 all
ssl_bump splice step3 allowed_https_connections
ssl_bump terminate all

where allowed_https_connections is an ACL checking ssl::server_name.

How can I get the splicing setup working when mixing TLS 1.3 servers and
TLS 1.2 clients?

Many thanks!
Nikolaus
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

Alex Rousskov
On 9/20/19 10:53 AM, Nikolaus wrote:

> If server and squid use TLS 1.3, but client only supports TLS 1.2: The
> client terminates the connection due to certificate verification errors.
>
> I have had a look at what happens at TLS protocol level using wireshark,
> and it seems that in the latter case, squid - for some reason - performs
> (something similar to) bumping instead of splicing!

Bumping happens when a splicing Squid wants to report an SslBump-related
error to the client.


> How can I get the splicing setup working when mixing TLS 1.3 servers and
> TLS 1.2 clients?

I do not know the exact answer to that question, but I would start by
figuring out what error Squid is trying to serve to the client. You may
be able to figure it out by looking at the corresponding access.log
records, especially if you log %err_code and %err_detail. In the worst
case, enabling and looking at debugging info in cache.log may be
necessary, but I would start with access.log anyway.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
dc
Reply | Threaded
Open this post in threaded view
|

Re: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

dc

> You may
> be able to figure it out by looking at the corresponding access.log
> records, especially if you log %err_code and %err_detail. In the worst
> case, enabling and looking at debugging info in cache.log may be
> necessary, but I would start with access.log anyway.

Thank you for the suggestion Alex!

The access.log contains error code / detail "ERR_SECURE_CONNECT_FAIL /
SQUID_ERR_SSL_HANDSHAKE" - which is not too helpful - but the cache.log
contains the more detailed "ERROR: negotiating TLS on FD 19:
error:1425F175:SSL routines:ssl_choose_client_version:inappropriate
fallback (1/-1/0)".

Is a TLS fallback prevention mechanism kicking in by error? If so, how
to fix it?
Please let me know if additional log output (the debug log around the
error location did not seem helpful to me though) or a configuration to
reproduce the error are needed.

Nikolaus
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

Alex Rousskov
On 9/22/19 9:18 AM, Nikolaus wrote:

> The access.log contains error code / detail "ERR_SECURE_CONNECT_FAIL /
> SQUID_ERR_SSL_HANDSHAKE" - which is not too helpful - but the cache.log
> contains the more detailed "ERROR: negotiating TLS on FD 19:
> error:1425F175:SSL routines:ssl_choose_client_version:inappropriate
> fallback (1/-1/0)".

> Is a TLS fallback prevention mechanism kicking in by error? If so, how
> to fix it?

I do not know the answers to your questions, but I am sure that it is
possible to figure it out by looking at either packet captures or
detailed debugging logs. Unfortunately, I do not have enough free time
to guide you through this triage. There were several similar complains
about "inappropriate fallback" errors on this list recently. I would
start by revisiting those threads for more clues.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

John Sweet-Escott


On 22 Sep 2019, at 14:41, Alex Rousskov <[hidden email]> wrote:

On 9/22/19 9:18 AM, Nikolaus wrote:

The access.log contains error code / detail "ERR_SECURE_CONNECT_FAIL /
SQUID_ERR_SSL_HANDSHAKE" - which is not too helpful - but the cache.log
contains the more detailed "ERROR: negotiating TLS on FD 19:
error:1425F175:SSL routines:ssl_choose_client_version:inappropriate
fallback (1/-1/0)".

Is a TLS fallback prevention mechanism kicking in by error? If so, how
to fix it?

I do not know the answers to your questions, but I am sure that it is
possible to figure it out by looking at either packet captures or
detailed debugging logs. Unfortunately, I do not have enough free time
to guide you through this triage. There were several similar complains
about "inappropriate fallback" errors on this list recently. I would
start by revisiting those threads for more clues.

Alex
Unfortunately we have not been able to work out the inappropriate fallback issue described http://lists.squid-cache.org/pipermail/squid-users/2019-September/021047.html. If you do fix your issue, please do share. 
John

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
dc
Reply | Threaded
Open this post in threaded view
|

Resolved: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

dc
I was able to solve the issue, fixing both squid-side "error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)" and client-side certificate verification errors when attempting to contact TLS 1.3 server over a TLS 1.3-enabled squid from a TLS 1.2 client. I will first explain what causes the issue before presenting my solution, which involves changes of the squid code base, for anybody who is affected by the same problem.

I have inspected the squid source code and noticed that TLS peeking works roughly like this:

1. The client sends a client_hello, which is parsed by squid using a custom handshake parser.  <-- Uses TLS 1.2
2. Squid creates an OpenSSL TLS session for the peeked connection  <-- Uses TLS 1.3!
3. Squid forwards the original client_hello handshake message to the server <-- TLS 1.2
4. Squid passes the received server_hello response to the OpenSSL session created previously  <-- Response uses TLS 1.2 - Problem!

Now, the "problem" is that TLS 1.3 defines a set of new protocol downgrade attack prevention mechanisms (can be found e.g. here: https://blog.gypsyengineer.com/en/security/how-does-tls-1-3-protect-against-downgrade-attacks.html). Both OpenSSL and most likely the server implement these. This includes that the server random in the TLS 1.2 server_hello contains an indicator that the server is TL 1.3-capable. The OpenSSL session created by squid detects this, notices that it is TLS 1.3-capable itself, and closes the connection because it assumes a protocol downgrade attack! Little does it know, that our client actually only supports TLS 1.2, so we *want* the lower protocol version.

My solution includes setting the maximum TLS version of the OpenSSL session to the version received from the client. This proved a little bit difficult, since the way TLS versions are negotiated has also been changed by the TLS 1.3 specification, and the squid handshake parser was not yet able to detect TLS 1.3 correctly - I have therefore also implemented parsing of the SupportedVersions TLS Extension and a preliminary support for sparse version ranges. You can find all these changes at https://github.com/nthuemmel/squid/tree/tls_downgrade_compatibility , which is a fork of squid 4.9. Feel free to compile & test it if you have a transparent peek-and-splice setup and are affected by the "inappropriate fallback" problem.

I would of course be glad if the fix could be merged into the main squid repository. If you are a dev, please let me know what you think and if I should open a pull request. There are still some TODOs left, because I wasn't sure what the best way is to integrate some of the changes. Notably, there was also a comment which discourages setting a maximum version for the OpenSSL session to improve peek+bump compatibility - I don't have a setup to which this applies, so I don't know how big of an impact this is or if it is still relevant.

Best Regards
Nikolaus

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Resolved: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

Alex Rousskov
On 12/7/19 8:54 AM, Nikolaus wrote:

> https://github.com/nthuemmel/squid/tree/tls_downgrade_compatibility
>
> I would of course be glad if the fix could be merged into the main squid
> repository. If you are a dev, please let me know what you think and if I
> should open a pull request.


FYI: There are two other ongoing and independent efforts related to TLS
v1.3 version handling:

[1] Fix stalled SslBump-peeked connections from older browsers
    https://github.com/measurement-factory/squid/pull/60/

[2] Bug 5011: TLS 1.3 connection get stuck when parsing ServerHello
    https://bugs.squid-cache.org/show_bug.cgi?id=5011

My team is responsible for [1]. Our unofficial (and currently very
unpolished) code should be ready for the official review in a couple of
weeks. AFAICT from a quick look through your changes, we are working on
the same or a very similar problem. If you can test [1] in your
environment, please let me know whether it works in your environment.

I am not sure what is the best way to minimize further duplication of
effort here. Here is one option: If [1] works in your environment, and
you would rather avoid porting your changes to master, then perhaps you
can help with reviewing and backporting [1] (after it is officially
reviewed) to v4 instead.

If you decide to improve your branch towards its official submission,
please see https://wiki.squid-cache.org/MergeProcedure and keep in mind
that you will need to port your changes to master. Please also consider
_not_ storing the entire array of parsed supported versions if storing
just a couple of them (or storing their implications) is sufficient.
Please also note that SSL_set_max_proto_version() is not available in
OpenSSL v1.0. If Squid still supports that older OpenSSL version, it
would be best to avoid dropping that support because of this change.

If you have technical/development comments regarding [1], they are
probably best handled as pull request comments on GitHub (or a
discussion on the squid-dev@ mailing list). The squid-users@ mailing
list is not a good place to discuss code.


Thank you,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Resolved: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

John Sweet-Escott
In reply to this post by dc
Hi Nikolaus 

This sounds exactly like the symptoms we have encountered. Will build from your patch & test to see if it works in our situation. 

John. 


On 7 Dec 2019, at 13:54, Nikolaus <[hidden email]> wrote:

 I was able to solve the issue, fixing both squid-side "error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)" and client-side certificate verification errors when attempting to contact TLS 1.3 server over a TLS 1.3-enabled squid from a TLS 1.2 client. I will first explain what causes the issue before presenting my solution, which involves changes of the squid code base, for anybody who is affected by the same problem.

I have inspected the squid source code and noticed that TLS peeking works roughly like this:

1. The client sends a client_hello, which is parsed by squid using a custom handshake parser.  <-- Uses TLS 1.2
2. Squid creates an OpenSSL TLS session for the peeked connection  <-- Uses TLS 1.3!
3. Squid forwards the original client_hello handshake message to the server <-- TLS 1.2
4. Squid passes the received server_hello response to the OpenSSL session created previously  <-- Response uses TLS 1.2 - Problem!

Now, the "problem" is that TLS 1.3 defines a set of new protocol downgrade attack prevention mechanisms (can be found e.g. here: https://blog.gypsyengineer.com/en/security/how-does-tls-1-3-protect-against-downgrade-attacks.html). Both OpenSSL and most likely the server implement these. This includes that the server random in the TLS 1.2 server_hello contains an indicator that the server is TL 1.3-capable. The OpenSSL session created by squid detects this, notices that it is TLS 1.3-capable itself, and closes the connection because it assumes a protocol downgrade attack! Little does it know, that our client actually only supports TLS 1.2, so we *want* the lower protocol version.

My solution includes setting the maximum TLS version of the OpenSSL session to the version received from the client. This proved a little bit difficult, since the way TLS versions are negotiated has also been changed by the TLS 1.3 specification, and the squid handshake parser was not yet able to detect TLS 1.3 correctly - I have therefore also implemented parsing of the SupportedVersions TLS Extension and a preliminary support for sparse version ranges. You can find all these changes at https://github.com/nthuemmel/squid/tree/tls_downgrade_compatibility , which is a fork of squid 4.9. Feel free to compile & test it if you have a transparent peek-and-splice setup and are affected by the "inappropriate fallback" problem.

I would of course be glad if the fix could be merged into the main squid repository. If you are a dev, please let me know what you think and if I should open a pull request. There are still some TODOs left, because I wasn't sure what the best way is to integrate some of the changes. Notably, there was also a comment which discourages setting a maximum version for the OpenSSL session to improve peek+bump compatibility - I don't have a setup to which this applies, so I don't know how big of an impact this is or if it is still relevant.

Best Regards
Nikolaus
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Resolved: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

John Sweet-Escott
In reply to this post by dc
Hi Nikolaus

I have taken your patch, applied it to squid
http://http.debian.net/debian/pool/main/s/squid/squid_4.9-2.dsc, built
it as a deb package and tested it.

In use I am getting many errors of the form:
2019/12/12 16:50:19 kid1| Error parsing SSL Server Hello Message on FD 15

Turning on debug, the relevant section of the log seems to be:
2019/12/12 16:50:16.711 kid1| 24,7| BinaryTokenizer.cc(65) got:
Handshake.msg_type=2 occupying 1 bytes @0 in 0x55cd92e91c18.
2019/12/12 16:50:16.711 kid1| 24,7| BinaryTokenizer.cc(65) got:
Handshake.msg_body.length=87 occupying 3 bytes @1 in 0x55cd92e91c18.
2019/12/12 16:50:16.711 kid1| 24,7| BinaryTokenizer.cc(74) got:
Handshake.msg_body.octets= occupying 87 bytes @4 in 0x55cd92e91c18.
2019/12/12 16:50:16.711 kid1| 24,7| BinaryTokenizer.cc(57) got:
Handshake occupying 91 bytes @0 in 0x55cd92e91c18.
2019/12/12 16:50:16.711 kid1| 0,3| Handshake.cc(324)
parseHandshakeMessage: check failed: state < atHelloReceived
    exception location: Handshake.cc(324) parseHandshakeMessage

2019/12/12 16:50:16.711 kid1| 83,2| bio.cc(318) readAndParse: parsing
error on FD 15: check failed: state < atHelloReceived
    exception location: Handshake.cc(324) parseHandshakeMessage

2019/12/12 16:50:16.711 kid1| 83,7| bio.cc(324) readAndParse: Hold
flag is set, retry latter. (Hold 5bytes)
2019/12/12 16:50:16.711 kid1| 83,7| bio.cc(166) stateChanged: FD 15
now: 0x1002 TWCH (SSLv3/TLS write client hello)
2019/12/12 16:50:16.711 kid1| 83,5| PeerConnector.cc(462)
noteWantRead: local=10.12.255.133:54576 remote=52.94.56.114:443 FD 15
flags=1
2019/12/12 16:50:16.711 kid1| Error parsing SSL Server Hello Message on FD 15

However... functionally, the patch does seem to be working, allowing
TLS 1.2 traffic from the client inside my network to connect to a TLS
1.3 server (www.google.com) when the site is in the whitelist and
terminating the connection when it is removed from the whitelist.

It is unclear to me if the "Error parsing SSL Server Hello Message"
are benign or not.

John

On Sat, 7 Dec 2019 at 13:54, Nikolaus <[hidden email]> wrote:

>
> I was able to solve the issue, fixing both squid-side "error:1425F175:SSL routines:ssl_choose_client_version:inappropriate fallback (1/-1/0)" and client-side certificate verification errors when attempting to contact TLS 1.3 server over a TLS 1.3-enabled squid from a TLS 1.2 client. I will first explain what causes the issue before presenting my solution, which involves changes of the squid code base, for anybody who is affected by the same problem.
>
> I have inspected the squid source code and noticed that TLS peeking works roughly like this:
>
> 1. The client sends a client_hello, which is parsed by squid using a custom handshake parser.  <-- Uses TLS 1.2
> 2. Squid creates an OpenSSL TLS session for the peeked connection  <-- Uses TLS 1.3!
> 3. Squid forwards the original client_hello handshake message to the server <-- TLS 1.2
> 4. Squid passes the received server_hello response to the OpenSSL session created previously  <-- Response uses TLS 1.2 - Problem!
>
> Now, the "problem" is that TLS 1.3 defines a set of new protocol downgrade attack prevention mechanisms (can be found e.g. here: https://blog.gypsyengineer.com/en/security/how-does-tls-1-3-protect-against-downgrade-attacks.html). Both OpenSSL and most likely the server implement these. This includes that the server random in the TLS 1.2 server_hello contains an indicator that the server is TL 1.3-capable. The OpenSSL session created by squid detects this, notices that it is TLS 1.3-capable itself, and closes the connection because it assumes a protocol downgrade attack! Little does it know, that our client actually only supports TLS 1.2, so we *want* the lower protocol version.
>
> My solution includes setting the maximum TLS version of the OpenSSL session to the version received from the client. This proved a little bit difficult, since the way TLS versions are negotiated has also been changed by the TLS 1.3 specification, and the squid handshake parser was not yet able to detect TLS 1.3 correctly - I have therefore also implemented parsing of the SupportedVersions TLS Extension and a preliminary support for sparse version ranges. You can find all these changes at https://github.com/nthuemmel/squid/tree/tls_downgrade_compatibility , which is a fork of squid 4.9. Feel free to compile & test it if you have a transparent peek-and-splice setup and are affected by the "inappropriate fallback" problem.
>
> I would of course be glad if the fix could be merged into the main squid repository. If you are a dev, please let me know what you think and if I should open a pull request. There are still some TODOs left, because I wasn't sure what the best way is to integrate some of the changes. Notably, there was also a comment which discourages setting a maximum version for the OpenSSL session to improve peek+bump compatibility - I don't have a setup to which this applies, so I don't know how big of an impact this is or if it is still relevant.
>
> Best Regards
> Nikolaus
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Resolved: Peek-and-splice not working when mixing TLS1.3 servers and TLS1.2 clients

tannmann
I've also compiled from Nikolaus's branch and get the same results as John.
It appears to fix the issues with inappropriate fallback, but I get the same
"Error parsing SSL Server Hello Message on FD 15".

Interestingly, I also get those errors when I compile from the branch
suggested by Alex at https://github.com/measurement-factory/squid/pull/60/.
Building from this branch, or from Nikolaus's gives me the same results, no
more inappropriate fallback errors (yay!), but now those "Error parsing SSL
Server Hello Message on FD XX" errors. I haven't found any problems related
to these new errors, everything going through the proxy appears to work
fine.



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users