TCP FIN,ACK after ServerHelloDone with pcmag.com

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

TCP FIN,ACK after ServerHelloDone with pcmag.com

Ahmad, Sarfaraz

Hi Folks,

 

I am using Squid as a HTTPS interception proxy. When I try to access https://www.pcmag.com , (which is supposed to be bumped in my environment ), I get

“unable to forward request at this time” even though the website is perfectly accessible outside of the proxy.

 

A packet capture suggests that after Client Hello -> ServerHello -> ServerCertificate,Server Key Exchange, ServerHelloDone, the remote server just sends a FIN,ACK packet, killing off the TCP connection. Nothing else looks out of the ordinary.  ( Without squid, firefox successfully opens the site and the negotiation is TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS1.2)

 

The only weird thing that stands out about that website is that the list of SubjectAlternateNames is huge. Could this be a possible bug with Squid ?

 

My TLS options in Squid.conf :

 

tls_outgoing_options cafile=/etc/pki/tls/certs/ca-bundle.crt \

    options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \

    cipher=HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!EXPORT:!DES:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

 

https_port :

 

https_port 23129 intercept ssl-bump \

    generate-host-certificates=on \

    dynamic_cert_mem_cache_size=4MB \

    cert=/etc/squid/InternetCA/InternetCA.pem \

    key=/etc/squid/InternetCA/InternetCA.key \

    tls-cafile=/etc/squid/InternetCA/InternetCA.chain.pem \

    capath=/etc/pki/tls/certs/certs.d \

    options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \

    tls-dh=prime256v1:/etc/squid/dhparam.pem

 

Please advise.

 

Regards,

Sarfaraz


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Marcus Kool
pcmag.com also does not load here, although my config parameters are slightly different.
The certificate is indeed huge...
Do you have
    ERROR: negotiating TLS on FD NNN: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)
or other errors in cache.log ?

Marcus

On 15/05/18 10:15, Ahmad, Sarfaraz wrote:

> Hi Folks,
>
> I am using Squid as a HTTPS interception proxy. When I try to access https://www.pcmag.com , (which is supposed to be bumped in my environment ), I get
>
> “unable to forward request at this time” even though the website is perfectly accessible outside of the proxy.
>
> A packet capture suggests that after Client Hello -> ServerHello -> ServerCertificate,Server Key Exchange, ServerHelloDone, the remote server just sends a FIN,ACK packet, killing off the TCP
> connection. Nothing else looks out of the ordinary.  ( Without squid, firefox successfully opens the site and the negotiation is TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS1.2)
>
> The only weird thing that stands out about that website is that the list of SubjectAlternateNames is huge. Could this be a possible bug with Squid ?
>
> My TLS options in Squid.conf :
>
> tls_outgoing_options cafile=/etc/pki/tls/certs/ca-bundle.crt \
>
>      options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \
>
>      cipher=HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!EXPORT:!DES:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
>
> https_port :
>
> https_port 23129 intercept ssl-bump \
>
>      generate-host-certificates=on \
>
>      dynamic_cert_mem_cache_size=4MB \
>
>      cert=/etc/squid/InternetCA/InternetCA.pem \
>
>      key=/etc/squid/InternetCA/InternetCA.key \
>
>      tls-cafile=/etc/squid/InternetCA/InternetCA.chain.pem \
>
>      capath=/etc/pki/tls/certs/certs.d \
>
>      options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE \
>
>      tls-dh=prime256v1:/etc/squid/dhparam.pem
>
> Please advise.
>
> Regards,
>
> Sarfaraz
>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Amos Jeffries
Administrator
On 16/05/18 01:32, Marcus Kool wrote:

> pcmag.com also does not load here, although my config parameters are
> slightly different.
> The certificate is indeed huge...
> Do you have
>    ERROR: negotiating TLS on FD NNN: error:14090086:SSL
> routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)
> or other errors in cache.log ?
>
> Marcus
>

Are these Squid-4.0.24 ? There is a regression[1] in the cafile=
parameter handling in the latest release.
 <https://bugs.squid-cache.org/show_bug.cgi?id=4831>

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Marcus Kool
The proxies that I used for the test have Squid 4.0.22 and Squid 4.0.23.

Marcus


On 15/05/18 15:40, Amos Jeffries wrote:

> On 16/05/18 01:32, Marcus Kool wrote:
>> pcmag.com also does not load here, although my config parameters are
>> slightly different.
>> The certificate is indeed huge...
>> Do you have
>>     ERROR: negotiating TLS on FD NNN: error:14090086:SSL
>> routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)
>> or other errors in cache.log ?
>>
>> Marcus
>>
>
> Are these Squid-4.0.24 ? There is a regression[1] in the cafile=
> parameter handling in the latest release.
>   <https://bugs.squid-cache.org/show_bug.cgi?id=4831>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Ahmad, Sarfaraz
I see a message similar to Marcus' in cache.log.

2018/05/16 00:20:10 kid1| ERROR: negotiating TLS on FD 77: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)

And I am running squid-4.0.24.

Sarfaraz

-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Marcus Kool
Sent: Wednesday, May 16, 2018 1:41 AM
To: [hidden email]
Subject: Re: [squid-users] TCP FIN,ACK after ServerHelloDone with pcmag.com

The proxies that I used for the test have Squid 4.0.22 and Squid 4.0.23.

Marcus


On 15/05/18 15:40, Amos Jeffries wrote:

> On 16/05/18 01:32, Marcus Kool wrote:
>> pcmag.com also does not load here, although my config parameters are
>> slightly different.
>> The certificate is indeed huge...
>> Do you have
>>     ERROR: negotiating TLS on FD NNN: error:14090086:SSL
>> routines:ssl3_get_server_certificate:certificate verify failed
>> (1/-1/0) or other errors in cache.log ?
>>
>> Marcus
>>
>
> Are these Squid-4.0.24 ? There is a regression[1] in the cafile=
> parameter handling in the latest release.
>   <https://bugs.squid-cache.org/show_bug.cgi?id=4831>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Ahmad, Sarfaraz
In reply to this post by Marcus Kool
Guys,

Any thoughts ?

Regards,
Sarfaraz

-----Original Message-----
From: Ahmad, Sarfaraz
Sent: Wednesday, May 16, 2018 10:36 AM
To: 'Marcus Kool' <[hidden email]>; [hidden email]
Subject: RE: [squid-users] TCP FIN,ACK after ServerHelloDone with pcmag.com

I see a message similar to Marcus' in cache.log.

2018/05/16 00:20:10 kid1| ERROR: negotiating TLS on FD 77: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)

And I am running squid-4.0.24.

Sarfaraz

-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Marcus Kool
Sent: Wednesday, May 16, 2018 1:41 AM
To: [hidden email]
Subject: Re: [squid-users] TCP FIN,ACK after ServerHelloDone with pcmag.com

The proxies that I used for the test have Squid 4.0.22 and Squid 4.0.23.

Marcus


On 15/05/18 15:40, Amos Jeffries wrote:

> On 16/05/18 01:32, Marcus Kool wrote:
>> pcmag.com also does not load here, although my config parameters are
>> slightly different.
>> The certificate is indeed huge...
>> Do you have
>>     ERROR: negotiating TLS on FD NNN: error:14090086:SSL
>> routines:ssl3_get_server_certificate:certificate verify failed
>> (1/-1/0) or other errors in cache.log ?
>>
>> Marcus
>>
>
> Are these Squid-4.0.24 ? There is a regression[1] in the cafile=
> parameter handling in the latest release.
>   <https://bugs.squid-cache.org/show_bug.cgi?id=4831>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Ahmad, Sarfaraz
In reply to this post by Marcus Kool
I was wrong. It is not the remote server but Squid itself which is sending a FIN,ACK after ServerHelloDone.
At 8 seconds, ServerKeyExchange, ServerHelloDone is received by Squid. The cipher suite looks like (ECDHE+RSA+SHA512 ,wireshark shows rsa_pkcs_sha512.)
After about 60 more seconds (there is no activity on the wire during this period), Squid sends a FIN/ACK to the remote server effectively closing the connection.
What debug_options should I be using for more relevant logging in cache.log ? 26,9 11,9 and 5,9 are not helping much.

I am adding few loglines anyways.

2018/05/28 07:20:13.603 kid1| 5,4| AsyncCall.cc(26) AsyncCall: The AsyncCall clientLifetimeTimeout constructed, this=0x1c5e5f0 [call136782]
2018/05/28 07:20:13.603 kid1| 5,3| comm.cc(559) commSetConnTimeout: local=<Squid_IP>:3128 remote=<Client_IP>:64774 FD 13 flags=1 timeout 86400
2018/05/28 07:20:13.603 kid1| 11,5| HttpRequest.cc(460) detailError: current error details: 12/-2
2018/05/28 07:20:13.603 kid1| 11,2| Stream.cc(266) sendStartOfMessage: HTTP Client local=<Squid_IP>:3128 remote=<Client_IP>:64774 FD 13 flags=1
2018/05/28 07:20:13.603 kid1| 11,2| Stream.cc(267) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 503 Service Unavailable

Post splicing the webpage opens just fine. That website (www.pcmag.com) has over 750 DNS names added to SAN field. The RFC does not set an upper bound on the number of DNS names you can have in there.

Regards,
Sarfaraz

-----Original Message-----
From: Ahmad, Sarfaraz
Sent: Thursday, May 17, 2018 4:18 PM
To: '[hidden email]' <[hidden email]>
Cc: 'Marcus Kool' <[hidden email]>
Subject: RE: [squid-users] TCP FIN,ACK after ServerHelloDone with pcmag.com

Guys,

Any thoughts ?

Regards,
Sarfaraz

-----Original Message-----
From: Ahmad, Sarfaraz
Sent: Wednesday, May 16, 2018 10:36 AM
To: 'Marcus Kool' <[hidden email]>; [hidden email]
Subject: RE: [squid-users] TCP FIN,ACK after ServerHelloDone with pcmag.com

I see a message similar to Marcus' in cache.log.

2018/05/16 00:20:10 kid1| ERROR: negotiating TLS on FD 77: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)

And I am running squid-4.0.24.

Sarfaraz

-----Original Message-----
From: squid-users <[hidden email]> On Behalf Of Marcus Kool
Sent: Wednesday, May 16, 2018 1:41 AM
To: [hidden email]
Subject: Re: [squid-users] TCP FIN,ACK after ServerHelloDone with pcmag.com

The proxies that I used for the test have Squid 4.0.22 and Squid 4.0.23.

Marcus


On 15/05/18 15:40, Amos Jeffries wrote:

> On 16/05/18 01:32, Marcus Kool wrote:
>> pcmag.com also does not load here, although my config parameters are
>> slightly different.
>> The certificate is indeed huge...
>> Do you have
>>     ERROR: negotiating TLS on FD NNN: error:14090086:SSL
>> routines:ssl3_get_server_certificate:certificate verify failed
>> (1/-1/0) or other errors in cache.log ?
>>
>> Marcus
>>
>
> Are these Squid-4.0.24 ? There is a regression[1] in the cafile=
> parameter handling in the latest release.
>   <https://bugs.squid-cache.org/show_bug.cgi?id=4831>
>
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: TCP FIN,ACK after ServerHelloDone with pcmag.com

Amos Jeffries
Administrator
On 29/05/18 00:17, Ahmad, Sarfaraz wrote:
> I was wrong. It is not the remote server but Squid itself which is sending a FIN,ACK after ServerHelloDone.
> At 8 seconds, ServerKeyExchange, ServerHelloDone is received by Squid. The cipher suite looks like (ECDHE+RSA+SHA512 ,wireshark shows rsa_pkcs_sha512.)
> After about 60 more seconds (there is no activity on the wire during this period), Squid sends a FIN/ACK to the remote server effectively closing the connection.
> What debug_options should I be using for more relevant logging in cache.log ? 26,9 11,9 and 5,9 are not helping much.

If in doubt ALL,9 has everything.


Sounds normal symptoms for a verify failure, except odd that there is
still a 60sec timeout happening. It should FIN immediately on the verify
failure.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users