Quantcast

HTTPS reverse proxy: SSL Certficate verification failed

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

HTTPS reverse proxy: SSL Certficate verification failed

Eric Veiras Galisson
Hello,

I want to setup Squid as a HTTPS reverse proxy for several of our websites, but I have a certificate verification problem on Squid access.log.
Our upstream webservers are behind a VPN tunnel and only the Squid server can access it. (We actually use Nginx for the same purpose but want to switch to Squid)

                              HTTPS                           HTTPS
[client browser] -----------------------> [Squid] --------------------------> [upstream server]


I run squid 3.4.8-6+deb8u4 recompiled with --enable-ssl --with-open-ssl="/etc/ssl/openssl.cnf" on Debian Jessie.

The certificate presented to the client is the same as on the upstream server, a wildcard one signed by GeoTrust (with intermediate CA). It appears correctly in the browser.
The problem comes from squid verification of upstream certificate.

My basic squid.conf looks like

https_port <squid IP>:443 accel defaultsite=upstream1.domain.tld vhost cert=<path to SSL cert>

cache_peer <upstream IP> parent 443 0 no-query originserver name=upstream1 ssl 

acl upstream1 dstdomain upstream1.domain.tld
cache_peer_access upstream1 allow upstream1

And logs are full of

fwdNegotiateSSL: Error negotiating SSL connection on FD 14: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
TCP connection to <upstream IP> failed


If I verify with openssl the upstream server, I got an error but if I give it the intermediary CA certificate (to be precise I give it the full chain concatenated in one file), it's OK.

$ openssl s_client -showcerts -connect upstream.domain.tld:443 -CAfile <path to full cert chain>.pem 
CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
verify return:1
depth=0 CN = *.fraudbuster.mobi
verify return:1

...

    Timeout   : 300 (sec)
    Verify return code: 0 (ok)


In squid, I tried several options for cache_peer (sslcapath and sslcafile...) but I keep having this error. Of course the sslflags=DONT_VERIFY_PEER,DONT_VERIFY_DOMAIN options solve the problem, but I don't want to use this solution (my certificate is legitimate and want to validate it normally).

What am I doing wrong? and what should I do to make squid work in this setup?

Thanks.

Eric.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Amos Jeffries
Administrator
On 31/03/2017 4:01 a.m., Eric Veiras Galisson wrote:

> Hello,
>
> I want to setup Squid as a HTTPS reverse proxy for several of our websites,
> but I have a certificate verification problem on Squid access.log.
> Our upstream webservers are behind a VPN tunnel and only the Squid server
> can access it. (*We actually use Nginx for the same purpose but want to
> switch to Squid)*
>
>                               HTTPS                           HTTPS
> [client browser] -----------------------> [Squid]
> --------------------------> [upstream server]
>
>
> I run squid 3.4.8-6+deb8u4 recompiled with --enable-ssl
> --with-open-ssl="/etc/ssl/openssl.cnf" on Debian Jessie.
>
> The certificate presented to the client is the same as on the upstream
> server, a wildcard one signed by GeoTrust (with intermediate CA). It
> appears correctly in the browser.
> The problem comes from squid verification of upstream certificate.
>
...
>
> What am I doing wrong? and what should I do to make squid work in this
> setup?

The server (and Squid) should be supplying the intermediate cert along
with its own cert for best validation behaviour.

If it cannot, use the cache_peer sslcafile= option to provide Squid with
a PEM file containing the public certs of the whole chain (excluding the
server cert itself). Order of those certs in the file is important. I've
forgotten which end of the chain to start with sorry, but it is one or
the other and definitely sequential.

When that is working you should use the sslflags=NO_DEFAULT_CA option to
prevent MITM on those connections altering the chain - and saves a huge
amount of memory :-).

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Eric Veiras Galisson


On Fri, Mar 31, 2017 at 4:44 AM, Amos Jeffries <[hidden email]> wrote:
On 31/03/2017 4:01 a.m., Eric Veiras Galisson wrote:
> Hello,
>
> I want to setup Squid as a HTTPS reverse proxy for several of our websites,
> but I have a certificate verification problem on Squid access.log.
> Our upstream webservers are behind a VPN tunnel and only the Squid server
> can access it. (*We actually use Nginx for the same purpose but want to
> switch to Squid)*
>
>                               HTTPS                           HTTPS
> [client browser] -----------------------> [Squid]
> --------------------------> [upstream server]
>
>
> I run squid 3.4.8-6+deb8u4 recompiled with --enable-ssl
> --with-open-ssl="/etc/ssl/openssl.cnf" on Debian Jessie.
>
> The certificate presented to the client is the same as on the upstream
> server, a wildcard one signed by GeoTrust (with intermediate CA). It
> appears correctly in the browser.
> The problem comes from squid verification of upstream certificate.
>
...
>
> What am I doing wrong? and what should I do to make squid work in this
> setup?

The server (and Squid) should be supplying the intermediate cert along
with its own cert for best validation behaviour.

Both the server and squid (https_port cert= option) are actually using the same certificate: a single file with priv key, server certificate and intermediary cert CA.

 
If it cannot, use the cache_peer sslcafile= option to provide Squid with
a PEM file containing the public certs of the whole chain (excluding the
server cert itself). Order of those certs in the file is important. I've
forgotten which end of the chain to start with sorry, but it is one or
the other and definitely sequential.


I changed cache_peer directive to add sslcafile option to a PEM file containing root and intermediary CA certificate, in one order or the other.

And when verifying with openssl s_client -showcerts -connect upstream1.domain.tld directly (no squid) or via squid, it's OK [1]

$ openssl s_client -showcerts -connect upstream.domain.tld:443
CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
verify return:1
depth=0 CN = *.domain.tld
verify return:1
---
    Verify return code: 0 (ok)


But I still have the same error when connecting to the website with a browser.


 
When that is working you should use the sslflags=NO_DEFAULT_CA option to
prevent MITM on those connections altering the chain - and saves a huge
amount of memory :-).

Amos
 

[1] for this server at least, which is Apache 2.4 on Debian Jessie, old Apache 2.2 on Debian Wheezy don't work, but it's another problem.


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Amos Jeffries
Administrator
On 1/04/2017 4:42 a.m., Eric Veiras Galisson wrote:

> On Fri, Mar 31, 2017 at 4:44 AM, Amos Jeffries wrote:
>
>> On 31/03/2017 4:01 a.m., Eric Veiras Galisson wrote:
>>> Hello,
>>>
>>> I want to setup Squid as a HTTPS reverse proxy for several of our
>> websites,
>>> but I have a certificate verification problem on Squid access.log.
>>> Our upstream webservers are behind a VPN tunnel and only the Squid server
>>> can access it. (*We actually use Nginx for the same purpose but want to
>>> switch to Squid)*
>>>
>>>                               HTTPS                           HTTPS
>>> [client browser] -----------------------> [Squid]
>>> --------------------------> [upstream server]
>>>
>>>
>>> I run squid 3.4.8-6+deb8u4 recompiled with --enable-ssl
>>> --with-open-ssl="/etc/ssl/openssl.cnf" on Debian Jessie.
>>>
>>> The certificate presented to the client is the same as on the upstream
>>> server, a wildcard one signed by GeoTrust (with intermediate CA). It
>>> appears correctly in the browser.
>>> The problem comes from squid verification of upstream certificate.
>>>
>> ...
>>>
>>> What am I doing wrong? and what should I do to make squid work in this
>>> setup?
>>
>> The server (and Squid) should be supplying the intermediate cert along
>> with its own cert for best validation behaviour.
>>
>
> Both the server and squid (https_port cert= option) are actually using the
> same certificate: a single file with priv key, server certificate and
> intermediary cert CA.
>

That does not matter. TLS is point-to-point.

What matters is that *any* software operating as the server endpoint of
a TLS connection sends the right details along with its cert.

Your setup is special in that it has multiple pieces of software using
the same cert (Squid and the peer web service). That is not great for
security (reduces the effective trust lifetime of the cert), but is doable.


>
>> If it cannot, use the cache_peer sslcafile= option to provide Squid with
>> a PEM file containing the public certs of the whole chain (excluding the
>> server cert itself). Order of those certs in the file is important. I've
>> forgotten which end of the chain to start with sorry, but it is one or
>> the other and definitely sequential.
>>
>>
> I changed cache_peer directive to add sslcafile option to a PEM file
> containing root and intermediary CA certificate, in one order or the other.
>
> And when verifying with openssl s_client -showcerts -connect
> upstream1.domain.tld directly (no squid) or via squid, it's OK [1]
>
> $ openssl s_client -showcerts -connect upstream.domain.tld:443
> CONNECTED(00000003)
> depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
> verify return:1
> depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
> verify return:1
> depth=0 CN = *.domain.tld
> verify return:1
> ---
>     Verify return code: 0 (ok)
>
>
> But I still have the same error when connecting to the website with a
> browser.
>

Exact same error? Since Squid is juggling multiple TLS connections for
the transaction I am looking at "fwdNegotiateSSL" in the log message
which is the only detail indicating what connection is having the issue.

That Squid->server connection has zero difference between the browser
and the command line tool connecting to a reverse-proxy, or when both
are using opaque (non-Bumped) CONNECT tunnels. So one working and the
other not is impossible.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Eric Veiras Galisson
On Sun, Apr 2, 2017 at 10:27 AM, Amos Jeffries <[hidden email]> wrote:
On 1/04/2017 4:42 a.m., Eric Veiras Galisson wrote:
> On Fri, Mar 31, 2017 at 4:44 AM, Amos Jeffries wrote:
>
>> On 31/03/2017 4:01 a.m., Eric Veiras Galisson wrote:
>>> Hello,
>>>
>>> I want to setup Squid as a HTTPS reverse proxy for several of our
>> websites,
>>> but I have a certificate verification problem on Squid access.log.
>>> Our upstream webservers are behind a VPN tunnel and only the Squid server
>>> can access it. (*We actually use Nginx for the same purpose but want to
>>> switch to Squid)*
>>>
>>>                               HTTPS                           HTTPS
>>> [client browser] -----------------------> [Squid]
>>> --------------------------> [upstream server]
>>>
>>>
>>> I run squid 3.4.8-6+deb8u4 recompiled with --enable-ssl
>>> --with-open-ssl="/etc/ssl/openssl.cnf" on Debian Jessie.
>>>
>>> The certificate presented to the client is the same as on the upstream
>>> server, a wildcard one signed by GeoTrust (with intermediate CA). It
>>> appears correctly in the browser.
>>> The problem comes from squid verification of upstream certificate.
>>>
>> ...
>>>
>>> What am I doing wrong? and what should I do to make squid work in this
>>> setup?
>>
>> The server (and Squid) should be supplying the intermediate cert along
>> with its own cert for best validation behaviour.
>>
>
> Both the server and squid (https_port cert= option) are actually using the
> same certificate: a single file with priv key, server certificate and
> intermediary cert CA.
>

That does not matter. TLS is point-to-point.

What matters is that *any* software operating as the server endpoint of
a TLS connection sends the right details along with its cert.

I think that is what I'm doing: priv key, server cert + intermediate CA cert.

And openssl s_client is OK with TLS chain.

 
Your setup is special in that it has multiple pieces of software using
the same cert (Squid and the peer web service). That is not great for
security (reduces the effective trust lifetime of the cert), but is doable.
 
I understand that.


>
>> If it cannot, use the cache_peer sslcafile= option to provide Squid with
>> a PEM file containing the public certs of the whole chain (excluding the
>> server cert itself). Order of those certs in the file is important. I've
>> forgotten which end of the chain to start with sorry, but it is one or
>> the other and definitely sequential.
>>
>>
> I changed cache_peer directive to add sslcafile option to a PEM file
> containing root and intermediary CA certificate, in one order or the other.
>
> And when verifying with openssl s_client -showcerts -connect
> upstream1.domain.tld directly (no squid) or via squid, it's OK [1]
>
> $ openssl s_client -showcerts -connect upstream.domain.tld:443
> CONNECTED(00000003)
> depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
> verify return:1
> depth=1 C = US, O = GeoTrust Inc., CN = RapidSSL SHA256 CA
> verify return:1
> depth=0 CN = *.domain.tld
> verify return:1
> ---
>     Verify return code: 0 (ok)
>
>
> But I still have the same error when connecting to the website with a
> browser.
>

Exact same error? Since Squid is juggling multiple TLS connections for
the transaction I am looking at "fwdNegotiateSSL" in the log message
which is the only detail indicating what connection is having the issue.


Yes, same error. If I try to access upstream1.domain.tld via a browser via squid, I got this in squid/cache.log

2017/04/03 09:29:57 kid1| fwdNegotiateSSL: Error negotiating SSL connection on FD 14: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
2017/04/03 09:29:57 kid1| TCP connection to <upstream IP>/443 failed
2017/04/03 09:29:57 kid1| fwdNegotiateSSL: Error negotiating SSL connection on FD 14: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
2017/04/03 09:29:57 kid1| TCP connection to <upstream IP>/443 failed

No request is visible in upstream Apache logs.

 
That Squid->server connection has zero difference between the browser
and the command line tool connecting to a reverse-proxy, or when both
are using opaque (non-Bumped) CONNECT tunnels. So one working and the
other not is impossible.

Yes, I understand this. My problem now is finding what is failing in my setup.

Eric.


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Amos Jeffries
Administrator
On 3/04/2017 8:38 p.m., Eric Veiras Galisson wrote:

> On Sun, Apr 2, 2017 at 10:27 AM, Amos Jeffries wrote:
>
>> That Squid->server connection has zero difference between the browser
>> and the command line tool connecting to a reverse-proxy, or when both
>> are using opaque (non-Bumped) CONNECT tunnels. So one working and the
>> other not is impossible.
>>
>
> Yes, I understand this. My problem now is finding what is failing in my
> setup.
>
> Eric.
>

I think you are going to have to resort to packet tracing with wireshark
on the Squid->server connection. :-( good luck.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Eric Veiras Galisson
I'm back with more information about my problem.

I put squid in front of https://fr.wikipedia.org, I generated a false certificate for my test to avoid problems with my browser and... I still have a problem with squid, the same as before.

I'm thinking that my problem does not come from the upstream certificate itself (which could be the case with ours, but I don't think about wikipedia's ;) and that the root cause is my custom squid build.

I'm running squid Debian Jessie version 3.4.8-6+deb8u4 and I recompiled adding the following options:
- --enable-ssl --with-open-ssl="/etc/ssl/openssl.cnf"
- --enable-ssl --with-open-ssl
- --enable-ssl
- --enable-ssl --with-open-ssl --with-ssl-crtd

I tried these combinations and none of them solve my problem. I think I may be missing some important compilation option but I can't find which.

Output of squid -v

Squid Cache: Version 3.4.8
Debian linux
configure options:  '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,MSNT,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--enable-ssl' '--with-open-ssl' '--enable-ssl-crtd' '--disable-translation' '--with-swapdir=/var/spool/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-build-info=Debian linux' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wall' 'LDFLAGS=-fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security'


On Tue, Apr 4, 2017 at 2:14 AM, Amos Jeffries <[hidden email]> wrote:
On 3/04/2017 8:38 p.m., Eric Veiras Galisson wrote:
> On Sun, Apr 2, 2017 at 10:27 AM, Amos Jeffries wrote:
>
>> That Squid->server connection has zero difference between the browser
>> and the command line tool connecting to a reverse-proxy, or when both
>> are using opaque (non-Bumped) CONNECT tunnels. So one working and the
>> other not is impossible.
>>
>
> Yes, I understand this. My problem now is finding what is failing in my
> setup.
>
> Eric.
>

I think you are going to have to resort to packet tracing with wireshark
on the Squid->server connection. :-( good luck.

Amos




--
Eric Veiras Galisson

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: HTTPS reverse proxy: SSL Certficate verification failed

Amos Jeffries
Administrator
On 18/04/17 21:29, Eric Veiras Galisson wrote:

> I'm back with more information about my problem.
>
> I put squid in front of https://fr.wikipedia.org, I generated a false
> certificate for my test to avoid problems with my browser and... I
> still have a problem with squid, the same as before.
>
> I'm thinking that my problem does not come from the upstream
> certificate itself (which could be the case with ours, but I don't
> think about wikipedia's ;) and that the root cause is my custom squid
> build.
>
> I'm running squid Debian Jessie version 3.4.8-6+deb8u4 and I
> recompiled adding the following options:
> - --enable-ssl --with-open-ssl="/etc/ssl/openssl.cnf"
> - --enable-ssl --with-open-ssl
> - --enable-ssl
> - --enable-ssl --with-open-ssl --with-ssl-crtd
>
> I tried these combinations and none of them solve my problem. I think
> I may be missing some important compilation option but I can't find which.

You should use: --enable-ssl-crtd --with-openssl


The --enable-ssl option is obsolete.

The --with-openssl option takes a path to where the openssl development
files are installed. Somehow I doubt that you have a library installed
as /etc/ssl/openssl.cnf/openssl/libssl.a. When building against the
systems default openssl installation you can omit the path. You only
need it if you are building a custom Squid against a custom openssl.


Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...