Squid 4.4 security_file_certgen helpers crashing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid 4.4 security_file_certgen helpers crashing

johnr
Hi,

I am having trouble running squid 4.4 on ubuntu 14.04. I have successfully
built squid, and it runs fine if I'm not trying to SSL bump, but once I SSL
bump traffic, it starts crashing.

I've tried various ssl bump configurations with the same net result, so I
don't believe the configuration is relevant, but here it is:
sslcrtd_children 2 startup=2 idle=1
http_port 3129 ssl-bump generate-host-certificates=on
cert=/home/ssl_bump.pem
acl step1 at_step SslBump1
ssl_bump stare step1
ssl_bump bump all

After browsing to a https site, squid crashes and I find the following in
the cache log:
2018/12/27 21:15:40 kid1| WARNING:
/usr/local/squid/libexec/security_file_certgen -s
/usr/local/squid/var/cache/squid/ssl_db -M 4MB #Hlpr1 exited
2018/12/27 21:15:40 kid1| FATAL: The
/usr/local/squid/libexec/security_file_certgen -s
/usr/local/squid/var/cache/squid/ssl_db -M 4MB helpers are crashing too
rapidly, need help!

I ran the security_gen_helper under GDB and it seems to be crashing here:
https://github.com/squid-cache/squid/blob/master/src/ssl/gadgets.cc#L218

My squid version output is as follows:
Squid Cache: Version 4.4
Service Name: squid

This binary uses OpenSSL 1.0.1f 6 Jan 2014. For legal restrictions on
distribution see https://www.openssl.org/source/license.html

configure options:  '--disable-arch-native' '--disable-dependency-tracking'
'--disable-eui' '--enable-auth'
'--enable-basic-auth-helpers=getpwnam,LDAP,PAM'
'--enable-digest-auth-helpers=password'
'--enable-external-acl-helpers=file_userip,unix_group'
'--enable-follow-x-forwarded-for' '--enable-ssl-crtd' '--with-openssl'
'--disable-ipv6'

I specifically mention ubuntu 14.04, because I compiled and ran squid 4.4 on
ubuntu 18.04 with the same config and it ran successfully. I was
successfully able to run squid 4.3 on ubuntu 14.04 and 18.04, so I think
this might be something newly introduced in the code? I saw a commit
supporting a newer version of openssl, I wonder if that may have mistakenly
broken support for older versions of openssl?

Thank you for any help!



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 4.4 security_file_certgen helpers crashing

ziprasidone146939277
> I've tried various ssl bump configurations with the same net result, so I don't
> believe the configuration is relevant, but here it is:

> sslcrtd_children 2 startup=2 idle=1

Try comment this line and then see if it fails/crash.
Note that this directive has a default value which is 32/5/1.

See: http://www.squid-cache.org/Doc/config/sslcrtd_children/

HTH,

Julian


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 4.4 security_file_certgen helpers crashing

johnr
Thanks for the reply Julian.


ziprasidone146939277 wrote
> Try comment this line and then see if it fails/crash.
> Note that this directive has a default value which is 32/5/1.

I tried commenting out this line and saw the same behavior.




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 4.4 security_file_certgen helpers crashing

Alex Rousskov
In reply to this post by johnr
On 12/27/18 2:30 PM, johnr wrote:

> I find the following in the cache log:

> 2018/12/27 21:15:40 kid1| WARNING:
> /usr/local/squid/libexec/security_file_certgen -s
> /usr/local/squid/var/cache/squid/ssl_db -M 4MB #Hlpr1 exited

We need to figure out why the helper is exiting. If there are no error
messages in cache.log, then your system log may have additional
information such as the process signal that killed the helper. If it was
a crash, then your core dump directory should have the corresponding
core dump (make sure you enable core dumps!) that you can examine with gdb.


> I ran the security_gen_helper under GDB and it seems to be crashing here:
> https://github.com/squid-cache/squid/blob/master/src/ssl/gadgets.cc#L218

If you can reproduce helper crash while it has gdb attached, please post
the stack trace.


> I saw a commit supporting a newer version of openssl, I wonder if
> that may have mistakenly broken support for older versions of
> openssl?

Sure, it may have. Most likely, the changes are not tested in an
environment matching yours, and the bug may be environment-driven.


If you get more details such as a backtrace, please consider filing a
bug report with all the details.


Thank you,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 4.4 security_file_certgen helpers crashing

Eliezer Croitoru
Hey Alex,

I didn't had the time to sit and compose a STDINT/OUT input and output that can be used to test the security_file_certgen.
Can you or anyone of the related developers post in the wiki a simple "example" input that can be sent over STDIN to debug this type of issues?
I can just load the software as squid or proxy user but...
Another option is to point us towards the debug options that will give the testing admins(or me) an option to copy and paste the data that squid is sending to the helper.

Thanks,
Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
Sent: Friday, December 28, 2018 05:29
To: [hidden email]
Subject: Re: [squid-users] Squid 4.4 security_file_certgen helpers crashing

On 12/27/18 2:30 PM, johnr wrote:

> I find the following in the cache log:

> 2018/12/27 21:15:40 kid1| WARNING:
> /usr/local/squid/libexec/security_file_certgen -s
> /usr/local/squid/var/cache/squid/ssl_db -M 4MB #Hlpr1 exited

We need to figure out why the helper is exiting. If there are no error
messages in cache.log, then your system log may have additional
information such as the process signal that killed the helper. If it was
a crash, then your core dump directory should have the corresponding
core dump (make sure you enable core dumps!) that you can examine with gdb.


> I ran the security_gen_helper under GDB and it seems to be crashing here:
> https://github.com/squid-cache/squid/blob/master/src/ssl/gadgets.cc#L218

If you can reproduce helper crash while it has gdb attached, please post
the stack trace.


> I saw a commit supporting a newer version of openssl, I wonder if
> that may have mistakenly broken support for older versions of
> openssl?

Sure, it may have. Most likely, the changes are not tested in an
environment matching yours, and the bug may be environment-driven.


If you get more details such as a backtrace, please consider filing a
bug report with all the details.


Thank you,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 4.4 security_file_certgen helpers crashing

johnr
In reply to this post by Alex Rousskov
I can open a bug if you think I should/can based on the backtrace here:

Core was generated by `(security_file_certgen) -s
/usr/local/squid/var/cache/squid/ssl_db -M 4MB'.
Program terminated with signal SIGSEGV, Segmentation fault.


Dec 28 22:15:20 vagrant-ubuntu-trusty-64 kernel: [ 4314.045153]
security_file_c[2876]: segfault at 100000000 ip 000000000040d5d8 sp
00007ffd810b4010 error 4 in security_file_certgen[400000+13000]

Program received signal SIGSEGV, Segmentation fault.
printX509Signature (out="", cert=...) at gadgets.cc:225
225   gadgets.cc: No such file or directory.
(gdb) bt
#0  printX509Signature (out="", cert=...) at gadgets.cc:225
#1  Ssl::OnDiskCertificateDbKey (properties=...) at gadgets.cc:238
#2  0x0000000000405f8d in processNewRequest (fs_block_size=4096,
max_db_size=4194304,
    db_path="/var/lib/squid-pnr-proxy/ssl_db_v4", request_message=...) at
security_file_certgen.cc:195
#3  main (argc=<optimized out>, argv=0x7fffffffe608) at
security_file_certgen.cc:345



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 4.4 security_file_certgen helpers crashing

Alex Rousskov
On 1/2/19 12:19 PM, johnr wrote:
> I can open a bug if you think I should/can based on the backtrace here:

I think that you should/can, but it is your call. If you decide to open
a bug report, please also post gdb output of the following commands
(sanitized if needed):

-------------
set print pretty
set print-static-members off

bt full

frame 0
print cert
print *cert.raw
print *sig
print s

frame 1
print properties

frame 2
print request_message
-------------


Thank you,

Alex.


> Core was generated by `(security_file_certgen) -s
> /usr/local/squid/var/cache/squid/ssl_db -M 4MB'.
> Program terminated with signal SIGSEGV, Segmentation fault.
>
>
> Dec 28 22:15:20 vagrant-ubuntu-trusty-64 kernel: [ 4314.045153]
> security_file_c[2876]: segfault at 100000000 ip 000000000040d5d8 sp
> 00007ffd810b4010 error 4 in security_file_certgen[400000+13000]
>
> Program received signal SIGSEGV, Segmentation fault.
> printX509Signature (out="", cert=...) at gadgets.cc:225
> 225   gadgets.cc: No such file or directory.
> (gdb) bt
> #0  printX509Signature (out="", cert=...) at gadgets.cc:225
> #1  Ssl::OnDiskCertificateDbKey (properties=...) at gadgets.cc:238
> #2  0x0000000000405f8d in processNewRequest (fs_block_size=4096,
> max_db_size=4194304,
>     db_path="/var/lib/squid-pnr-proxy/ssl_db_v4", request_message=...) at
> security_file_certgen.cc:195
> #3  main (argc=<optimized out>, argv=0x7fffffffe608) at
> security_file_certgen.cc:345
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users