SSLBump, system requirements ?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

SSLBump, system requirements ?

FredB
Hi all,

I'm testing SSLBump and Squid eats up all my CPU, maybe I made something wrong or maybe some updates are required ? Any advice would be greatly appreciated.

Debian 8.10 64 bits, Squid 3.5.27 + 64 Go ram + SSD + 15 Cores Xeon(R) CPU E5-2637 v2 @ 3.50GHz
FI, I don't see anything about limit reached in kern.log (File descriptor or network)

acl nobump dstdomain "/home/squid/domains" -> Some very used websites (google, fb, etc) otherwise the system dies after less 1 minute
http_port 3128 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on dynamic_cert_mem_cache_size=500MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 100MB
sslcrtd_children 2000 startup=100 idle=20
sslproxy_capath /etc/ssl/certs/
sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem
acl step1 at_step SslBump1
ssl_bump peek step1 all
ssl_bump splice nobump
ssl_bump bump all

The sslcrtd_children increases quickly and permanently

root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
1321
root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
1341
root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
1341
root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
1380
root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
1381
root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
1382
root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
1395

Of course after a while 2000 is reached and the system becomes completely mad, but I already tried 200, 500, 1000, etc

Right after squid start CPU and load average values are very, very, high

top - 16:06:17 up 13 days,  2:46,  3 users,  load average: 102,02, 56,67, 30,75
Tasks: 1964 total,   3 running, 1961 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15,3 us,  3,7 sy,  0,0 ni, 80,2 id,  0,4 wa,  0,0 hi,  0,4 si,  0,0 st
KiB Mem:  66086692 total, 52378248 used, 13708444 free,  2899764 buffers
KiB Swap:  1952764 total,        0 used,  1952764 free. 32798948 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  
23711 squid     20   0 3438832 2,976g  13784 R 100,0  4,7   6:01.02 squid                                                    
23724 squid     20   0   24868   8552   4340 S   3,6  0,0   0:02.46 ssl_crtd                                                
23712 squid     20   0   25132   8896   4428 R   3,0  0,0   0:02.62 ssl_crtd                                                
23714 squid     20   0   24868   8556   4344 S   2,3  0,0   0:02.43 ssl_crtd                                                
23716 squid     20   0   24868   8636   4428 S   2,3  0,0   0:02.26 ssl_crtd                                                
23720 squid     20   0   24868   8612   4400 S   2,3  0,0   0:02.58 ssl_crtd                                                
23771 squid     20   0   24868   8580   4368 S   2,0  0,0   0:01.86 ssl_crtd                                                
23780 squid     20   0   24872   8484   4268 S   2,0  0,0   0:01.86 ssl_crtd                                                
23787 squid     20   0   24868   8612   4404 S   2,0  0,0   0:01.92 ssl_crtd  

The same system without SSLBump and e2guardian (web filtering) added (I tried without more or less 10% CPU )

Tasks: 304 total,   2 running, 302 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2,0 us,  1,1 sy,  0,0 ni, 95,9 id,  0,1 wa,  0,0 hi,  0,9 si,  0,0 st
KiB Mem:  66086700 total, 65627952 used,   458748 free,  2652264 buffers
KiB Swap:  1952764 total,    20884 used,  1931880 free. 32639208 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                        
20389 e2guard+  20   0  0,122t 1,133g   6144 S  28,6  1,8 191:06.50 e2guardian                      
20283 squid     20   0 21,761g 0,021t   8128 R  24,2 34,0 145:00.09 squid                          
  101 root      20   0       0      0      0 S   1,3  0,0  19:05.09 kswapd1                        
  100 root      20   0       0      0      0 S   1,0  0,0  22:41.82 kswapd0                        
    8 root      20   0       0      0      0 S   0,7  0,0  68:49.48 rcu_sched                      
   24 root      20   0       0      0      0 S   0,3  0,0   8:37.14 ksoftirqd/3                    
   65 root      20   0       0      0      0 S   0,3  0,0   8:05.02 ksoftirqd/11                    
  929 root      20   0   71928   6984   4716 S   0,3  0,0  17:53.57 syslog-ng                      
 8069 root      20   0       0      0      0 S   0,3  0,0   0:22.35 kworker/0:0                    
16624 root      20   0   25868   3236   2592 R   0,3  0,0   0:00.19 top                            
20291 squid     20   0   59504   5228   4568 S   0,3  0,0   0:03.41 digest_
 
FredB
   
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Amos Jeffries
Administrator
On 21/03/18 04:30, FredB wrote:
> Hi all,
>
> I'm testing SSLBump and Squid eats up all my CPU, maybe I made something wrong or maybe some updates are required ? Any advice would be greatly appreciated.

Not sure about CPU consumption. AFAIK that is related to traffic loading
on the crypto library, mitigated by whether it is using hardware support
for the intensive math parts.


>
> Debian 8.10 64 bits, Squid 3.5.27 + 64 Go ram + SSD + 15 Cores Xeon(R) CPU E5-2637 v2 @ 3.50GHz
> FI, I don't see anything about limit reached in kern.log (File descriptor or network)
>
> acl nobump dstdomain "/home/squid/domains" -> Some very used websites (google, fb, etc) otherwise the system dies after less 1 minute
> http_port 3128 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on dynamic_cert_mem_cache_size=500MB

Definitely use sslflags=NO_DEFAULT_CA to avoid memory bloat, whether
that is your problem now or not.

> sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 100MB

FYI: 100MB x 2000 helpers is larger than your 64GB. Even just the 100
helpers being initialized on startup is a significant chunk out of memory.


> sslcrtd_children 2000 startup=100 idle=20
> sslproxy_capath /etc/ssl/certs/
> sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem
> acl step1 at_step SslBump1
> ssl_bump peek step1 all
> ssl_bump splice nobump
> ssl_bump bump all
>
> The sslcrtd_children increases quickly and permanently
>
> root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
> 1321
...
> root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
> 1395
>
> Of course after a while 2000 is reached and the system becomes completely mad, but I already tried 200, 500, 1000, etc
>


Can you tell how fast (or not) they are responding?
 If it is particularly slow you may benefit from the memory-only mode in
the Squid-4 helper (or might not).

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov
In reply to this post by FredB


20.03.2018 21:30, FredB пишет:
> Hi all,
>
> I'm testing SSLBump and Squid eats up all my CPU, maybe I made something wrong or maybe some updates are required ? Any advice would be greatly appreciated.
>
> Debian 8.10 64 bits, Squid 3.5.27 + 64 Go ram + SSD + 15 Cores Xeon(R) CPU E5-2637 v2 @ 3.50GHz
Big box. How much users and traffic?
>  
> FI, I don't see anything about limit reached in kern.log (File descriptor or network)
>
> acl nobump dstdomain "/home/squid/domains" -> Some very used websites (google, fb, etc) otherwise the system dies after less 1 minute
> http_port 3128 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on dynamic_cert_mem_cache_size=500MB
> sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 100MB
Disbalanced config.

dynamic_cert_mem_cache_size=500MB

and only 100 MB on disk?

sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 100MB


> sslcrtd_children 2000 startup=100 idle=20
Why so much children? Again - for what workload?

> sslproxy_capath /etc/ssl/certs/
> sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem
> acl step1 at_step SslBump1
> ssl_bump peek step1 all
> ssl_bump splice nobump
> ssl_bump bump all
>
> The sslcrtd_children increases quickly and permanently
>
> root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
> 1321
> root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
> 1341
> root@proxyorion5:/tmp# ps -edf | grep ssl | wc -l
> 1341
> root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
> 1380
> root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
> 1381
> root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
> 1382
> root@proxyorion5:/tmp# ps -edf | grep ssl_crt | wc -l
> 1395
>
> Of course after a while 2000 is reached and the system becomes completely mad, but I already tried 200, 500, 1000, etc
>
> Right after squid start CPU and load average values are very, very, high
>
> top - 16:06:17 up 13 days,  2:46,  3 users,  load average: 102,02, 56,67, 30,75
> Tasks: 1964 total,   3 running, 1961 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 15,3 us,  3,7 sy,  0,0 ni, 80,2 id,  0,4 wa,  0,0 hi,  0,4 si,  0,0 st
> KiB Mem:  66086692 total, 52378248 used, 13708444 free,  2899764 buffers
> KiB Swap:  1952764 total,        0 used,  1952764 free. 32798948 cached Mem
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  
> 23711 squid     20   0 3438832 2,976g  13784 R 100,0  4,7   6:01.02 squid                                                    
> 23724 squid     20   0   24868   8552   4340 S   3,6  0,0   0:02.46 ssl_crtd                                                
> 23712 squid     20   0   25132   8896   4428 R   3,0  0,0   0:02.62 ssl_crtd                                                
> 23714 squid     20   0   24868   8556   4344 S   2,3  0,0   0:02.43 ssl_crtd                                                
> 23716 squid     20   0   24868   8636   4428 S   2,3  0,0   0:02.26 ssl_crtd                                                
> 23720 squid     20   0   24868   8612   4400 S   2,3  0,0   0:02.58 ssl_crtd                                                
> 23771 squid     20   0   24868   8580   4368 S   2,0  0,0   0:01.86 ssl_crtd                                                
> 23780 squid     20   0   24872   8484   4268 S   2,0  0,0   0:01.86 ssl_crtd                                                
> 23787 squid     20   0   24868   8612   4404 S   2,0  0,0   0:01.92 ssl_crtd
.... what means some bottlenecks. Obviously.

>  
>
> The same system without SSLBump and e2guardian (web filtering) added (I tried without more or less 10% CPU )
>
> Tasks: 304 total,   2 running, 302 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  2,0 us,  1,1 sy,  0,0 ni, 95,9 id,  0,1 wa,  0,0 hi,  0,9 si,  0,0 st
> KiB Mem:  66086700 total, 65627952 used,   458748 free,  2652264 buffers
> KiB Swap:  1952764 total,    20884 used,  1931880 free. 32639208 cached Mem
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                        
> 20389 e2guard+  20   0  0,122t 1,133g   6144 S  28,6  1,8 191:06.50 e2guardian                      
> 20283 squid     20   0 21,761g 0,021t   8128 R  24,2 34,0 145:00.09 squid                          
>   101 root      20   0       0      0      0 S   1,3  0,0  19:05.09 kswapd1                        
>   100 root      20   0       0      0      0 S   1,0  0,0  22:41.82 kswapd0                        
>     8 root      20   0       0      0      0 S   0,7  0,0  68:49.48 rcu_sched                      
>    24 root      20   0       0      0      0 S   0,3  0,0   8:37.14 ksoftirqd/3                    
>    65 root      20   0       0      0      0 S   0,3  0,0   8:05.02 ksoftirqd/11                    
>   929 root      20   0   71928   6984   4716 S   0,3  0,0  17:53.57 syslog-ng                      
>  8069 root      20   0       0      0      0 S   0,3  0,0   0:22.35 kworker/0:0                    
> 16624 root      20   0   25868   3236   2592 R   0,3  0,0   0:00.19 top                            
> 20291 squid     20   0   59504   5228   4568 S   0,3  0,0   0:03.41 digest_
>  
> FredB
>    
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

FredB
Hi Yuri,

200 mbits, more or less 1000/2000 simultaneous users

I increase children value, because the limit is reached very quickly

> and only 100 MB on disk?

100 MB by process, no ? I think I should reduce this value and rather increase the max of children

Maybe such load is just impossible because I reached a limit with a single core
Perhaps I should retry SMP but unfortunately in the past I had many issues with, and some features I'm using still SMP-unaware
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov


20.03.2018 23:03, FredB пишет:
> Hi Yuri,
>
> 200 mbits, more or less 1000/2000 simultaneous users
>
> I increase children value, because the limit is reached very quickly
Because of SSL processing to slow. Investigate, why. Simple increasing
number of children exghausting your RAM.
>
>> and only 100 MB on disk?
> 100 MB by process, no ? I think I should reduce this value and rather increase the max of children
No. This is overall fs limit to store.
>
> Maybe such load is just impossible because I reached a limit with a single core
Hardly. SSL helper children should spread across cores by OS scheduler.
> Perhaps I should retry SMP but unfortunately in the past I had many issues with, and some features I'm using still SMP-unaware
Squid's SMP itself does not solves SSL Bump issues. It's about different
things, and, IMHO, irrelevant your load profile.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

--
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov


20.03.2018 23:10, Yuri пишет:

>
> 20.03.2018 23:03, FredB пишет:
>> Hi Yuri,
>>
>> 200 mbits, more or less 1000/2000 simultaneous users
>>
>> I increase children value, because the limit is reached very quickly
> Because of SSL processing to slow. Investigate, why. Simple increasing
> number of children exghausting your RAM.
>>> and only 100 MB on disk?
>> 100 MB by process, no ? I think I should reduce this value and rather increase the max of children
> No. This is overall fs limit to store.
Look on my relatively big server (Squid 5.0) config snippet:

https_port 3127 intercept ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=10MB cert=/usr/local/squid/etc/rootCA2.crt
key=/usr/local/squid/etc/rootCA2.key
tls-cafile=/usr/local/squid/etc/rootCA12.crt
options=SINGLE_DH_USE:SINGLE_ECDH_USE
tls-dh=secp384r1:/usr/local/squid/etc/dhparam.pem
cipher=HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS
tls-no-npn sslflags=NO_DEFAULT_CA:VERIFY_CRL_ALL
http_port 3128 ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=10MB cert=/usr/local/squid/etc/rootCA2.crt
key=/usr/local/squid/etc/rootCA2.key
tls-cafile=/usr/local/squid/etc/rootCA12.crt
options=SINGLE_DH_USE:SINGLE_ECDH_USE
tls-dh=secp384r1:/usr/local/squid/etc/dhparam.pem
cipher=HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS
tls-no-npn sslflags=NO_DEFAULT_CA:VERIFY_CRL_ALL
tls_outgoing_options cafile=/usr/local/squid/etc/ca-bundle.crt
cipher=HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS

# Cert database on ramdisk
sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s
/ramdisk1/ssl_db -M 1GB
sslcrtd_children 32 startup=10 idle=5

Pay attention - I've put SSL db on RAM disk. :)
>> Maybe such load is just impossible because I reached a limit with a single core
> Hardly. SSL helper children should spread across cores by OS scheduler.
>> Perhaps I should retry SMP but unfortunately in the past I had many issues with, and some features I'm using still SMP-unaware
> Squid's SMP itself does not solves SSL Bump issues. It's about different
> things, and, IMHO, irrelevant your load profile.
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users

--
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov

Forgot about:

My server is relatively modest (more resources just do not need :))

Just 8 cores (Xeon 2.3 GHz), 16 Gb RAM, SAS HDD's 10k RPM (~300 Gb in RAID-10)  :)

Overall CPU usage is ~3% (with SSL Bump). And half of RAM is free :)


20.03.2018 23:14, Yuri пишет:

20.03.2018 23:10, Yuri пишет:
20.03.2018 23:03, FredB пишет:
Hi Yuri,

200 mbits, more or less 1000/2000 simultaneous users 

I increase children value, because the limit is reached very quickly 
Because of SSL processing to slow. Investigate, why. Simple increasing
number of children exghausting your RAM.
and only 100 MB on disk?
100 MB by process, no ? I think I should reduce this value and rather increase the max of children
No. This is overall fs limit to store.
Look on my relatively big server (Squid 5.0) config snippet:

https_port 3127 intercept ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=10MB cert=/usr/local/squid/etc/rootCA2.crt
key=/usr/local/squid/etc/rootCA2.key
tls-cafile=/usr/local/squid/etc/rootCA12.crt
options=SINGLE_DH_USE:SINGLE_ECDH_USE
tls-dh=secp384r1:/usr/local/squid/etc/dhparam.pem
cipher=HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS
tls-no-npn sslflags=NO_DEFAULT_CA:VERIFY_CRL_ALL
http_port 3128 ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=10MB cert=/usr/local/squid/etc/rootCA2.crt
key=/usr/local/squid/etc/rootCA2.key
tls-cafile=/usr/local/squid/etc/rootCA12.crt
options=SINGLE_DH_USE:SINGLE_ECDH_USE
tls-dh=secp384r1:/usr/local/squid/etc/dhparam.pem
cipher=HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS
tls-no-npn sslflags=NO_DEFAULT_CA:VERIFY_CRL_ALL
tls_outgoing_options cafile=/usr/local/squid/etc/ca-bundle.crt
cipher=HIGH:MEDIUM:RC4:3DES:!aNULL:!eNULL:!LOW:!MD5:!EXP:!PSK:!SRP:!DSS

# Cert database on ramdisk
sslcrtd_program /usr/local/squid/libexec/security_file_certgen -s
/ramdisk1/ssl_db -M 1GB
sslcrtd_children 32 startup=10 idle=5

Pay attention - I've put SSL db on RAM disk. :)
Maybe such load is just impossible because I reached a limit with a single core 
Hardly. SSL helper children should spread across cores by OS scheduler.
Perhaps I should retry SMP but unfortunately in the past I had many issues with, and some features I'm using still SMP-unaware 
Squid's SMP itself does not solves SSL Bump issues. It's about different
things, and, IMHO, irrelevant your load profile.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

    

-- 
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

FredB
In reply to this post by Yuri Voinov

> > Perhaps I should retry SMP but unfortunately in the past I had many
> > issues with, and some features I'm using still SMP-unaware
> Squid's SMP itself does not solves SSL Bump issues. It's about
> different
> things, and, IMHO, irrelevant your load profile.


I'm thinking about that, because the single squid core is 100% CPU
I tried with 900MB and 50MB without more success, I also added sslflags-NO_DEFAULT_CA

How much simultaneous users do you have ? and bandwidth ?

I'm using this right now, the number of process used is very better now but still an issue with CPU  

acl nobump dstdomain "/home/squid/domains"

http_port 8080 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on sslflags=NO_DEFAULT_CA dynamic_cert_mem_cache_size=500MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 500MB
sslcrtd_children 1000 startup=100 idle=5

sslproxy_capath /etc/ssl/certs/
sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem

acl step1 at_step SslBump1
ssl_bump peek step1 all
ssl_bump splice nobump
ssl_bump bump all

Maybe there is a problem with memory, but as you can see here CPU is the point

top - 09:50:04 up 16:16,  1 user,  load average: 1,72, 1,78, 1,39
Tasks: 393 total,   3 running, 390 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8,4 us,  1,2 sy,  0,0 ni, 89,6 id,  0,3 wa,  0,0 hi,  0,5 si,  0,0 st
KiB Mem:  66086692 total, 28654240 used, 37432452 free,  2974568 buffers
KiB Swap:  1952764 total,        0 used,  1952764 free. 17653336 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 9803 squid     20   0 3913044 3,452g  13464 R  99,9  5,5   7:47.47 squid                                                                                                                                                                    
10051 e2guard+  20   0  0,122t 284392   5124 S  25,6  0,4   1:33.10 e2guardian                                                                                                                                                              
 9804 squid     20   0   21956   5628   4420 S   7,3  0,0   0:48.93 ssl_crtd                                                                                                                                                                
 9805 squid     20   0   21952   5672   4372 S   6,3  0,0   0:31.25 ssl_crtd                                                                                                                                                                
 9806 squid     20   0   21952   5476   4252 S   2,7  0,0   0:19.10 ssl_crtd                                                                                                                                                                
 9807 squid     20   0   21952   5616   4408 S   2,3  0,0   0:13.88 ssl_crtd                                                                                                                                                                
 9808 squid     20   0   21952   5540   4332 S   2,3  0,0   0:10.59 ssl_crtd                                                                                                                                                                
 9810 squid     20   0   21956   5536   4332 S   2,0  0,0   0:05.61 ssl_crtd                                                                                                                                                                
 9809 squid     20   0   21952   5584   4372 S   1,7  0,0   0:07.40 ssl_crtd                                                                                                                                                                
 9996 squid     20   0   25612   2924   2696 S   1,3  0,0   0:05.47 diskd                                                                                                                                                                    
 9995 squid     20   0   25612   2744   2516 S   1,0  0,0   0:04.41 diskd                                                                                                                                                                    
 9811 squid     20   0   21964   5588   4372 S   0,7  0,0   0:03.72 ssl_crtd                                                                                                                                                                
 9813 squid     20   0   21848   5660   4464 S   0,7  0,0   0:01.96 ssl_crtd    

Amos, there is way to add the domain requested in message like this ?

2018/03/21 09:45:30| Error negotiating SSL on FD 1835: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
2018/03/21 09:45:30| Error negotiating SSL on FD 4782: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)

It can be very, very, useful for analysis

Thanks

FredB
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov



21.03.2018 14:55, FredB пишет:

      
Perhaps I should retry SMP but unfortunately in the past I had many
issues with, and some features I'm using still SMP-unaware
Squid's SMP itself does not solves SSL Bump issues. It's about
different
things, and, IMHO, irrelevant your load profile.

I'm thinking about that, because the single squid core is 100% CPU
I tried with 900MB and 50MB without more success, I also added sslflags-NO_DEFAULT_CA

How much simultaneous users do you have ? and bandwidth ? 
200 users, 2 Gbps downstream.

I'm using this right now, the number of process used is very better now but still an issue with CPU  

acl nobump dstdomain "/home/squid/domains"

http_port 8080 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on sslflags=NO_DEFAULT_CA dynamic_cert_mem_cache_size=500MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 500MB
sslcrtd_children 1000 startup=100 idle=5
It does not work that way. There are still many processes. Scaling is rarely linear.

Pls keep in mind, squid itself is thread-unaware. So, with
thousands of children you make serialization point by yourself from squid's size.
Another serialization point is SSL db on disk, due to it uses file locking mechanism.

Both reasons leads to make bottleneck on SSL certgen processes.

So, you can't simple set 1000 children and expect good performing.

Just for record:
Performance tuning/scaling makes different. Much different.

1. You require to set good initial approximation. Not by proportion "1 user - 1 child instance".
2. Then run your load and get performance statistics.
3. Analyze results.
4. Based on step 3, increasing/decreasing parameter value.

General rule: change only one parameter during tuning/scaling iteration.

sslproxy_capath /etc/ssl/certs/
sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem

acl step1 at_step SslBump1
ssl_bump peek step1 all
ssl_bump splice nobump
ssl_bump bump all

Maybe there is a problem with memory, but as you can see here CPU is the point 
Yes, indeed. You eat up too much RAM due to misconfiguration. But also you have 2 waiting points descrived above.

I can recommend you get wait CPU/IO performance events to make sure.

Wait IO events can increasing CPU consumption, when such structures of queues overflows etc. Usually this occurs on thread-aware apps with spin-count synchronization mech, however, often can occurs on single-threaded applications depenging implementation.

top - 09:50:04 up 16:16,  1 user,  load average: 1,72, 1,78, 1,39
Tasks: 393 total,   3 running, 390 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8,4 us,  1,2 sy,  0,0 ni, 89,6 id,  0,3 wa,  0,0 hi,  0,5 si,  0,0 st
KiB Mem:  66086692 total, 28654240 used, 37432452 free,  2974568 buffers
KiB Swap:  1952764 total,        0 used,  1952764 free. 17653336 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 9803 squid     20   0 3913044 3,452g  13464 R  99,9  5,5   7:47.47 squid                                                                                                                                                                    
10051 e2guard+  20   0  0,122t 284392   5124 S  25,6  0,4   1:33.10 e2guardian                                                                                                                                                               
 9804 squid     20   0   21956   5628   4420 S   7,3  0,0   0:48.93 ssl_crtd                                                                                                                                                                 
 9805 squid     20   0   21952   5672   4372 S   6,3  0,0   0:31.25 ssl_crtd                                                                                                                                                                 
 9806 squid     20   0   21952   5476   4252 S   2,7  0,0   0:19.10 ssl_crtd                                                                                                                                                                 
 9807 squid     20   0   21952   5616   4408 S   2,3  0,0   0:13.88 ssl_crtd                                                                                                                                                                 
 9808 squid     20   0   21952   5540   4332 S   2,3  0,0   0:10.59 ssl_crtd                                                                                                                                                                 
 9810 squid     20   0   21956   5536   4332 S   2,0  0,0   0:05.61 ssl_crtd                                                                                                                                                                 
 9809 squid     20   0   21952   5584   4372 S   1,7  0,0   0:07.40 ssl_crtd                                                                                                                                                                 
 9996 squid     20   0   25612   2924   2696 S   1,3  0,0   0:05.47 diskd                                                                                                                                                                    
 9995 squid     20   0   25612   2744   2516 S   1,0  0,0   0:04.41 diskd                                                                                                                                                                    
 9811 squid     20   0   21964   5588   4372 S   0,7  0,0   0:03.72 ssl_crtd                                                                                                                                                                 
 9813 squid     20   0   21848   5660   4464 S   0,7  0,0   0:01.96 ssl_crtd  
As you can see, your Squid's consumes most CPU. And __not__ ssl_crtd. So, most probably you have bottleneck between squid and ssl_crtd due to reasons above.
  

Amos, there is way to add the domain requested in message like this ?

2018/03/21 09:45:30| Error negotiating SSL on FD 1835: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
2018/03/21 09:45:30| Error negotiating SSL on FD 4782: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)

It can be very, very, useful for analysis 

Thanks

FredB
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

-- 
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov
In reply to this post by FredB



21.03.2018 14:55, FredB пишет:

      
Perhaps I should retry SMP but unfortunately in the past I had many
issues with, and some features I'm using still SMP-unaware
Squid's SMP itself does not solves SSL Bump issues. It's about
different
things, and, IMHO, irrelevant your load profile.

I'm thinking about that, because the single squid core is 100% CPU
I tried with 900MB and 50MB without more success, I also added sslflags-NO_DEFAULT_CA

How much simultaneous users do you have ? and bandwidth ? 

I'm using this right now, the number of process used is very better now but still an issue with CPU  

acl nobump dstdomain "/home/squid/domains"

http_port 8080 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on sslflags=NO_DEFAULT_CA dynamic_cert_mem_cache_size=500MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 500MB
sslcrtd_children 1000 startup=100 idle=5
Still misconfiguration. Pay attention. You set
dynamic_cert_mem_cache_size=500MB
Again - why so much?

Do not think that a lot of RAM will not make anything worse.

For some unknown reason, you set dynamic_cert_mem_cache_size equal to -M on-disk fs limit. It is enough to set dynamic_cert_mem_cache_size to 1/10-1/20 of overall SSL db on-disk size.

And still too high upper children limit. Just imagine, how much RAM will eat by 1000 processes. Each with own heap.

It seems for me, in your case good initial approximation will be

sslcrtd_children 256 startup=100 idle=200

No more. Other changes will do only based on performance stats and diagnostics.

sslproxy_capath /etc/ssl/certs/
sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem

acl step1 at_step SslBump1
ssl_bump peek step1 all
ssl_bump splice nobump
ssl_bump bump all

Maybe there is a problem with memory, but as you can see here CPU is the point 

top - 09:50:04 up 16:16,  1 user,  load average: 1,72, 1,78, 1,39
Tasks: 393 total,   3 running, 390 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8,4 us,  1,2 sy,  0,0 ni, 89,6 id,  0,3 wa,  0,0 hi,  0,5 si,  0,0 st
KiB Mem:  66086692 total, 28654240 used, 37432452 free,  2974568 buffers
KiB Swap:  1952764 total,        0 used,  1952764 free. 17653336 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 9803 squid     20   0 3913044 3,452g  13464 R  99,9  5,5   7:47.47 squid                                                                                                                                                                    
10051 e2guard+  20   0  0,122t 284392   5124 S  25,6  0,4   1:33.10 e2guardian                                                                                                                                                               
 9804 squid     20   0   21956   5628   4420 S   7,3  0,0   0:48.93 ssl_crtd                                                                                                                                                                 
 9805 squid     20   0   21952   5672   4372 S   6,3  0,0   0:31.25 ssl_crtd                                                                                                                                                                 
 9806 squid     20   0   21952   5476   4252 S   2,7  0,0   0:19.10 ssl_crtd                                                                                                                                                                 
 9807 squid     20   0   21952   5616   4408 S   2,3  0,0   0:13.88 ssl_crtd                                                                                                                                                                 
 9808 squid     20   0   21952   5540   4332 S   2,3  0,0   0:10.59 ssl_crtd                                                                                                                                                                 
 9810 squid     20   0   21956   5536   4332 S   2,0  0,0   0:05.61 ssl_crtd                                                                                                                                                                 
 9809 squid     20   0   21952   5584   4372 S   1,7  0,0   0:07.40 ssl_crtd                                                                                                                                                                 
 9996 squid     20   0   25612   2924   2696 S   1,3  0,0   0:05.47 diskd                                                                                                                                                                    
 9995 squid     20   0   25612   2744   2516 S   1,0  0,0   0:04.41 diskd                                                                                                                                                                    
 9811 squid     20   0   21964   5588   4372 S   0,7  0,0   0:03.72 ssl_crtd                                                                                                                                                                 
 9813 squid     20   0   21848   5660   4464 S   0,7  0,0   0:01.96 ssl_crtd    

Amos, there is way to add the domain requested in message like this ?

2018/03/21 09:45:30| Error negotiating SSL on FD 1835: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
2018/03/21 09:45:30| Error negotiating SSL on FD 4782: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)

It can be very, very, useful for analysis 

Thanks

FredB
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

-- 
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov

Finally.

Premature optimization is the root of all evils.

Never start new setups from your assumptions only. Set good enough starting values and monitor. Increase only if required.

And, pls, don't think all performance problems can solves with giant RAM.

It does not matter how big your RAM is. It's important how you use it.

Scaling is also done differently.


21.03.2018 19:08, Yuri пишет:



21.03.2018 14:55, FredB пишет:
Perhaps I should retry SMP but unfortunately in the past I had many
issues with, and some features I'm using still SMP-unaware
Squid's SMP itself does not solves SSL Bump issues. It's about
different
things, and, IMHO, irrelevant your load profile.
I'm thinking about that, because the single squid core is 100% CPU
I tried with 900MB and 50MB without more success, I also added sslflags-NO_DEFAULT_CA

How much simultaneous users do you have ? and bandwidth ? 

I'm using this right now, the number of process used is very better now but still an issue with CPU  

acl nobump dstdomain "/home/squid/domains"

http_port 8080 ssl-bump cert=/etc/squid/ca_orion/cert generate-host-certificates=on sslflags=NO_DEFAULT_CA dynamic_cert_mem_cache_size=500MB
sslcrtd_program /usr/lib/squid/ssl_crtd -s /usr/lib/squid/ssl_db -M 500MB
sslcrtd_children 1000 startup=100 idle=5
Still misconfiguration. Pay attention. You set
dynamic_cert_mem_cache_size=500MB
Again - why so much?

Do not think that a lot of RAM will not make anything worse.

For some unknown reason, you set dynamic_cert_mem_cache_size equal to -M on-disk fs limit. It is enough to set dynamic_cert_mem_cache_size to 1/10-1/20 of overall SSL db on-disk size.

And still too high upper children limit. Just imagine, how much RAM will eat by 1000 processes. Each with own heap.

It seems for me, in your case good initial approximation will be

sslcrtd_children 256 startup=100 idle=200

No more. Other changes will do only based on performance stats and diagnostics.

sslproxy_capath /etc/ssl/certs/
sslproxy_foreign_intermediate_certs /etc/squid/ssl_certs/imtermediate.ca.pem

acl step1 at_step SslBump1
ssl_bump peek step1 all
ssl_bump splice nobump
ssl_bump bump all

Maybe there is a problem with memory, but as you can see here CPU is the point 

top - 09:50:04 up 16:16,  1 user,  load average: 1,72, 1,78, 1,39
Tasks: 393 total,   3 running, 390 sleeping,   0 stopped,   0 zombie
%Cpu(s):  8,4 us,  1,2 sy,  0,0 ni, 89,6 id,  0,3 wa,  0,0 hi,  0,5 si,  0,0 st
KiB Mem:  66086692 total, 28654240 used, 37432452 free,  2974568 buffers
KiB Swap:  1952764 total,        0 used,  1952764 free. 17653336 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                  
 9803 squid     20   0 3913044 3,452g  13464 R  99,9  5,5   7:47.47 squid                                                                                                                                                                    
10051 e2guard+  20   0  0,122t 284392   5124 S  25,6  0,4   1:33.10 e2guardian                                                                                                                                                               
 9804 squid     20   0   21956   5628   4420 S   7,3  0,0   0:48.93 ssl_crtd                                                                                                                                                                 
 9805 squid     20   0   21952   5672   4372 S   6,3  0,0   0:31.25 ssl_crtd                                                                                                                                                                 
 9806 squid     20   0   21952   5476   4252 S   2,7  0,0   0:19.10 ssl_crtd                                                                                                                                                                 
 9807 squid     20   0   21952   5616   4408 S   2,3  0,0   0:13.88 ssl_crtd                                                                                                                                                                 
 9808 squid     20   0   21952   5540   4332 S   2,3  0,0   0:10.59 ssl_crtd                                                                                                                                                                 
 9810 squid     20   0   21956   5536   4332 S   2,0  0,0   0:05.61 ssl_crtd                                                                                                                                                                 
 9809 squid     20   0   21952   5584   4372 S   1,7  0,0   0:07.40 ssl_crtd                                                                                                                                                                 
 9996 squid     20   0   25612   2924   2696 S   1,3  0,0   0:05.47 diskd                                                                                                                                                                    
 9995 squid     20   0   25612   2744   2516 S   1,0  0,0   0:04.41 diskd                                                                                                                                                                    
 9811 squid     20   0   21964   5588   4372 S   0,7  0,0   0:03.72 ssl_crtd                                                                                                                                                                 
 9813 squid     20   0   21848   5660   4464 S   0,7  0,0   0:01.96 ssl_crtd    

Amos, there is way to add the domain requested in message like this ?

2018/03/21 09:45:30| Error negotiating SSL on FD 1835: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)
2018/03/21 09:45:30| Error negotiating SSL on FD 4782: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)

It can be very, very, useful for analysis 

Thanks

FredB
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

-- 
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************

-- 
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

FredB
In reply to this post by Yuri Voinov
Sorry, it was just a wrong cut/paste cache_size=50MB the previous result still the same
About children I tried with 256, unfortunately squid is still stuck at 100%

Regards

Fred

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov
Aha, this is better.

So, next step should be detailed performance statistics to identify
bottleneck.

As I've said - check wait events first.


21.03.2018 19:23, FredB пишет:

> Sorry, it was just a wrong cut/paste cache_size=50MB the previous result still the same
> About children I tried with 256, unfortunately squid is still stuck at 100%
>
> Regards
>
> Fred
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

FredB
In reply to this post by Yuri Voinov
I agree, to be honest I started with low values updated again and again, I should have post my previous tests rather than the latest :)
 

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SSLBump, system requirements ?

Yuri Voinov
Use OS performance tools. Require to identify bottleneck. Pay attention
on wait events.


21.03.2018 20:05, FredB пишет:
> I agree, to be honest I started with low values updated again and again, I should have post my previous tests rather than the latest :)
>  
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

--
"C++ seems like a language suitable for firing other people's legs."

*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment