Re: Squid memory consumption problem

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Alex Rousskov
On 6/5/20 2:38 PM, DIXIT Ankit wrote:

> We are facing memory issues on Squid proxy. We have squid proxy server
> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM
> and 100 GB hard disk.
>
> The problem is that squid is eating all of system memory and not freeing
> up the objects. Server memory lasts for maximum 15 days and after that
> squid process start crashing. Please suggest.

I would start by making sure that you are not suffering from Squid bug
#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

After that, if you are sure that there is a memory leak, then I suggest
upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that
somebody would volunteer to triage a v3.5 memory leak these days.


HTH,

Alex.


> [root@eaa-lpx-003-p ~]# squid -version
>
> Squid Cache: Version 3.5.27-20180318-r1330042
>
> Service Name: squid
>
> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

Hey,

 

Squid 4 is tested on Amazon Linux 2.

I have tested it in the lat year.. and I believe that there is no reason to run a full set of tests now.

 

Squid 4 only needs a basic “dry-run” to make sure what you need to change in your squid.conf.

 

I do suggest to first run it whiteout any cache-dir for maybe an hour.

 

Do not go back to CentOS 7.

 

To my knowledge Amazon Linux 2 receives a better over-all support then CentOS 7.

 

From my tests in the past Amazon Linux 2 is faster and is a LTS distribution so you will have someone to contact in any case of trouble.

Compared to CentOS X which you will be in the grace of your own efforts and “google foo”.

 

I believe that you would not want to find your self in a situation which you try to contact upstream RH for help.

 

Feel free to send me or the mailing lists any email you want.

I will try to at-least read them if I cannot respond on the spot.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 2:56 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

Thanks for providing the rpm information but I was having some questions as per my last email.

==============================================================================

Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

 

How much time, testing will take?

 

If it will take time , then I am thinking to change the Operating system to Centos 7 from Amazon Linux 2 and then install Squid 4?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 12:26 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Alex Rousskov
In reply to this post by Eliezer Croitoru-3
On 6/10/20 7:56 AM, DIXIT Ankit wrote:

> Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

Officially, no Squid version is tested on Amazon Linux. With enough
Squid Project donations, that may change in the foreseeable future.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3
In reply to this post by Eliezer Croitoru-3

Hey,

 

What you should do is:

yum localinstall squid-helpers-4.12-1.amzn2.x86_64.rpm perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

and it should resolve the dependencies automatically.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Thursday, June 11, 2020 7:49 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

When I am installing these packages, its not able to resolve dependencies. Below is the error.

 

[root@ squid_package]# rpm -ivh squid-helpers-4.12-1.amzn2.x86_64.rpm

error: Failed dependencies:

        perl(DBI) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Data::Dumper) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::MD5) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::SHA) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(URI::URL) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

 

Before above error, I had to resolve one more dependency by installing below lip package.

 

yum install libtool-ltdl-2.4.2-22.2.amzn2.0.2.x86_64

 

Please suggest.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 1:14 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

Squid 4 is tested on Amazon Linux 2.

I have tested it in the lat year.. and I believe that there is no reason to run a full set of tests now.

 

Squid 4 only needs a basic “dry-run” to make sure what you need to change in your squid.conf.

 

I do suggest to first run it whiteout any cache-dir for maybe an hour.

 

Do not go back to CentOS 7.

 

To my knowledge Amazon Linux 2 receives a better over-all support then CentOS 7.

 

From my tests in the past Amazon Linux 2 is faster and is a LTS distribution so you will have someone to contact in any case of trouble.

Compared to CentOS X which you will be in the grace of your own efforts and “google foo”.

 

I believe that you would not want to find your self in a situation which you try to contact upstream RH for help.

 

Feel free to send me or the mailing lists any email you want.

I will try to at-least read them if I cannot respond on the spot.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 2:56 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

Thanks for providing the rpm information but I was having some questions as per my last email.

==============================================================================

Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

 

How much time, testing will take?

 

If it will take time , then I am thinking to change the Operating system to Centos 7 from Amazon Linux 2 and then install Squid 4?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 12:26 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

Since you are using ssl-bump, you would need to run it manually from CLI as squid user and see what happens.

 

You will need to reinitialize the certificate directory and test at again.

 

Take a peek at:

https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit#Create_and_initialize_TLS_certificates_cache_directory

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 16, 2020 11:23 AM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

Dependencies are resolved and squid installed successfully but during squid process start, we are getting below error. Screen shot also attached. Please suggest.

 

FATAL: The /usr/lib64/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 4MB helpers are crashi...eed help!

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Thursday, June 11, 2020 7:18 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

What you should do is:

yum localinstall squid-helpers-4.12-1.amzn2.x86_64.rpm perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

and it should resolve the dependencies automatically.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Thursday, June 11, 2020 7:49 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

When I am installing these packages, its not able to resolve dependencies. Below is the error.

 

[root@ squid_package]# rpm -ivh squid-helpers-4.12-1.amzn2.x86_64.rpm

error: Failed dependencies:

        perl(DBI) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Data::Dumper) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::MD5) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::SHA) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(URI::URL) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

 

Before above error, I had to resolve one more dependency by installing below lip package.

 

yum install libtool-ltdl-2.4.2-22.2.amzn2.0.2.x86_64

 

Please suggest.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 1:14 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

Squid 4 is tested on Amazon Linux 2.

I have tested it in the lat year.. and I believe that there is no reason to run a full set of tests now.

 

Squid 4 only needs a basic “dry-run” to make sure what you need to change in your squid.conf.

 

I do suggest to first run it whiteout any cache-dir for maybe an hour.

 

Do not go back to CentOS 7.

 

To my knowledge Amazon Linux 2 receives a better over-all support then CentOS 7.

 

From my tests in the past Amazon Linux 2 is faster and is a LTS distribution so you will have someone to contact in any case of trouble.

Compared to CentOS X which you will be in the grace of your own efforts and “google foo”.

 

I believe that you would not want to find your self in a situation which you try to contact upstream RH for help.

 

Feel free to send me or the mailing lists any email you want.

I will try to at-least read them if I cannot respond on the spot.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 2:56 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

Thanks for providing the rpm information but I was having some questions as per my last email.

==============================================================================

Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

 

How much time, testing will take?

 

If it will take time , then I am thinking to change the Operating system to Centos 7 from Amazon Linux 2 and then install Squid 4?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 12:26 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3
In reply to this post by Eliezer Croitoru-3

If for some reason you are having trouble initializing ssl-bump use the next script:

https://gist.github.com/elico/206857f628088c2359c10d58278b7bcd

 

And cleanup before the relevant directories:

/var/lib/ssl_db

/etc/squid/

/etc/squid/ssl_cert/

 

 

I have tested this setup script a lot of times(k’s) and it works with all of my packages.

 

 

All The Bests,

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 16, 2020 11:23 AM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

Dependencies are resolved and squid installed successfully but during squid process start, we are getting below error. Screen shot also attached. Please suggest.

 

FATAL: The /usr/lib64/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 4MB helpers are crashi...eed help!

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Thursday, June 11, 2020 7:18 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

What you should do is:

yum localinstall squid-helpers-4.12-1.amzn2.x86_64.rpm perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

and it should resolve the dependencies automatically.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Thursday, June 11, 2020 7:49 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

When I am installing these packages, its not able to resolve dependencies. Below is the error.

 

[root@ squid_package]# rpm -ivh squid-helpers-4.12-1.amzn2.x86_64.rpm

error: Failed dependencies:

        perl(DBI) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Data::Dumper) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::MD5) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::SHA) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(URI::URL) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

 

Before above error, I had to resolve one more dependency by installing below lip package.

 

yum install libtool-ltdl-2.4.2-22.2.amzn2.0.2.x86_64

 

Please suggest.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 1:14 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

Squid 4 is tested on Amazon Linux 2.

I have tested it in the lat year.. and I believe that there is no reason to run a full set of tests now.

 

Squid 4 only needs a basic “dry-run” to make sure what you need to change in your squid.conf.

 

I do suggest to first run it whiteout any cache-dir for maybe an hour.

 

Do not go back to CentOS 7.

 

To my knowledge Amazon Linux 2 receives a better over-all support then CentOS 7.

 

From my tests in the past Amazon Linux 2 is faster and is a LTS distribution so you will have someone to contact in any case of trouble.

Compared to CentOS X which you will be in the grace of your own efforts and “google foo”.

 

I believe that you would not want to find your self in a situation which you try to contact upstream RH for help.

 

Feel free to send me or the mailing lists any email you want.

I will try to at-least read them if I cannot respond on the spot.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 2:56 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

Thanks for providing the rpm information but I was having some questions as per my last email.

==============================================================================

Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

 

How much time, testing will take?

 

If it will take time , then I am thinking to change the Operating system to Centos 7 from Amazon Linux 2 and then install Squid 4?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 12:26 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

Hey,

(Is your private name Dixit?)

 


Since you have a ready to use setup you should only look at the next lines from the script:
### START OF SCRIPT
SSLCRTD="/usr/lib64/squid/security_file_certgen"

SSLCRTDDB="/var/spool/squid/ssl_db"


echo “cleaning up the current ssl_crtd_db directory”

rm -rf "${SSLCRTDDB}"


echo "initializing ssl_crtd_db"

${SSLCRTD} -c -s "${SSLCRTDDB}" -M 16MB

 

echo "changing ownership for ssl_db"

chown -R nobody "${SSLCRTDDB}"
### END OF SCRIPT

I do believe that the above lines should help you to resolve the issues you are having now.
If for any reason you are still having any technical issues contact me directly on my mobile at my working hours
so we would be able to resolve this issue on-line without so much delay between email to email.

My usual working hours are usually from 10:00 AM till 05:00 PM IST.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Friday, June 19, 2020 7:26 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Eliezer,

 

If I summarize last 2-3 emails, below is my understanding.

 

  1. Install the rpms provided by you.

 

  1. If we face below error,

FATAL: The /usr/lib64/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 4MB helpers are crashi...eed help!

 

Then, we will need to reinitialize the certificate directory as per below link

 

Take a peek at:

https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit#Create_and_initialize_TLS_certificates_cache_directory

 

(We tried above link to initialize ssl cache but I don’t see anything inside /var/spool/squid/ssl_db , its blank)

 

  1. So if ssl_db is blank, then we need to run below script:

https://gist.github.com/elico/206857f628088c2359c10d58278b7bcd

 

And as you mentioned cleanup below directories before running above script. I am concerned about cleaning /etc/squid/ because main installation is present inside this path and we may lose it.

 

/var/lib/ssl_db

/etc/squid/

/etc/squid/ssl_cert/

 

Please go through above points and let me know if my understanding is correct.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Thursday, June 18, 2020 1:17 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

If for some reason you are having trouble initializing ssl-bump use the next script:

https://gist.github.com/elico/206857f628088c2359c10d58278b7bcd

 

And cleanup before the relevant directories:

/var/lib/ssl_db

/etc/squid/

/etc/squid/ssl_cert/

 

 

I have tested this setup script a lot of times(k’s) and it works with all of my packages.

 

 

All The Bests,

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 16, 2020 11:23 AM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

Dependencies are resolved and squid installed successfully but during squid process start, we are getting below error. Screen shot also attached. Please suggest.

 

FATAL: The /usr/lib64/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 4MB helpers are crashi...eed help!

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Thursday, June 11, 2020 7:18 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

What you should do is:

yum localinstall squid-helpers-4.12-1.amzn2.x86_64.rpm perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

and it should resolve the dependencies automatically.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Thursday, June 11, 2020 7:49 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

When I am installing these packages, its not able to resolve dependencies. Below is the error.

 

[root@ squid_package]# rpm -ivh squid-helpers-4.12-1.amzn2.x86_64.rpm

error: Failed dependencies:

        perl(DBI) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Data::Dumper) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::MD5) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::SHA) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(URI::URL) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

 

Before above error, I had to resolve one more dependency by installing below lip package.

 

yum install libtool-ltdl-2.4.2-22.2.amzn2.0.2.x86_64

 

Please suggest.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 1:14 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

Squid 4 is tested on Amazon Linux 2.

I have tested it in the lat year.. and I believe that there is no reason to run a full set of tests now.

 

Squid 4 only needs a basic “dry-run” to make sure what you need to change in your squid.conf.

 

I do suggest to first run it whiteout any cache-dir for maybe an hour.

 

Do not go back to CentOS 7.

 

To my knowledge Amazon Linux 2 receives a better over-all support then CentOS 7.

 

From my tests in the past Amazon Linux 2 is faster and is a LTS distribution so you will have someone to contact in any case of trouble.

Compared to CentOS X which you will be in the grace of your own efforts and “google foo”.

 

I believe that you would not want to find your self in a situation which you try to contact upstream RH for help.

 

Feel free to send me or the mailing lists any email you want.

I will try to at-least read them if I cannot respond on the spot.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 2:56 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

Thanks for providing the rpm information but I was having some questions as per my last email.

==============================================================================

Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

 

How much time, testing will take?

 

If it will take time , then I am thinking to change the Operating system to Centos 7 from Amazon Linux 2 and then install Squid 4?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 12:26 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

Hey,

 

I am unsure on how to answer your questions.

Squid-Cache is a proxy server “source code”.

It comes bundled with sources for a set of utilities (helpers) that extends Squid ACLs capabilities by a very simple STDIN\STDOUT api.

 

I know that SquidGuard is the most “famous” external ACL helper for allying big blacklists.

However there are tools and technologies to enforce these.

 

I have also seen ufdbguard which is the successor of SquidGuard in this area.

 

Depends on you environment these might or might not be enough.

Both SquidGuard and ufdbguard are based on a “compiled” DB.

When you update or compile the DB you have what so called “down time”.

For some it’s acceptable while for others it’s not.

 

Squid by itself allows and gives you what ever you need to apply couple black or white lists, both customized and others.

 

Depends on your workload and your ideals or vision you might decide to apply different list.

 

I don’t know if this:

https://github.com/andihofmeister/squidGuard

 

is the most up-to-date version of SquidGuard.

 

These days the most and up-to-date blacklists are via DNS services such as:

 

With the help of a dns cahing proxy service you can define 1 to 3 layers of protection on you clients.

1 - static lists(black or white) located on the server(SquidGuard or other)

2 –helper which checks against external DNS services

3 – Others

 

All these layers can help you to get the best shield you might need.

Squid have the ability to cache individual result for a domain/url/client etc .

 

My suggestion is for you to start with something that makes sense for your business and move forward.

 

If you need help to setup something let me know what do you prefer.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Thursday, June 25, 2020 5:08 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Eliezer,

 

I have some queries for Blacklist DB on Squid.

 

  1. Does Squid 4.12 version provides its own list of default Blacklisted domain/URLs?
  2. What if we apply custom blacklist on top of existing default blacklist, so that we can update it regularly?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: DIXIT Ankit
Sent: Thursday, June 25, 2020 2:49 PM
To: Eliezer Croitoru <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

Eliezer,

 

I have some queries for Blacklist DB on Squid.

 

  1. Does Squid 4.12 version provides its own list of default Blacklisted domain/URLs?
  2. What if we apply custom blacklist on top of existing default blacklist, so that we can update it regularly?

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Saturday, June 20, 2020 8:46 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

(Is your private name Dixit?)

 

  • Install the RPM’s to use a newer version of squid
  • This specific issue is happening mostly because of:
    - permissions
    - runtime errors
    - Other things
  • To overcome the above specific issue you need to cleanup the /var/spool/squid/ssl_db directory and re-create
    it and change ownership of this directory  to make sure that it was not damaged in any way.
  • The wiki link: https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit#Create_and_initialize_TLS_certificates_cache_directory
    (Which for some reason sometimes shows 504 errors)
    Helps to recreate the directory manually and I hope that the proxy installer understands this specific article.
  • It is expected that this /var/.. directory to be empty but after initialization it should be populated with at least one file.
  • The link to the script at:

    demonstrates a full ssl-bump initialization on a an empty squid-cache setup based on my RPMs.


Since you have a ready to use setup you should only look at the next lines from the script:
### START OF SCRIPT
SSLCRTD="/usr/lib64/squid/security_file_certgen"

SSLCRTDDB="/var/spool/squid/ssl_db"


echo “cleaning up the current ssl_crtd_db directory”

rm -rf "${SSLCRTDDB}"


echo "initializing ssl_crtd_db"

${SSLCRTD} -c -s "${SSLCRTDDB}" -M 16MB

 

echo "changing ownership for ssl_db"

chown -R nobody "${SSLCRTDDB}"
### END OF SCRIPT


I do believe that the above lines should help you to resolve the issues you are having now.
If for any reason you are still having any technical issues contact me directly on my mobile at my working hours
so we would be able to resolve this issue on-line without so much delay between email to email.

My usual working hours are usually from 10:00 AM till 05:00 PM IST.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Friday, June 19, 2020 7:26 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Eliezer,

 

If I summarize last 2-3 emails, below is my understanding.

 

  1. Install the rpms provided by you.

 

  1. If we face below error,

FATAL: The /usr/lib64/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 4MB helpers are crashi...eed help!

 

Then, we will need to reinitialize the certificate directory as per below link

 

Take a peek at:

https://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit#Create_and_initialize_TLS_certificates_cache_directory

 

(We tried above link to initialize ssl cache but I don’t see anything inside /var/spool/squid/ssl_db , its blank)

 

  1. So if ssl_db is blank, then we need to run below script:

https://gist.github.com/elico/206857f628088c2359c10d58278b7bcd

 

And as you mentioned cleanup below directories before running above script. I am concerned about cleaning /etc/squid/ because main installation is present inside this path and we may lose it.

 

/var/lib/ssl_db

/etc/squid/

/etc/squid/ssl_cert/

 

Please go through above points and let me know if my understanding is correct.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Thursday, June 18, 2020 1:17 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

If for some reason you are having trouble initializing ssl-bump use the next script:

https://gist.github.com/elico/206857f628088c2359c10d58278b7bcd

 

And cleanup before the relevant directories:

/var/lib/ssl_db

/etc/squid/

/etc/squid/ssl_cert/

 

 

I have tested this setup script a lot of times(k’s) and it works with all of my packages.

 

 

All The Bests,

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 16, 2020 11:23 AM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

Dependencies are resolved and squid installed successfully but during squid process start, we are getting below error. Screen shot also attached. Please suggest.

 

FATAL: The /usr/lib64/squid/security_file_certgen -s /var/spool/squid/ssl_db -M 4MB helpers are crashi...eed help!

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Thursday, June 11, 2020 7:18 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

What you should do is:

yum localinstall squid-helpers-4.12-1.amzn2.x86_64.rpm perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

and it should resolve the dependencies automatically.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Thursday, June 11, 2020 7:49 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Croitoru,

 

When I am installing these packages, its not able to resolve dependencies. Below is the error.

 

[root@ squid_package]# rpm -ivh squid-helpers-4.12-1.amzn2.x86_64.rpm

error: Failed dependencies:

        perl(DBI) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Data::Dumper) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::MD5) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(Digest::SHA) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

        perl(URI::URL) is needed by squid-helpers-7:4.12-1.amzn2.x86_64

 

Before above error, I had to resolve one more dependency by installing below lip package.

 

yum install libtool-ltdl-2.4.2-22.2.amzn2.0.2.x86_64

 

Please suggest.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 1:14 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Cc: UPADHYAY Neeraj <[hidden email]>; SETHI Konica <[hidden email]>; DWIVEDI Gaurav Kumar <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

Hey,

 

Squid 4 is tested on Amazon Linux 2.

I have tested it in the lat year.. and I believe that there is no reason to run a full set of tests now.

 

Squid 4 only needs a basic “dry-run” to make sure what you need to change in your squid.conf.

 

I do suggest to first run it whiteout any cache-dir for maybe an hour.

 

Do not go back to CentOS 7.

 

To my knowledge Amazon Linux 2 receives a better over-all support then CentOS 7.

 

From my tests in the past Amazon Linux 2 is faster and is a LTS distribution so you will have someone to contact in any case of trouble.

Compared to CentOS X which you will be in the grace of your own efforts and “google foo”.

 

I believe that you would not want to find your self in a situation which you try to contact upstream RH for help.

 

Feel free to send me or the mailing lists any email you want.

I will try to at-least read them if I cannot respond on the spot.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 2:56 PM
To: [hidden email]; [hidden email]; [hidden email]
Cc: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

Thanks for providing the rpm information but I was having some questions as per my last email.

==============================================================================

Does it mean, Squid 4 is not tested on Amazon Linux 2, yet?

 

How much time, testing will take?

 

If it will take time , then I am thinking to change the Operating system to Centos 7 from Amazon Linux 2 and then install Squid 4?

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 12:26 PM
To: DIXIT Ankit <[hidden email]>; Alex Rousskov <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

## The last email got corrupted for some reason so..

 

First save the existing config files.

 

The RPM files are at:

https://1drv.ms/u/s!AoiLG1Jyh7JqoiBbOtBMb9wuAfjS?e=9XeXCS

 

You will need to install:

perl-Crypt-OpenSSL-X509-0.1-1.amzn2.noarch.rpm

 

 

It’s a dummy recursive package that installs basic dependencies.

You will not be able to install the squid-helpers packages without it.

 

Then install the RPM’s:

squid-4.12-1.amzn2.x86_64.rpm

squid-helpers-4.12-1.amzn2.x86_64.rpm

 

If you have installed squid RPM’s from other sources/repositories you will need to first uninstall/remove then and only then install the ngtech RPM’s.

 

If for any reason you need the sources for the RPM’s take a peek at:

https://github.com/elico/squid-docker-build-nodes

https://gist.github.com/elico/5fa3050ed1a7247fbf2f6810c74933cc

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Wednesday, June 10, 2020 11:59 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Hi,

 

We have two squid servers running in our environment.

 

  1. Squid Cache: Version 4.2-20180806-r6d8f397 running on CentOS Linux release 7.7.1908 (Core)---- Its running perfectly with no CPU and Memory issue

 

  1. Squid Cache: Version 3.5.27-20180318-r1330042 running on Amazon Linux 2 --- It’s not running perfectly with high CPU and Memory issue

 

I think you are right, please provide the Squid rpm version 4 for Amazon Linux 2 Or link to download.

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Wednesday, June 10, 2020 9:26 AM
To: Alex Rousskov <[hidden email]>; DIXIT Ankit <[hidden email]>; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

 

I have tested v4 to work properly on Amazon Linux 2 in the past.

If RPM’s are needed, let me know.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 9, 2020 11:33 PM
To: [hidden email]; [hidden email]
Subject: Re: [squid-users] Squid memory consumption problem

 

On 6/5/20 2:38 PM, DIXIT Ankit wrote:

 

> We are facing memory issues on Squid proxy. We have squid proxy server

> running on AWS Cloud(Amazon Linux 2). Server is having total 8 GB RAM

> and 100 GB hard disk.

>

> The problem is that squid is eating all of system memory and not freeing

> up the objects. Server memory lasts for maximum 15 days and after that

> squid process start crashing. Please suggest.

 

I would start by making sure that you are not suffering from Squid bug

#4005: https://bugs.squid-cache.org/show_bug.cgi?id=4005

 

After that, if you are sure that there is a memory leak, then I suggest

upgrading to v4 (or v5) as the next step. IMHO, it is unlikely that

somebody would volunteer to triage a v3.5 memory leak these days.

 

 

HTH,

 

Alex.

 

 

> [root@eaa-lpx-003-p ~]# squid -version

>

> Squid Cache: Version 3.5.27-20180318-r1330042

>

> Service Name: squid

>

> This binary uses OpenSSL 1.0.2k-fips  26 Jan 2017.

 

_______________________________________________

squid-users mailing list

[hidden email]

http://lists.squid-cache.org/listinfo/squid-users

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 

 

 

This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001

 


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

The first thing to do is look at:

https://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery

 

It should clear couple doubts for you.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 30, 2020 10:46 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Elizer,

 

We installed Squid 4.12 on production server, amazon Linux 2, successfully but I could see below messages in the logs for SECURITY ALERT: Host header forgery detected. These are getting generated very frequently.

Can we ignore this Or is it advised to suppress these alerts?

 

kid2| SECURITY ALERT: on URL: 5-25-3-app.agent.datadoghq.com:443

2020/06/30 07:41:29 kid1| SECURITY ALERT: Host header forgery detected on local=IP remote=IP FD 97 flags=33 (local IP does not match any domain IP)

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3

Hey Dixit is it your first name?,

 

I want to help you but you are jumping between subjects and issues over and over again.

Try to open a single thread for a subject and follow it.

 

You are missing details on the clients.

Clients can vary from PC to MOBILE etc..

To clear out any doubts about the setup more details are required if you want any public help.

 

I can try to help you specifically with installation level issues.

The topics you are writing about are a more advanced topic rather then basic..

 

Did you had any chance reading the next docs:

·         https://stackoverflow.com/questions/30057104/squid-ssl-bump-3-5-4-error-error-negotiating-ssl-connection-on-fd-10-success

·         http://squid-web-proxy-cache.1019090.n4.nabble.com/Error-negotiating-SSL-connection-on-FD-12-Success-td4671090.html

 

I want to help you but it’s not always so simple.

 

Take into account that some servers/websites use different versions of TLS/SSL and Squid 4 is not compatible with all of them.

 

What version of OpenSSL do you have installed on your OS?

yum list installed

 

might give you the right answer.

 

 

Also, if you have specific clients which are having issues try to get the relevant details on their software and device.

I cannot do this work for you but can try to help you based on what you might send the public list.

 

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

 

 

 

From: DIXIT Ankit [mailto:[hidden email]]
Sent: Tuesday, June 30, 2020 12:25 PM
To: Eliezer Croitoru; Squid Users
Cc: SETHI Konica
Subject: RE: [squid-users] Squid memory consumption problem

 

Eliezer,

 

Clients are facing some SSL related issues after upgrade. I could see below error. Please suggest, its little urgent.

 

quid[6706]: Error negotiating SSL connection on FD 167: error:00000001:lib(0):func(0):reason(1) (1/0)
Jun 30 09:17:38 squid[6706]: Error parsing SSL Server Hello Message on FD 77
Jun 30 09:17:38 squid[6706]: Error negotiating SSL connection on FD 75: error:00000001:lib(0):func(0):reason(1) (1/0)

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Tuesday, June 30, 2020 9:10 AM
To: Squid Users <[hidden email]>; DIXIT Ankit <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

The first thing to do is look at:

https://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery

 

It should clear couple doubts for you.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 30, 2020 10:46 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Elizer,

 

We installed Squid 4.12 on production server, amazon Linux 2, successfully but I could see below messages in the logs for SECURITY ALERT: Host header forgery detected. These are getting generated very frequently.

Can we ignore this Or is it advised to suppress these alerts?

 

kid2| SECURITY ALERT: on URL: 5-25-3-app.agent.datadoghq.com:443

2020/06/30 07:41:29 kid1| SECURITY ALERT: Host header forgery detected on local=IP remote=IP FD 97 flags=33 (local IP does not match any domain IP)

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

 


This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid memory consumption problem

Eliezer Croitoru-3
In reply to this post by Eliezer Croitoru-3

Hey,

 

What happen with this issue?

I am waiting for any input about this issue to understand with what I can try to help.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: DIXIT Ankit [mailto:[hidden email]]
Sent: Tuesday, June 30, 2020 12:35 PM
To: Eliezer Croitoru; Squid Users
Cc: SETHI Konica
Subject: RE: [squid-users] Squid memory consumption problem

 

For your information, we have added below configurations but again same issue.

 

tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

 

tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: DIXIT Ankit
Sent: Tuesday, June 30, 2020 10:25 AM
To: Eliezer Croitoru <[hidden email]>; Squid Users <[hidden email]>
Cc: SETHI Konica <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

Eliezer,

 

Clients are facing some SSL related issues after upgrade. I could see below error. Please suggest, its little urgent.

 

quid[6706]: Error negotiating SSL connection on FD 167: error:00000001:lib(0):func(0):reason(1) (1/0)
Jun 30 09:17:38 squid[6706]: Error parsing SSL Server Hello Message on FD 77
Jun 30 09:17:38 squid[6706]: Error negotiating SSL connection on FD 75: error:00000001:lib(0):func(0):reason(1) (1/0)

 

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

From: Eliezer Croitoru <[hidden email]>
Sent: Tuesday, June 30, 2020 9:10 AM
To: Squid Users <[hidden email]>; DIXIT Ankit <[hidden email]>
Subject: RE: [squid-users] Squid memory consumption problem

 

 

The first thing to do is look at:

https://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery

 

It should clear couple doubts for you.

 

Eliezer

 

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: [hidden email]

 

From: [hidden email]
Sent: Tuesday, June 30, 2020 10:46 AM
To: [hidden email]; [hidden email]; [hidden email]
Subject: RE: [squid-users] Squid memory consumption problem

 

Elizer,

 

We installed Squid 4.12 on production server, amazon Linux 2, successfully but I could see below messages in the logs for SECURITY ALERT: Host header forgery detected. These are getting generated very frequently.

Can we ignore this Or is it advised to suppress these alerts?

 

kid2| SECURITY ALERT: on URL: 5-25-3-app.agent.datadoghq.com:443

2020/06/30 07:41:29 kid1| SECURITY ALERT: Host header forgery detected on local=IP remote=IP FD 97 flags=33 (local IP does not match any domain IP)

 

Regards,

Ankit Dixit|IS Cloud Team

Eurostar International Ltd

Times House | Bravingtons Walk | London N1 9AW

Office: +44 (0)207 84 35550 (Extension– 35530)

 

 


This email (including any attachments) is intended only for the addressee(s), is confidential and may be legally privileged. If you are not the intended recipient, do not use, disclose, copy, or forward this email. Please notify the sender immediately and then delete it. Eurostar International Limited and its affiliates ("EIL") do not accept any liability for action taken in reliance on this email. EIL makes no representation that this email is free of viruses and addressees should check this email for viruses. The comments or statements expressed in this email are not necessarily those of EIL.

Eurostar International Ltd
Times House, Bravingtons Walk, London N1 9AW Registered in England and Wales No. 2462001



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users