SMP mode, workers, cache_dir with conditional

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SMP mode, workers, cache_dir with conditional

juju
Hello,
Sorry if this question has already been posted ( I can't check as the "Search Function" is currently not working )
I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
My squid is currently working but only on 1 core.
I which to use the 2 cores. I understood that I have to use workers in squid.conf with separate cache directory. 
So I modified default  squid.conf  with :
---
[root@JujuServerTest2 ~]# egrep -v "^#|^$" /etc/squid/squid.conf
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
http_access deny all
http_port 3128
workers 2
if ${process_number} = 1
cache_dir rock /var/spool/squid1 1024
endif
if ${process_number} = 2
cache_dir rock /var/spool/squid2 1024
endif
coredump_dir /var/spool/squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320
[root@JujuServerTest2 ~]#

---
but
/var/log/squid/squid.out :
[...]
2018/10/10 13:52:15 kid2| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid2| Creating missing swap directories
2018/10/10 13:52:15 kid1| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid1| Creating missing swap directories
2018/10/10 13:52:15 kid3| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid3| Creating missing swap directories
2018/10/10 13:52:15 kid3| No cache_dir stores are configured.
2018/10/10 13:52:15 kid3| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid2| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid1| Set Current Directory to /var/spool/squid
2018/10/10 13:52:15 kid2| Creating missing swap directories
2018/10/10 13:52:15 kid1| Creating missing swap directories
2018/10/10 13:52:15 kid3| Creating missing swap directories
2018/10/10 13:52:15 kid3| No cache_dir stores are configured    <------
---
ps faux |grep squid
root      4172  0.0  0.1 110512  2012 pts/1    S+   14:00   0:00                          \_ grep --color=auto squid
root      4133  0.0  0.6 346844  6848 ?        Ss   13:52   0:00 squid -f /etc/squid/squid.conf
squid     4136  0.0  2.1 359828 21292 ?        S    13:52   0:00  \_ (squid-coord-3) -f /etc/squid/squid.conf
squid     4140  0.0  0.1  28004  1892 ?        S    13:52   0:00      \_ (logfile-daemon) /var/log/squid/access.log



Looks like, my configuration is not good because "No cache_dir stores are configured"
Coud you tell me where is my mistake ? 

Thank you very much.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP mode, workers, cache_dir with conditional

Amos Jeffries
Administrator
On 11/10/18 3:01 AM, juju wrote:
> Hello,
> Sorry if this question has already been posted ( I can't check as the
> "Search Function" is currently not working )
> I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
> My squid is currently working but only on 1 core.
> I which to use the 2 cores. I understood that I have to use workers in
> squid.conf with separate cache directory.

This is not quite correct.

Per-worker configuration is only required for features which are not yet
SMP-aware.

The rock cache type *is* SMP-aware, so one directory can be shared by
workers.

The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
separate directories per-worker currently.



Since you are using only rock type caches you do not have to do anything
special for SMP to work.


> So I modified default  squid.conf  with :
> ---
...
> workers 2
> if ${process_number} = 1
> cache_dir rock /var/spool/squid1 1024
> endif
> if ${process_number} = 2
> cache_dir rock /var/spool/squid2 1024
> endif
...

>
> ---
> but
> /var/log/squid/squid.out :
> [...]
> 2018/10/10 13:52:15 kid3| Set Current Directory to /var/spool/squid
> 2018/10/10 13:52:15 kid2| Set Current Directory to /var/spool/squid
> 2018/10/10 13:52:15 kid1| Set Current Directory to /var/spool/squid
> 2018/10/10 13:52:15 kid2| Creating missing swap directories
> 2018/10/10 13:52:15 kid1| Creating missing swap directories
> 2018/10/10 13:52:15 kid3| Creating missing swap directories
> 2018/10/10 13:52:15 kid3| No cache_dir stores are configured    <------

Notice the kid (process) numbers in those log lines.

You only defined cache directories for process #1 and #2. This line is
being reported by process #3 - which is the process coordinator process.
So it is correct for your config.

Also, this is informative message - not an error.


Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP mode, workers, cache_dir with conditional

Alex Rousskov
On 10/10/2018 09:10 AM, Amos Jeffries wrote:
> On 11/10/18 3:01 AM, juju wrote:
>> Hello,
>> Sorry if this question has already been posted ( I can't check as the
>> "Search Function" is currently not working )
>> I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
>> My squid is currently working but only on 1 core.
>> I which to use the 2 cores. I understood that I have to use workers in
>> squid.conf with separate cache directory.

> This is not quite correct.

> Per-worker configuration is only required for features which are not yet
> SMP-aware.


Just a couple of additional clarifications, not contradicting what Amos
has said:


> The rock cache type *is* SMP-aware, so one directory can be shared by
> workers.

"can be" and "must be". We do not support worker-specific cache_dirs.

In case of rock cache_dirs, I do not know for sure what would happen if
cache_dirs are configured on a per-worker basis, but I suspect that
there will be no disker processes to serve them. At best, you will get
no disk caching at all.


> The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
> separate directories per-worker currently.

... but worker-specific cache_dirs are not supported, will violate HTTP
rules, etc.


> Since you are using only rock type caches you do not have to do anything
> special for SMP to work.

... with your simple configuration that does not use SMP-unaware features.

In fact, with rock cache_dirs, you do not even have to specify multiple
workers to get SMP Squid -- by default, Squid will run in SMP mode if
you have one worker (default) and at least one rock cache_dir. However,
multiple workers may be necessary to better utilize multicore CPUs.

Beware of hyperthreading: Do not expect two workers, each showing 80+%
CPU utilization to be able to share a single physical CPU core
efficiently. I do not know whether your two cores are hyperthreaded or real.

N.B. To search the mailing list archives, consider using your favorite
search engine. Many would accept a site:lists.squid-cache.org restriction.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP mode, workers, cache_dir with conditional

juju
Hello !

Thank you very much Alex and Amos
I have 2 real cores so it will be good.

on http://www.squid-cache.org/Doc/config/cache_dir/  (available in: 4   3.5   3.4   3.3   3.2   2.7   3.1   3.0   2.6)   :
"In SMP configurations, cache_dir must not precede the workers option and should use configuration macros or conditionals to give each worker interested in disk caching a dedicated cache directory"

Perhaps it would be good to specify for which version and which cache_dir type this note is for.

Best regards 

Le mercredi 10 octobre 2018 à 19:38:55 UTC+2, Alex Rousskov <[hidden email]> a écrit :


On 10/10/2018 09:10 AM, Amos Jeffries wrote:
> On 11/10/18 3:01 AM, juju wrote:
>> Hello,
>> Sorry if this question has already been posted ( I can't check as the
>> "Search Function" is currently not working )
>> I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
>> My squid is currently working but only on 1 core.
>> I which to use the 2 cores. I understood that I have to use workers in
>> squid.conf with separate cache directory.

> This is not quite correct.

> Per-worker configuration is only required for features which are not yet
> SMP-aware.


Just a couple of additional clarifications, not contradicting what Amos
has said:


> The rock cache type *is* SMP-aware, so one directory can be shared by
> workers.

"can be" and "must be". We do not support worker-specific cache_dirs.

In case of rock cache_dirs, I do not know for sure what would happen if
cache_dirs are configured on a per-worker basis, but I suspect that
there will be no disker processes to serve them. At best, you will get
no disk caching at all.


> The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
> separate directories per-worker currently.

... but worker-specific cache_dirs are not supported, will violate HTTP
rules, etc.


> Since you are using only rock type caches you do not have to do anything
> special for SMP to work.

... with your simple configuration that does not use SMP-unaware features.

In fact, with rock cache_dirs, you do not even have to specify multiple
workers to get SMP Squid -- by default, Squid will run in SMP mode if
you have one worker (default) and at least one rock cache_dir. However,
multiple workers may be necessary to better utilize multicore CPUs.

Beware of hyperthreading: Do not expect two workers, each showing 80+%
CPU utilization to be able to share a single physical CPU core
efficiently. I do not know whether your two cores are hyperthreaded or real.

N.B. To search the mailing list archives, consider using your favorite
search engine. Many would accept a site:lists.squid-cache.org restriction.

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP mode, workers, cache_dir with conditional

Alex Rousskov
On 10/11/2018 02:15 AM, juju wrote:

> on http://www.squid-cache.org/Doc/config/cache_dir/:
> "In SMP configurations, cache_dir must not precede the workers option
> and should use configuration macros or conditionals to give each worker
> interested in disk caching a dedicated cache directory"

> Perhaps it would be good to specify for which version and which
> cache_dir type this note is for.

Yes, that stale note should be rephrased to discourage unsupported
configurations or even replaced with Squid warnings about unsupported
use of ufs-based cache_dirs in SMP configurations.

I am not sure, but I suspect the "must not precede" part is also stale
or, at the very least, should be unnecessary -- if the right order is
still required, Squid should detect wrong order and quit with an error
message.


Quality patches welcomed.

Alex.



> Le mercredi 10 octobre 2018 à 19:38:55 UTC+2, Alex Rousskov a écrit :
>
>
> On 10/10/2018 09:10 AM, Amos Jeffries wrote:
>> On 11/10/18 3:01 AM, juju wrote:
>>> Hello,
>>> Sorry if this question has already been posted ( I can't check as the
>>> "Search Function" is currently not working )
>>> I'm running squid-3.5.20 on Amazon Linux 2018.03. 2 cores with 8 GB
>>> My squid is currently working but only on 1 core.
>>> I which to use the 2 cores. I understood that I have to use workers in
>>> squid.conf with separate cache directory.
>
>> This is not quite correct.
>
>> Per-worker configuration is only required for features which are not yet
>> SMP-aware.
>
>
> Just a couple of additional clarifications, not contradicting what Amos
> has said:
>
>
>> The rock cache type *is* SMP-aware, so one directory can be shared by
>> workers.
>
> "can be" and "must be". We do not support worker-specific cache_dirs.
>
> In case of rock cache_dirs, I do not know for sure what would happen if
> cache_dirs are configured on a per-worker basis, but I suspect that
> there will be no disker processes to serve them. At best, you will get
> no disk caching at all.
>
>
>> The UFS/AUFS/diskd cache types are not SMP-aware yet, so they do need
>> separate directories per-worker currently.
>
> ... but worker-specific cache_dirs are not supported, will violate HTTP
> rules, etc.
>
>
>> Since you are using only rock type caches you do not have to do anything
>> special for SMP to work.
>
> ... with your simple configuration that does not use SMP-unaware features.
>
> In fact, with rock cache_dirs, you do not even have to specify multiple
> workers to get SMP Squid -- by default, Squid will run in SMP mode if
> you have one worker (default) and at least one rock cache_dir. However,
> multiple workers may be necessary to better utilize multicore CPUs.
>
> Beware of hyperthreading: Do not expect two workers, each showing 80+%
> CPU utilization to be able to share a single physical CPU core
> efficiently. I do not know whether your two cores are hyperthreaded or real.
>
> N.B. To search the mailing list archives, consider using your favorite
> search engine. Many would accept a site:lists.squid-cache.org restriction.
>
> Alex.
>
> _______________________________________________
> squid-users mailing list
> [hidden email] <mailto:[hidden email]>
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users