SMP mode and StoreID rewriter

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

SMP mode and StoreID rewriter

Ivan Larionov
Hello.

We're currently testing squid in SMP mode. One of our services uses Store ID feature. The interesting thing we see is that store_id_program started for every squid process (except main). Process tree looks like this:

> squid
>  \_ (squid-coord-4)
>  |   \_ (rewriter_3)
>  |   \_ (rewriter_3)
>  \_ (squid-disk-3)
>  |   \_ (rewriter_3)
>  |   \_ (rewriter_3)
>  \_ (squid-2)
>  |   \_ (rewriter_3)
>  |   \_ (rewriter_3)
>  \_ (squid-1)
>      \_ (rewriter_3)
>      \_ (rewriter_3)

From my brief testing it seems like rewrite is working as expected, but I just wanted to make sure it's ok to see store_id_program started for every child or may be it's sort of a bug.

May be it should be started only for "worker" or only for "disk"?

Relevant parts of the config:

> workers 2
> store_id_program /mnt/services/squid-url-rewriter/rewriter_3
> store_id_children 5 startup=2 idle=2 concurrency=10

From the log file:

2018/01/10 16:56:12 kid4| helperOpenServers: Starting 2/5 'rewriter_3' processes
2018/01/10 16:56:12 kid2| helperOpenServers: Starting 2/5 'rewriter_3' processes
2018/01/10 16:56:12 kid3| helperOpenServers: Starting 2/5 'rewriter_3' processes
2018/01/10 16:56:12 kid1| helperOpenServers: Starting 2/5 'rewriter_3' processes

Squid Cache: Version 3.5.27
Service Name: squid
configure options:  '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid' '--includedir=/usr/include' '--datadir=/usr/share' '--sharedstatedir=/usr/com' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll' '--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock' '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' '--with-large-files' '--with-maxfd=16384' '--enable-htcp'

--
With best regards, Ivan Larionov.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP mode and StoreID rewriter

Alex Rousskov
On 01/10/2018 06:31 PM, Ivan Larionov wrote:

> I just wanted to make sure it's ok to see store_id_program started for
> every child or may be it's sort of a bug.

It is "sort of a bug" -- Squid should not start store_id_program (and
many other helpers!) for some kids. Many Squid features are not made
SMP-aware yet, and features that "work fine" in SMP mode are often
ignored by developers and their sponsors. Quality patches welcome!

Ideally, workers should even be able to share helper processes, but that
is a huge change compared to simply not starting unnecessary helpers.


> May be it should be started only for "worker" or only for "disk"?

Yes, only for "worker" IIRC: Diskers do not deal with HTTP, and Store ID
is an HTTP-request-to-Store-ID translator.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP mode and StoreID rewriter

Eliezer Croitoru
Hey,

For most StoreID helpers you won't care about this but the other option is StoreID ICAP server integration but I didn't had enough time to work on it.
If you can run "ps aux" and share the relevant StoreID helpers lines it would help to understand how significant it is.
For most systems couple MB of RES\RSS memory is fine but if it grows over time or for some reason consumes lots of memory you should rethink\re-design the helper.

Eliezer

* I'm not following the users list these days so FW the post to my email directly

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
Sent: Thursday, January 11, 2018 06:31
To: Ivan Larionov <[hidden email]>; [hidden email]
Subject: Re: [squid-users] SMP mode and StoreID rewriter

On 01/10/2018 06:31 PM, Ivan Larionov wrote:

> I just wanted to make sure it's ok to see store_id_program started for
> every child or may be it's sort of a bug.

It is "sort of a bug" -- Squid should not start store_id_program (and
many other helpers!) for some kids. Many Squid features are not made
SMP-aware yet, and features that "work fine" in SMP mode are often
ignored by developers and their sponsors. Quality patches welcome!

Ideally, workers should even be able to share helper processes, but that
is a huge change compared to simply not starting unnecessary helpers.


> May be it should be started only for "worker" or only for "disk"?

Yes, only for "worker" IIRC: Diskers do not deal with HTTP, and Store ID
is an HTTP-request-to-Store-ID translator.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users