SMP and AUFS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

SMP and AUFS

senor
Hi All,
In a thread with subject "squid workers question" Alex Rousskov has
stated that AUFS is not compatible with SMP mode. I need this clarified.

I understand that AUFS is not SMP aware but if each worker has its own
AUFS cache is there any problem other than the inefficiencies of
duplicate cache?

I'm pretty sure that AUFS is used with squid running in SMP mode a lot.
The squid wiki even has a CARP configuration example for this combination.

I apologize if I am misunderstanding the statements.

Senor
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Alex Rousskov
On 03/16/2017 10:42 PM, senor wrote:

> I understand that AUFS is not SMP aware but if each worker has its own
> AUFS cache is there any problem other than the inefficiencies of
> duplicate cache?

Yes. Clients may get stale cached entries, possibly breaking advanced
HTTP transactions that rely on a more-or-less compliant proxy cache.

Also, I do not know exactly how local and shared cache indexes interact
when SMP-unaware store updates its local index without updating the
shared one. Most likely, such partial updates lead to bugs. You may
reduce bugs probability by not mixing shared and ufs-based stores in SMP
mode, but I doubt you can eliminate all problems that way.


> I'm pretty sure that AUFS is used with squid running in SMP mode a lot.

I can think of many examples where a lot of people do things they should
not be doing and do not do things they should be doing. Just because
many use X to solve some problem, does not make using X a good idea and
certainly does not make it the best solution available.


> The squid wiki even has a CARP configuration example for this combination.

I hope there are no official examples advertising SMP AUFS
configurations. If there are, they should be removed IMO.

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

senor
On 3/17/2017 7:45, Alex Rousskov wrote:

> On 03/16/2017 10:42 PM, senor wrote:
>
>> I understand that AUFS is not SMP aware but if each worker has its own
>> AUFS cache is there any problem other than the inefficiencies of
>> duplicate cache?
>
> Yes. Clients may get stale cached entries, possibly breaking advanced
> HTTP transactions that rely on a more-or-less compliant proxy cache.
>
> Also, I do not know exactly how local and shared cache indexes interact
> when SMP-unaware store updates its local index without updating the
> shared one. Most likely, such partial updates lead to bugs. You may
> reduce bugs probability by not mixing shared and ufs-based stores in SMP
> mode, but I doubt you can eliminate all problems that way.
>
>
>> I'm pretty sure that AUFS is used with squid running in SMP mode a lot.
>
> I can think of many examples where a lot of people do things they should
> not be doing and do not do things they should be doing. Just because
> many use X to solve some problem, does not make using X a good idea and
> certainly does not make it the best solution available.
>
>
>> The squid wiki even has a CARP configuration example for this combination.
>
> I hope there are no official examples advertising SMP AUFS
> configurations. If there are, they should be removed IMO.
>
> Alex.
>
There are many references in the squid wiki, FAQ and Knowlegebase about
SMP but I don't see any of them reflecting the concerns you have brought
up. My point in mentioning that there are a lot of installations using
SMP and AUFS is that something widely used but buggy tends to be brought
up on this email list and I haven't seen it.

I'm not trying to claim there are no problems. I'm just making sure my
expectations are realistic. Your comments were the first I became aware
anyone thought poorly about the combination of AUFS with SMP. Rock is of
course preferred but it comes with more baggage than AUFS. My own
experience has been pretty good. Maybe just lucky.

Senor
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Eliezer Croitoru
I think that some warning message like "WARNING: be sure you know that UFS\AUFS doesn't support SMP\MultiWorkers" should be added to the stderr or cache.log.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of senor
Sent: Sunday, March 19, 2017 7:12 AM
To: [hidden email]
Subject: Re: [squid-users] SMP and AUFS

On 3/17/2017 7:45, Alex Rousskov wrote:

> On 03/16/2017 10:42 PM, senor wrote:
>
>> I understand that AUFS is not SMP aware but if each worker has its own
>> AUFS cache is there any problem other than the inefficiencies of
>> duplicate cache?
>
> Yes. Clients may get stale cached entries, possibly breaking advanced
> HTTP transactions that rely on a more-or-less compliant proxy cache.
>
> Also, I do not know exactly how local and shared cache indexes interact
> when SMP-unaware store updates its local index without updating the
> shared one. Most likely, such partial updates lead to bugs. You may
> reduce bugs probability by not mixing shared and ufs-based stores in SMP
> mode, but I doubt you can eliminate all problems that way.
>
>
>> I'm pretty sure that AUFS is used with squid running in SMP mode a lot.
>
> I can think of many examples where a lot of people do things they should
> not be doing and do not do things they should be doing. Just because
> many use X to solve some problem, does not make using X a good idea and
> certainly does not make it the best solution available.
>
>
>> The squid wiki even has a CARP configuration example for this combination.
>
> I hope there are no official examples advertising SMP AUFS
> configurations. If there are, they should be removed IMO.
>
> Alex.
>
There are many references in the squid wiki, FAQ and Knowlegebase about
SMP but I don't see any of them reflecting the concerns you have brought
up. My point in mentioning that there are a lot of installations using
SMP and AUFS is that something widely used but buggy tends to be brought
up on this email list and I haven't seen it.

I'm not trying to claim there are no problems. I'm just making sure my
expectations are realistic. Your comments were the first I became aware
anyone thought poorly about the combination of AUFS with SMP. Rock is of
course preferred but it comes with more baggage than AUFS. My own
experience has been pretty good. Maybe just lucky.

Senor
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Marcus Kool
The root cause of why admins configure SMP + [A]UFS is the lack of good documentation.
A few lines in the wiki and squid.conf.documented should be enough.

Marcus


On 19/03/17 06:11, Eliezer  Croitoru wrote:

> I think that some warning message like "WARNING: be sure you know that UFS\AUFS doesn't support SMP\MultiWorkers" should be added to the stderr or cache.log.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of senor
> Sent: Sunday, March 19, 2017 7:12 AM
> To: [hidden email]
> Subject: Re: [squid-users] SMP and AUFS
>
> On 3/17/2017 7:45, Alex Rousskov wrote:
>> On 03/16/2017 10:42 PM, senor wrote:
>>
>>> I understand that AUFS is not SMP aware but if each worker has its own
>>> AUFS cache is there any problem other than the inefficiencies of
>>> duplicate cache?
>>
>> Yes. Clients may get stale cached entries, possibly breaking advanced
>> HTTP transactions that rely on a more-or-less compliant proxy cache.
>>
>> Also, I do not know exactly how local and shared cache indexes interact
>> when SMP-unaware store updates its local index without updating the
>> shared one. Most likely, such partial updates lead to bugs. You may
>> reduce bugs probability by not mixing shared and ufs-based stores in SMP
>> mode, but I doubt you can eliminate all problems that way.
>>
>>
>>> I'm pretty sure that AUFS is used with squid running in SMP mode a lot.
>>
>> I can think of many examples where a lot of people do things they should
>> not be doing and do not do things they should be doing. Just because
>> many use X to solve some problem, does not make using X a good idea and
>> certainly does not make it the best solution available.
>>
>>
>>> The squid wiki even has a CARP configuration example for this combination.
>>
>> I hope there are no official examples advertising SMP AUFS
>> configurations. If there are, they should be removed IMO.
>>
>> Alex.
>>
> There are many references in the squid wiki, FAQ and Knowlegebase about
> SMP but I don't see any of them reflecting the concerns you have brought
> up. My point in mentioning that there are a lot of installations using
> SMP and AUFS is that something widely used but buggy tends to be brought
> up on this email list and I haven't seen it.
>
> I'm not trying to claim there are no problems. I'm just making sure my
> expectations are realistic. Your comments were the first I became aware
> anyone thought poorly about the combination of AUFS with SMP. Rock is of
> course preferred but it comes with more baggage than AUFS. My own
> experience has been pretty good. Maybe just lucky.
>
> Senor
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Eliezer Croitoru
I believe that some admins do not have enough time to sit and find the documentation.
I have seen it more than once that an admin just don't want to read the docs since it's like the "ocean" or something similar.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Marcus Kool
Sent: Sunday, March 19, 2017 2:36 PM
To: [hidden email]
Subject: Re: [squid-users] SMP and AUFS

The root cause of why admins configure SMP + [A]UFS is the lack of good documentation.
A few lines in the wiki and squid.conf.documented should be enough.

Marcus


On 19/03/17 06:11, Eliezer  Croitoru wrote:

> I think that some warning message like "WARNING: be sure you know that UFS\AUFS doesn't support SMP\MultiWorkers" should be added to the stderr or cache.log.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: [hidden email]
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]]
> On Behalf Of senor
> Sent: Sunday, March 19, 2017 7:12 AM
> To: [hidden email]
> Subject: Re: [squid-users] SMP and AUFS
>
> On 3/17/2017 7:45, Alex Rousskov wrote:
>> On 03/16/2017 10:42 PM, senor wrote:
>>
>>> I understand that AUFS is not SMP aware but if each worker has its
>>> own AUFS cache is there any problem other than the inefficiencies of
>>> duplicate cache?
>>
>> Yes. Clients may get stale cached entries, possibly breaking advanced
>> HTTP transactions that rely on a more-or-less compliant proxy cache.
>>
>> Also, I do not know exactly how local and shared cache indexes
>> interact when SMP-unaware store updates its local index without
>> updating the shared one. Most likely, such partial updates lead to
>> bugs. You may reduce bugs probability by not mixing shared and
>> ufs-based stores in SMP mode, but I doubt you can eliminate all problems that way.
>>
>>
>>> I'm pretty sure that AUFS is used with squid running in SMP mode a lot.
>>
>> I can think of many examples where a lot of people do things they
>> should not be doing and do not do things they should be doing. Just
>> because many use X to solve some problem, does not make using X a
>> good idea and certainly does not make it the best solution available.
>>
>>
>>> The squid wiki even has a CARP configuration example for this combination.
>>
>> I hope there are no official examples advertising SMP AUFS
>> configurations. If there are, they should be removed IMO.
>>
>> Alex.
>>
> There are many references in the squid wiki, FAQ and Knowlegebase
> about SMP but I don't see any of them reflecting the concerns you have
> brought up. My point in mentioning that there are a lot of
> installations using SMP and AUFS is that something widely used but
> buggy tends to be brought up on this email list and I haven't seen it.
>
> I'm not trying to claim there are no problems. I'm just making sure my
> expectations are realistic. Your comments were the first I became
> aware anyone thought poorly about the combination of AUFS with SMP.
> Rock is of course preferred but it comes with more baggage than AUFS.
> My own experience has been pretty good. Maybe just lucky.
>
> Senor
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Alex Rousskov
In reply to this post by senor
On 03/18/2017 11:11 PM, senor wrote:

> There are many references in the squid wiki, FAQ and Knowlegebase about
> SMP but I don't see any of them reflecting the concerns you have brought
> up.

There is a paragraph about these problems at [1] (search for "ufs") but
I agree that better documentation, including wiki and
squid.conf.documented changes/additions would be nice.

  [1] http://wiki.squid-cache.org/Features/SmpScale


> My point in mentioning that there are a lot of installations using
> SMP and AUFS is that something widely used but buggy tends to be brought
> up on this email list and I haven't seen it.

IIRC, it has been brought up several times on the mailing lists and in
Bugzilla. Once you dedicate each ufs-based store to each individual
worker, most of the problems become subtle, often "invisible" to an
admin because they "break" transactions, not Squid, especially if you do
not use a mixture of ufs-based and rock stores. Using mailing list as an
indicator that as subtle problem does _not_ exist is a risky strategy IMO.

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Matus UHLAR - fantomas
On 19.03.17 11:08, Alex Rousskov wrote:

>On 03/18/2017 11:11 PM, senor wrote:
>
>> There are many references in the squid wiki, FAQ and Knowlegebase about
>> SMP but I don't see any of them reflecting the concerns you have brought
>> up.
>
>There is a paragraph about these problems at [1] (search for "ufs") but
>I agree that better documentation, including wiki and
>squid.conf.documented changes/additions would be nice.
>
>  [1] http://wiki.squid-cache.org/Features/SmpScale
>
>
>> My point in mentioning that there are a lot of installations using
>> SMP and AUFS is that something widely used but buggy tends to be brought
>> up on this email list and I haven't seen it.
>
>IIRC, it has been brought up several times on the mailing lists and in
>Bugzilla. Once you dedicate each ufs-based store to each individual
>worker, most of the problems become subtle, often "invisible" to an
>admin because they "break" transactions, not Squid, especially if you do
>not use a mixture of ufs-based and rock stores. Using mailing list as an
>indicator that as subtle problem does _not_ exist is a risky strategy IMO.

Well, I personally will still be curious how much does SMP affect the case of
one worker and one or more diskers...

do diskers only provide I/O to the requestor?

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Depression is merely anger without enthusiasm.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Alex Rousskov
On 03/20/2017 10:15 AM, Matus UHLAR - fantomas wrote:

> Well, I personally will still be curious how much does SMP affect the
> case of one worker and one or more diskers...

I do not understand why you are asking this question in AUFS context.
AUFS does not use diskers! Today, only Rock store uses diskers (in SMP
mode). Some other [ufs-based] cache stores use various helper threads
and processes for I/O as well, but those helper processes are not
diskers or even kids in SMP terminology.


> do diskers only provide I/O to the requestor?

Diskers primary function is low-level disk cache I/O. Like all kids,
diskers respond to cache manager requests and Squid management events
(e.g. shutdown and reconfiguration). IIRC, diskers also build in-RAM
cache_dir index.

    http://wiki.squid-cache.org/Features/SmpScale#Terminology

HTH,

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Matus UHLAR - fantomas
>On 03/20/2017 10:15 AM, Matus UHLAR - fantomas wrote:
>> Well, I personally will still be curious how much does SMP affect the
>> case of one worker and one or more diskers...

On 20.03.17 12:19, Alex Rousskov wrote:
>I do not understand why you are asking this question in AUFS context.
>AUFS does not use diskers!

because it's diskers (rock store) who switch to SMP mode when "workers 1" is
used, and this may cause troubles to *ufs.

> Today, only Rock store uses diskers (in SMP
>mode). Some other [ufs-based] cache stores use various helper threads
>and processes for I/O as well, but those helper processes are not
>diskers or even kids in SMP terminology.

this is something I'd like to understand better...

>> do diskers only provide I/O to the requestor?

>Diskers primary function is low-level disk cache I/O. Like all kids,
>diskers respond to cache manager requests and Squid management events
>(e.g. shutdown and reconfiguration). IIRC, diskers also build in-RAM
>cache_dir index.
>
>    http://wiki.squid-cache.org/Features/SmpScale#Terminology

their own or also the shared one?

--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watson.  -- Daffy Duck & Porky Pig
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Amos Jeffries
Administrator
On 30/03/2017 9:43 p.m., Matus UHLAR - fantomas wrote:

>> On 03/20/2017 10:15 AM, Matus UHLAR - fantomas wrote:
>>> Well, I personally will still be curious how much does SMP affect the
>>> case of one worker and one or more diskers...
>
> On 20.03.17 12:19, Alex Rousskov wrote:
>> I do not understand why you are asking this question in AUFS context.
>> AUFS does not use diskers!
>
> because it's diskers (rock store) who switch to SMP mode when "workers
> 1" is
> used, and this may cause troubles to *ufs.
>
>> Today, only Rock store uses diskers (in SMP
>> mode). Some other [ufs-based] cache stores use various helper threads
>> and processes for I/O as well, but those helper processes are not
>> diskers or even kids in SMP terminology.
>
> this is something I'd like to understand better...

Alex is referring to;

a) "diskd" cache_dir type, which uses a helper named 'diskd' to offload
parts of the file manipulations to that helper. (I've not looked closely
to see which, sorry).

b) squid builds with 'unlinkd' feature enabled, which uses an helper
called 'unlinkd' to perform file deletion outside the squid process.

c) AUFS which starts threads to perform the read/write I/O and file
deletion stuff.

... and I'm still learning of other such odd things in weird rarely
touched corners. There is something thread-like in digest creation for
walking the store index quickly and swap log rebuilding for example.


>
>>> do diskers only provide I/O to the requestor?
>
>> Diskers primary function is low-level disk cache I/O. Like all kids,
>> diskers respond to cache manager requests and Squid management events
>> (e.g. shutdown and reconfiguration). IIRC, diskers also build in-RAM
>> cache_dir index.
>>
>>    http://wiki.squid-cache.org/Features/SmpScale#Terminology
>
> their own or also the shared one?

Alex may correct me but AFAIK there is currently only the shared index.
Diskers just do better with it through generic parallel-CPU efficiencies
than a non-SMP Squid would spreading the index actions between other
transaction work.

Amos

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SMP and AUFS

Alex Rousskov
In reply to this post by Matus UHLAR - fantomas
On 03/30/2017 02:43 AM, Matus UHLAR - fantomas wrote:
>> On 03/20/2017 10:15 AM, Matus UHLAR - fantomas wrote:
>>> Well, I personally will still be curious how much does SMP affect the
>>> case of one worker and one or more diskers...
>
> On 20.03.17 12:19, Alex Rousskov wrote:
>> I do not understand why you are asking this question in AUFS context.
>> AUFS does not use diskers!

> because it's diskers (rock store) who switch to SMP mode when "workers
> 1" is used, and this may cause troubles to *ufs.

Diskers do not cause trouble -- they are doing exactly what they have
been designed to do. It is ufs (being SMP-unaware) that causes trouble
in SMP mode when admins enable ufs-based cache_dirs in SMP configurations.


>> Diskers primary function is low-level disk cache I/O. Like all kids,
>> diskers respond to cache manager requests and Squid management events
>> (e.g. shutdown and reconfiguration). IIRC, diskers also build in-RAM
>> cache_dir index.
>>
>>    http://wiki.squid-cache.org/Features/SmpScale#Terminology

> their own or also the shared one?

Their own, which is shared among all workers. In many contexts, one can
view the set of all disk-specific shared indexes as one big shared disk
index. Please note that my recollection that diskers build the in-RAM
cache_dir index may be wrong -- this behavior may have changed, and I
have not checked the code or commit logs. It would certainly make sense
for diskers to build their indexes, but I do not remember whether we
have implemented that optimization.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users