SIGBUS attempting to use rock

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

SIGBUS attempting to use rock

Antonio SJ Musumeci
I'm sending this out for those that might run into this in the future.

After a lot of tinkering and turning on full debug I realized the reason
rock was failing for me in my container was due to the small default SHM
size allocated by Docker. Increasing the SHM size with `--shm-size`
fixed the issue.

It'd be significantly more helpful if the Squid reported precisely what
the issue and exited gracefully rather than SIGBUS'ing.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Alex Rousskov
On 10/17/19 12:28 PM, Antonio SJ Musumeci wrote:

> After a lot of tinkering and turning on full debug I realized the reason
> rock was failing for me in my container was due to the small default SHM
> size allocated by Docker. Increasing the SHM size with `--shm-size`
> fixed the issue.
>
> It'd be significantly more helpful if the Squid reported precisely what
> the issue and exited gracefully rather than SIGBUS'ing.

Does enabling shared_memory_locking in squid.conf trigger the behavior
you want?

If yes, then you may be wondering why this is not the default setting.
The answer is in the following two squid-dev messages. Message [1] is a
high-level description. Message [2] contains more low-level details and
examples of ~60 second startup delays that locking may cause.

[1] http://lists.squid-cache.org/pipermail/squid-dev/2016-March/005260.html

[2]
http://lists.squid-cache.org/pipermail/squid-dev/2015-December/004112.html

Enhancing shared memory management (e.g., implementing ideas mentioned
in the "As we gain more experience" paragraph of [1]) is welcomed.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Antonio SJ Musumeci
I'm currently unable to test fully due to my environment lacking
CAP_IPC_LOCK. It does fail more gracefully saying it can't use mlock at all.

That said... the option is listed in relation to SMP and not referenced
by rock docs. Also it should be possible to check /dev/shm's free space
and compare that against the file size needed. Clearly it knows based on
the requested storage and slot sizes. Even a guess with a warning rather
than leaving it to fail otherwise silently with a SIGBUS.

On 10/17/2019 2:53 PM, Alex Rousskov wrote:

> On 10/17/19 12:28 PM, Antonio SJ Musumeci wrote:
>
>> After a lot of tinkering and turning on full debug I realized the reason
>> rock was failing for me in my container was due to the small default SHM
>> size allocated by Docker. Increasing the SHM size with `--shm-size`
>> fixed the issue.
>>
>> It'd be significantly more helpful if the Squid reported precisely what
>> the issue and exited gracefully rather than SIGBUS'ing.
> Does enabling shared_memory_locking in squid.conf trigger the behavior
> you want?
>
> If yes, then you may be wondering why this is not the default setting.
> The answer is in the following two squid-dev messages. Message [1] is a
> high-level description. Message [2] contains more low-level details and
> examples of ~60 second startup delays that locking may cause.
>
> [1] http://lists.squid-cache.org/pipermail/squid-dev/2016-March/005260.html
>
> [2]
> http://lists.squid-cache.org/pipermail/squid-dev/2015-December/004112.html
>
> Enhancing shared memory management (e.g., implementing ideas mentioned
> in the "As we gain more experience" paragraph of [1]) is welcomed.
>
>
> HTH,
>
> Alex.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Alex Rousskov
On 10/17/19 5:07 PM, Antonio SJ Musumeci wrote:

> the option is listed in relation to SMP and not referenced by rock docs.

The second paragraph in cache_dir rock directive documentation implies
SMP -- it talks about various processes that rock uses to avoid locking
(if it can). SMP usage is the primary rock scalability feature. There is
more info at https://wiki.squid-cache.org/Features/RockStore (and
https://wiki.squid-cache.org/Features/LargeRockStore).


> Also it should be possible to check /dev/shm's free space
> and compare that against the file size needed. Clearly it knows based on
> the requested storage and slot sizes. Even a guess with a warning rather
> than leaving it to fail otherwise silently with a SIGBUS.

Unfortunately, a simple implementation may produce a lot of false
warnings in some environments while a quality implementation may not be
as easy as you think: Accessing free space info may require special
permissions and correctly accounting for the existing shared memory
segments in that partition would be tricky (they can be leftovers from
the previous Squid run that will be overwritten or something completely
unrelated to Squid). Even finding the right partition name in a portable
way may be tricky!

IMHO, the future development directions outlined when adding
shared_memory_locking are more promising in general, but I would be
happy to learn that there are even better options.


Cheers,

Alex.


> On 10/17/2019 2:53 PM, Alex Rousskov wrote:
>> On 10/17/19 12:28 PM, Antonio SJ Musumeci wrote:
>>
>>> After a lot of tinkering and turning on full debug I realized the reason
>>> rock was failing for me in my container was due to the small default SHM
>>> size allocated by Docker. Increasing the SHM size with `--shm-size`
>>> fixed the issue.
>>>
>>> It'd be significantly more helpful if the Squid reported precisely what
>>> the issue and exited gracefully rather than SIGBUS'ing.

>> Does enabling shared_memory_locking in squid.conf trigger the behavior
>> you want?
>>
>> If yes, then you may be wondering why this is not the default setting.
>> The answer is in the following two squid-dev messages. Message [1] is a
>> high-level description. Message [2] contains more low-level details and
>> examples of ~60 second startup delays that locking may cause.
>>
>> [1]
>> http://lists.squid-cache.org/pipermail/squid-dev/2016-March/005260.html
>>
>> [2]
>> http://lists.squid-cache.org/pipermail/squid-dev/2015-December/004112.html
>>
>>
>> Enhancing shared memory management (e.g., implementing ideas mentioned
>> in the "As we gain more experience" paragraph of [1]) is welcomed.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Antonio SJ Musumeci
On 10/17/2019 5:47 PM, Alex Rousskov wrote:

> Unfortunately, a simple implementation may produce a lot of false
> warnings in some environments while a quality implementation may not be
> as easy as you think: Accessing free space info may require special
> permissions and correctly accounting for the existing shared memory
> segments in that partition would be tricky (they can be leftovers from
> the previous Squid run that will be overwritten or something completely
> unrelated to Squid). Even finding the right partition name in a portable
> way may be tricky!
>
> IMHO, the future development directions outlined when adding
> shared_memory_locking are more promising in general, but I would be
> happy to learn that there are even better options.

Clearly Squid is aware of the path where these temp files are being
created and can simply statvfs the base path can it not? If it's
creating new files in /dev/shm it can statvfs and compare the available
space to what it intends to create and in the least warn.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Alex Rousskov
On 10/17/19 7:13 PM, Antonio SJ Musumeci wrote:

> On 10/17/2019 5:47 PM, Alex Rousskov wrote:
>> Unfortunately, a simple implementation may produce a lot of false
>> warnings in some environments while a quality implementation may not be
>> as easy as you think: Accessing free space info may require special
>> permissions and correctly accounting for the existing shared memory
>> segments in that partition would be tricky (they can be leftovers from
>> the previous Squid run that will be overwritten or something completely
>> unrelated to Squid). Even finding the right partition name in a portable
>> way may be tricky!
>>
>> IMHO, the future development directions outlined when adding
>> shared_memory_locking are more promising in general, but I would be
>> happy to learn that there are even better options.

> Clearly Squid is aware of the path where these temp files are being
> created

Actually, the location of shared memory segments is chosen by the OS.
Not all OSes do what Linux does. The code dealing with [naming] shared
memory segments is more complicated than you probably imagine. However,
let's not spend time arguing about these low-level specifics here: If
you can contribute an improvement, please post a plan on squid-dev.
Otherwise, let's leave these low-level details to those contributing
improvements.


Cheers,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Antonio SJ Musumeci
On 10/18/2019 9:49 AM, Alex Rousskov wrote:
> Actually, the location of shared memory segments is chosen by the OS.
> Not all OSes do what Linux does. The code dealing with [naming] shared
> memory segments is more complicated than you probably imagine. However,
> let's not spend time arguing about these low-level specifics here: If
> you can contribute an improvement, please post a plan on squid-dev.
> Otherwise, let's leave these low-level details to those contributing
> improvements.

You don't need to know where the OS put the file (though /dev/shm is a
well known location and easily could be checked explicitly).

$ man fstatvfs

The attached example seems to work fine and could be incorporated into
ipc/mem/Segment.cc.

I'd be happy to post this on squid-dev / submit a pull request.



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

shm_statvfs.c (940 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SIGBUS attempting to use rock

Alex Rousskov
On 10/18/19 10:25 AM, Antonio SJ Musumeci wrote:

> I'd be happy to post this on squid-dev / submit a pull request.

Please do, assuming you are willing to do the legwork necessary to go
from a proof of concept (that has already changed several times during
this short discussion!) to the final officially-acceptable solution.

    https://wiki.squid-cache.org/MergeProcedure

You do not have to get the overall design approved on squid-dev first,
but I would recommend that step to reduce code rewrites in this case.
The design RFC needs no code. I would focus on how the new feature will
avoid false warnings (especially in cases where the partition may have
stale Squid files that this instance will overwrite). BTW, please keep
in mind that Squid creates many segments of various sizes, in
decentralized fashion.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users