squid-5 lost my cached dir

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

squid-5 lost my cached dir

joseph
using squid-5.0.0-20180218-r3b65960 release kill my cache dir some how
swap.state is empty  and the size of the cached dir has 30 geg
if i run for couple HR  and the swap.state grow like couple meg  then
restart squid
it become empty swap.state  but the cached item in storage stay
i removed patch by patch the  killer was
http://www.squid-cache.org/Versions/v5/changesets/squid-5-4310f8b00dd574542dcec4208112bb89ef403528.patch
so please fix it



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Alex Rousskov
On 02/20/2018 04:22 PM, joseph wrote:
> using squid-5.0.0-20180218-r3b65960 release kill my cache dir some how
> swap.state is empty  and the size of the cached dir has 30 geg
> if i run for couple HR  and the swap.state grow like couple meg  then
> restart squid
> it become empty swap.state  but the cached item in storage stay
> i removed patch by patch the  killer was
> http://www.squid-cache.org/Versions/v5/changesets/squid-5-4310f8b00dd574542dcec4208112bb89ef403528.patch
> so please fix it

Since our tests for that change were successful, we may need to know
more about your environment to fix this problem. Please share your Squid
configuration (at least the SMP-, cache_mem-, and cache_dir-related
directives). You may want to file a bug report to track this fix better.


Thank you,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
alex tested on 2 environment        debian 9 and solariss
same shame  just keep for 30 minute running caching and do squid restart and
chek your swap.state  you will notice its empty almost on 2 environment
totally different ok
i removed that patch and all fine swap.state stay perfect and all fine
so i guess one of the 2 scenario  its detecting swap file on start never
exist or detecting wrong size that all



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
In reply to this post by Alex Rousskov
>>Since our tests for that change were successful
did you restart wile your test was success i guess not



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Yuri Voinov
In reply to this post by Alex Rousskov
We're never use SMP. Just regular SMP-unaware configuration.


21.02.2018 05:28, Alex Rousskov пишет:

> On 02/20/2018 04:22 PM, joseph wrote:
>> using squid-5.0.0-20180218-r3b65960 release kill my cache dir some how
>> swap.state is empty  and the size of the cached dir has 30 geg
>> if i run for couple HR  and the swap.state grow like couple meg  then
>> restart squid
>> it become empty swap.state  but the cached item in storage stay
>> i removed patch by patch the  killer was
>> http://www.squid-cache.org/Versions/v5/changesets/squid-5-4310f8b00dd574542dcec4208112bb89ef403528.patch
>> so please fix it
> Since our tests for that change were successful, we may need to know
> more about your environment to fix this problem. Please share your Squid
> configuration (at least the SMP-, cache_mem-, and cache_dir-related
> directives). You may want to file a bug report to track this fix better.
>
>
> Thank you,
>
> Alex.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Alex Rousskov
In reply to this post by joseph
On 02/20/2018 04:38 PM, joseph wrote:
> alex tested on 2 environment        debian 9 and solariss
> same shame  just keep for 30 minute running caching and do squid restart and
> chek your swap.state  you will notice its empty almost on 2 environment
> totally different ok
> i removed that patch and all fine swap.state stay perfect and all fine
> so i guess one of the 2 scenario  its detecting swap file on start never
> exist or detecting wrong size that all

Thank you for providing these additional snippets of information. We
will, of course, try to reproduce the problem with the info you have
already provided. If you would like to increase our chances of
reproducing this problem, please see my original response (quoted below).

Alex.


> On 02/20/2018 04:28 PM, Alex Rousskov wrote:

>> ... Please share your Squid
>> configuration (at least the SMP-, cache_mem-, and cache_dir-related
>> directives). You may want to file a bug report to track this fix better.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Yuri Voinov
# -------------------------------------
# Store parameters
# -------------------------------------
minimum_object_size 10 bytes
maximum_object_size 4 GB

cache_dir aufs /data/cache/d1 48000 64 512
cache_dir aufs /data/cache/d2 48000 64 512
cache_dir aufs /data/cache/d3 48000 64 512
cache_dir aufs /data/cache/d4 48000 64 512

# -------------------------------------
# Memory parameters
# -------------------------------------
cache_mem 512 MB

memory_pools off
#memory_pools_limit 1024 MB

maximum_object_size_in_memory 1 MB


21.02.2018 05:48, Alex Rousskov пишет:

> On 02/20/2018 04:38 PM, joseph wrote:
>> alex tested on 2 environment        debian 9 and solariss
>> same shame  just keep for 30 minute running caching and do squid restart and
>> chek your swap.state  you will notice its empty almost on 2 environment
>> totally different ok
>> i removed that patch and all fine swap.state stay perfect and all fine
>> so i guess one of the 2 scenario  its detecting swap file on start never
>> exist or detecting wrong size that all
> Thank you for providing these additional snippets of information. We
> will, of course, try to reproduce the problem with the info you have
> already provided. If you would like to increase our chances of
> reproducing this problem, please see my original response (quoted below).
>
> Alex.
>
>
>> On 02/20/2018 04:28 PM, Alex Rousskov wrote:
>>> ... Please share your Squid
>>> configuration (at least the SMP-, cache_mem-, and cache_dir-related
>>> directives). You may want to file a bug report to track this fix better.
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
In reply to this post by Alex Rousskov
NO SMP just normal configuration
cache_mem 500 MB
memory_pools off


cache_dir aufs /mnt/cache-a 500000 128 512
cache_dir aufs /mnt/cache-b 500000 128 512




-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Yuri Voinov
In reply to this post by Yuri Voinov
It's immediately lost swap.state contents after any restart and starts
up with empty swap.state.

After removal of swap.state with stopped squid, it's correctly
re-indexing cache contents and then run normally.


21.02.2018 05:50, Yuri пишет:

> # -------------------------------------
> # Store parameters
> # -------------------------------------
> minimum_object_size 10 bytes
> maximum_object_size 4 GB
>
> cache_dir aufs /data/cache/d1 48000 64 512
> cache_dir aufs /data/cache/d2 48000 64 512
> cache_dir aufs /data/cache/d3 48000 64 512
> cache_dir aufs /data/cache/d4 48000 64 512
>
> # -------------------------------------
> # Memory parameters
> # -------------------------------------
> cache_mem 512 MB
>
> memory_pools off
> #memory_pools_limit 1024 MB
>
> maximum_object_size_in_memory 1 MB
>
>
> 21.02.2018 05:48, Alex Rousskov пишет:
>> On 02/20/2018 04:38 PM, joseph wrote:
>>> alex tested on 2 environment        debian 9 and solariss
>>> same shame  just keep for 30 minute running caching and do squid restart and
>>> chek your swap.state  you will notice its empty almost on 2 environment
>>> totally different ok
>>> i removed that patch and all fine swap.state stay perfect and all fine
>>> so i guess one of the 2 scenario  its detecting swap file on start never
>>> exist or detecting wrong size that all
>> Thank you for providing these additional snippets of information. We
>> will, of course, try to reproduce the problem with the info you have
>> already provided. If you would like to increase our chances of
>> reproducing this problem, please see my original response (quoted below).
>>
>> Alex.
>>
>>
>>> On 02/20/2018 04:28 PM, Alex Rousskov wrote:
>>>> ... Please share your Squid
>>>> configuration (at least the SMP-, cache_mem-, and cache_dir-related
>>>> directives). You may want to file a bug report to track this fix better.
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
--
*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Yuri Voinov
Yes, after re-indexing, every next restart cleans up swap.state again
and again.


21.02.2018 05:57, Yuri пишет:

> It's immediately lost swap.state contents after any restart and starts
> up with empty swap.state.
>
> After removal of swap.state with stopped squid, it's correctly
> re-indexing cache contents and then run normally.
>
>
> 21.02.2018 05:50, Yuri пишет:
>> # -------------------------------------
>> # Store parameters
>> # -------------------------------------
>> minimum_object_size 10 bytes
>> maximum_object_size 4 GB
>>
>> cache_dir aufs /data/cache/d1 48000 64 512
>> cache_dir aufs /data/cache/d2 48000 64 512
>> cache_dir aufs /data/cache/d3 48000 64 512
>> cache_dir aufs /data/cache/d4 48000 64 512
>>
>> # -------------------------------------
>> # Memory parameters
>> # -------------------------------------
>> cache_mem 512 MB
>>
>> memory_pools off
>> #memory_pools_limit 1024 MB
>>
>> maximum_object_size_in_memory 1 MB
>>
>>
>> 21.02.2018 05:48, Alex Rousskov пишет:
>>> On 02/20/2018 04:38 PM, joseph wrote:
>>>> alex tested on 2 environment        debian 9 and solariss
>>>> same shame  just keep for 30 minute running caching and do squid restart and
>>>> chek your swap.state  you will notice its empty almost on 2 environment
>>>> totally different ok
>>>> i removed that patch and all fine swap.state stay perfect and all fine
>>>> so i guess one of the 2 scenario  its detecting swap file on start never
>>>> exist or detecting wrong size that all
>>> Thank you for providing these additional snippets of information. We
>>> will, of course, try to reproduce the problem with the info you have
>>> already provided. If you would like to increase our chances of
>>> reproducing this problem, please see my original response (quoted below).
>>>
>>> Alex.
>>>
>>>
>>>> On 02/20/2018 04:28 PM, Alex Rousskov wrote:
>>>>> ... Please share your Squid
>>>>> configuration (at least the SMP-, cache_mem-, and cache_dir-related
>>>>> directives). You may want to file a bug report to track this fix better.
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
--
*****************************
* C++20 : Bug to the future *
*****************************



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Alex Rousskov
In reply to this post by Yuri Voinov
Thank you and joseph for providing these important details. If we can
reproduce the problem, then we will post a pull request with a fix once
it is is available. If we cannot, we will post more questions here
(unless there is a report in Squid Bugzilla by then). --Alex.

On 02/20/2018 04:57 PM, Yuri wrote:

> It's immediately lost swap.state contents after any restart and starts
> up with empty swap.state.
>
> After removal of swap.state with stopped squid, it's correctly
> re-indexing cache contents and then run normally.
>
>
> 21.02.2018 05:50, Yuri пишет:
>> # -------------------------------------
>> # Store parameters
>> # -------------------------------------
>> minimum_object_size 10 bytes
>> maximum_object_size 4 GB
>>
>> cache_dir aufs /data/cache/d1 48000 64 512
>> cache_dir aufs /data/cache/d2 48000 64 512
>> cache_dir aufs /data/cache/d3 48000 64 512
>> cache_dir aufs /data/cache/d4 48000 64 512
>>
>> # -------------------------------------
>> # Memory parameters
>> # -------------------------------------
>> cache_mem 512 MB
>>
>> memory_pools off
>> #memory_pools_limit 1024 MB
>>
>> maximum_object_size_in_memory 1 MB
>>
>>
>> 21.02.2018 05:48, Alex Rousskov пишет:
>>> On 02/20/2018 04:38 PM, joseph wrote:
>>>> alex tested on 2 environment        debian 9 and solariss
>>>> same shame  just keep for 30 minute running caching and do squid restart and
>>>> chek your swap.state  you will notice its empty almost on 2 environment
>>>> totally different ok
>>>> i removed that patch and all fine swap.state stay perfect and all fine
>>>> so i guess one of the 2 scenario  its detecting swap file on start never
>>>> exist or detecting wrong size that all
>>> Thank you for providing these additional snippets of information. We
>>> will, of course, try to reproduce the problem with the info you have
>>> already provided. If you would like to increase our chances of
>>> reproducing this problem, please see my original response (quoted below).
>>>
>>> Alex.
>>>
>>>
>>>> On 02/20/2018 04:28 PM, Alex Rousskov wrote:
>>>>> ... Please share your Squid
>>>>> configuration (at least the SMP-, cache_mem-, and cache_dir-related
>>>>> directives). You may want to file a bug report to track this fix better.
>>> _______________________________________________
>>> squid-users mailing list
>>> [hidden email]
>>> http://lists.squid-cache.org/listinfo/squid-users
>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
found where the bug   and i made it work but i dont know wat is the best

in UFSSwapDir.cc function Fs::Ufs::UFSSwapDir::openTmpSwapLog(int
*clean_flag, int *zero_flag)

was -->    *zero_flag = log_sb.st_size == 0 ? 1 : 0;
change to  *zero_flag = log_sb.st_size == 0 ? 0 : 1;
best to change int to bool  in futur

the change make it work in RebuildState.cc function
Fs::Ufs::RebuildState::RebuildState(RefCount<UFSSwapDir> aSwapDir) :

now this work correctly
    if (fp && !zeroLengthLog)
        LogParser = Fs::Ufs::UFSSwapLogParser::GetUFSSwapLogParser(fp);



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
sorry this fix the start rebuild if swap.state is empty
but there is more issue  wen you stop squid   it delete the swap.state and
re create new one empty




-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
In reply to this post by Alex Rousskov
testing https://github.com/squid-cache/squid/pull/155
working fine :)
i will report if something else came up but so fare so good tks



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Alex Rousskov
In reply to this post by Alex Rousskov
On 02/20/2018 05:04 PM, Alex Rousskov wrote:
> Thank you and joseph for providing these important details. If we can
> reproduce the problem, then we will post a pull request with a fix once
> it is is available.

Done: https://github.com/squid-cache/squid/pull/155

Alex.


> On 02/20/2018 04:57 PM, Yuri wrote:
>> It's immediately lost swap.state contents after any restart and starts
>> up with empty swap.state.
>>
>> After removal of swap.state with stopped squid, it's correctly
>> re-indexing cache contents and then run normally.
>>
>>
>> 21.02.2018 05:50, Yuri пишет:
>>> # -------------------------------------
>>> # Store parameters
>>> # -------------------------------------
>>> minimum_object_size 10 bytes
>>> maximum_object_size 4 GB
>>>
>>> cache_dir aufs /data/cache/d1 48000 64 512
>>> cache_dir aufs /data/cache/d2 48000 64 512
>>> cache_dir aufs /data/cache/d3 48000 64 512
>>> cache_dir aufs /data/cache/d4 48000 64 512
>>>
>>> # -------------------------------------
>>> # Memory parameters
>>> # -------------------------------------
>>> cache_mem 512 MB
>>>
>>> memory_pools off
>>> #memory_pools_limit 1024 MB
>>>
>>> maximum_object_size_in_memory 1 MB
>>>
>>>
>>> 21.02.2018 05:48, Alex Rousskov пишет:
>>>> On 02/20/2018 04:38 PM, joseph wrote:
>>>>> alex tested on 2 environment        debian 9 and solariss
>>>>> same shame  just keep for 30 minute running caching and do squid restart and
>>>>> chek your swap.state  you will notice its empty almost on 2 environment
>>>>> totally different ok
>>>>> i removed that patch and all fine swap.state stay perfect and all fine
>>>>> so i guess one of the 2 scenario  its detecting swap file on start never
>>>>> exist or detecting wrong size that all
>>>> Thank you for providing these additional snippets of information. We
>>>> will, of course, try to reproduce the problem with the info you have
>>>> already provided. If you would like to increase our chances of
>>>> reproducing this problem, please see my original response (quoted below).
>>>>
>>>> Alex.
>>>>
>>>>
>>>>> On 02/20/2018 04:28 PM, Alex Rousskov wrote:
>>>>>> ... Please share your Squid
>>>>>> configuration (at least the SMP-, cache_mem-, and cache_dir-related
>>>>>> directives). You may want to file a bug report to track this fix better.
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> [hidden email]
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
thank you and thanks to eduard bagdasaryan
no more memory overrun on his big patch

hoop someone soon work on  real range caching  since the real world ar https
mostly now
so adding real range caching will save us  more bandwidth on http squid
server



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

Amos Jeffries
Administrator
On 22/02/18 11:56, joseph wrote:
> thank you and thanks to eduard bagdasaryan
> no more memory overrun on his big patch
>
> hoop someone soon work on  real range caching  since the real world ar https
> mostly now
> so adding real range caching will save us  more bandwidth on http squid
> server

Huh?  HTTPS has nothing to do with the HTTP Range mechanism, and neither
does any of the caching work Eduard just finished.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: squid-5 lost my cached dir

joseph
what i was trying to say is about partial range support bro lol and eduard
work is perfect
like making squid cache partial of the files :)



-----
**************************
***** Crash to the future  ****
**************************
--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
************************** ***** Crash to the future **** **************************