hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

L. A. Walsh
This caught my attention as my housemate tends to watch alot of
youtube videos, and caching some of them might speed up their
access, so was trying to understand what was meant in your post:

Yuri Voinov wrote:
> Things are changed in the web on regular basis. Nothing permanent in the
> world.
>
> So, store ID rules lost relevance and no longer work.
>  
----
    Is the problem that "Store ID rules lost relevance" caused by a
change from squid 3.5.19 -> 3.5.20?

    That doesn't sound so much like a change on the web, but a change
in squid.

    Were storeID rules removed in 3.5.20?  If that's the case,
what might have replaced them?

@Eduardo: am I to understand that this plugin worked in 3.5.19, but
not in 3.5.20 and above?  (trying to get the versions right)

Also, Eduardo -- what specific features above 3.5.19 were you hoping
to include by upgrading?  I.e. is 3.5.19 not working for you for some
reason?  It might be easier to cherry pick changes that you needed from
some later version back into 3.5.19 if some major feature
(like storeID rules?) was removed after that version...

Thanks Yuri!
Linda


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

Yuri Voinov


28.03.2017 1:26, L A Walsh пишет:
> This caught my attention as my housemate tends to watch alot of
> youtube videos, and caching some of them might speed up their
> access, so was trying to understand what was meant in your post:
>
> Yuri Voinov wrote:
>> Things are changed in the web on regular basis. Nothing permanent in the
>> world.
>>
>> So, store ID rules lost relevance and no longer work.
What word is not clear here?
>>  
> ----
>    Is the problem that "Store ID rules lost relevance" caused by a
> change from squid 3.5.19 -> 3.5.20?
Caused independently during "things changes time to time".
>
>    That doesn't sound so much like a change on the web, but a change
> in squid.
Heh? Really?
>
>    Were storeID rules removed in 3.5.20?  If that's the case,
> what might have replaced them?
Why did it happen? What does the documentation tell us?
>
> @Eduardo: am I to understand that this plugin worked in 3.5.19, but
> not in 3.5.20 and above?  (trying to get the versions right)
>
> Also, Eduardo -- what specific features above 3.5.19 were you hoping
> to include by upgrading?  I.e. is 3.5.19 not working for you for some
> reason?  It might be easier to cherry pick changes that you needed from
> some later version back into 3.5.19 if some major feature
> (like storeID rules?) was removed after that version...
Right now I'm running Squid v5.x. With store ID. So?
>
> Thanks Yuri!
> Linda
>
>

--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

Eliezer Croitoru
In reply to this post by L. A. Walsh
Hey Linda,

As the pathcer\author of StoreID I will try to clarify what might seems odd.
StoreID is a "static" rule which is one of the squid cache fundamentals.
The feature is the option to tweak this internal cache object ID.
This is a very static feature and will not be changed for a very long time from now on.
Most of the public helpers I have seen are very "simple" and rely on very simple things.
The main idea is that the admin can know and predict what would be the StoreID.
It's like "knowing" what the kid is going to say next from the past experience with him.
In a similar way StoreID allows an admin that can predict that a specific URL will contain the same content as another one that was previously cached.
In a more simple way, it gives that admin the option to run a "de-duplication" script.

But since many systems these days are aware of the option to predict what would be the next url(think about an exe or other binary that can be replaced on-the-fly\in-transit) the developers changed and change(like a Diffe Hellman) their way of transporting content to the end client.
Due to this "Diffe Hellman" feature that many added it makes the more simple scripts useless since the developers became much smarter.
And back to the analogy, the kid grows and talks in a certain language.
Now the kids is much more mature after couple years and he became smart enough so you would need to actually answer the question "where do kids come from" or some other questions.
Then the answer can be creative enough to leave the child with a satisfying and a good one. If you "miss" this opportunity and will answer a very non-realistic answer(compared to the kid's age) the next time he would demand from you a more demanding answer and eventually if you would answer too many bad responses he will eventually go and research by himself in other and maybe a more reliable sources.
(the above analogy is what gave squid a very bad reputation by many admins)

Today Squid-Cache is smart enough to at-least answer the right answer for a demanding client even if the admin does something wrong in his StoreID helper.
Indeed you will see TCP_MISS and it won't cache but this is only since the admin might have the illusion that an encrypted content can be predicted when a Diffie  Helman cryptography is creating the plain url's these days.

Hope It helped,
Eliezer

* For youtube to be cached the best way is to learn more about content adaptation and to have a look at some youtube video downloaders scripts.

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of L A Walsh
Sent: Monday, March 27, 2017 10:26 PM
To: Yuri Voinov <[hidden email]>
Cc: [hidden email]
Subject: [squid-users] hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

This caught my attention as my housemate tends to watch alot of
youtube videos, and caching some of them might speed up their
access, so was trying to understand what was meant in your post:

Yuri Voinov wrote:
> Things are changed in the web on regular basis. Nothing permanent in the
> world.
>
> So, store ID rules lost relevance and no longer work.
>  
----
    Is the problem that "Store ID rules lost relevance" caused by a
change from squid 3.5.19 -> 3.5.20?

    That doesn't sound so much like a change on the web, but a change
in squid.

    Were storeID rules removed in 3.5.20?  If that's the case,
what might have replaced them?

@Eduardo: am I to understand that this plugin worked in 3.5.19, but
not in 3.5.20 and above?  (trying to get the versions right)

Also, Eduardo -- what specific features above 3.5.19 were you hoping
to include by upgrading?  I.e. is 3.5.19 not working for you for some
reason?  It might be easier to cherry pick changes that you needed from
some later version back into 3.5.19 if some major feature
(like storeID rules?) was removed after that version...

Thanks Yuri!
Linda


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

Eduardo Carneiro
In reply to this post by L. A. Walsh
Hello everyone.

Answering Linda's questions:

- Yes, I tried to use the hsc helper in versions above 3.5.19. But I did not succeed.
- There are no specific features in the versions above 3.5.19 that I need to use. But, for security reasons, I'd rather use the newer versions.

I posted on this just because, as I said before, I'd rather use newer versions. Anyway, when I have some free time, I'll try to fix it.

My knowledge on this is limited, but I also think this problem is something wrong with Squid. Following the logic, today in 3.5.19 still works normally. I can cache Youtube, Facebook, etc. However, when I use a version above 3.5.19 it simply stops working.

Finally, I repeat: My knowledge of this is limited. But I'll try to fix that in the future.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

L. A. Walsh
In reply to this post by Eliezer Croitoru
Eliezer Croitoru wrote:
> Hey Linda,
>
> As the pathcer\author of StoreID I will try to clarify what might seems odd.
> StoreID is a "static" rule which is one of the squid cache fundamentals.
> The feature is the option to tweak this internal cache object ID.
> This is a very static feature and will not be changed for a very long time from now on.
> Most of the public helpers I have seen are very "simple" and rely on very simple things.
----
        Makes sense, otherwise too prone to breakage.



>
> But since many systems these days are aware of the option to predict what would be the next url(think about an exe or other binary that can be replaced on-the-fly\in-transit) the developers changed and change(like a Diffe Hellman) their way of transporting content to the end client.
> Due to this "Diffe Hellman" feature that many added it makes the more simple scripts useless since the developers became much smarter.
----
        yeah, my use case was fairly simple -- same
person w/same browser, watching same vid a 2nd time.

        They gave me many "kudos" and noticed that
youtube was noticeably faster to browse through when I
implemented the SSL interception on the squid proxy
that web traffic goes through.  In that case, it was mainly
the caching of the video-thumbs that noticeably sped up
moving through YT pages.

> Indeed you will see TCP_MISS and it won't cache but this is only since the admin might have the illusion that an encrypted content can be predicted when a Diffie  Helman cryptography is creating the plain url's these days.
----
        Oh yeah.  Have noted that there are an infinite number
of ways to access the same URL and have thought about ways
I might collapse them to 1 URL, but it's just idle thinking as
other things on the plate.

        One good idea that didn't get updated was an
extension in FF, that tried to store some of the latest
Javascript libs that sites used so if they asked for the lib
from a common site (like jquery), it might return the result
from a local cache.  

It wouldn't help for those sites that merge
multiple JS files and minify them.

But many sites have 15-20 different websites that are
"included" to get different elements (fonts, stylesheets,
JS libs, etc) from different sources.  They seem to
include URL's like a developer would use
#include files...(and often take forever to load).

multiple elements from different URLs like they would
use multiple header include files in a local compilation.


> Hope It helped,
> Eliezer

Thanks for the explanation, certainly more useful
than just telling someone:

 "the web broke it"...
:-)





_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hsc-dynamic-cache: relied on storeID rules? Removed in 3.5.20?

Yuri Voinov


29.03.2017 5:55, L A Walsh пишет:

> Eliezer Croitoru wrote:
>> Hey Linda,
>>
>> As the pathcer\author of StoreID I will try to clarify what might
>> seems odd.
>> StoreID is a "static" rule which is one of the squid cache fundamentals.
>> The feature is the option to tweak this internal cache object ID.
>> This is a very static feature and will not be changed for a very long
>> time from now on.
>> Most of the public helpers I have seen are very "simple" and rely on
>> very simple things.
> ----
>     Makes sense, otherwise too prone to breakage.
Don't think so. More effective (and complex) solutions just not free
(and, of course, open-source). I don't think that C++ code is easy to
breakage. :)

>
>
>
>>
>> But since many systems these days are aware of the option to predict
>> what would be the next url(think about an exe or other binary that
>> can be replaced on-the-fly\in-transit) the developers changed and
>> change(like a Diffe Hellman) their way of transporting content to the
>> end client.
>> Due to this "Diffe Hellman" feature that many added it makes the more
>> simple scripts useless since the developers became much smarter.
> ----
>     yeah, my use case was fairly simple -- same
> person w/same browser, watching same vid a 2nd time.
>
>     They gave me many "kudos" and noticed that
> youtube was noticeably faster to browse through when I
> implemented the SSL interception on the squid proxy
> that web traffic goes through.  In that case, it was mainly
> the caching of the video-thumbs that noticeably sped up
> moving through YT pages.
YT now is very different thing. As I told much times, YT is actively
opposite caching (to force ISP uses Google Cache) by shuffling
underlying CDN URLs and/or encrypts parts of URL. So, it is very
difficult to make YT videos cacheable in runtime this time. Just
possible to cache static YT pages (and, so, not at all).

>
>> Indeed you will see TCP_MISS and it won't cache but this is only
>> since the admin might have the illusion that an encrypted content can
>> be predicted when a Diffie  Helman cryptography is creating the plain
>> url's these days.
> ----
>     Oh yeah.  Have noted that there are an infinite number
> of ways to access the same URL and have thought about ways I might
> collapse them to 1 URL, but it's just idle thinking as
> other things on the plate.
>
>     One good idea that didn't get updated was an
> extension in FF, that tried to store some of the latest
> Javascript libs that sites used so if they asked for the lib
> from a common site (like jquery), it might return the result
> from a local cache.
> It wouldn't help for those sites that merge
> multiple JS files and minify them.
>
> But many sites have 15-20 different websites that are "included" to
> get different elements (fonts, stylesheets,
> JS libs, etc) from different sources.  They seem to
In this case Store ID is useful. You simple write regex which combines
this several URLs to one.

> include URL's like a developer would use
> #include files...(and often take forever to load).
>
> multiple elements from different URLs like they would
> use multiple header include files in a local compilation.
>
>
>> Hope It helped,
>> Eliezer
>
> Thanks for the explanation, certainly more useful
> than just telling someone:
>
> "the web broke it"... :-)
It is hardly an explanation that will help to solve a specific question.
It takes some effort, often very big. And - yes, the web actively
opposes caching, or at least does not worry about caching at all. Is it
any wonder that those who could solve this problem not hurry to spread
the results of hard works free of charge around the world? :)
>
>
>
>
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Loading...