a bit off topic. New user question

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

a bit off topic. New user question

George Diaz

Hi

sorry this off-topic question ... 

I want pre-cache some object from some interest host with wget. 

My question is : I want: the wget download the object to the /dev/null but 
I'm not found this switches.... 
(GNU Wget 1.5.3) 

I'm probe this : 
export http_proxy=http://mycache.com:8080/ 
wget -r http://sobredinero.com -P /dev/null -nH -nd -Y on -b -l5 -t1 -o /dev/null 
  
but this is create a /dev/null directory :) and download the files into this. 

any suggestions ? 

Thanks in advance. 

George


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: a bit off topic. New user question

Amos Jeffries
Administrator
On 23/05/17 11:39, George Diaz wrote:

>
> Hi
>
> sorry this off-topic question ...
>
> I want pre-cache some object from some interest host with wget.
>
> My question is : I want: the wget download the object to the /dev/null
> but
> I'm not found this switches....
> (GNU Wget 1.5.3)
>
> I'm probe this :
> export http_proxy=http://mycache.com:8080/
> wget -r http://sobredinero.com -P /dev/null -nH -nd -Y on -b -l5 -t1
> -o /dev/null
>
> but this is create a /dev/null directory :) and download the files
> into this.
>
> any suggestions ?
>

Some advice before you get too far into this project;

  Pre-caching was an good idea back in the days of HTTP/1.0 and static
websites where the URL was all that mattered. In todays HTTP/1.1 and
HTTP/2 world dynamic content and variants are much more common things,
and both make pre-caching pretty much useless.

Before you attempt it for any domain I recommend passing a few of its
URLs through the tool at <https://redbot.org>. If that tool indicates
the site uses content negotiation or conditional HTTP features then
pre-caching is just going to be causing problems.

For example; that sobredinero domain above produces these details:


      Content Negotiation

  * The response body is different when content negotiation happens.


      Caching

  * Vary: User-Agent can cause cache inefficiency.


This means that anything you pre-cache with wget will be ignored and
probably replaced when any non-wget agent (ie a browser) is used to
fetch through the proxy. So you just waste all the bandwidth, time, and
storage space used pre-caching it.


Vary:User-Agent is particularly bad since any single character
difference in the User-Agent header will cause a different object to be
referred to in the cache storage. If you wish to pre-cache these objects
in any useful way you have to know and mimic the *exact* User-Agent
header values that will be used to fetch it. For example; two different
version of Chrome -> different User-Agent header. Internet Explorer with
different Windows Updates applied -> different User-Agent header. As you
can imagine that is a very hard thing to predict.


Note: if you have come to this idea after seeing objects from that
domain getting a lot of MISS records, the problem is very much that Vary
header causing so many different objects to be needed that objects being
stored are often not the right one(s) for any later client request.
pre-caching will not solve this but make it worse as wget is just
another different User-Agent.


Amos


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: a bit off topic. New user question

Yuri Voinov
In reply to this post by George Diaz

       --delete-after              delete files locally after downloading them

I know this by simple execution wget --help :-D

Feel free to do RTFM :-D


23.05.2017 5:39, George Diaz пишет:

Hi

sorry this off-topic question ... 

I want pre-cache some object from some interest host with wget. 

My question is : I want: the wget download the object to the /dev/null but 
I'm not found this switches.... 
(GNU Wget 1.5.3) 

I'm probe this : 
export http_proxy=http://mycache.com:8080/ 
wget -r http://sobredinero.com -P /dev/null -nH -nd -Y on -b -l5 -t1 -o /dev/null 
  
but this is create a /dev/null directory :) and download the files into this. 

any suggestions ? 

Thanks in advance. 

George



_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: a bit off topic. New user question

726mediallc
In reply to this post by George Diaz
Reply | Threaded
Open this post in threaded view
|

Re: a bit off topic. New user question

726mediallc
In reply to this post by George Diaz