Quantcast

Squid with download manager

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Squid with download manager

Mohsen Saeedi
Hello all

I had very experience with squid caching server for 8 years. but i
have a question. now i'm using squid 3.1 on the RHEL under 100Mbit/s
bandwidth for large university. it's great in performance. but how can
i cache some content is downloading with some download manager such as
IDman or some other.
I think squid couldn't able to cache file has been downloaded with
download manager application.

anyone can help me?

thank.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Amos Jeffries-2
On 07/01/11 23:00, Mohsen Saeedi wrote:

> Hello all
>
> I had very experience with squid caching server for 8 years. but i
> have a question. now i'm using squid 3.1 on the RHEL under 100Mbit/s
> bandwidth for large university. it's great in performance. but how can
> i cache some content is downloading with some download manager such as
> IDman or some other.
> I think squid couldn't able to cache file has been downloaded with
> download manager application.
>

Whether the object is cached by Squid depends on the headers the app and
the server send out. Usually they allow caching.
The problem is more likely that the download app is fetching multiple
ranges simultaneously, which is a problem with Squid which may try to
download the whole file for each part so that it can send cached copies
in future. Squid-3 does not yet do collapsed forwarding which hid the
problem in Squid-2.

We will need details of the problem you are having in order to help.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.10
   Beta testers wanted for 3.2.0.4
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Mohsen Saeedi
Dear Amos
I'm mohsen saeedi as translator of squid for persian language. We chat
on squid IRC with each other many times already have.
I want to cache some files are being downloaded with download managers
( for example IDMan. ) as you know download manager split file to
multiple part for accelerate with simultaneously connection to http
server. how we can cache files when they are being downloaded with
squid.

thanks.

On Fri, Jan 7, 2011 at 2:34 PM, Amos Jeffries <[hidden email]> wrote:

> On 07/01/11 23:00, Mohsen Saeedi wrote:
>>
>> Hello all
>>
>> I had very experience with squid caching server for 8 years. but i
>> have a question. now i'm using squid 3.1 on the RHEL under 100Mbit/s
>> bandwidth for large university. it's great in performance. but how can
>> i cache some content is downloading with some download manager such as
>> IDman or some other.
>> I think squid couldn't able to cache file has been downloaded with
>> download manager application.
>>
>
> Whether the object is cached by Squid depends on the headers the app and the
> server send out. Usually they allow caching.
> The problem is more likely that the download app is fetching multiple ranges
> simultaneously, which is a problem with Squid which may try to download the
> whole file for each part so that it can send cached copies in future.
> Squid-3 does not yet do collapsed forwarding which hid the problem in
> Squid-2.
>
> We will need details of the problem you are having in order to help.
>
> Amos
> --
> Please be using
>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>  Beta testers wanted for 3.2.0.4
>



--
Seyyed Mohsen Saeedi
سید محسن سعیدی
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Amos Jeffries-2
On 08/01/11 02:20, Mohsen Saeedi wrote:

> Dear Amos
> I'm mohsen saeedi as translator of squid for persian language. We chat
> on squid IRC with each other many times already have.
> I want to cache some files are being downloaded with download managers
> ( for example IDMan. ) as you know download manager split file to
> multiple part for accelerate with simultaneously connection to http
> server. how we can cache files when they are being downloaded with
> squid.
>
> thanks.

You will have to play around with the quick_abort_* and range_offset
directives to force-download a while file when a range is requested.
That way the file gets cached and later requests get the cached version.

This may need to be combined with maxconn ACL to limit the number of
simultaneous connections from each client to prevent bandwidth waste.

Amos

>
> On Fri, Jan 7, 2011 at 2:34 PM, Amos Jeffries<[hidden email]>  wrote:
>> On 07/01/11 23:00, Mohsen Saeedi wrote:
>>>
>>> Hello all
>>>
>>> I had very experience with squid caching server for 8 years. but i
>>> have a question. now i'm using squid 3.1 on the RHEL under 100Mbit/s
>>> bandwidth for large university. it's great in performance. but how can
>>> i cache some content is downloading with some download manager such as
>>> IDman or some other.
>>> I think squid couldn't able to cache file has been downloaded with
>>> download manager application.
>>>
>>
>> Whether the object is cached by Squid depends on the headers the app and the
>> server send out. Usually they allow caching.
>> The problem is more likely that the download app is fetching multiple ranges
>> simultaneously, which is a problem with Squid which may try to download the
>> whole file for each part so that it can send cached copies in future.
>> Squid-3 does not yet do collapsed forwarding which hid the problem in
>> Squid-2.
>>
>> We will need details of the problem you are having in order to help.
>>
>> Amos
>> --
>> Please be using
>>   Current Stable Squid 2.7.STABLE9 or 3.1.10
>>   Beta testers wanted for 3.2.0.4
>>
>
>
>


--
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.10
   Beta testers wanted for 3.2.0.4
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Mohsen Saeedi
Thank.
but which value should be set for quick_abort?? i don't know relation
between quick_abort and download manager.
can you explain it or give me some useful links?
now i'm reading about range_offset_limit.

> On Fri, Jan 7, 2011 at 6:11 PM, Amos Jeffries <[hidden email]> wrote:
>>
>> On 08/01/11 02:20, Mohsen Saeedi wrote:
>>>
>>> Dear Amos
>>> I'm mohsen saeedi as translator of squid for persian language. We chat
>>> on squid IRC with each other many times already have.
>>> I want to cache some files are being downloaded with download managers
>>> ( for example IDMan. ) as you know download manager split file to
>>> multiple part for accelerate with simultaneously connection to http
>>> server. how we can cache files when they are being downloaded with
>>> squid.
>>>
>>> thanks.
>>
>> You will have to play around with the quick_abort_* and range_offset directives to force-download a while file when a range is requested. That way the file gets cached and later requests get the cached version.
>>
>> This may need to be combined with maxconn ACL to limit the number of simultaneous connections from each client to prevent bandwidth waste.
>>
>> Amos
>>
>>>
>>> On Fri, Jan 7, 2011 at 2:34 PM, Amos Jeffries<[hidden email]>  wrote:
>>>>
>>>> On 07/01/11 23:00, Mohsen Saeedi wrote:
>>>>>
>>>>> Hello all
>>>>>
>>>>> I had very experience with squid caching server for 8 years. but i
>>>>> have a question. now i'm using squid 3.1 on the RHEL under 100Mbit/s
>>>>> bandwidth for large university. it's great in performance. but how can
>>>>> i cache some content is downloading with some download manager such as
>>>>> IDman or some other.
>>>>> I think squid couldn't able to cache file has been downloaded with
>>>>> download manager application.
>>>>>
>>>>
>>>> Whether the object is cached by Squid depends on the headers the app and the
>>>> server send out. Usually they allow caching.
>>>> The problem is more likely that the download app is fetching multiple ranges
>>>> simultaneously, which is a problem with Squid which may try to download the
>>>> whole file for each part so that it can send cached copies in future.
>>>> Squid-3 does not yet do collapsed forwarding which hid the problem in
>>>> Squid-2.
>>>>
>>>> We will need details of the problem you are having in order to help.
>>>>
>>>> Amos
>>>> --
>>>> Please be using
>>>>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>>>>  Beta testers wanted for 3.2.0.4
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> Please be using
>>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>>  Beta testers wanted for 3.2.0.4
>
>
>
> --
> Seyyed Mohsen Saeedi
> سید محسن سعیدی
>



--
Seyyed Mohsen Saeedi
سید محسن سعیدی
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Amos Jeffries-2
On 08/01/11 03:51, Mohsen Saeedi wrote:
> Thank.
> but which value should be set for quick_abort?? i don't know relation
> between quick_abort and download manager.
> can you explain it or give me some useful links?

Abort needs to be disabled. Range offset needs to force full-download.

> now i'm reading about range_offset_limit.
>

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.10
   Beta testers wanted for 3.2.0.4
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Mohsen Saeedi
Do you mean range_offset_limit -1 and quick_abort_* 0 ?
that's true?


On Fri, Jan 7, 2011 at 7:19 PM, Amos Jeffries <[hidden email]> wrote:

> On 08/01/11 03:51, Mohsen Saeedi wrote:
>>
>> Thank.
>> but which value should be set for quick_abort?? i don't know relation
>> between quick_abort and download manager.
>> can you explain it or give me some useful links?
>
> Abort needs to be disabled. Range offset needs to force full-download.
>
>> now i'm reading about range_offset_limit.
>>
>
> Amos
> --
> Please be using
>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>  Beta testers wanted for 3.2.0.4
>



--
Seyyed Mohsen Saeedi
سید محسن سعیدی
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Mohsen Saeedi
Amos i set these:
quick_abort_min -1
quick_abort_max -1
range_offset_limit -1
but when clients are being downloaded, it's very slow. about 14KB/s
but normaly they can fill 256KB/s.
is anything wrong?

On Fri, Jan 7, 2011 at 7:25 PM, Mohsen Saeedi <[hidden email]> wrote:

> Do you mean range_offset_limit -1 and quick_abort_* 0 ?
> that's true?
>
>
> On Fri, Jan 7, 2011 at 7:19 PM, Amos Jeffries <[hidden email]> wrote:
>> On 08/01/11 03:51, Mohsen Saeedi wrote:
>>>
>>> Thank.
>>> but which value should be set for quick_abort?? i don't know relation
>>> between quick_abort and download manager.
>>> can you explain it or give me some useful links?
>>
>> Abort needs to be disabled. Range offset needs to force full-download.
>>
>>> now i'm reading about range_offset_limit.
>>>
>>
>> Amos
>> --
>> Please be using
>>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>>  Beta testers wanted for 3.2.0.4
>>
>
>
>
> --
> Seyyed Mohsen Saeedi
> سید محسن سعیدی
>



--
Seyyed Mohsen Saeedi
سید محسن سعیدی
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Hasanen AL-Bana
In reply to this post by Amos Jeffries-2
This will cause a bigger problem , if user downloading a file with
download manager ,let's say in 4 segments , squid will start 4
download threads for the same file each one from its beginning. will
consume 4 times bandwidth than really needed.

On Fri, Jan 7, 2011 at 6:49 PM, Amos Jeffries <[hidden email]> wrote:

> On 08/01/11 03:51, Mohsen Saeedi wrote:
>>
>> Thank.
>> but which value should be set for quick_abort?? i don't know relation
>> between quick_abort and download manager.
>> can you explain it or give me some useful links?
>
> Abort needs to be disabled. Range offset needs to force full-download.
>
>> now i'm reading about range_offset_limit.
>>
>
> Amos
> --
> Please be using
>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>  Beta testers wanted for 3.2.0.4
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Mohsen Saeedi
Dea Hasanen
which setting is better than for it? can you give me some helps?

On Fri, Jan 7, 2011 at 7:36 PM, Hasanen AL-Bana <[hidden email]> wrote:

> This will cause a bigger problem , if user downloading a file with
> download manager ,let's say in 4 segments , squid will start 4
> download threads for the same file each one from its beginning. will
> consume 4 times bandwidth than really needed.
>
> On Fri, Jan 7, 2011 at 6:49 PM, Amos Jeffries <[hidden email]> wrote:
>> On 08/01/11 03:51, Mohsen Saeedi wrote:
>>>
>>> Thank.
>>> but which value should be set for quick_abort?? i don't know relation
>>> between quick_abort and download manager.
>>> can you explain it or give me some useful links?
>>
>> Abort needs to be disabled. Range offset needs to force full-download.
>>
>>> now i'm reading about range_offset_limit.
>>>
>>
>> Amos
>> --
>> Please be using
>>  Current Stable Squid 2.7.STABLE9 or 3.1.10
>>  Beta testers wanted for 3.2.0.4
>>
>



--
Seyyed Mohsen Saeedi
سید محسن سعیدی
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Mohsen Saeedi
Hi
another problem is when some users try to download file without
download manager application. they rate is very slow. for example with
download manager you can download about 120KB/s and without it speed
reduce to 10KB/s. how can i configure squid to solve this problem?
Thanks

On Fri, Jan 7, 2011 at 7:39 PM, Mohsen Saeedi <[hidden email]> wrote:

>
> Dea Hasanen
> which setting is better than for it? can you give me some helps?
>
> On Fri, Jan 7, 2011 at 7:36 PM, Hasanen AL-Bana <[hidden email]> wrote:
> > This will cause a bigger problem , if user downloading a file with
> > download manager ,let's say in 4 segments , squid will start 4
> > download threads for the same file each one from its beginning. will
> > consume 4 times bandwidth than really needed.
> >
> > On Fri, Jan 7, 2011 at 6:49 PM, Amos Jeffries <[hidden email]> wrote:
> >> On 08/01/11 03:51, Mohsen Saeedi wrote:
> >>>
> >>> Thank.
> >>> but which value should be set for quick_abort?? i don't know relation
> >>> between quick_abort and download manager.
> >>> can you explain it or give me some useful links?
> >>
> >> Abort needs to be disabled. Range offset needs to force full-download.
> >>
> >>> now i'm reading about range_offset_limit.
> >>>
> >>
> >> Amos
> >> --
> >> Please be using
> >>  Current Stable Squid 2.7.STABLE9 or 3.1.10
> >>  Beta testers wanted for 3.2.0.4
> >>
> >
>
>
>
> --
> Seyyed Mohsen Saeedi
> سید محسن سعیدی



--
Seyyed Mohsen Saeedi
سید محسن سعیدی
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid with download manager

Eliezer Croitoru
On 01/05/2011 01:14, Mohsen Saeedi wrote:



> Hi
> another problem is when some users try to download file without
> download manager application. they rate is very slow. for example with
> download manager you can download about 120KB/s and without it speed
> reduce to 10KB/s. how can i configure squid to solve this problem?
> Thanks
>

well it's not always squid fault..
if the server limits the connection speed or there is some other problem
or other infrastructure limits on your speed that not on squid fault...
you can't do anything else then using another ISP or any other
infrastructure element that causes the speed problem.
keep in mind that it can be the web server itself.
HP web site was one of the slowest ever for downloads (i dont know what
today).
people all over the world was downloading from HP download servers 15KB avg.
a client that was one isp hop from the servers got it in 15KB.
in my country the opensource local mirrors are now downloading files on
23KB because of mass update of firefox and ubuntu users.
so for now im using france mirrors and it's almost 3-4 Mbps.

so make sure that squid is not the one at low speed fault and after you
will find the cause start searching for a solution.

Regards
Eliezer


> On Fri, Jan 7, 2011 at 7:39 PM, Mohsen Saeedi<[hidden email]>  wrote:
>> Dea Hasanen
>> which setting is better than for it? can you give me some helps?
>>
>> On Fri, Jan 7, 2011 at 7:36 PM, Hasanen AL-Bana<[hidden email]>  wrote:
>>> This will cause a bigger problem , if user downloading a file with
>>> download manager ,let's say in 4 segments , squid will start 4
>>> download threads for the same file each one from its beginning. will
>>> consume 4 times bandwidth than really needed.
>>>
>>> On Fri, Jan 7, 2011 at 6:49 PM, Amos Jeffries<[hidden email]>  wrote:
>>>> On 08/01/11 03:51, Mohsen Saeedi wrote:
>>>>> Thank.
>>>>> but which value should be set for quick_abort?? i don't know relation
>>>>> between quick_abort and download manager.
>>>>> can you explain it or give me some useful links?
>>>> Abort needs to be disabled. Range offset needs to force full-download.
>>>>
>>>>> now i'm reading about range_offset_limit.
>>>>>
>>>> Amos
>>>> --
>>>> Please be using
>>>>   Current Stable Squid 2.7.STABLE9 or 3.1.10
>>>>   Beta testers wanted for 3.2.0.4
>>>>
>>
>>
>> --
>> Seyyed Mohsen Saeedi
>> سید محسن سعیدی
>
>
> --
> Seyyed Mohsen Saeedi
> سید محسن سعیدی

Loading...