Problems with Internet Download Manager and Squid 2.7 Stable 9

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems with Internet Download Manager and Squid 2.7 Stable 9

Saiful Alam

Hi,
I am using Squid 2.7 Stable on Ubuntu 10.10 x64

Files like
 mp3 which have a refresh_pattern defined, and downloaded within the
browser download manager is cached, but if I download the file with
Internet Download Manager 6.05, the file is not cached. Note that IDM by
 default uses 8 connections to download a single file. Again, if I
reduce the default connection number to 1, and try to download again
with IDM, then the file is cached instantly.

Most users in our
network have IDM as their primary download manager, and if we can't
cache objects downloaded with IDM, then :((((((((((((((((

Any help or ideas are welcomed.

Regards,
Saiful    
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Internet Download Manager and Squid 2.7 Stable 9

Amos Jeffries
Administrator
On 19.01.2012 07:16, Saiful Alam wrote:

> Hi,
> I am using Squid 2.7 Stable on Ubuntu 10.10 x64
>
> Files like
>  mp3 which have a refresh_pattern defined, and downloaded within the
> browser download manager is cached, but if I download the file with
> Internet Download Manager 6.05, the file is not cached. Note that IDM
> by
>  default uses 8 connections to download a single file. Again, if I
> reduce the default connection number to 1, and try to download again
> with IDM, then the file is cached instantly.
>

"cached instantly" is a bit of a strange description. Surely it
requires download time before caching? or did you means something else
entirely?


Consider what that download manager is doing. Splitting the file into
8+ pieces and requesting each of those pieces as different HTTP
requests. Squid cannot cache partial files, only whole files.


You can check for the download manager User-Agent value with a
"browser" type ACL and also a "maxconn" type ACL to block more than 1
connection at a time by it.
  The range_offset_limit and abort settings will also help with these
partial file requests. Their use is best known for WU, but applies to
any big partial file. http://wiki.squid-cache.org/SquidFaq/WindowsUpdate


Alternatively you can balance the users mistaken perception of DL
manager benefit with delay_pools which use the "browser" ACL type and
slow its rate of download so that users see it as worse than normal
traffic when they use it. Migration away from the manager

It is polite to inform your users about the proxy and the effect these
managers are having before going to such extremes. If you can make them
understand that they get *faster* downloads by working with the proxy
cache sharing. You can perhapse advise alternative methods of getting
fast download at the same time to encourage the change. see below.


> Most users in our
> network have IDM as their primary download manager, and if we can't
> cache objects downloaded with IDM, then :((((((((((((((((
>

This is a strong sign that they perceive the multiple connections the
manager provides as a faster network connection than the proxied
traffic. I've usually seen this sort of perception growing out of the
old browsers limitation of only opening ~2 connections to a proxy, which
makes things appear really slow when big objects are filling one of the
connections.
  There is a "connections to server" setting in browsers which can be
raised to 8-10 to double or quadruple the bandwidth availability for
each user without needing a manager to do it specifically. NP: be
careful you have enough FD available on the Squid before letting them
know about that.

Amos
Reply | Threaded
Open this post in threaded view
|

Re: Problems with Internet Download Manager and Squid 2.7 Stable 9

Matus UHLAR - fantomas
In reply to this post by Saiful Alam
On 19.01.12 05:16, Saiful Alam wrote:

>I am using Squid 2.7 Stable on Ubuntu 10.10 x64
>
>Files like
> mp3 which have a refresh_pattern defined, and downloaded within the
>browser download manager is cached, but if I download the file with
>Internet Download Manager 6.05, the file is not cached. Note that IDM by
> default uses 8 connections to download a single file. Again, if I
>reduce the default connection number to 1, and try to download again
>with IDM, then the file is cached instantly.
>
>Most users in our
>network have IDM as their primary download manager, and if we can't
>cache objects downloaded with IDM, then :((((((((((((((((

I'm afraid this is due to suid not being to cache partial files at this
time. Other problem can be causes by current squid not able to do
collapsed forwarding (fetch the same file only once even when there's
multiple requests for that) which is related to this problem and afaik
in progress for 3.2.

Download manager that download multiple chunks in parallel encounter
this behaviour and often cause data not being cached.

You can configure squid to work around this behaviour by configuring
quick_abort_* values to fetch while files even if it's not requested in
hope that it will be requested later.

Problems are in cases like windows updates that are configured in large
files of which only small parts are needed, so you would fetch data not
needed (you can avoid this by having local WSUS server).

Other problems can be download managers fetching the same file from
multiple sources, but that's again problem with download managers.

Note that most download managers open multiple connections to get more
of bandwidth, which results in keeping less of the bandwidth to others.

That results in increased overhead (multiple connections, ack packets,
etc) and when more people use download managers, they only cause more
overhead flow through youur company's line.

I hope that any sane network admin configures networks so users can not
steal bandwitdh from others, only from themselves.

The most sane solution to this problem is configure download managers
to fetch each file only once.
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Support bacteria - they're the only culture some people have.