simultanous requests: collapsed_forwarding

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

simultanous requests: collapsed_forwarding

fansari
In our scenario it might/will happen that clients will request the same
resources simultaneously.

I ran a test where I start one download with curl and with short delay
(about 1s) I request the same content by another client.

1567673769.781  12123 xxx.xxx.0.1 TCP_MISS/200 15553655 GET
https://xxx/1.mp4 - HIER_NONE/- video/mp4
1567673769.782  13992 xxx.xxx.0.239 TCP_MISS/200 15553655 GET
https://xxx/1.mp4 - HIER_DIRECT/xxx.xxx.xxx.xxx video/mp4

From the documentation I found that there is a setting called
"collapsed_forwarding" which when enabled gives the behaviour we want to
achieve: let the first request finish with download and then handle the
others from squid cash.

Since our scenario is not time critical this would be the behaviour we want.

I am just wondering about one thing: I did not enable this setting (I also
tried to set it to "off" explicitly) and nevertheless it already behaves
this way.

When I interpret the logs correctly the first line indicates a waiting and
the second shows the download.

With "collapsed_forwarding on" it behaves in the same way as with "off". To
me it looks like as if the collapsing is used regardless whether it is
enabled or not.




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: simultanous requests: collapsed_forwarding

Amos Jeffries
Administrator
On 5/09/19 9:52 pm, fansari wrote:

> In our scenario it might/will happen that clients will request the same
> resources simultaneously.
>
> I ran a test where I start one download with curl and with short delay
> (about 1s) I request the same content by another client.
>
> 1567673769.781  12123 xxx.xxx.0.1 TCP_MISS/200 15553655 GET
> https://xxx/1.mp4 - HIER_NONE/- video/mp4
> 1567673769.782  13992 xxx.xxx.0.239 TCP_MISS/200 15553655 GET
> https://xxx/1.mp4 - HIER_DIRECT/xxx.xxx.xxx.xxx video/mp4
>
> From the documentation I found that there is a setting called
> "collapsed_forwarding" which when enabled gives the behaviour we want to
> achieve: let the first request finish with download and then handle the
> others from squid cash.

Not quite, collapsing does not let the first request complete. It starts
sending to all waiting clients as soon as it has response data to
deliver them.


>
> Since our scenario is not time critical this would be the behaviour we want.
>
> I am just wondering about one thing: I did not enable this setting (I also
> tried to set it to "off" explicitly) and nevertheless it already behaves
> this way.
>
> When I interpret the logs correctly the first line indicates a waiting and
> the second shows the download.

I agree, that is what it looks like. The request that finished+logged
first (by 0.001 seconds) took 0.2 seconds less time and did not involve
server contact (HIER_NONE), yet was a "MISS".

>
> With "collapsed_forwarding on" it behaves in the same way as with "off". To
> me it looks like as if the collapsing is used regardless whether it is
> enabled or not.
>

Looks to me like you may have found a bug. What exact version of Squid
are you using? (output of "squid -v" please).

If you are using an older Squid can you try the latest v4 and/or
development version to see if the odd behaviour still happens?

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: simultanous requests: collapsed_forwarding

fansari
I have compiled squid 3.5.23 on a Debian Stretch (because I need SSL).

This means i could enable the collapsed_forwarding feature because this is
what we need (we don't have a case where we need to disable it - this could
be interesting if you have a scenario where speed is more important than
traffic limitation).

squid -v gives this output:

Squid Cache: Version 3.5.23
Service Name: squid
Raspbian linux
configure options:  '--build=arm-linux-gnueabihf' '--prefix=/usr'
'--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
'--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
'--libexecdir=${prefix}/lib/squid3' '--srcdir=.' '--disable-maintainer-mode'
'--disable-dependency-tracking' '--disable-silent-rules' 'BUILDCXXFLAGS=-g
-O2 -fdebug-prefix-map=/home/pi/squid3-3.5.23=. -fstack-protector-strong
-Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
-Wl,-z,relro -Wl,-z,now -Wl,--as-needed' '--datadir=/usr/share/squid'
'--sysconfdir=/etc/squid' '--libexecdir=/usr/lib/squid'
'--mandir=/usr/share/man' '--enable-inline' '--disable-arch-native'
'--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock'
'--enable-removal-policies=lru,heap' '--enable-delay-pools'
'--enable-cache-digests' '--enable-icap-client'
'--enable-follow-x-forwarded-for'
'--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB'
'--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper'
'--enable-auth-ntlm=fake,smb_lm'
'--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group'
'--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi'
'--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation'
'--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid'
'--with-pidfile=/var/run/squid.pid' '--with-filedescriptors=65536'
'--with-large-files' '--with-default-user=proxy' '--enable-ssl'
'--enable-ssl-crtd' '--with-openssl' '--enable-build-info=Raspbian linux'
'--enable-linux-netfilter' 'build_alias=arm-linux-gnueabihf' 'CFLAGS=-g -O2
-fdebug-prefix-map=/home/pi/squid3-3.5.23=. -fstack-protector-strong
-Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now
-Wl,--as-needed' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2
-fdebug-prefix-map=/home/pi/squid3-3.5.23=. -fstack-protector-strong
-Wformat -Werror=format-security'





--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: simultanous requests: collapsed_forwarding

Matus UHLAR - fantomas
On 05.09.19 06:46, fansari wrote:
>I have compiled squid 3.5.23 on a Debian Stretch (because I need SSL).
>
>This means i could enable the collapsed_forwarding feature because this is
>what we need (we don't have a case where we need to disable it - this could
>be interesting if you have a scenario where speed is more important than
>traffic limitation).

in fact, you can use collapsed forwarding with stock debian 9 squid.

However, I miss your question here.

>
>squid -v gives this output:
>
>Squid Cache: Version 3.5.23
>Service Name: squid
>Raspbian linux

doesn't look like debian, does it? :)
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety. -- Benjamin Franklin, 1759
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: simultanous requests: collapsed_forwarding

Alex Rousskov
In reply to this post by fansari
On 9/5/19 5:52 AM, fansari wrote:

> I ran a test where I start one download with curl and with short delay
> (about 1s) I request the same content by another client.
>
> 1567673769.781 12123 xxx.xxx.0.1   TCP_MISS/200 15553655 ... HIER_NONE
> 1567673769.782 13992 xxx.xxx.0.239 TCP_MISS/200 15553655 ... HIER_DIRECT
>
> From the documentation I found that there is a setting called
> "collapsed_forwarding" which when enabled gives the behaviour we want to
> achieve: let the first request finish with download and then handle the
> others from squid cash.

Regardless of the collapsed_forwarding setting, the second request never
waits for the first request to finish downloading the response. Instead:

* When collapsed forwarding is prohibited: There is no waiting at all.

* When collapsed forwarding is allowed: The second request waits until
the response headers arrive.


> I am just wondering about one thing: I did not enable this setting (I also
> tried to set it to "off" explicitly) and nevertheless it already behaves
> this way.

You cannot make that conclusion by examining v3-4 access log records.
Latest (unreleased) Squids mark collapsed requests using a CF tag (e.g.
TCP_CF_MISS): https://github.com/squid-cache/squid/commit/d2a6dcb


> When I interpret the logs correctly the first line indicates a waiting and
> the second shows the download.

Not necessarily:

1. The first access.log record could be a cache hit (by the second
transaction) of an in-progress download initiated by the first
transaction. You build Squid with --enable-delay-pools which marks such
transactions as misses after bug 1001 changes:
https://bugs.squid-cache.org/show_bug.cgi?id=1001

2. The second access.log record can be a regular cache miss (by the
first transaction). It just happened to end a bit later so it was logged
second. You can check request:record mapping by looking at logged client
IPs.



> With "collapsed_forwarding on" it behaves in the same way as with "off". To
> me it looks like as if the collapsing is used regardless whether it is
> enabled or not.

Yes, but probably because there are no collapsable (in Squid
terminology) requests in your tests.

To test collapsed forwarding, you need Squid to receive the second
request _before_ Squid receives the response headers for the first request.


HTH,

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users