Icap preview size

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Icap preview size

Niels Hofmans
Hi guys,

One more: I believe ICAP is not respecting the Preview header for REQMOD nor RESPMOD.

For the REQMOD OPTIONS requests, I respond with:

ICAP/1.0 200 OK
Allow: 200,204
Connection: close
Date: Fri, 05 Mar 2021 07:34:56 GMT
Encapsulated: null-body=0
Methods: REQMOD
Preview: 64000

However in my ICAP service, I often receive a request body over 64000 bytes, e.g. 1000000 request bytes.

I also noticed that a "Preview: 0” will not have any effect and the ICAP service will still receive a complete HTTP REQMOD+REQRESP body.
Any idea if I’m doing something wrong here?

icap_enable on
icap_connect_timeout 1 second
icap_service_failure_limit -1
icap_send_client_ip on
icap_preview_size 0
icap_persistent_connections on
icap_service service_req reqmod_precache bypass=0 <a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request
icap_service service_res respmod_precache bypass=0 <a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response
icap_preview_enable on
adaptation_access service_req allow all
adaptation_access service_res allow all

Man thanks.

Regards,
Best regards
Niels Hofmans

SITE   https://ironpeak.be
BTW   BE0694785660
BANK BE76068909740795


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Alex Rousskov
On 3/5/21 2:55 AM, Niels Hofmans wrote:

> One more: I believe ICAP is not respecting the Preview header for REQMOD
> nor RESPMOD.

> For the REQMOD OPTIONS requests, I respond with:
>
> ICAP/1.0 200 OK
> Allow: 200,204
> Connection: close
> Date: Fri, 05 Mar 2021 07:34:56 GMT
> Encapsulated: null-body=0
> Methods: REQMOD
> Preview: 64000
>
> However in my ICAP service, I often receive a request body over 64000
> bytes, e.g. 1000000 request bytes.

Receive when? During preview, instead of Preview, and/or after Preview?


> I also noticed that a "Preview: 0” will not have any effect and the ICAP
> service will still receive a complete HTTP REQMOD+REQRESP body.

Receive when? During preview, instead of Preview, and/or after Preview?

Alex.



> icap_enable on
> icap_connect_timeout 1 second
> icap_service_failure_limit -1
> icap_send_client_ip    on
> icap_preview_size 0
> icap_persistent_connections on
> icap_service service_req reqmod_precache bypass=0 icap://172.17.0.2:1344/request <icap://172.17.0.2:1344/request>
> icap_service service_res respmod_precache bypass=0 icap://172.17.0.2:1344/response <icap://172.17.0.2:1344/response>
> icap_preview_enable on
> adaptation_access service_req allow all
> adaptation_access service_res allow all


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Niels Hofmans
Hi Alex,

I receive that large payload right after an OPTIONS call to my ICAP endpoint. It is the preview.

Regards,
Niels

Sent from my mobile

On 5 Mar 2021, at 17:21, Alex Rousskov <[hidden email]> wrote:

On 3/5/21 2:55 AM, Niels Hofmans wrote:

> One more: I believe ICAP is not respecting the Preview header for REQMOD
> nor RESPMOD.

> For the REQMOD OPTIONS requests, I respond with:
>
> ICAP/1.0 200 OK
> Allow: 200,204
> Connection: close
> Date: Fri, 05 Mar 2021 07:34:56 GMT
> Encapsulated: null-body=0
> Methods: REQMOD
> Preview: 64000
>
> However in my ICAP service, I often receive a request body over 64000
> bytes, e.g. 1000000 request bytes.

Receive when? During preview, instead of Preview, and/or after Preview?


> I also noticed that a "Preview: 0” will not have any effect and the ICAP
> service will still receive a complete HTTP REQMOD+REQRESP body.

Receive when? During preview, instead of Preview, and/or after Preview?

Alex.



> icap_enable on
> icap_connect_timeout 1 second
> icap_service_failure_limit -1
> icap_send_client_ip    on
> icap_preview_size 0
> icap_persistent_connections on
> icap_service service_req reqmod_precache bypass=0 icap://172.17.0.2:1344/request <icap://172.17.0.2:1344/request>
> icap_service service_res respmod_precache bypass=0 icap://172.17.0.2:1344/response <icap://172.17.0.2:1344/response>
> icap_preview_enable on
> adaptation_access service_req allow all
> adaptation_access service_res allow all


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Alex Rousskov
On 3/5/21 5:21 PM, Niels Hofmans wrote:

> I receive that large payload right after an OPTIONS call to my ICAP endpoint. It is the preview.

The timing of an ICAP request does not determine whether that request
has a Preview stage and whether the "unexpected" body bytes were sent
during that Preview stage. I am asking these questions because I want to
determine whether Squid increases Preview size beyond what your server
has requested or does not send Preview at all.

Please share the ICAP headers of the problematic REQMOD request and the
last chunk metadata of that request body. You can get the latter from a
packet capture if your ICAP server does not report it in a convenient
form. In fact, sharing (a pointer to) the packet capture of the whole
problematic ICAP request is probably a good idea!

Alex.



> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>
> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>
>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>> nor RESPMOD.
>
>> For the REQMOD OPTIONS requests, I respond with:
>>
>> ICAP/1.0 200 OK
>> Allow: 200,204
>> Connection: close
>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>> Encapsulated: null-body=0
>> Methods: REQMOD
>> Preview: 64000
>>
>> However in my ICAP service, I often receive a request body over 64000
>> bytes, e.g. 1000000 request bytes.
>
> Receive when? During preview, instead of Preview, and/or after Preview?
>
>
>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>> service will still receive a complete HTTP REQMOD+REQRESP body.
>
> Receive when? During preview, instead of Preview, and/or after Preview?
>
> Alex.
>
>
>
>> icap_enable on
>> icap_connect_timeout 1 second
>> icap_service_failure_limit -1
>> icap_send_client_ip    on
>> icap_preview_size 0
>> icap_persistent_connections on
>> icap_service service_req reqmod_precache bypass=0 icap://172.17.0.2:1344/request <icap://172.17.0.2:1344/request>
>> icap_service service_res respmod_precache bypass=0 icap://172.17.0.2:1344/response <icap://172.17.0.2:1344/response>
>> icap_preview_enable on
>> adaptation_access service_req allow all
>> adaptation_access service_res allow all
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Niels Hofmans
Hi Alex,

I’ve uploaded a tcpdump sample of where I’m requesting an image through squid + icap and where you can see the image response being posted fully to the ICAP service.
Headers passed on are Preview: 0 for RESPMOD and Transfer-Preview: *. (I only need the headers for the response)


Many thanks for co-debugging. :-)
Enjoy your weekend!

Regards,
Niels Hofmans

SITE   https://ironpeak.be
BTW   BE0694785660
BANK BE76068909740795

On 5 Mar 2021, at 23:32, Alex Rousskov <[hidden email]> wrote:

On 3/5/21 5:21 PM, Niels Hofmans wrote:

I receive that large payload right after an OPTIONS call to my ICAP endpoint. It is the preview.

The timing of an ICAP request does not determine whether that request
has a Preview stage and whether the "unexpected" body bytes were sent
during that Preview stage. I am asking these questions because I want to
determine whether Squid increases Preview size beyond what your server
has requested or does not send Preview at all.

Please share the ICAP headers of the problematic REQMOD request and the
last chunk metadata of that request body. You can get the latter from a
packet capture if your ICAP server does not report it in a convenient
form. In fact, sharing (a pointer to) the packet capture of the whole
problematic ICAP request is probably a good idea!

Alex.



On 5 Mar 2021, at 17:21, Alex Rousskov wrote:

On 3/5/21 2:55 AM, Niels Hofmans wrote:

One more: I believe ICAP is not respecting the Preview header for REQMOD
nor RESPMOD.

For the REQMOD OPTIONS requests, I respond with:

ICAP/1.0 200 OK
Allow: 200,204
Connection: close
Date: Fri, 05 Mar 2021 07:34:56 GMT
Encapsulated: null-body=0
Methods: REQMOD
Preview: 64000

However in my ICAP service, I often receive a request body over 64000
bytes, e.g. 1000000 request bytes.

Receive when? During preview, instead of Preview, and/or after Preview?


I also noticed that a "Preview: 0” will not have any effect and the ICAP
service will still receive a complete HTTP REQMOD+REQRESP body.

Receive when? During preview, instead of Preview, and/or after Preview?

Alex.



icap_enable on
icap_connect_timeout 1 second
icap_service_failure_limit -1
icap_send_client_ip    on
icap_preview_size 0
icap_persistent_connections on
icap_service service_req reqmod_precache bypass=0 <a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request <<a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request>
icap_service service_res respmod_precache bypass=0 <a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response <<a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response>
icap_preview_enable on
adaptation_access service_req allow all
adaptation_access service_res allow all




_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Alex Rousskov
On 3/6/21 5:09 AM, Niels Hofmans wrote:

> I’ve uploaded a tcpdump sample of where I’m requesting an image through
> squid + icap and where you can see the image response being posted fully
> to the ICAP service.

AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
zero-size Preview containing just the HTTP headers (e.g., see frame
#44). In one case, Squid also sends the HTTP response body but only
because your ICAP service explicitly requests that HTTP response body by
responding to Squid Preview with ICAP 100 Continue control message
(e.g., see frame #48)!

If your ICAP service does not want to see an HTTP body, then it should
not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
based on the Preview alone, without sending ICAP 100 Continue control
message first.


HTH,

Alex.


> On 5 Mar 2021, at 23:32, Alex Rousskov wrote:
>
> On 3/5/21 5:21 PM, Niels Hofmans wrote:
>
>> I receive that large payload right after an OPTIONS call to my ICAP
>> endpoint. It is the preview.
>
> The timing of an ICAP request does not determine whether that request
> has a Preview stage and whether the "unexpected" body bytes were sent
> during that Preview stage. I am asking these questions because I want to
> determine whether Squid increases Preview size beyond what your server
> has requested or does not send Preview at all.
>
> Please share the ICAP headers of the problematic REQMOD request and the
> last chunk metadata of that request body. You can get the latter from a
> packet capture if your ICAP server does not report it in a convenient
> form. In fact, sharing (a pointer to) the packet capture of the whole
> problematic ICAP request is probably a good idea!
>
> Alex.
>
>
>
>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>>
>> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>>
>>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>>> nor RESPMOD.
>>
>>> For the REQMOD OPTIONS requests, I respond with:
>>>
>>> ICAP/1.0 200 OK
>>> Allow: 200,204
>>> Connection: close
>>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>>> Encapsulated: null-body=0
>>> Methods: REQMOD
>>> Preview: 64000
>>>
>>> However in my ICAP service, I often receive a request body over 64000
>>> bytes, e.g. 1000000 request bytes.
>>
>> Receive when? During preview, instead of Preview, and/or after Preview?
>>
>>
>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>>> service will still receive a complete HTTP REQMOD+REQRESP body.
>>
>> Receive when? During preview, instead of Preview, and/or after Preview?
>>
>> Alex.
>>
>>
>>
>>> icap_enable on
>>> icap_connect_timeout 1 second
>>> icap_service_failure_limit -1
>>> icap_send_client_ip    on
>>> icap_preview_size 0
>>> icap_persistent_connections on
>>> icap_service service_req reqmod_precache bypass=0
>>> icap://172.17.0.2:1344/request <icap://172.17.0.2:1344/request>
>>> <icap://172.17.0.2:1344/request <icap://172.17.0.2:1344/request>>
>>> icap_service service_res respmod_precache bypass=0
>>> icap://172.17.0.2:1344/response <icap://172.17.0.2:1344/response>
>>> <icap://172.17.0.2:1344/response <icap://172.17.0.2:1344/response>>
>>> icap_preview_enable on
>>> adaptation_access service_req allow all
>>> adaptation_access service_res allow all
>>
>
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Niels Hofmans


On 6 Mar 2021, at 21:33, Niels Hofmans <[hidden email]> wrote:

Hi Alex,

I fixed a bug in the go-icap/icap library, seen here:
But this made me wonder about the following: https://i.imgur.com/bA3w8YT.png

Here you can see squid issuing a RESPMOD to my ICAP server with a Preview of “0”, but it does contain a HTTP response.
And since my ICAP service will not return a HTTP response since the Preview size is 0, my http client receives nothing from squid.
I checked but I don’t think I see a 100 continuation from ICAP in the dump anymore.

(I’ve revoked my session cookies for this dump)

One step at a time closer to the truth! :-)

Regards,
Niels Hofmans

SITE   https://ironpeak.be
BTW   BE0694785660
BANK BE76068909740795

On 6 Mar 2021, at 19:29, Alex Rousskov <[hidden email]> wrote:

On 3/6/21 5:09 AM, Niels Hofmans wrote:

I’ve uploaded a tcpdump sample of where I’m requesting an image through
squid + icap and where you can see the image response being posted fully
to the ICAP service.

AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
zero-size Preview containing just the HTTP headers (e.g., see frame
#44). In one case, Squid also sends the HTTP response body but only
because your ICAP service explicitly requests that HTTP response body by
responding to Squid Preview with ICAP 100 Continue control message
(e.g., see frame #48)!

If your ICAP service does not want to see an HTTP body, then it should
not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
based on the Preview alone, without sending ICAP 100 Continue control
message first.


HTH,

Alex.


On 5 Mar 2021, at 23:32, Alex Rousskov wrote:

On 3/5/21 5:21 PM, Niels Hofmans wrote:

I receive that large payload right after an OPTIONS call to my ICAP
endpoint. It is the preview.

The timing of an ICAP request does not determine whether that request
has a Preview stage and whether the "unexpected" body bytes were sent
during that Preview stage. I am asking these questions because I want to
determine whether Squid increases Preview size beyond what your server
has requested or does not send Preview at all.

Please share the ICAP headers of the problematic REQMOD request and the
last chunk metadata of that request body. You can get the latter from a
packet capture if your ICAP server does not report it in a convenient
form. In fact, sharing (a pointer to) the packet capture of the whole
problematic ICAP request is probably a good idea!

Alex.



On 5 Mar 2021, at 17:21, Alex Rousskov wrote:

On 3/5/21 2:55 AM, Niels Hofmans wrote:

One more: I believe ICAP is not respecting the Preview header for REQMOD
nor RESPMOD.

For the REQMOD OPTIONS requests, I respond with:

ICAP/1.0 200 OK
Allow: 200,204
Connection: close
Date: Fri, 05 Mar 2021 07:34:56 GMT
Encapsulated: null-body=0
Methods: REQMOD
Preview: 64000

However in my ICAP service, I often receive a request body over 64000
bytes, e.g. 1000000 request bytes.

Receive when? During preview, instead of Preview, and/or after Preview?


I also noticed that a "Preview: 0” will not have any effect and the ICAP
service will still receive a complete HTTP REQMOD+REQRESP body.

Receive when? During preview, instead of Preview, and/or after Preview?

Alex.



icap_enable on
icap_connect_timeout 1 second
icap_service_failure_limit -1
icap_send_client_ip    on
icap_preview_size 0
icap_persistent_connections on
icap_service service_req reqmod_precache bypass=0
<a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request <<a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request>
<<a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request <<a href="icap://172.17.0.2:1344/request" class="">icap://172.17.0.2:1344/request>>
icap_service service_res respmod_precache bypass=0
<a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response <<a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response>
<<a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response <<a href="icap://172.17.0.2:1344/response" class="">icap://172.17.0.2:1344/response>>
icap_preview_enable on
adaptation_access service_req allow all
adaptation_access service_res allow all


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Alex Rousskov
In reply to this post by Alex Rousskov
On 3/6/21 3:33 PM, Niels Hofmans wrote:

> I fixed a bug in the go-icap/icap library, seen here:

FYI: Whatever you were trying to share in this paragraph, did not come
through the mailing list. However, we should not be discussing some ICAP
library bugs on this mailing list, so perhaps it is not important.


> But this made me wonder about the following: <PastedGraphic-1.tiff>

In the future, please paste message _text_ and/or provide frame/stream
references to pieces of shared captures. Working with images is very
inefficient when it comes to protocol analysis.


> Here you can see squid issuing a RESPMOD to my ICAP server with a
> Preview of “0”,

Yes.


> but it does contain a HTTP response.

It contains an HTTP request header (shown in the PastedGraphic-1.tiff
image you have attached), probably an HTTP response header (not shown),
and probably no HTTP response body. The HTTP response does have a body,
but the ICAP client does not send it during the Preview:0 stage. This is
how ICAP Preview works. I see nothing wrong here.


> And since my ICAP service will not return a HTTP response since the
> Preview size is 0, my http client receives nothing from squid.

AFAICT, your ICAP service is butchering HTTP responses by responding
with ICAP 200 responses that encapsulate HTTP messages without bodies. A
standard ICAP client will _not_ re-assemble the HTTP message by
combining your header with client-cached body parts. The ICAP standard
just does not work that way. When the ICAP service sends an ICAP 200
response, the service must return the _entire_ HTTP message, headers and
body.

As a consequence, an ICAP service cannot use standard ICAP to adapt HTTP
headers without receiving and returning the HTTP body as well. This
applies to REQMOD and RESPMOD transactions. You are just lucky that your
test REQMOD transactions have requests without bodies.


> I checked but I don’t think I see a 100 continuation from ICAP in the
> dump anymore.

Yes, that is correct. However, the new trace shows another ICAP service
problem as detailed above.


HTH,

Alex.


> On 6 Mar 2021, at 19:29, Alex Rousskov wrote:
>
> On 3/6/21 5:09 AM, Niels Hofmans wrote:
>
>> I’ve uploaded a tcpdump sample of where I’m requesting an image through
>> squid + icap and where you can see the image response being posted fully
>> to the ICAP service.
>
> AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
> zero-size Preview containing just the HTTP headers (e.g., see frame
> #44). In one case, Squid also sends the HTTP response body but only
> because your ICAP service explicitly requests that HTTP response body by
> responding to Squid Preview with ICAP 100 Continue control message
> (e.g., see frame #48)!
>
> If your ICAP service does not want to see an HTTP body, then it should
> not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
> based on the Preview alone, without sending ICAP 100 Continue control
> message first.
>
>
> HTH,
>
> Alex.
>
>
>> On 5 Mar 2021, at 23:32, Alex Rousskov wrote:
>>
>> On 3/5/21 5:21 PM, Niels Hofmans wrote:
>>
>>> I receive that large payload right after an OPTIONS call to my ICAP
>>> endpoint. It is the preview.
>>
>> The timing of an ICAP request does not determine whether that request
>> has a Preview stage and whether the "unexpected" body bytes were sent
>> during that Preview stage. I am asking these questions because I want to
>> determine whether Squid increases Preview size beyond what your server
>> has requested or does not send Preview at all.
>>
>> Please share the ICAP headers of the problematic REQMOD request and the
>> last chunk metadata of that request body. You can get the latter from a
>> packet capture if your ICAP server does not report it in a convenient
>> form. In fact, sharing (a pointer to) the packet capture of the whole
>> problematic ICAP request is probably a good idea!
>>
>> Alex.
>>
>>
>>
>>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>>>
>>> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>>>
>>>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>>>> nor RESPMOD.
>>>
>>>> For the REQMOD OPTIONS requests, I respond with:
>>>>
>>>> ICAP/1.0 200 OK
>>>> Allow: 200,204
>>>> Connection: close
>>>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>>>> Encapsulated: null-body=0
>>>> Methods: REQMOD
>>>> Preview: 64000
>>>>
>>>> However in my ICAP service, I often receive a request body over 64000
>>>> bytes, e.g. 1000000 request bytes.
>>>
>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>
>>>
>>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>>>> service will still receive a complete HTTP REQMOD+REQRESP body.
>>>
>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>
>>> Alex.
>>>
>>>
>>>
>>>> icap_enable on
>>>> icap_connect_timeout 1 second
>>>> icap_service_failure_limit -1
>>>> icap_send_client_ip    on
>>>> icap_preview_size 0
>>>> icap_persistent_connections on
>>>> icap_service service_req reqmod_precache bypass=0
>>>> icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request>>
>>>> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request>>>
>>>> icap_service service_res respmod_precache bypass=0
>>>> icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response>>
>>>> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response>>>
>>>> icap_preview_enable on
>>>> adaptation_access service_req allow all
>>>> adaptation_access service_res allow all
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Niels Hofmans
Hi Alex,

Missed that in the RFC, I was hoping I could only send the headers across and modify those, without having to send over everything, including very large responses for performance reasons.
Thanks for following along.

Regards,
Niels

Sent from my mobile

On 6 Mar 2021, at 23:22, Alex Rousskov <[hidden email]> wrote:

On 3/6/21 3:33 PM, Niels Hofmans wrote:

> I fixed a bug in the go-icap/icap library, seen here:

FYI: Whatever you were trying to share in this paragraph, did not come
through the mailing list. However, we should not be discussing some ICAP
library bugs on this mailing list, so perhaps it is not important.


> But this made me wonder about the following: <PastedGraphic-1.tiff>

In the future, please paste message _text_ and/or provide frame/stream
references to pieces of shared captures. Working with images is very
inefficient when it comes to protocol analysis.


> Here you can see squid issuing a RESPMOD to my ICAP server with a
> Preview of “0”,

Yes.


> but it does contain a HTTP response.

It contains an HTTP request header (shown in the PastedGraphic-1.tiff
image you have attached), probably an HTTP response header (not shown),
and probably no HTTP response body. The HTTP response does have a body,
but the ICAP client does not send it during the Preview:0 stage. This is
how ICAP Preview works. I see nothing wrong here.


> And since my ICAP service will not return a HTTP response since the
> Preview size is 0, my http client receives nothing from squid.

AFAICT, your ICAP service is butchering HTTP responses by responding
with ICAP 200 responses that encapsulate HTTP messages without bodies. A
standard ICAP client will _not_ re-assemble the HTTP message by
combining your header with client-cached body parts. The ICAP standard
just does not work that way. When the ICAP service sends an ICAP 200
response, the service must return the _entire_ HTTP message, headers and
body.

As a consequence, an ICAP service cannot use standard ICAP to adapt HTTP
headers without receiving and returning the HTTP body as well. This
applies to REQMOD and RESPMOD transactions. You are just lucky that your
test REQMOD transactions have requests without bodies.


> I checked but I don’t think I see a 100 continuation from ICAP in the
> dump anymore.

Yes, that is correct. However, the new trace shows another ICAP service
problem as detailed above.


HTH,

Alex.


> On 6 Mar 2021, at 19:29, Alex Rousskov wrote:
>
> On 3/6/21 5:09 AM, Niels Hofmans wrote:
>
>> I’ve uploaded a tcpdump sample of where I’m requesting an image through
>> squid + icap and where you can see the image response being posted fully
>> to the ICAP service.
>
> AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
> zero-size Preview containing just the HTTP headers (e.g., see frame
> #44). In one case, Squid also sends the HTTP response body but only
> because your ICAP service explicitly requests that HTTP response body by
> responding to Squid Preview with ICAP 100 Continue control message
> (e.g., see frame #48)!
>
> If your ICAP service does not want to see an HTTP body, then it should
> not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
> based on the Preview alone, without sending ICAP 100 Continue control
> message first.
>
>
> HTH,
>
> Alex.
>
>
>> On 5 Mar 2021, at 23:32, Alex Rousskov wrote:
>> On 3/5/21 5:21 PM, Niels Hofmans wrote:
>>> I receive that large payload right after an OPTIONS call to my ICAP
>>> endpoint. It is the preview.
>> The timing of an ICAP request does not determine whether that request
>> has a Preview stage and whether the "unexpected" body bytes were sent
>> during that Preview stage. I am asking these questions because I want to
>> determine whether Squid increases Preview size beyond what your server
>> has requested or does not send Preview at all.
>> Please share the ICAP headers of the problematic REQMOD request and the
>> last chunk metadata of that request body. You can get the latter from a
>> packet capture if your ICAP server does not report it in a convenient
>> form. In fact, sharing (a pointer to) the packet capture of the whole
>> problematic ICAP request is probably a good idea!
>> Alex.
>>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>>> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>>>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>>>> nor RESPMOD.
>>>> For the REQMOD OPTIONS requests, I respond with:
>>>> ICAP/1.0 200 OK
>>>> Allow: 200,204
>>>> Connection: close
>>>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>>>> Encapsulated: null-body=0
>>>> Methods: REQMOD
>>>> Preview: 64000
>>>> However in my ICAP service, I often receive a request body over 64000
>>>> bytes, e.g. 1000000 request bytes.
>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>>>> service will still receive a complete HTTP REQMOD+REQRESP body.
>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>> Alex.
>>>> icap_enable on
>>>> icap_connect_timeout 1 second
>>>> icap_service_failure_limit -1
>>>> icap_send_client_ip    on
>>>> icap_preview_size 0
>>>> icap_persistent_connections on
>>>> icap_service service_req reqmod_precache bypass=0
>>>> icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request>>
>>>> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>> <icap://172.17.0.2:1344/request>>>
>>>> icap_service service_res respmod_precache bypass=0
>>>> icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response>>
>>>> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>> <icap://172.17.0.2:1344/response>>>
>>>> icap_preview_enable on
>>>> adaptation_access service_req allow all
>>>> adaptation_access service_res allow all

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Icap preview size

Alex Rousskov
On 3/6/21 5:38 PM, Niels Hofmans wrote:

> Missed that in the RFC, I was hoping I could only send the headers across and modify those, without having to send over everything, including very large responses for performance reasons.

There are ICAP extensions that allow header-only adaptation, but, IMO,
it is best to get the basics working before adding support for
experimental protocol extensions.

Alex.


> On 6 Mar 2021, at 23:22, Alex Rousskov <[hidden email]> wrote:
>
> On 3/6/21 3:33 PM, Niels Hofmans wrote:
>
>> I fixed a bug in the go-icap/icap library, seen here:
>
> FYI: Whatever you were trying to share in this paragraph, did not come
> through the mailing list. However, we should not be discussing some ICAP
> library bugs on this mailing list, so perhaps it is not important.
>
>
>> But this made me wonder about the following: <PastedGraphic-1.tiff>
>
> In the future, please paste message _text_ and/or provide frame/stream
> references to pieces of shared captures. Working with images is very
> inefficient when it comes to protocol analysis.
>
>
>> Here you can see squid issuing a RESPMOD to my ICAP server with a
>> Preview of “0”,
>
> Yes.
>
>
>> but it does contain a HTTP response.
>
> It contains an HTTP request header (shown in the PastedGraphic-1.tiff
> image you have attached), probably an HTTP response header (not shown),
> and probably no HTTP response body. The HTTP response does have a body,
> but the ICAP client does not send it during the Preview:0 stage. This is
> how ICAP Preview works. I see nothing wrong here.
>
>
>> And since my ICAP service will not return a HTTP response since the
>> Preview size is 0, my http client receives nothing from squid.
>
> AFAICT, your ICAP service is butchering HTTP responses by responding
> with ICAP 200 responses that encapsulate HTTP messages without bodies. A
> standard ICAP client will _not_ re-assemble the HTTP message by
> combining your header with client-cached body parts. The ICAP standard
> just does not work that way. When the ICAP service sends an ICAP 200
> response, the service must return the _entire_ HTTP message, headers and
> body.
>
> As a consequence, an ICAP service cannot use standard ICAP to adapt HTTP
> headers without receiving and returning the HTTP body as well. This
> applies to REQMOD and RESPMOD transactions. You are just lucky that your
> test REQMOD transactions have requests without bodies.
>
>
>> I checked but I don’t think I see a 100 continuation from ICAP in the
>> dump anymore.
>
> Yes, that is correct. However, the new trace shows another ICAP service
> problem as detailed above.
>
>
> HTH,
>
> Alex.
>
>
>> On 6 Mar 2021, at 19:29, Alex Rousskov wrote:
>>
>> On 3/6/21 5:09 AM, Niels Hofmans wrote:
>>
>>> I’ve uploaded a tcpdump sample of where I’m requesting an image through
>>> squid + icap and where you can see the image response being posted fully
>>> to the ICAP service.
>>
>> AFAICT, all ICAP REQMOD and RESPMOD requests in your packet capture send
>> zero-size Preview containing just the HTTP headers (e.g., see frame
>> #44). In one case, Squid also sends the HTTP response body but only
>> because your ICAP service explicitly requests that HTTP response body by
>> responding to Squid Preview with ICAP 100 Continue control message
>> (e.g., see frame #48)!
>>
>> If your ICAP service does not want to see an HTTP body, then it should
>> not ask for it. It should respond (usually with ICAP 200 or ICAP 204)
>> based on the Preview alone, without sending ICAP 100 Continue control
>> message first.
>>
>>
>> HTH,
>>
>> Alex.
>>
>>
>>> On 5 Mar 2021, at 23:32, Alex Rousskov wrote:
>>> On 3/5/21 5:21 PM, Niels Hofmans wrote:
>>>> I receive that large payload right after an OPTIONS call to my ICAP
>>>> endpoint. It is the preview.
>>> The timing of an ICAP request does not determine whether that request
>>> has a Preview stage and whether the "unexpected" body bytes were sent
>>> during that Preview stage. I am asking these questions because I want to
>>> determine whether Squid increases Preview size beyond what your server
>>> has requested or does not send Preview at all.
>>> Please share the ICAP headers of the problematic REQMOD request and the
>>> last chunk metadata of that request body. You can get the latter from a
>>> packet capture if your ICAP server does not report it in a convenient
>>> form. In fact, sharing (a pointer to) the packet capture of the whole
>>> problematic ICAP request is probably a good idea!
>>> Alex.
>>>> On 5 Mar 2021, at 17:21, Alex Rousskov wrote:
>>>> On 3/5/21 2:55 AM, Niels Hofmans wrote:
>>>>> One more: I believe ICAP is not respecting the Preview header for REQMOD
>>>>> nor RESPMOD.
>>>>> For the REQMOD OPTIONS requests, I respond with:
>>>>> ICAP/1.0 200 OK
>>>>> Allow: 200,204
>>>>> Connection: close
>>>>> Date: Fri, 05 Mar 2021 07:34:56 GMT
>>>>> Encapsulated: null-body=0
>>>>> Methods: REQMOD
>>>>> Preview: 64000
>>>>> However in my ICAP service, I often receive a request body over 64000
>>>>> bytes, e.g. 1000000 request bytes.
>>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>>> I also noticed that a "Preview: 0” will not have any effect and the ICAP
>>>>> service will still receive a complete HTTP REQMOD+REQRESP body.
>>>> Receive when? During preview, instead of Preview, and/or after Preview?
>>>> Alex.
>>>>> icap_enable on
>>>>> icap_connect_timeout 1 second
>>>>> icap_service_failure_limit -1
>>>>> icap_send_client_ip    on
>>>>> icap_preview_size 0
>>>>> icap_persistent_connections on
>>>>> icap_service service_req reqmod_precache bypass=0
>>>>> icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request>>
>>>>> <icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request> <icap://172.17.0.2:1344/request
>>>>> <icap://172.17.0.2:1344/request>>>
>>>>> icap_service service_res respmod_precache bypass=0
>>>>> icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response>>
>>>>> <icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response> <icap://172.17.0.2:1344/response
>>>>> <icap://172.17.0.2:1344/response>>>
>>>>> icap_preview_enable on
>>>>> adaptation_access service_req allow all
>>>>> adaptation_access service_res allow all

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users