Not Implemented error returned I believe due to chunking

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Not Implemented error returned I believe due to chunking

Russel McDonald
Hi,

On Windows I'm getting an error returned when attempting to upload a file to an Amazon AWS S3 bucket, using either the WinSCP app or just using the S3 upload interface via browser, pointing to my squid proxy.
I can download just fine, but not upload. ECAP is enabled for both the request and the response.

Here is the error:
A header you provided implies functionality that is not implemented
Extra Details: Header: Transfer-Encoding, RequestId: 3ED7C576056435A0, HostId: SU/9a+qN5rL79STErKgziygc2A34jLhWJ110qbUB1fele67wIZ5thMmgtNnKczBFVxuOy810HB4=

If I instead only specify to pass the response to my adapter via ECAP then the issue goes away.
So I captured the HTTP header for the success case and the failure case and narrowed it down the the failure case not specifying the length and specifying ",chunking" at the end. I assume if the request is passed to the adapter then the header has to be constructed at a time when the length of the file contents is not known? Hence has to use chunking.

I've indeed read a few forum links about just this issue, but confused as to whether chunking is supposed to finally work for 3.5.28 that I have or not. I compared to 4.x source and it looked basically the same.

I'll be needing to upload large files, too large to fit in memory, hence pretty sure chunking will be a requirement.

Is this is known issue? Any thoughts on solving it?

Thanks! I'm a Windows kernel filter driver developer so working open source is new to me. But with this forum's help I've had quite a lot of success. So thanks!

Russel McDonald

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Not Implemented error returned I believe due to chunking

Alex Rousskov
On 3/6/19 1:00 PM, Russel McDonald wrote:

> I've indeed read a few forum links about just this issue, but confused
> as to whether chunking is supposed to finally work for 3.5.28

Chunked requests should be supported in all modern Squid versions,
including v3.5.28. There are some corner cases where dechunking code
fails, but they are very unlikely to affect you. I do not know what
forum links you have read, but I do not recall any general long-standing
problems with chunking support in modern Squids.


> If I instead only specify to pass the response to my adapter via
> ECAP then the issue goes away. So I captured the HTTP header for the
> success case and the failure case and narrowed it down the the
> failure case not specifying the length and specifying ",chunking" at
> the end.

Many adapters strip Content-Length header and effectively switch to
chunked encoding because it is easier and cheaper to adapt message
bodies that way. Does your eCAP adapter remove the Content-Length header
(possibly replacing it with Transfer-Encoding:chunked)? Such actions are
OK from HTTP point of view, but they may expose lack of chunked request
support at the _origin server_.


> I assume if the request is passed to the adapter then the header has
> to be constructed at a time when the length of the file contents is
> not known? Hence has to use chunking.

Squid does not "construct" that request header from scratch -- it
essentially forwards the request header sent by the client. If I
interpret your description correctly, the client sends a
Content-Length:N header, so the size of the request body (i.e. N bytes)
is known to Squid and, hence, to the REQMOD eCAP adapter in advance.

It is possible that your adapter strips that Content-Length header, and
then the origin server complains because it cannot handle chunked
requests. If that is what's happening, then you will need to seek
support from the adapter makers.

Alex.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

icap example server not working

steven
hi,


after going trough rfc3507 (icap) im not really sure how a icap server
is supposed to work together with squid, for example the python icap
servers that are available via github have method names like:


example_OPTIONS, example_REQMOD, example_RESPMOD

yara_OPTIONS_, yara_....



or the example on this page: https://github.com/netom/pyicap

echo_OPTIONS, echo_RESPMOD


with:

icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/respmod
adaptation_access service_resp allow all

127.0.0.1 - - [06/b'Mar'/2019 23:22:58] respmod_OPTIONS not found
127.0.0.1 - - [06/b'Mar'/2019 23:22:58] code 404, message b'Not Found'


with:

icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/echo
adaptation_access service_resp allow all

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 46458)
Traceback (most recent call last):
   File "/usr/lib/python3.7/socketserver.py", line 650, in
process_request_thread
     self.finish_request(request, client_address)
   File "/usr/lib/python3.7/socketserver.py", line 360, in finish_request
     self.RequestHandlerClass(request, client_address, self)
   File "/usr/lib/python3.7/socketserver.py", line 720, in __init__
     self.handle()
   File
"/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
line 443, in handle
     self.handle_one_request()
   File
"/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
line 494, in handle_one_request
     method()
   File "icaptest.py", line 22, in echo_OPTIONS
     self.send_headers(False)
   File
"/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
line 326, in send_headers
     icap_header_str += k + b': ' + v + b'\r\n'
TypeError: can only concatenate str (not "bytes") to str
----------------------------------------




squid config:

icap_enable on

icap_preview_enable off
icap_send_client_ip on
icap_send_client_username on
icap_service service_req reqmod_precache bypass=1
icap://127.0.0.1:1344/respmod
adaptation_access service_req allow all
#icap_service service_resp respmod_precache bypass=0
icap://127.0.0.1:1344/response
#adaptation_access service_resp allow all




why is this simple example not working?

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: icap example server not working

Alex Rousskov
On 3/6/19 3:33 PM, steven wrote:

> after going trough rfc3507 (icap) im not really sure how a icap server
> is supposed to work together with squid,

At the protocol level, an ICAP server works pretty much how RFC 3507
describes it. At the ICAP server software/application level, each server
"works" in its own way, irrelevant to Squid.


> the example on this page: https://github.com/netom/pyicap

> 127.0.0.1 - - [06/b'Mar'/2019 23:22:58] respmod_OPTIONS not found
> 127.0.0.1 - - [06/b'Mar'/2019 23:22:58] code 404, message b'Not Found'

I know nothing about that ICAP framework, but you probably need to
follow its README directions and define the respmod_OPTIONS method
(among other things):

> You can use a framework by importing stuff from the pyicap package,
> extending the protocol handler class and starting the server, passing
> your handler to it: ...

In other words, that specific ICAP code is not a ready-to-use ICAP
server but a _framework_ for building an ICAP server (that does what
_you_ want it to do).

If you need an ICAP server to do X, then you should either find an
existing ICAP server that already does X or build a new ICAP server that
does X. The latter should be accomplished either by using an existing
ICAP library/framework (like the one you found above) or by writing a
loadable module/plugin/adapter (that does X) for an existing ICAP server
that accepts such adapters.

Squid wiki has a bit more info about ICAP:

* https://wiki.squid-cache.org/SquidFaq/ContentAdaptation#ICAP
* https://wiki.squid-cache.org/Features/ICAP

The first page also discusses (and compares) various ICAP alternatives.


HTH,

Alex.


> icap_service service_resp respmod_precache bypass=0
> icap://127.0.0.1:1344/echo
> adaptation_access service_resp allow all
>
> ----------------------------------------
> Exception happened during processing of request from ('127.0.0.1', 46458)
> Traceback (most recent call last):
>   File "/usr/lib/python3.7/socketserver.py", line 650, in
> process_request_thread
>     self.finish_request(request, client_address)
>   File "/usr/lib/python3.7/socketserver.py", line 360, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.7/socketserver.py", line 720, in __init__
>     self.handle()
>   File
> "/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
> line 443, in handle
>     self.handle_one_request()
>   File
> "/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
> line 494, in handle_one_request
>     method()
>   File "icaptest.py", line 22, in echo_OPTIONS
>     self.send_headers(False)
>   File
> "/home/julius/code/python/icap/venv/lib/python3.7/site-packages/pyicap.py",
> line 326, in send_headers
>     icap_header_str += k + b': ' + v + b'\r\n'
> TypeError: can only concatenate str (not "bytes") to str
> ----------------------------------------
>
>
>
>
> squid config:
>
> icap_enable on
>
> icap_preview_enable off
> icap_send_client_ip on
> icap_send_client_username on
> icap_service service_req reqmod_precache bypass=1
> icap://127.0.0.1:1344/respmod
> adaptation_access service_req allow all
> #icap_service service_resp respmod_precache bypass=0
> icap://127.0.0.1:1344/response
> #adaptation_access service_resp allow all
>
>
>
>
> why is this simple example not working?
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users