Another "Forwarding loop detected" issue

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Another "Forwarding loop detected" issue

Nick Howitt-2
I am trying to help someone who is running squid-3.5.20-12 on a
standalone server with the dansguardian content filter and suddenly
recently has been getting a lot of messages like:

    2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
    HEAD / HTTP/1.0
    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
    Cache-Control: max-age=259200
    Connection: keep-alive
    X-Forwarded-For: 10.10.1.2
    Host: 10.10.1.2:8080


The access log looks something like:

    1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
    1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
    1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -

(but these are for different transactions - they are all the same apart
from the timestamps)

The content filter listens on port 8080 and squid on 3128. The machine
is on 10.10.1.2

All the other posts I've seen seem to be for transparent mode or where
there is a User Agent string. I have found nothing to cover this
scenario. How can I troubleshoot to fix it and what information do you
need from me to help diagnose?

The only thing I have thought of is to put in a firewall rule blocking
traffic from 10.10.1.2 to 10.10.1.2:8080 but I fear shooting myself in
the foot.

Any help would be greatly appreciated.

Nick

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Another "Forwarding loop detected" issue

Amos Jeffries
Administrator
On 5/11/19 10:40 pm, Nick Howitt wrote:

> I am trying to help someone who is running squid-3.5.20-12 on a
> standalone server with the dansguardian content filter and suddenly
> recently has been getting a lot of messages like:
>
>    2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>    HEAD / HTTP/1.0
>    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>    Cache-Control: max-age=259200
>    Connection: keep-alive
>    X-Forwarded-For: 10.10.1.2
>    Host: 10.10.1.2:8080
>
>
> The access log looks something like:
>
>    1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>    1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>    1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>
> (but these are for different transactions - they are all the same apart
> from the timestamps)
>

That is what a forwarding loop looks like in the access.log.


> The content filter listens on port 8080 and squid on 3128. The machine
> is on 10.10.1.2
>
> All the other posts I've seen seem to be for transparent mode or where
> there is a User Agent string. I have found nothing to cover this
> scenario. How can I troubleshoot to fix it and what information do you
> need from me to help diagnose?
>

Something is telling Squid the origin server being contacted exists at
10.10.1.2:8080. You can see that in the Host header of the message.

I would trace the traffic flow from the client to Squid.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Another "Forwarding loop detected" issue

Nick Howitt-2


On 05/11/2019 10:44, Amos Jeffries wrote:

> On 5/11/19 10:40 pm, Nick Howitt wrote:
>> I am trying to help someone who is running squid-3.5.20-12 on a
>> standalone server with the dansguardian content filter and suddenly
>> recently has been getting a lot of messages like:
>>
>>     2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>>     HEAD / HTTP/1.0
>>     Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>>     Cache-Control: max-age=259200
>>     Connection: keep-alive
>>     X-Forwarded-For: 10.10.1.2
>>     Host: 10.10.1.2:8080
>>
>>
>> The access log looks something like:
>>
>>     1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>     1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>     1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>
>> (but these are for different transactions - they are all the same apart
>> from the timestamps)
>>
> That is what a forwarding loop looks like in the access.log.
>
>
>> The content filter listens on port 8080 and squid on 3128. The machine
>> is on 10.10.1.2
>>
>> All the other posts I've seen seem to be for transparent mode or where
>> there is a User Agent string. I have found nothing to cover this
>> scenario. How can I troubleshoot to fix it and what information do you
>> need from me to help diagnose?
>>
> Something is telling Squid the origin server being contacted exists at
> 10.10.1.2:8080. You can see that in the Host header of the message.
>
> I would trace the traffic flow from the client to Squid.
>
But isn't everything coming to 8080 as that is the proxy you'd set up in
the browser? I'm afraid I don't understand how proxying works at the
packet level. I see nothing before these messages to indicate the
packets are coming from elsewhere. A cut down startup log looks like:

    <snip>
    2019/10/31 13:47:40 kid1| helperOpenServers: Starting 5/5
    'ext_unix_group_acl' processes
    2019/10/31 13:47:40 kid1| HTCP Disabled.
    2019/10/31 13:47:40 kid1| Finished loading MIME types and icons.
    2019/10/31 13:47:40 kid1| Accepting HTTP Socket connections at
    local=[::1]:3128 remote=[::] FD 2021 flags=9
    2019/10/31 13:47:40 kid1| Accepting HTTP Socket connections at
    local=127.0.0.1:3128 remote=[::] FD 2022 flags=9
    2019/10/31 13:47:40 kid1| Accepting HTTP Socket connections at
    local=10.10.1.2:3128 remote=[::] FD 2023 flags=9
    2019/10/31 13:48:12 kid1| WARNING: Forwarding loop detected for:
    HEAD / HTTP/1.0
    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
    Cache-Control: max-age=259200
    Connection: keep-alive
    X-Forwarded-For: 10.10.1.2
    Host: 10.10.1.2:8080


    2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
    HEAD / HTTP/1.0
    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
    Cache-Control: max-age=259200
    Connection: keep-alive
    X-Forwarded-For: 10.10.1.2
    Host: 10.10.1.2:8080


Is there anything I can look for in my logs or do I need to do some sort
of tcpdump with some filters?

Thanks,

Nick


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Another "Forwarding loop detected" issue

Nick Howitt-2
On 05/11/2019 11:07, Nick Howitt wrote:

>
>
> On 05/11/2019 10:44, Amos Jeffries wrote:
>> On 5/11/19 10:40 pm, Nick Howitt wrote:
>>> I am trying to help someone who is running squid-3.5.20-12 on a
>>> standalone server with the dansguardian content filter and suddenly
>>> recently has been getting a lot of messages like:
>>>
>>>     2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>>>     HEAD / HTTP/1.0
>>>     Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>>>     Cache-Control: max-age=259200
>>>     Connection: keep-alive
>>>     X-Forwarded-For: 10.10.1.2
>>>     Host: 10.10.1.2:8080
>>>
>>>
>>> The access log looks something like:
>>>
>>>     1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>     1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>     1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>
>>> (but these are for different transactions - they are all the same apart
>>> from the timestamps)
>>>
>> That is what a forwarding loop looks like in the access.log.
>>
>>
>>> The content filter listens on port 8080 and squid on 3128. The machine
>>> is on 10.10.1.2
>>>
>>> All the other posts I've seen seem to be for transparent mode or where
>>> there is a User Agent string. I have found nothing to cover this
>>> scenario. How can I troubleshoot to fix it and what information do you
>>> need from me to help diagnose?
>>>
>> Something is telling Squid the origin server being contacted exists at
>> 10.10.1.2:8080. You can see that in the Host header of the message.
>>
>> I would trace the traffic flow from the client to Squid.
>>
> But isn't everything coming to 8080 as that is the proxy you'd set up
> in the browser? I'm afraid I don't understand how proxying works at
> the packet level. I see nothing before these messages to indicate the
> packets are coming from elsewhere. A cut down startup log looks like:
>
>    <snip>
>    2019/10/31 13:47:40 kid1| helperOpenServers: Starting 5/5
>    'ext_unix_group_acl' processes
>    2019/10/31 13:47:40 kid1| HTCP Disabled.
>    2019/10/31 13:47:40 kid1| Finished loading MIME types and icons.
>    2019/10/31 13:47:40 kid1| Accepting HTTP Socket connections at
>    local=[::1]:3128 remote=[::] FD 2021 flags=9
>    2019/10/31 13:47:40 kid1| Accepting HTTP Socket connections at
>    local=127.0.0.1:3128 remote=[::] FD 2022 flags=9
>    2019/10/31 13:47:40 kid1| Accepting HTTP Socket connections at
>    local=10.10.1.2:3128 remote=[::] FD 2023 flags=9
>    2019/10/31 13:48:12 kid1| WARNING: Forwarding loop detected for:
>    HEAD / HTTP/1.0
>    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>    Cache-Control: max-age=259200
>    Connection: keep-alive
>    X-Forwarded-For: 10.10.1.2
>    Host: 10.10.1.2:8080
>
>
>    2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>    HEAD / HTTP/1.0
>    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>    Cache-Control: max-age=259200
>    Connection: keep-alive
>    X-Forwarded-For: 10.10.1.2
>    Host: 10.10.1.2:8080
>
>
> Is there anything I can look for in my logs or do I need to do some
> sort of tcpdump with some filters?
>
> Thanks,
>
> Nick
At the moment the wpad file is not pointing to the proxy server so no
machines should be using it. I have tried a:

    tcpdump -vvvnnn -A -i eth0 port 8080 -s 1500


This gives me bursts of:

    07:50:47.569305 IP (tos 0x0, ttl 128, id 56718, offset 0, flags
    [DF], proto TCP (6), length 52)
         10.10.11.215.64857 > 10.10.1.2.8080: Flags [S], cksum 0x389b
    (correct), seq 625662051, win 64240, options [mss 1460,nop,wscale
    8,nop,nop,sackOK], length 0
    E..4..@....H

    ..

    ...Y..%J.c........8...............
    07:50:47.569419 IP (tos 0x0, ttl 64, id 7161, offset 0, flags [DF],
    proto TCP (6), length 40)
         10.10.1.2.8080 > 10.10.11.215.64857: Flags [R.], cksum 0x744b
    (correct), seq 0, ack 1, win 0, length 0
    E..(..@.@...

    ..

    .....Y....%J.dP...tK..


 From what I've researched so far there are no http headers in these
packets. The proxy is 10.10.1.2. Does this mean 10.10.11.215 could be
the offending machine if no other machines should be using the proxy? Or
do I need to do something cleverer with my tcpdump?

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Another "Forwarding loop detected" issue

Matus UHLAR - fantomas
>>>On 5/11/19 10:40 pm, Nick Howitt wrote:
>>>>I am trying to help someone who is running squid-3.5.20-12 on a
>>>>standalone server with the dansguardian content filter and suddenly
>>>>recently has been getting a lot of messages like:
>>>>
>>>>    2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>>>>    HEAD / HTTP/1.0
>>>>    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>>>>    Cache-Control: max-age=259200
>>>>    Connection: keep-alive
>>>>    X-Forwarded-For: 10.10.1.2
>>>>    Host: 10.10.1.2:8080
>>>>
>>>>
>>>>The access log looks something like:
>>>>
>>>>    1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>    1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>    1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>
>>>>(but these are for different transactions - they are all the same apart
>>>>from the timestamps)


>>On 05/11/2019 10:44, Amos Jeffries wrote:
>>>That is what a forwarding loop looks like in the access.log.

>>>>The content filter listens on port 8080 and squid on 3128. The machine
>>>>is on 10.10.1.2

\On 05.11.19 12:57, Nick Howitt wrote:

>At the moment the wpad file is not pointing to the proxy server so no
>machines should be using it. I have tried a:
>
>   tcpdump -vvvnnn -A -i eth0 port 8080 -s 1500
>
>
>This gives me bursts of:
>
>   07:50:47.569305 IP (tos 0x0, ttl 128, id 56718, offset 0, flags
>   [DF], proto TCP (6), length 52)
>        10.10.11.215.64857 > 10.10.1.2.8080: Flags [S], cksum 0x389b

>From what I've researched so far there are no http headers in these
>packets. The proxy is 10.10.1.2. Does this mean 10.10.11.215 could be
>the offending machine if no other machines should be using the proxy?
>Or do I need to do something cleverer with my tcpdump?

I don't think so.

How does your schema look like?
How does your content filter work?

The logs above show that someone from local machins (content-filter) is
using squid to access local machine port 8080, which should be your content
filter.

That looks much like a loop, connections from squid or content filter that
are going back to content filter via squid



--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Chernobyl was an Windows 95 beta test site.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Another "Forwarding loop detected" issue

Nick Howitt-2


On 06/11/2019 09:39, Matus UHLAR - fantomas wrote:

>>>> On 5/11/19 10:40 pm, Nick Howitt wrote:
>>>>> I am trying to help someone who is running squid-3.5.20-12 on a
>>>>> standalone server with the dansguardian content filter and suddenly
>>>>> recently has been getting a lot of messages like:
>>>>>
>>>>>     2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>>>>>     HEAD / HTTP/1.0
>>>>>     Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>>>>>     Cache-Control: max-age=259200
>>>>>     Connection: keep-alive
>>>>>     X-Forwarded-For: 10.10.1.2
>>>>>     Host: 10.10.1.2:8080
>>>>>
>>>>>
>>>>> The access log looks something like:
>>>>>
>>>>>     1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>>     1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>>     1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>>     http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>>
>>>>> (but these are for different transactions - they are all the same
>>>>> apart
>>>>> from the timestamps)
>
>
>>> On 05/11/2019 10:44, Amos Jeffries wrote:
>>>> That is what a forwarding loop looks like in the access.log.
>
>>>>> The content filter listens on port 8080 and squid on 3128. The
>>>>> machine
>>>>> is on 10.10.1.2
>
> \On 05.11.19 12:57, Nick Howitt wrote:
>> At the moment the wpad file is not pointing to the proxy server so no
>> machines should be using it. I have tried a:
>>
>>   tcpdump -vvvnnn -A -i eth0 port 8080 -s 1500
>>
>>
>> This gives me bursts of:
>>
>>   07:50:47.569305 IP (tos 0x0, ttl 128, id 56718, offset 0, flags
>>   [DF], proto TCP (6), length 52)
>>        10.10.11.215.64857 > 10.10.1.2.8080: Flags [S], cksum 0x389b
>
>> From what I've researched so far there are no http headers in these
>> packets. The proxy is 10.10.1.2. Does this mean 10.10.11.215 could be
>> the offending machine if no other machines should be using the proxy?
>> Or do I need to do something cleverer with my tcpdump?
>
> I don't think so.
>
> How does your schema look like?
> How does your content filter work?
>
> The logs above show that someone from local machins (content-filter) is
> using squid to access local machine port 8080, which should be your
> content
> filter.
> That looks much like a loop, connections from squid or content filter
> that
> are going back to content filter via squid
>
>
>
The set up is eth0 (10.10.1.2:8080) -> Content filter (dansguardian) ->
Squid (port 3128) -> eth0 -> gateway

If what you are saying is right then a firewall rule blocking source
10.10.1.2 to 10.10.1.2:8080 may work. I am not sure if it would be in
the FORWARD or INPUT chain and I don't know if it would cause collateral
damage. It also does not explain why only recently it started going
wrong. The machine has been rebuilt now and I am waiting for it to
trigger again, upgrading from ClearOS6.x (a Centos derivative) to
ClearOS 7.6 (which will soon update to 7.7).

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Another "Forwarding loop detected" issue

Matus UHLAR - fantomas
>On 06/11/2019 09:39, Matus UHLAR - fantomas wrote:
>>>>>On 5/11/19 10:40 pm, Nick Howitt wrote:
>>>>>>I am trying to help someone who is running squid-3.5.20-12 on a
>>>>>>standalone server with the dansguardian content filter and suddenly
>>>>>>recently has been getting a lot of messages like:
>>>>>>
>>>>>>    2019/10/31 13:48:14 kid1| WARNING: Forwarding loop detected for:
>>>>>>    HEAD / HTTP/1.0
>>>>>>    Via: 1.0 HSFilterHyperos7.haftr.local (squid/3.5.20)
>>>>>>    Cache-Control: max-age=259200
>>>>>>    Connection: keep-alive
>>>>>>    X-Forwarded-For: 10.10.1.2
>>>>>>    Host: 10.10.1.2:8080
>>>>>>
>>>>>>
>>>>>>The access log looks something like:
>>>>>>
>>>>>>    1572545946.383 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>>>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>>>    1572545946.477 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>>>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>>>    1572545946.493 120000 10.10.1.2 TCP_MISS_ABORTED/000 0 HEAD
>>>>>>    http://10.10.1.2:8080/ - HIER_DIRECT/10.10.1.2 -
>>>>>>
>>>>>>(but these are for different transactions - they are all the
>>>>>>same apart
>>>>>>from the timestamps)
>>
>>
>>>>On 05/11/2019 10:44, Amos Jeffries wrote:
>>>>>That is what a forwarding loop looks like in the access.log.
>>
>>>>>>The content filter listens on port 8080 and squid on 3128.
>>>>>>The machine
>>>>>>is on 10.10.1.2

>>How does your schema look like?
>>How does your content filter work?
>>
>>The logs above show that someone from local machins (content-filter) is
>>using squid to access local machine port 8080, which should be your
>>content
>>filter.
>>That looks much like a loop, connections from squid or content
>>filter that
>>are going back to content filter via squid

On 06.11.19 09:54, Nick Howitt wrote:
>The set up is eth0 (10.10.1.2:8080) -> Content filter (dansguardian)
>-> Squid (port 3128) -> eth0 -> gateway

I understand this as:

client
->
10.10.1.2:8080 aka Content filter (dansguardian)
->
10.10.1.2:3128 aka squid
->
the net.


>If what you are saying is right then a firewall rule blocking source
>10.10.1.2 to 10.10.1.2:8080 may work

apparently, but I don't understand why would anyone from 10.10.1.2 to
10.10.1.2:8080.
Is it any HTTP client running on 10.10.1.2 ? Then it's ok.

Is it squid or dansguardian ?  Then something is broken in your setup, or,
any client is requesting 10.10.1.2:8080 which should apparently be disabled
in squid config.

> I am not sure if it would be in
>the FORWARD or INPUT chain

INPUT chain, since it's connection from to local IP, unless it's redirected
connection.

But IIRC you have said your clients have proxy configured.
--
Matus UHLAR - fantomas, [hidden email] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Emacs is a complicated operating system without good text editor.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users