transparent tproxy: routing issue or my own problem ?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

transparent tproxy: routing issue or my own problem ?

Ming-Ching Tiew


This is long I appreciate you patience.

I am using squid in a Linux box setting up as a bridge, and have
set up ebtables and iptables following the documentation
available on the Net :-

ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
  --ip-destination-port 80 -j redirect --redirect-target ACCEPT

iptables -t tproxy -A PREROUTING -i br0 -p tcp --dport 80 \
  -j TPROXY --on-port 80

# this don't seem to have impact by I have put in anyway
for i in /proc/sys/net/ipv4/conf/*/rp_filter
do
 echo 0 > $i
done

On a brief glance it seems it's working properly but upon detail
investigation,
there are some issues.

This is my observation :-

If I place the Bridge/Squid S in a subnet A  before the default internet
gateway D, then all the machines inside the same subnet A can be
serviced by the squid cache engine. Sniffing confirmed that the source
IP has been spoofed by Bridge/Squid S.

However, if there is a subnet B, which is connected to subnet A, via
a router R, then all the machines inside subnet B will have problem
getting the http reply packets but http request packets have no
problem going out.

Note that none-http packets because it has not been redirected by the
ebtable rules, have no problem at all. This shows that the routing
outside of the Bridge/Squid, have all been set up correctly.

Then I added a route inside the Bridge/Squid S for the subnet B via
router R, then the web request/reply problem is solved.

It seems then to me that the http reply ( source port 80 ) has also be
directed ***INTO*** the Bridge/Squid S. Why is that so ? Why didn't the
Bridge/Squid forward the reply packet to the other side of the
interface ?

I am looking for something more transparent. Any insight is much
appreciated.

p/s :-

The logs I capture using tcpdump on the squid machine before and after I
added the route. Network B 10.6.1.0/24, Network A 192.168.128.0/18,
Router R  10.6.1.1<-->192.168.128.50,  Squid 192.168.128.20.

Before :-

squid:~> tcpdump -ni br0 host 10.6.1.2 and port 80
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 68 bytes
09:06:12.974206 IP 10.6.1.2.39895 > 192.168.128.20.80: S
3302818155:3302818155(0) win 5840 <mss 1460,sackOK,timestamp 13603778[|tcp]>
09:06:12.974252 IP 66.249.89.99.80 > 10.6.1.2.39895: S
3648928734:3648928734(0) ack 3302818156 win 5792 <mss 1460,sackOK,timestamp
18102136[|tcp]>
09:06:15.974464 IP 10.6.1.2.39895 > 192.168.128.20.80: S
3302818155:3302818155(0) win 5840 <mss 1460,sackOK,timestamp 13604528[|tcp]>
09:06:15.974492 IP 66.249.89.99.80 > 10.6.1.2.39895: S
3648928734:3648928734(0) ack 3302818156 win 5792 <mss 1460,sackOK,timestamp
18102886[|tcp]>
09:06:16.233344 IP 66.249.89.99.80 > 10.6.1.2.39893: S
3551948981:3551948981(0) ack 3215288824 win 5792 <mss 1460,sackOK,timestamp
18102951[|tcp]>
0


squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:03:46.982444 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 <mss 1460,sackOK,timestamp
18065645[|tcp]>
09:03:49.982585 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 <mss 1460,sackOK,timestamp
18066395[|tcp]>
09:03:50.334072 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0)

squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:03:46.982444 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 <mss 1460,sackOK,timestamp
18065645[|tcp]>
09:03:49.982585 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0) ack 3133545990 win 5792 <mss 1460,sackOK,timestamp
18066395[|tcp]>
09:03:50.334072 IP 66.249.89.104.80 > 10.6.1.2.48082: S
3479803592:3479803592(0)


After I added a route :-

squid:~> ip route add 10.6.1.0/24 via 192.168.128.50

squid:~> tcpdump -ni br0 host 10.6.1.2 and port 80
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 68 bytes
09:12:55.957274 IP 10.6.1.2.47574 > 192.168.128.20.80: S
3726051898:3726051898(0) win 5840 <mss 1460,sackOK,timestamp 13704510[|tcp]>
09:12:55.957398 IP 66.249.89.147.80 > 10.6.1.2.47574: S
4058179260:4058179260(0) ack 3726051899 win 5792 <mss 1460,sackOK,timestamp
18202862[|tcp]>
09:12:55.957777 IP 10.6.1.2.47574 > 192.168.128.20.80: . ack 4058179261 win
92 <nop,nop,timestamp 13704510 18202862>


squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:12:55.962016 IP 10.6.1.2.43328 > 66.249.89.99.80: S
4071804540:4071804540(0) win 5840 <mss 1460,sackOK,timestamp 18202863[|tcp]>
09:12:56.403123 IP 66.249.89.99.80 > 10.6.1.2.43328: S
3907206245:3907206245(0) ack 4071804541 win 8472 <mss
1412,nop,nop,sackOK,nop,wscale 0,nop,nop,[|tcp]>

squid:~> tcpdump -ni eth0 host 10.6.1.2 and port 80 tcpdump: WARNING: eth0:
no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 68 bytes
09:12:55.962016 IP 10.6.1.2.43328 > 66.249.89.99.80: S
4071804540:4071804540(0) win 5840 <mss 1460,sackOK,timestamp 18202863[|tcp]>
09:12:56.403123 IP 66.249.89.99.80 > 10.6.1.2.43328: S
3907206245:3907206245(0) ack 4071804541 win 8472 <mss
1412,nop,nop,sackOK,nop,wscale 0,nop,nop,[|tcp]>
09:12:56.403155 IP 10.6.1.2.43328 > 66.249.89.99.80: . ack 1 win 46
<nop,nop,timestamp 18202973 41623216>
09:12:56.403560 IP 10.6.1.2.43328 > 66.249.89.99.80: P 1:1400(1399) ack 1
win 46 <nop,nop,timestamp 18202974 41623216>
0



Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my own problem ?

Ming-Ching Tiew

From: "Ming-Ching Tiew" <[hidden email]>

>
> I am using squid in a Linux box setting up as a bridge, and have
> set up ebtables and iptables following the documentation
> available on the Net :-
>
> ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
>   --ip-destination-port 80 -j redirect --redirect-target ACCEPT
>
> iptables -t tproxy -A PREROUTING -i br0 -p tcp --dport 80 \
>   -j TPROXY --on-port 80
>
>
> On a brief glance it seems it's working properly but upon detail
> investigation,
> there are some issues.
> ....
> I am looking for something more transparent. Any insight is much
> appreciated.


I think I fixed the issue by changing the ebtables rule to :-

ebtables -t broute -A BROUTING --logical-in br0 -p IPv4 --ip-protocol 6 \
   --ip-destination-port 80 -j redirect --redirect-target DROP

Note that subtle changes. With that I don't need to add routes and other
shits.
I would appreciate feedback from others to see if this is a better rule than
the original one.

Regards.

Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my own problem ?

Ming-Ching Tiew
In reply to this post by Ming-Ching Tiew
> I think I fixed the issue by changing the ebtables rule to :-
>
> ebtables -t broute -A BROUTING --logical-in br0 -p IPv4 --ip-protocol 6 \
>    --ip-destination-port 80 -j redirect --redirect-target DROP
>
> Note that subtle changes. With that I don't need to add routes and other
> shits.
> I would appreciate feedback from others to see if this is a better rule
than
> the original one.
>

Sorry false alarm. The new rule bypasses all traffic from squid, that's why
it is working. Back to square ones. Need to work harder on it.

:-(

Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my own problem ?

Ming-Ching Tiew
In reply to this post by Ming-Ching Tiew

From: "Ming-Ching Tiew" <[hidden email]>
>
> It seems then to me that the http reply ( source port 80 ) has also be
> directed ***INTO*** the Bridge/Squid S. Why is that so ? Why didn't the
> Bridge/Squid forward the reply packet to the other side of the
> interface ?
>
> I am looking for something more transparent. Any insight is much
> appreciated.
>

Sorry for taking up your bandwidth it looks like I am looking for something
impossible at this moment.

The http reply has to go back **INTO** the Bridge/Squid box, so that it can
make
a cache copy, as such the http reply to the http request will have to ROUTE
out
from the bridge/squid box ( verses  BRIDGE ).

Unless some enhancement is made to do some kind of "connection tracking",
and thus reply the packet back to the mac address of the original requests.

Regards.



Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my own problem ?

Henrik Nordström
In reply to this post by Ming-Ching Tiew
fre 2007-07-06 klockan 09:41 +0800 skrev Ming-Ching Tiew:

> However, if there is a subnet B, which is connected to subnet A, via
> a router R, then all the machines inside subnet B will have problem
> getting the http reply packets but http request packets have no
> problem going out.

Do your proxy have a return path route for subnet B?

> Then I added a route inside the Bridge/Squid S for the subnet B via
> router R, then the web request/reply problem is solved.

Ah, you didn't.. You need routing for all sessions you intercept, or the
proxy server won't know where to return traffic..

> It seems then to me that the http reply ( source port 80 ) has also be
> directed ***INTO*** the Bridge/Squid S. Why is that so ? Why didn't the
> Bridge/Squid forward the reply packet to the other side of the
> interface ?

I'd say that your ebtables rules is perhaps a bit too broad..

a packet matched by the ebtables redirect rule will be diverted from the
bridge into the TCP/IP stack to be routed, NAT:ed etc..

Regards
Henrik

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my own problem ?

Henrik Nordström
In reply to this post by Ming-Ching Tiew
fre 2007-07-06 klockan 11:07 +0800 skrev Ming-Ching Tiew:

> I think I fixed the issue by changing the ebtables rule to :-
>
> ebtables -t broute -A BROUTING --logical-in br0 -p IPv4 --ip-protocol 6 \
>    --ip-destination-port 80 -j redirect --redirect-target DROP

Should be

ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
        -i eth0 --ip-source your.lan.network/mask \
        --ip-destination-port 80 -j redirect --redirect-target ACCEPT

with eth0 being the interface connected to your LAN, and
your.lan.network/mask the IP network used on your LAN.

Do NOT redirects networks for which you do not have routing configured,
doing so will not work.

If you are to use TPROXY then I'd recommend using the bridge-netfilter
integration instead of ebtables. This because TPROXY needs to intercept
the return traffic as well, not just lan->internet traffic. It's
possible to add ebtables rules for this by doing rules inverse to the
above.

ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
        --ip-destination your.lan.network/mask \
        --ip-source-port 80 -j redirect --redirect-target ACCEPT


Regards
Henrik

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: transparent tproxy: routing issue or my own problem ?

Henrik Nordström
In reply to this post by Ming-Ching Tiew
fre 2007-07-06 klockan 15:08 +0800 skrev Ming-Ching Tiew:

> Sorry for taking up your bandwidth it looks like I am looking for something
> impossible at this moment.

Not impossible, but quite hard. A lot easier to make sure you have
proper routing set up on the bridge...

Regards
Henrik

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my ownproblem ?

Ming-Ching Tiew
In reply to this post by Henrik Nordström
From: "Henrik Nordstrom" <[hidden email]>

>
> ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
> -i eth0 --ip-source your.lan.network/mask \
> --ip-destination-port 80 -j redirect --redirect-target ACCEPT

If you look at the http://ebtables.sourceforge.net/examples.html#easy,
it says when re-direct on ethX, it should be DROP instead of accept,
while doing it on brX, then it should be ACCEPT. I am no ebtables
expert, correctly if I am wrong. :-)

> If you are to use TPROXY then I'd recommend using the bridge-netfilter
> integration instead of ebtables.

I lost you, what do you mean by bridge-netfilter integration. Any URL ?

> This because TPROXY needs to intercept
> the return traffic as well, not just lan->internet traffic. It's
> possible to add ebtables rules for this by doing rules inverse to the
> above.
>
>
> ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 \
> --ip-destination your.lan.network/mask \
> --ip-source-port 80 -j redirect --redirect-target ACCEPT
>

Hmmm interesting. I do not  have this rule in my system and I am
able to surf the NET via the bridge/squid ( if I set up proper routing ).
Now you make me wonder if I have set it up correctly. It seems to
me that the internet-->lan traffic is already heading into the bridge,
so there is no need to hijack it again. Am I missing something ?

Regards.






Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or my ownproblem ?

Henrik Nordström
tis 2007-07-10 klockan 00:14 +0800 skrev Ming-Ching Tiew:

> I lost you, what do you mean by bridge-netfilter integration. Any URL ?

It's a kernel option.

> Hmmm interesting. I do not  have this rule in my system and I am
> able to surf the NET via the bridge/squid ( if I set up proper routing ).

It will work fine until you use TPROXY to have Squid fake the source IP
on the requests it sends..

> Now you make me wonder if I have set it up correctly. It seems to
> me that the internet-->lan traffic is already heading into the bridge,
> so there is no need to hijack it again. Am I missing something ?

The bridge needs to know to forward that traffic to Squid if it's a
response to the request sent by Squid..

Regards
Henrik

signature.asc (316 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or myownproblem ?

Ming-Ching Tiew

From: "Henrik Nordstrom" <[hidden email]>

>
>> I lost you, what do you mean by bridge-netfilter integration. Any URL ?
>
> It's a kernel option.

Did you mean

CONFIG_BRIDGE_NETFILTER=y

and all these :-

#
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m

I have plenty of those inside many kernel and modules. How do I use it
instead of TPROXY ?

>> Hmmm interesting. I do not  have this rule in my system and I am
>> able to surf the NET via the bridge/squid ( if I set up proper routing ).
>
> It will work fine until you use TPROXY to have Squid fake the source IP
> on the requests it sends..

As far as I can tell my system is already faking the source IP. But I might
be
wrong. :-)

Do you mean it is a result of some of the kernel CONFIGs which I had instead
of TPROXY module ?

Regards.


Reply | Threaded
Open this post in threaded view
|

Re: transparent tproxy: routing issue or myownproblem ?

Henrik Nordström
On tis, 2007-07-10 at 09:39 +0800, Ming-Ching Tiew wrote:

> From: "Henrik Nordstrom" <[hidden email]>
>
> >
> >> I lost you, what do you mean by bridge-netfilter integration. Any URL ?
> >
> > It's a kernel option.
>
> Did you mean
>
> CONFIG_BRIDGE_NETFILTER=y
Yes that one..

> and all these :-
>
> #
> CONFIG_BRIDGE_NF_EBTABLES=m

Not needed. Thats for ebtables.

> I have plenty of those inside many kernel and modules. How do I use it
> instead of TPROXY ?

It's not instead of..

Regards
Henrik

signature.asc (316 bytes) Download Attachment