Add a prefix/suffix if a domain is not resolved?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Add a prefix/suffix if a domain is not resolved?

Olivier Sannier
Hello all,

I'm using Seamonkey without a proxy for general surfing at home and one
thing that I came to like is the fact that if I type in an address that
does not match any DNS record, it will try again by adding ".com" at the
end if it's not already there. Similarly, it will add "www." in front if
it's not already there.
When I'm on the move, I often use my own squid proxy to connect to the
internet and I loose this behavior because Squid replies with a "not
found" page whenever there was no DNS record associated with the URL.
Hence my question today:

Is it possible to tell Squid to retry the same query with a given suffix
or prefix if said prefix/suffix is not already in the URL?
If yes, how can this be done?

I looked around using various search engines, but I must say that I
could not find anything on this subject.

Thanks in advance for your help
Olivier


--
Olivier Sannier
[hidden email]
http://www.obones.com/

Retrouvez-moi sur Viadeo :
http://www.viadeo.com/invitationpersonnelle/0023asu3wwgwpzx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Francesco Chemolli
Many browsers already do this.

With Squid you can use a redirector scriptan but you'll have to write
your custom script, such a functionality is not bundled with squid.

On 8/7/09, Olivier Sannier <[hidden email]> wrote:

> Hello all,
>
> I'm using Seamonkey without a proxy for general surfing at home and one
> thing that I came to like is the fact that if I type in an address that
> does not match any DNS record, it will try again by adding ".com" at the
> end if it's not already there. Similarly, it will add "www." in front if
> it's not already there.
> When I'm on the move, I often use my own squid proxy to connect to the
> internet and I loose this behavior because Squid replies with a "not
> found" page whenever there was no DNS record associated with the URL.
> Hence my question today:
>
> Is it possible to tell Squid to retry the same query with a given suffix
> or prefix if said prefix/suffix is not already in the URL?
> If yes, how can this be done?
>
> I looked around using various search engines, but I must say that I
> could not find anything on this subject.
>
> Thanks in advance for your help
> Olivier
>
>
> --
> Olivier Sannier
> [hidden email]
> http://www.obones.com/
>
> Retrouvez-moi sur Viadeo :
> http://www.viadeo.com/invitationpersonnelle/0023asu3wwgwpzx
>


--
    /kinkie
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Amos Jeffries
Administrator
In reply to this post by Olivier Sannier
Olivier Sannier wrote:
> Hello all,
>
> I'm using Seamonkey without a proxy for general surfing at home and one
> thing that I came to like is the fact that if I type in an address that
> does not match any DNS record, it will try again by adding ".com" at the
> end if it's not already there.

It's not realistic to try all 280 common TLDs and their ~3000 common
ccTLDs. What the browsers most often do is:
   a) assume that the user is living in North America and uses .com
   b) send a search query ie:  http://google.com/search?q=example and
redirect to the first FQDN that returns  (for q=example its often
example.com etc).


Squid uses the settings the resolv.conf of the system it's running on.
That file has a line starting with "search " followed by the possible
domains you want to assume in the order to be tested. The default
setting is to assume its part of the local network and use the local
network master domain so "http://foo/" goes to "foo.example.com" etc.
For example; to try .com then .net set it to "search .com .net".


> Similarly, it will add "www." in front if
> it's not already there.

This is not possible outside the browser. Only Internet Explorer is
broken this way AFAIK. Though firefox on non-windows will search your
past website history to hint for any URLs and auto-complete them for you.

www.* is often (to be RFC compliant now) only provided as a stub to
redirect old Internet Explorer and its people from the bad habit of
adding www to everything to the real website at the domain without www,
using either DNS or HTTP redirects.
Websites which actually need and use it as part of the full domain name
are few and slightly broken.

> When I'm on the move, I often use my own squid proxy to connect to the
> internet and I loose this behavior because Squid replies with a "not
> found" page whenever there was no DNS record associated with the URL.

The solution is to learn to enter URL properly. If IANA ever cave in and
enable unlimited .* TLD which are being discussed you are going to be
screwed by these bad habits. (Or if you ever need to use a website
without www. in it.)

> Hence my question today:
>
> Is it possible to tell Squid to retry the same query with a given suffix
> or prefix if said prefix/suffix is not already in the URL?
> If yes, how can this be done?

see above.

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE18
   Current Beta Squid 3.1.0.13
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Olivier Sannier
Hum, it seems there is a misunderstanding here.
I do not want to always add www. in front of the domain name, I want it
tried if it's not already there. Here is the scenario :

I type in "domain".
This does not resolve to anything, so I want Squid to try with ".com"
appended, that is try with "domain.com". Only .com is to be tried, not
all the possible suffixes. If "domain.com" resolves in DNS, then it
stops there and connects to it.
If the "domain.com" did not resolve to anything in DNS, then try again
with "www." prefixed this time, thus giving "www.domain.com". If that
does not resolve either, then stop there.

Now, if I type in "domain.eu", then I don't want it to try with .com
appended because there already is an extension. However, if that
"domain.eu" does not get DNS resolution, then I want the "www." prefix
part to apply.
And finally, if I type "sometld.domain.ext", I don't want it to try
anything, even if it does not resolve.

I hope this makes it clearer.

Regards
Olivier
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Amos Jeffries
Administrator
Olivier Sannier wrote:
> Hum, it seems there is a misunderstanding here.

Yes. You seem to misunderstand how domain names work...

> I do not want to always add www. in front of the domain name, I want it
> tried if it's not already there. Here is the scenario :
>
> I type in "domain".
> This does not resolve to anything, so I want Squid to try with ".com"
> appended, that is try with "domain.com". Only .com is to be tried, not
> all the possible suffixes. If "domain.com" resolves in DNS, then it
> stops there and connects to it.
> If the "domain.com" did not resolve to anything in DNS, then try again
> with "www." prefixed this time, thus giving "www.domain.com". If that
> does not resolve either, then stop there.
>
> Now, if I type in "domain.eu", then I don't want it to try with .com
> appended because there already is an extension. However, if that
> "domain.eu" does not get DNS resolution, then I want the "www." prefix
> part to apply.
> And finally, if I type "sometld.domain.ext", I don't want it to try
> anything, even if it does not resolve.

3 seconds additional loading time later you realize that the French
(Secretary general?) (http://domain.eu/) are having a little DNS trouble
and Squid is sending you to the domain squatter at http://domain.eu.com/ 
for the next three months.

   (see for yoruself,  visit both those websites).


The only three rules about domain names is that they contain certain
readable characters, separated by dot at each ownership level, and there
is no more than 255 letters between each dot.

* by "ownership level" it means:  my domain name (treenet.co.nz) is sold
by the NZ government (nz) to DomainNZ (co.nz) who sub-leased it to me
(treenet.co.nz) who runs a website (http://treenet.co.nz/)


Lets put it this way since you like examples...

Here are a couple of very real domain names:

   http://hk/          -- Hong Kong .hk domain registry
   http://www.com/     -- verisign .com registry
   http://www.com.au/  -- optus networks
   http://com.au/      -- optus networks
   http://www.co.nz/   -- a local bank

Luckily OptusNet own both domains, so it does not matter if you add www
to them.

The bank is a different story; co.nz is owned by another business, but
without a website there so far. That might change at any time. They have
been talking about it.


Here is a case where that has actually happened already.
3 _different_ websites with very different owners around the planet.

   http://tm/          - a Turkmenistan community help network
   http://tm.com/      - a North American advertising co.
   http://tm.com.my/   - a Malaysian ISP


I have just downloaded a web page from all URL links I mention above, so
they are currently operating. There used to be cool domain at just
"http://com/" as well but it seems to be dead now.


One of my most popular customers has 7 dots following the www.
Like this:
   www.a.b.c.d.e.f.g

... but only on some of their sites does www work, others it won't. Some
need it to be there, most don't.  The domain itself ends in .net, but
often contain .com it mid-way down.


So to repeat;
  Squid will _already_ try the exact domain you gave it, then try each
of a set of possible right-hand sides.
  Squid cannot determine automatically when a left-hand side is missing
what should or should not be there as part of the domain.
  Squid will _already_ add the http:// part under certain circumstances
if its missing.

We will not be making Squid do it any other way, because:
  * 280 possible right-hand side of a domain. resolv.conf offers a nice
clean way to solve the problem.
  * 987 possible left-hand sides. even when cut down to just the
registered services like www, ftp, http, _http, _http._tcp etc.
  * Both those numbers are growing.
  * A single failed DNS lookup can slow Squid by half a second.

now multiply:  280 * 987 * .5 seconds = possible web page loading time.
Please hold the line...


Amos
--
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE18
   Current Beta Squid 3.1.0.13
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Henrik Nordström
In reply to this post by Francesco Chemolli
lör 2009-08-08 klockan 15:54 +0200 skrev Kinkie:
> Many browsers already do this.
>
> With Squid you can use a redirector scriptan but you'll have to write
> your custom script, such a functionality is not bundled with squid.

Indeed.. and didn't I write such a script some many years ago? Or was it
someone else who posted one.. don't remember.

A better alternative is to use a PAC script in the browser, forwarding
only known domains to the proxy. Domains which do not resolve is then
handled by the browser like how it would do if not using a proxy (as
it's not for those domains...).

Drawback from the PAC approach is longer initial pageloads as the
clients also need to DNS lookup each visited domain, before forwarding
the request to the proxy which then also performs a DNS Lookup. But if
both is sharing the same caching dns resolver and clients have
reasonable connectivity to this DNS resolver then it should not be too
bad..  But if clients have limited connectivity then performing DNS
lookups in the clients is not something I would recommend unless
absolutely needed.

Regards
Henrik

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Olivier Sannier
Henrik Nordstrom wrote:

> lör 2009-08-08 klockan 15:54 +0200 skrev Kinkie:
>  
>> Many browsers already do this.
>>
>> With Squid you can use a redirector scriptan but you'll have to write
>> your custom script, such a functionality is not bundled with squid.
>>    
>
> Indeed.. and didn't I write such a script some many years ago? Or was it
> someone else who posted one.. don't remember.
>  
Well, I must not have used the proper search terms then. Would you have
any clues as to what I should look for to find this script?
>
>  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Henrik Nordström
mån 2009-08-10 klockan 10:40 +0200 skrev Olivier Sannier:

> > Indeed.. and didn't I write such a script some many years ago? Or was it
> > someone else who posted one.. don't remember.
> >  
> Well, I must not have used the proper search terms then. Would you have
> any clues as to what I should look for to find this script?

Good question. I don't remember. It's even possible it never got
written.

Have you investigater the PAC approach? It's my recommended way of doing
this.

Regards
Henrik

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Olivier Sannier
Henrik Nordstrom wrote:

> mån 2009-08-10 klockan 10:40 +0200 skrev Olivier Sannier:
>
>  
>>> Indeed.. and didn't I write such a script some many years ago? Or was it
>>> someone else who posted one.. don't remember.
>>>  
>>>      
>> Well, I must not have used the proper search terms then. Would you have
>> any clues as to what I should look for to find this script?
>>    
>
> Good question. I don't remember. It's even possible it never got
> written.
>
> Have you investigater the PAC approach? It's my recommended way of doing
> this.

Well, thank you for insisting on the PAC approach, at first I did not
think it could help me.
But as it turns out, it's pretty simple to use and it works quite well.
Just in case someone finds it useful, here is the script I wrote:

function FindProxyForURL(url, host) {

  // If URL has no dots in host name, send traffic direct.
    if (isPlainHostName(host))
        return "DIRECT";

  // If specific URL needs to bypass proxy, send traffic direct.
    if (shExpMatch(url,"*domain1.com*") ||
        shExpMatch(url,"*domain2.com*"))                
        return "DIRECT";
   
    // If host is not resolvable, send direct. This will allow browser
to retry with prefix/suffix
    if (!isResolvable(host)) return "DIRECT";

  // If IP address is internal or hostname resolves to internal IP, send
direct.
    var resolved_ip = dnsResolve(host);

    if (isInNet(resolved_ip, "10.0.0.0", "255.0.0.0") ||
        isInNet(resolved_ip, "172.16.0.0",  "255.240.0.0") ||
        isInNet(resolved_ip, "192.168.0.0", "255.255.0.0") ||
        isInNet(resolved_ip, "127.0.0.0", "255.255.255.0"))
        return "DIRECT";
       
    return "PROXY proxy_host:3128";
}

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Henrik Nordström
mån 2009-08-17 klockan 10:45 +0200 skrev Olivier Sannier:

>   // If specific URL needs to bypass proxy, send traffic direct.
>     if (shExpMatch(url,"*domain1.com*") ||
>         shExpMatch(url,"*domain2.com*"))                
>         return "DIRECT";

I think you meant to use host there, not url...

Regards
Henrik

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Olivier Sannier
Henrik Nordstrom wrote:

> mån 2009-08-17 klockan 10:45 +0200 skrev Olivier Sannier:
>
>  
>>   // If specific URL needs to bypass proxy, send traffic direct.
>>     if (shExpMatch(url,"*domain1.com*") ||
>>         shExpMatch(url,"*domain2.com*"))                
>>         return "DIRECT";
>>    
>
> I think you meant to use host there, not url...
Hum, yes, you are quite right, that would be safer indeed.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Adam Gaskins
In reply to this post by Amos Jeffries
Amos Jeffries,

While I truly can appreciate your border-line pathological idealism,
I think it's reasonable to desire the browser behaves as designed
when behind a proxy. For better or worse, people have grown use to
certain tools and features within their browser of choice. I do work
in the IT field, but imagine how much benefit the average Jane/Joe
can get from something like TDL guessing! Sorry to rant, but I am
also seeking an answer to this. Do you have any information on
actually making this work? There is almost certainly a work-around
for this issue, I just haven't found it yet either.

-Adam

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Add a prefix/suffix if a domain is not resolved?

Amos Jeffries
Administrator
 On Sun, 6 Nov 2011 08:47:04 +0000 (UTC), Adam Gaskins wrote:

> Amos Jeffries,
>
> While I truly can appreciate your border-line pathological idealism,
> I think it's reasonable to desire the browser behaves as designed
> when behind a proxy. For better or worse, people have grown use to
> certain tools and features within their browser of choice. I do work
> in the IT field, but imagine how much benefit the average Jane/Joe
> can get from something like TDL guessing! Sorry to rant, but I am
> also seeking an answer to this. Do you have any information on
> actually making this work? There is almost certainly a work-around
> for this issue, I just haven't found it yet either.
>
> -Adam

 I don't quite follow your argument. For an analogy, people grown used
 to using forks are encouraged to continue to use forks even when eating
 soup. Different jobs == different "best" tool, and being accustomed to
 something is no measure of best.


 As for this feature proposal and Squid. Follow my logic...

 So, (a) do many (a few million) DNS lookups for potential variations of
 the name?
  up to 1.6 days wait for a single page to load while Squid scans for
 potential URLs. The why of it is outlined in my earlier post. I guess
 this is the same reason why browsers don't do it this way. Yes the
 timings will usually be shorter, but even the less than 60 second delay
 with Squid today trying to locate IPv6 domain access is raising a lot of
 complaints.

 Browsers avoid that major problem by using a search engine to do the
 domain lookup instead of DNS.


 So, that brings us to (b): which corporate search engine do we force
 everybody to use to follow the IE behaviour?
  Serious question. None of the Squid dev team believe we have the right
 to decide that on behalf of the whole Internet community.


 Browsers get away with it because they can (and do) provide users with
 a selection of engines to use. With the user always fully able to change
 the engine choice. We can build in configuration and provide it to the
 admin, but that still forces a whole network of users onto one search
 engine. Not much better.


 Squid does provide plugin interfaces for third-party scripts to do
 anything they like to fill a certain operation (auth, URL redirect, ACL
 test, file erasure, SSL cert creation).

 One option is (c) using the redirector interface with a helper that
 does all the lookups and searching.
  As Kinkie posted earlier in the thread. That could be done with a
 local database of who the users are and what their preferences are.
 Combined with a URL redirector that does all the preference loading and
 domain searching. Or not, if you want to be mean and not give the users
 any choice.
  Either way a simple "no change" or the "proper" URL to redirect the
 user to is all that is relevant to Squid.


 Or, (d) you could try to convince the browser people to do this name
 search before contacting the proxy as well as before doing a direct
 website connection. The PAC approach favoured by Henrik earlier goes a
 way towards that without needing to wait for the browser people to add a
 new feature.



 In summary, we have a choice between (a) very annoying behaviour, (b)
 very nasty behaviour, or leaving Squid unchanged (c and d), with a
 third-party script do all the tricky work to make everybody happy. We
 opt to keep Squid simple whenever possible.

 You still get to choose from (c) and (d), and best of all, you get to
 select through them how much user choice is followed and what name
 variations you want to allow.

 Amos

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Lawn Mowing Services Sydney

lawnmowing
This post has NOT been accepted by the mailing list yet.
In reply to this post by Olivier Sannier
I am actually thankful to the holder of this web page who has shared this fantastic post at at this place.  Hedge trimming Wollongong   Home cleaning services Wollongong   Lawn mowing services Wollongong
Loading...