Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow
Hi,

I use the below:

external_acl_type nt_group ttl=20 children-startup=10 children-max=70
children-idle=10 %LOGIN /usr/lib/squid3/ext_wbinfo_group_acl

to be able to use NT groups in my squid config. This works fine in 3.2
and 3.3, but I recently tried to upgrade to 3.4 and this stopped
working. In the cache.log there are hundreds of entries like:

Could not get groups for user DOMAINuser

Whereas the correct user name should be DOMAIN\user. If I pass the
correct username to the wbinfo_group helper it works, so it seems squid
is dropping the backslash in my 3.4 install (squid-3.4.1-20131216-r13058).

Going back the 3.3.11 makes everything work as expected.

Thanks

Alex
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Amos Jeffries
Administrator
On 24/12/2013 2:28 a.m., Alex Crow wrote:

> Hi,
>
> I use the below:
>
> external_acl_type nt_group ttl=20 children-startup=10 children-max=70
> children-idle=10 %LOGIN /usr/lib/squid3/ext_wbinfo_group_acl
>
> to be able to use NT groups in my squid config. This works fine in 3.2
> and 3.3, but I recently tried to upgrade to 3.4 and this stopped
> working. In the cache.log there are hundreds of entries like:
>
> Could not get groups for user DOMAINuser
>
> Whereas the correct user name should be DOMAIN\user. If I pass the
> correct username to the wbinfo_group helper it works, so it seems squid
> is dropping the backslash in my 3.4 install (squid-3.4.1-20131216-r13058).
>
> Going back the 3.3.11 makes everything work as expected.
>
Can you test 3.4 latest snapshot with this patch on top please?

Amos


quote_unescape_mk1.patch (298 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow
On 24/12/13 02:39, Amos Jeffries wrote:

> On 24/12/2013 2:28 a.m., Alex Crow wrote:
>> Hi,
>>
>> I use the below:
>>
>> external_acl_type nt_group ttl=20 children-startup=10 children-max=70
>> children-idle=10 %LOGIN /usr/lib/squid3/ext_wbinfo_group_acl
>>
>> to be able to use NT groups in my squid config. This works fine in 3.2
>> and 3.3, but I recently tried to upgrade to 3.4 and this stopped
>> working. In the cache.log there are hundreds of entries like:
>>
>> Could not get groups for user DOMAINuser
>>
>> Whereas the correct user name should be DOMAIN\user. If I pass the
>> correct username to the wbinfo_group helper it works, so it seems squid
>> is dropping the backslash in my 3.4 install (squid-3.4.1-20131216-r13058).
>>
>> Going back the 3.3.11 makes everything work as expected.
>>
> Can you test 3.4 latest snapshot with this patch on top please?
>
> Amos
>

Hi Amos,

Yes, this works re: the helper, but unfortunately we get very high CPU
usage in 3.4.1 as opposed to 3.3.11. I was getting 80-100% after a few
minutes whereas when I reverted back to 3.3.11, I only saw the odd peak
at about 27%, and most of the time it was <10%.

No other change other than the version, config was identical.

Cheers

Alex
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Eliezer Croitoru
Hey Alex,

Can you by any chance get a top snapshot output to verify if this issue
is related to the subprocess or the parent process.

Thanks,
Eliezer

On 27/12/13 19:58, Alex Crow wrote:

>
> Hi Amos,
>
> Yes, this works re: the helper, but unfortunately we get very high CPU
> usage in 3.4.1 as opposed to 3.3.11. I was getting 80-100% after a few
> minutes whereas when I reverted back to 3.3.11, I only saw the odd peak
> at about 27%, and most of the time it was <10%.
>
> No other change other than the version, config was identical.
>
> Cheers
>
> Alex

Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow
Hi Eliezer,

I can confirm it is the subprocess, can't get a snapshot now as it's in
prod, but I did the same myself and it was definitely the kid (only 1
kid is configured).

Cheers

Alex

On 27/12/13 19:21, Eliezer Croitoru wrote:

> Hey Alex,
>
> Can you by any chance get a top snapshot output to verify if this
> issue is related to the subprocess or the parent process.
>
> Thanks,
> Eliezer
>
> On 27/12/13 19:58, Alex Crow wrote:
>>
>> Hi Amos,
>>
>> Yes, this works re: the helper, but unfortunately we get very high CPU
>> usage in 3.4.1 as opposed to 3.3.11. I was getting 80-100% after a few
>> minutes whereas when I reverted back to 3.3.11, I only saw the odd peak
>> at about 27%, and most of the time it was <10%.
>>
>> No other change other than the version, config was identical.
>>
>> Cheers
>>
>> Alex
>

Reply | Threaded
Open this post in threaded view
|

AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Rietzler, Markus (RZF, SG 324 / <RIETZLER_SOFTWARE>)
i want to join this discussion.
we are using squid 3.4.2 also with ntlm_auth and winbind. only difference is that we don't use wbinfo_group. we just "need" or "use" the username. we also have the problem, that after a few minutes squid uses 100% cpu and is getting very slow. in the cache-log I can see "increase ntmm-helper" as the max number is used. I also can see in the cache manager menu output (ntlmauthenticator) that all the configured helper are "busy".

any idea about the 100% cpu usage?

> -----Urspr√ľngliche Nachricht-----
> Von: Alex Crow [mailto:[hidden email]]
> Gesendet: Sonntag, 29. Dezember 2013 19:26
> An: Eliezer Croitoru; [hidden email]
> Betreff: Re: [squid-users] Squid 3.4 sends Windows username without
> backslash to external wbinfo_group helper
>
> Hi Eliezer,
>
> I can confirm it is the subprocess, can't get a snapshot now as it's in
> prod, but I did the same myself and it was definitely the kid (only 1
> kid is configured).
>
> Cheers
>
> Alex
>
> On 27/12/13 19:21, Eliezer Croitoru wrote:
> > Hey Alex,
> >
> > Can you by any chance get a top snapshot output to verify if this
> > issue is related to the subprocess or the parent process.
> >
> > Thanks,
> > Eliezer
> >
> > On 27/12/13 19:58, Alex Crow wrote:
> >>
> >> Hi Amos,
> >>
> >> Yes, this works re: the helper, but unfortunately we get very high CPU
> >> usage in 3.4.1 as opposed to 3.3.11. I was getting 80-100% after a few
> >> minutes whereas when I reverted back to 3.3.11, I only saw the odd peak
> >> at about 27%, and most of the time it was <10%.
> >>
> >> No other change other than the version, config was identical.
> >>
> >> Cheers
> >>
> >> Alex
> >

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Eliezer Croitoru
Hey,

There was someone in the past that asked about this ntlm helper issue.
I would try in a way to use only squid to make sure what is going on at
the same time.
If you do have a number of Requests Per Second rate it will help to
understand the basic issue.

There is a complexity issue when for example a proxy was hit by 400
requests in one sec and authentication is being used.
There is also the basic issue that when authentication is being done on
a network and the network is not fast enough or it has too much latency,
the process will grow and grow over time.

100% cpu usage can be discovered sometimes but in a case the cache-mgr
is not responsive the only tools available are:
top
netstat
ss
iptables
iptraf

And there are couple other nice tools which can verify the basic
assumption that this network might need more then it have or need.

Eliezer

On 06/01/14 11:53, Rietzler, Markus (RZF, SG 324 / <RIETZLER_SOFTWARE>)
wrote:
> i want to join this discussion.
> we are using squid 3.4.2 also with ntlm_auth and winbind. only difference is that we don't use wbinfo_group. we just "need" or "use" the username. we also have the problem, that after a few minutes squid uses 100% cpu and is getting very slow. in the cache-log I can see "increase ntmm-helper" as the max number is used. I also can see in the cache manager menu output (ntlmauthenticator) that all the configured helper are "busy".
>
> any idea about the 100% cpu usage?

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Amos Jeffries
Administrator
On 2014-01-07 09:14, Eliezer Croitoru wrote:

> Hey,
>
> There was someone in the past that asked about this ntlm helper issue.
> I would try in a way to use only squid to make sure what is going on
> at the same time.
> If you do have a number of Requests Per Second rate it will help to
> understand the basic issue.
>
> There is a complexity issue when for example a proxy was hit by 400
> requests in one sec and authentication is being used.
> There is also the basic issue that when authentication is being done
> on a network and the network is not fast enough or it has too much
> latency, the process will grow and grow over time.
>
> 100% cpu usage can be discovered sometimes but in a case the cache-mgr
> is not responsive the only tools available are:
> top
> netstat
> ss
> iptables
> iptraf
>
> And there are couple other nice tools which can verify the basic
> assumption that this network might need more then it have or need.


strace or gdb would seem to be more appropriate at this point to find
out what logics Squid is running that consume the CPU. Assuming the
Squid binary has been compiled with full debug/symbols information
retained.

Amos

>
> Eliezer
>
> On 06/01/14 11:53, Rietzler, Markus (RZF, SG 324 / <RIETZLER_SOFTWARE>)
> wrote:
>> i want to join this discussion.
>> we are using squid 3.4.2 also with ntlm_auth and winbind. only
>> difference is that we don't use wbinfo_group. we just "need" or "use"
>> the username. we also have the problem, that after a few minutes squid
>> uses 100% cpu and is getting very slow. in the cache-log I can see
>> "increase ntmm-helper" as the max number is used. I also can see in
>> the cache manager menu output (ntlmauthenticator) that all the
>> configured helper are "busy".
>>
>> any idea about the 100% cpu usage?
Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow
In reply to this post by Eliezer Croitoru
Hi,

Just noticed something in the changelogs for the nightly build that
might mean this is fixed - I'm optimistic anyway:

Tue 2014-01-21 20:29:15 -0700
<http://www.squid-cache.org/Versions/v3/3.4/changesets/squid-3.4-13079.patch>
        Amos Jeffries +10 -2
        Fix external_acl_type async loop failures



Now I just need to figure out the "Unhandled exception: c" errors that
kill my squid every so often. It seems to be a rare issue as from
googling only myself and two other people seem to have faced it.

Cheers

Alex

On 06/01/14 20:14, Eliezer Croitoru wrote:

> Hey,
>
> There was someone in the past that asked about this ntlm helper issue.
> I would try in a way to use only squid to make sure what is going on
> at the same time.
> If you do have a number of Requests Per Second rate it will help to
> understand the basic issue.
>
> There is a complexity issue when for example a proxy was hit by 400
> requests in one sec and authentication is being used.
> There is also the basic issue that when authentication is being done
> on a network and the network is not fast enough or it has too much
> latency, the process will grow and grow over time.
>
> 100% cpu usage can be discovered sometimes but in a case the cache-mgr
> is not responsive the only tools available are:
> top
> netstat
> ss
> iptables
> iptraf
>
> And there are couple other nice tools which can verify the basic
> assumption that this network might need more then it have or need.
>
> Eliezer
>
> On 06/01/14 11:53, Rietzler, Markus (RZF, SG 324 /
> <RIETZLER_SOFTWARE>) wrote:
>> i want to join this discussion.
>> we are using squid 3.4.2 also with ntlm_auth and winbind. only
>> difference is that we don't use wbinfo_group. we just "need" or "use"
>> the username. we also have the problem, that after a few minutes
>> squid uses 100% cpu and is getting very slow. in the cache-log I can
>> see "increase ntmm-helper" as the max number is used. I also can see
>> in the cache manager menu output (ntlmauthenticator) that all the
>> configured helper are "busy".
>>
>> any idea about the 100% cpu usage?
>

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Eliezer Croitoru
Hey Alex and Markus,

For now we do have a solution to more issues that we had in the past.
This is one of the greatest things!!
One step at a time we do see that we find every one of the issues.
I have tried to write some lightweight helper to verify couple issues.
The src code is at:
http://wiki.squid-cache.org/EliezerCroitoru/Drafts/AdaFakeHelper

The above code is one of the lightest helpers that do exist.
it is not too portable but it shows what is the difference between a
ruby\perl\python script to a code which is written in a compiled language.

the memory print of this software is below 512 Bytes but I am still not
sure if it's 512 bytes or KB in either it is one of the simplest scripts
that do exist and can be tested to verify couple issues.

Eliezer

On 22/01/14 17:12, Alex Crow wrote:

> Hi,
>
> Just noticed something in the changelogs for the nightly build that
> might mean this is fixed - I'm optimistic anyway:
>
> Tue 2014-01-21 20:29:15 -0700
> <http://www.squid-cache.org/Versions/v3/3.4/changesets/squid-3.4-13079.patch>
>      Amos Jeffries     +10 -2
>      Fix external_acl_type async loop failures
>
>
>
> Now I just need to figure out the "Unhandled exception: c" errors that
> kill my squid every so often. It seems to be a rare issue as from
> googling only myself and two other people seem to have faced it.
>
> Cheers
>
> Alex

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Amos Jeffries
Administrator
In reply to this post by Alex Crow
On 23/01/2014 4:12 a.m., Alex Crow wrote:

> Hi,
>
> Just noticed something in the changelogs for the nightly build that
> might mean this is fixed - I'm optimistic anyway:
>
> Tue 2014-01-21 20:29:15 -0700
> <http://www.squid-cache.org/Versions/v3/3.4/changesets/squid-3.4-13079.patch>
>     Amos Jeffries     +10 -2    
>     Fix external_acl_type async loop failures
>

If it does then we can peg the problem down to being the well-known
systemic issues in NTLM handshake.

Amos

>
>
> Now I just need to figure out the "Unhandled exception: c" errors that
> kill my squid every so often. It seems to be a rare issue as from
> googling only myself and two other people seem to have faced it.

That mysterious 'c' again :-(. Some part of Squid is using the Must()
exception mechanism when it should be using assert() instead.

If you have core dumps enabled you may be able to locate in there what
the details that 'c' variable contain about the exception and where its
coming from. Without that there is little hope of fixing it any time soon.

Amos

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow
Hi Amos,

I get the following:

# gdb squid3 core
GNU gdb (GDB) 7.0.1-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/squid3...Reading symbols from
/usr/lib/debug/usr/sbin/squid3...done.
(no debugging symbols found)...done.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libpthread.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libcap.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcap.so.2
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libattr.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libattr.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Core was generated by `(squid-1) -YC -f /etc/squid3/squid.conf'.
Program terminated with signal 6, Aborted.
#0  0x00007f0a2911a1b5 in raise () from /lib/libc.so.6


Not sure if that helps, it doesn't look too helpful.

Any ideas what else I can do?

Cheers

Alex

On 23/01/14 08:31, Amos Jeffries wrote:

> On 23/01/2014 4:12 a.m., Alex Crow wrote:
>> Hi,
>>
>> Just noticed something in the changelogs for the nightly build that
>> might mean this is fixed - I'm optimistic anyway:
>>
>> Tue 2014-01-21 20:29:15 -0700
>> <http://www.squid-cache.org/Versions/v3/3.4/changesets/squid-3.4-13079.patch>
>>      Amos Jeffries     +10 -2
>>      Fix external_acl_type async loop failures
>>
> If it does then we can peg the problem down to being the well-known
> systemic issues in NTLM handshake.
>
> Amos
>
>>
>> Now I just need to figure out the "Unhandled exception: c" errors that
>> kill my squid every so often. It seems to be a rare issue as from
>> googling only myself and two other people seem to have faced it.
> That mysterious 'c' again :-(. Some part of Squid is using the Must()
> exception mechanism when it should be using assert() instead.
>
> If you have core dumps enabled you may be able to locate in there what
> the details that 'c' variable contain about the exception and where its
> coming from. Without that there is little hope of fixing it any time soon.
>
> Amos
>

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Amos Jeffries
Administrator
On 6/02/2014 2:17 a.m., Alex Crow wrote:

> Hi Amos,
>
> I get the following:
>
> # gdb squid3 core
> GNU gdb (GDB) 7.0.1-debian
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/squid3...Reading symbols from
> /usr/lib/debug/usr/sbin/squid3...done.
> (no debugging symbols found)...done.


> Not sure if that helps, it doesn't look too helpful.
>
> Any ideas what else I can do?
>

Is the squid3-dbg package available? It has the debug symbols which are
needed to make these traces meaningful.

Amos

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow
Amos,

Yes, I compiled a Debian package and installed the squid3*dbg*.deb file.

This is a bit tricky as this is a production just from testing with a
few clients the problem does not appear.

I can definitely say that

/usr/lib/debug/usr/sbin/squid3

Is there as is fairly large so I don't know why there are missing symbols.

Cheers

Alex

On 05/02/14 15:10, Amos Jeffries wrote:

> On 6/02/2014 2:17 a.m., Alex Crow wrote:
>> Hi Amos,
>>
>> I get the following:
>>
>> # gdb squid3 core
>> GNU gdb (GDB) 7.0.1-debian
>> Copyright (C) 2009 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /usr/sbin/squid3...Reading symbols from
>> /usr/lib/debug/usr/sbin/squid3...done.
>> (no debugging symbols found)...done.
>
>> Not sure if that helps, it doesn't look too helpful.
>>
>> Any ideas what else I can do?
>>
> Is the squid3-dbg package available? It has the debug symbols which are
> needed to make these traces meaningful.
>
> Amos
>

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Amos Jeffries
Administrator
On 2014-02-06 11:09, Alex Crow wrote:

> Amos,
>
> Yes, I compiled a Debian package and installed the squid3*dbg*.deb
> file.
>
> This is a bit tricky as this is a production just from testing with a
> few clients the problem does not appear.
>
> I can definitely say that
>
> /usr/lib/debug/usr/sbin/squid3
>
> Is there as is fairly large so I don't know why there are missing
> symbols.



For production servers I use this minimal-downtime debugging script. It
or small variations have worked well for a few clients on production
server passing upwards of 10K req/sec for several hundred users.
http://wiki.squid-cache.org/SquidFaq/BugReporting#Using_gdb_debugger_on_a_live_proxy_.28with_minimal_downtime.29

Amos


>
> Cheers
>
> Alex
>
> On 05/02/14 15:10, Amos Jeffries wrote:
>> On 6/02/2014 2:17 a.m., Alex Crow wrote:
>>> Hi Amos,
>>>
>>> I get the following:
>>>
>>> # gdb squid3 core
>>> GNU gdb (GDB) 7.0.1-debian
>>> Copyright (C) 2009 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>>> copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-linux-gnu".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>...
>>> Reading symbols from /usr/sbin/squid3...Reading symbols from
>>> /usr/lib/debug/usr/sbin/squid3...done.
>>> (no debugging symbols found)...done.
>>
>>> Not sure if that helps, it doesn't look too helpful.
>>>
>>> Any ideas what else I can do?
>>>
>> Is the squid3-dbg package available? It has the debug symbols which
>> are
>> needed to make these traces meaningful.
>>
>> Amos
>>
Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow

On 06/02/14 07:56, Amos Jeffries wrote:

> On 2014-02-06 11:09, Alex Crow wrote:
>> Amos,
>>
>> Yes, I compiled a Debian package and installed the squid3*dbg*.deb file.
>>
>> This is a bit tricky as this is a production just from testing with a
>> few clients the problem does not appear.
>>
>> I can definitely say that
>>
>> /usr/lib/debug/usr/sbin/squid3
>>
>> Is there as is fairly large so I don't know why there are missing
>> symbols.
>
>
>
> For production servers I use this minimal-downtime debugging script.
> It or small variations have worked well for a few clients on
> production server passing upwards of 10K req/sec for several hundred
> users.
> http://wiki.squid-cache.org/SquidFaq/BugReporting#Using_gdb_debugger_on_a_live_proxy_.28with_minimal_downtime.29 
>
>
> Amos
>
>

Hi

Thanks for that - I did get a backtrace today...

Program terminated with signal 6, Aborted.
#0  0x00007fa89b3fb1b5 in raise () from /lib/libc.so.6
(gdb) backtrace
#0  0x00007fa89b3fb1b5 in raise () from /lib/libc.so.6
#1  0x00007fa89b3fdfc0 in abort () from /lib/libc.so.6
#2  0x00007fa89bc8fdc5 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/libstdc++.so.6
#3  0x00007fa89bc8e166 in ?? () from /usr/lib/libstdc++.so.6
#4  0x00007fa89bc8e193 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0x00007fa89bc8e216 in __cxa_rethrow () from /usr/lib/libstdc++.so.6
#6  0x00000000005d457c in SquidMainSafe (argc=<value optimized out>,
argv=<value optimized out>) at main.cc:1246
#7  main (argc=<value optimized out>, argv=<value optimized out>) at
main.cc:1234

Thanks

Alex

Reply | Threaded
Open this post in threaded view
|

Re: AW: [squid-users] Squid 3.4 sends Windows username without backslash to external wbinfo_group helper

Alex Crow

> Hi
>
> Thanks for that - I did get a backtrace today...
>
> Program terminated with signal 6, Aborted.
> #0  0x00007fa89b3fb1b5 in raise () from /lib/libc.so.6
> (gdb) backtrace
> #0  0x00007fa89b3fb1b5 in raise () from /lib/libc.so.6
> #1  0x00007fa89b3fdfc0 in abort () from /lib/libc.so.6
> #2  0x00007fa89bc8fdc5 in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/libstdc++.so.6
> #3  0x00007fa89bc8e166 in ?? () from /usr/lib/libstdc++.so.6
> #4  0x00007fa89bc8e193 in std::terminate() () from
> /usr/lib/libstdc++.so.6
> #5  0x00007fa89bc8e216 in __cxa_rethrow () from /usr/lib/libstdc++.so.6
> #6  0x00000000005d457c in SquidMainSafe (argc=<value optimized out>,
> argv=<value optimized out>) at main.cc:1246
> #7  main (argc=<value optimized out>, argv=<value optimized out>) at
> main.cc:1234
>
> Thanks
>
> Alex
>

Hi,

Is this of any help?

Also I've had to go back to 3.2.11 as 3.4.x is still using way too much
CPU, I get users (about 350) complaining about extreme slowness by
lunchtime, and squid is using >90% CPU. In 3.2.11 it's always around 15%.

Request are fairly low:

Average HTTP requests per minute since start:    2647.4

Cheers

Alex




Reply | Threaded
Open this post in threaded view
|

Unhandled exception: c

Alex Crow
HI all,

I forgot I still have the issue in the subject bugging me too. Is the
below backtrace of any use or do I need to provide more?

Thanks

Alex
On 07/02/14 10:41, Alex Crow wrote:

>
>> Hi
>>
>> Thanks for that - I did get a backtrace today...
>>
>> Program terminated with signal 6, Aborted.
>> #0  0x00007fa89b3fb1b5 in raise () from /lib/libc.so.6
>> (gdb) backtrace
>> #0  0x00007fa89b3fb1b5 in raise () from /lib/libc.so.6
>> #1  0x00007fa89b3fdfc0 in abort () from /lib/libc.so.6
>> #2  0x00007fa89bc8fdc5 in __gnu_cxx::__verbose_terminate_handler() ()
>> from /usr/lib/libstdc++.so.6
>> #3  0x00007fa89bc8e166 in ?? () from /usr/lib/libstdc++.so.6
>> #4  0x00007fa89bc8e193 in std::terminate() () from
>> /usr/lib/libstdc++.so.6
>> #5  0x00007fa89bc8e216 in __cxa_rethrow () from /usr/lib/libstdc++.so.6
>> #6  0x00000000005d457c in SquidMainSafe (argc=<value optimized out>,
>> argv=<value optimized out>) at main.cc:1246
>> #7  main (argc=<value optimized out>, argv=<value optimized out>) at
>> main.cc:1234
>>
>> Thanks
>>
>> Alex
>>
>
> Hi,
>
> Is this of any help?
>
> Also I've had to go back to 3.2.11 as 3.4.x is still using way too
> much CPU, I get users (about 350) complaining about extreme slowness
> by lunchtime, and squid is using >90% CPU. In 3.2.11 it's always
> around 15%.
>
> Request are fairly low:
>
> Average HTTP requests per minute since start:    2647.4
>
> Cheers
>
> Alex
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Unhandled exception: c

Amos Jeffries
Administrator
On 26/04/2014 5:38 a.m., Alex Crow wrote:
> HI all,
>
> I forgot I still have the issue in the subject bugging me too. Is the
> below backtrace of any use or do I need to provide more?

Unfortunately yes these unhandled excetion ones do not show where the
exception came from. cache.log should contain the error/fatal message
which is a better clue.

Amos
Reply | Threaded
Open this post in threaded view
|

Re: Unhandled exception: c

Eliezer Croitoru
I have written a basic external_acl helper in ruby which dies under
squid 3.4.4 very often.
Then it's kind of "loosing" the STDIN and I get couple messages about in
STDOUT\ERR.

The process continues to run but cannot be contacted.

I am almost sure this issue is kind of similar but not yet sure.
Are we sure about the source of the issue? helper?squid code?
I think I can write a "simulation" of network slowdowns by the ntlm
process etc.

What do you think?

Eliezer
On 04/26/2014 12:19 PM, Amos Jeffries wrote:
> Unfortunately yes these unhandled excetion ones do not show where the
> exception came from. cache.log should contain the error/fatal message
> which is a better clue.
>
> Amos

12