Quantcast

Squid stopped working after cache.log and access.log rotation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Squid stopped working after cache.log and access.log rotation

Chee M Gui

Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Antony Stone
On Wednesday 22 March 2017 at 16:17:32, Chee M Gui wrote:

> Hi All
>
> We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It
> ran fine at first but stopped working after a while.   telnet server 3128
> still works, i.e., opens a blank window, but Squid is just not accepting
> requests.    Then we realized that there is no new access.log file.   The
> access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like
> Squid wasn't able to create a new access.log?  We could not find any error
> message in syslog or the cache.log.      We haven't rebooted the server
> because we want to know what went wrong.   It isn't the firewall blocking
> Squid because Squid was working fine all the while until recently.  Also
> after it stopped working, we disabled the firewall to see if it would work
> but it still didn't work.
>
> root@paproxy:/var/log/squid# ls -alt
> total 15536
> drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
> -rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
> drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
> -rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
> -rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
> -rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
> -rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state
>
> Any ideas what went wrong?

Any chance you've run out of disk space?


Antony.

--
I bought a book about anti-gravity.  The reviews say you can't put it down.

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Jason B. Nance
In reply to this post by Chee M Gui
If you do "lsof /var/log | grep -i delete" does it show squid writing to a deleted access.log / cache.log?

j



From: "Chee M Gui" <[hidden email]>
To: [hidden email]
Sent: Wednesday, March 22, 2017 10:17:32 AM
Subject: [squid-users] Squid stopped working after cache.log and access.log        rotation


Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Chee M Gui
Thank you Jason.

lsof /var/log | grep -i delete  does not show anything
Likewise for  lsof /var/log/squid | grep -i delete

Thank you Antony
No, we have a lot of disk space.



root@paproxy:/var/log# lsof /var/log/squid | grep -i delete
root@paproxy:/var/log# lsof /var/log/squid | more
root@paproxy:/var/log# ps -ax | grep squid
 1708 pts/0    S+     0:00 grep --color=auto squid
25482 ?        Ss     0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf
25484 ?        S      2:27 (squid-1) -YC -f /etc/squid/squid.conf
25485 ?        S      0:03 (logfile-daemon) /var/log/squid/access.log
root@paproxy:/var/log# lsof /var/log/squid | grep -i delete
root@paproxy:/var/log# cd squid
root@paproxy:/var/log/squid# ls -al
total 15540
drwxr-xr-x 2 proxy proxy      4096 Mar 23 06:25 .
drwxrwxr-x 9 root  syslog     4096 Mar 23 06:25 ..
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy        63 Mar 23 06:25 cache.log
-rw-r----- 1 proxy proxy        63 Mar 22 06:25 cache.log.1
-rw-r----- 1 proxy proxy        63 Mar 19 06:25 cache.log.2
-rw-r----- 1 proxy proxy        83 Mar 21 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state
root@paproxy:/var/log/squid#


On Wed, Mar 22, 2017 at 8:23 AM, Jason B. Nance <[hidden email]> wrote:
If you do "lsof /var/log | grep -i delete" does it show squid writing to a deleted access.log / cache.log?

j



From: "Chee M Gui" <[hidden email]>
To: [hidden email]
Sent: Wednesday, March 22, 2017 10:17:32 AM
Subject: [squid-users] Squid stopped working after cache.log and access.log        rotation


Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users




_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Antony Stone
On Thursday 23 March 2017 at 17:36:01, Chee M Gui wrote:

> Thank you Jason.
>
> lsof /var/log | grep -i delete  does not show anything
> Likewise for  lsof /var/log/squid | grep -i delete
>
> Thank you Antony
> No, we have a lot of disk space.

What does your squid logfile rotation script contain?


Antony.

--
What makes you think I know what I'm talking about?
I just have more O'Reilly books than most people.

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Eliezer Croitoru
In reply to this post by Chee M Gui
There is another option!
The log rotate script is doing something nasty or the systemd service file start up squid in a weird way.
The output of:
$ systemctl status squid
$ top -n1 -b
$ ps aux
$ df -h
$ netstat -ntulp
$ lsof -n|egrep "proxy|squid"

How many clients this system has?
Is the system facing the Internet directly or behind some nat(aws or another provider)?

The above are the basic required data to understand the situation.

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On Behalf Of Chee M Gui
Sent: Wednesday, March 22, 2017 5:18 PM
To: [hidden email]
Subject: [squid-users] Squid stopped working after cache.log and access.log rotation



Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Chee M Gui
Hi Eliezer

Thank you for the response, and sorry for the late reply.

As requested, here are the output of the commands you suggested:
root@paproxy:/# systemctl status squid
● squid.service - LSB: Squid HTTP Proxy version 3.x
   Loaded: loaded (/etc/init.d/squid; bad; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-15 14:47:53 PDT; 1 months 5 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 4
   Memory: 54.4M
      CPU: 2min 38.957s
   CGroup: /system.slice/squid.service
           ├─25482 /usr/sbin/squid -YC -f /etc/squid/squid.conf
           ├─25484 (squid-1) -YC -f /etc/squid/squid.conf
           ├─25485 (logfile-daemon) /var/log/squid/access.log
           └─25486 (unlinkd)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@paproxy:/#



top - 09:11:45 up 54 days, 23:51,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  6043140 total,  4844728 free,   135292 used,  1063120 buff/cache
KiB Swap:  6222844 total,  6222844 free,        0 used.  5556300 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23355 root      20   0   41668   3776   3236 R   6.7  0.1   0:00.01 top
    1 root      20   0   37884   5968   4020 S   0.0  0.1   0:27.20 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.22 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.01 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   1:28.65 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:05.53 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:05.75 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:02.48 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   18 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
   19 root      20   0       0      0      0 S   0.0  0.0   0:01.26 khungtaskd
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback


root@paproxy:/# ps -aux | grep squid
root     23358  0.0  0.0  14224   976 pts/0    S+   09:12   0:00 grep --color=auto squid
root     25482  0.0  0.1 109272  6416 ?        Ss   Mar15   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf
proxy    25484  0.0  0.7 166684 45184 ?        S    Mar15   2:27 (squid-1) -YC -f /etc/squid/squid.conf
proxy    25485  0.0  0.0  13280  1648 ?        S    Mar15   0:03 (logfile-daemon) /var/log/squid/access.log
root@paproxy:/#


root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#



root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#


root@paproxy:/# lsof -n|egrep "proxy|squid" | more
squid     25482                  root  cwd       DIR                8,1     4096          2 /
squid     25482                  root  rtd       DIR                8,1     4096          2 /
squid     25482                  root  txt       REG                8,1  6430816   41946763 /usr/sbin/squid
squid     25482                  root  mem       REG                8,1    47648   10093387 /lib/x86_64-linux-gnu/libnss_nis-2.23.
so
squid     25482                  root  mem       REG                8,1    93128   10093399 /lib/x86_64-linux-gnu/libnsl-2.23.so
squid     25482                  root  mem       REG                8,1    35688   10093400 /lib/x86_64-linux-gnu/libnss_compat-2.
23.so
squid     25482                  root  mem       REG                8,1    47600   10093403 /lib/x86_64-linux-gnu/libnss_files-2.2
3.so
squid     25482                  root  mem       REG                8,1 25913104   41944015 /usr/lib/x86_64-linux-gnu/libicudata.s
o.55.1
squid     25482                  root  mem       REG                8,1    22520   10093098 /lib/x86_64-linux-gnu/libmnl.so.0.1.0
squid     25482                  root  mem       REG                8,1    26248   41948589 /usr/lib/x86_64-linux-gnu/libnfnetlink
.so.0.2.0


Thank you once again

CM



On Thu, Mar 23, 2017 at 1:59 PM, Eliezer Croitoru <[hidden email]> wrote:
There is another option!
The log rotate script is doing something nasty or the systemd service file start up squid in a weird way.
The output of:
$ systemctl status squid
$ top -n1 -b
$ ps aux
$ df -h
$ netstat -ntulp
$ lsof -n|egrep "proxy|squid"

How many clients this system has?
Is the system facing the Internet directly or behind some nat(aws or another provider)?

The above are the basic required data to understand the situation.

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: <a href="tel:%2B972-5-28704261" value="+972528704261">+972-5-28704261
Email: [hidden email]


From: squid-users [mailto:[hidden email]] On Behalf Of Chee M Gui
Sent: Wednesday, March 22, 2017 5:18 PM
To: [hidden email]
Subject: [squid-users] Squid stopped working after cache.log and access.log rotation



Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users



--
Chee Meng Gui
Function Engineering
650-833-0660

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Eliezer Croitoru
Hey CM,

From the output I understand that there wasn't a change in squid packaging for 16.04 and they still use rc\init.d startup scripts.
Also they probably use the same logrotate scripts from very long ago.
We first must understand if squid is running and it seems that systemd sees it as running.
I do not see in lsof output any port listening mentioned so I assume this is the reason for the issue in hands.
The first thing I would do is run a crontab that will check if squid is alive using a cache manager info page fetch and check if it's listening using netstat or ss.
(did you tried to see if squid is listening using netstat or ss??)
The next step would be to check your squid roatate script and to verify it's doing what it suppose to do.
After all this I would recommend changing from the rc\init.d startup script to a real system based one and abandon the old rotation scripts of Ubuntu or fix them.

If you are looking for a fix it's one path and if you are looking to get the work done properly by Ubuntu it's a whole new wagon.
I have been working on squid packages for Ubuntu and Debian that uses system scripts but the packages are not perfected yet.

Let me know the path you want to choose and also your approach to things and I will try to help you with which of the options you will choose to resolve the issues.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


From: Chee M Gui [mailto:[hidden email]]
Sent: Thursday, April 20, 2017 7:17 PM
To: Eliezer Croitoru <[hidden email]>
Cc: [hidden email]
Subject: Re: [squid-users] Squid stopped working after cache.log and access.log rotation

Hi Eliezer

Thank you for the response, and sorry for the late reply.

As requested, here are the output of the commands you suggested:
root@paproxy:/# systemctl status squid
● squid.service - LSB: Squid HTTP Proxy version 3.x
   Loaded: loaded (/etc/init.d/squid; bad; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-15 14:47:53 PDT; 1 months 5 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 4
   Memory: 54.4M
      CPU: 2min 38.957s
   CGroup: /system.slice/squid.service
           ├─25482 /usr/sbin/squid -YC -f /etc/squid/squid.conf
           ├─25484 (squid-1) -YC -f /etc/squid/squid.conf
           ├─25485 (logfile-daemon) /var/log/squid/access.log
           └─25486 (unlinkd)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@paproxy:/#



top - 09:11:45 up 54 days, 23:51,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  6043140 total,  4844728 free,   135292 used,  1063120 buff/cache
KiB Swap:  6222844 total,  6222844 free,        0 used.  5556300 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23355 root      20   0   41668   3776   3236 R   6.7  0.1   0:00.01 top
    1 root      20   0   37884   5968   4020 S   0.0  0.1   0:27.20 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.22 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.01 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   1:28.65 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:05.53 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:05.75 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:02.48 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   18 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
   19 root      20   0       0      0      0 S   0.0  0.0   0:01.26 khungtaskd
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback


root@paproxy:/# ps -aux | grep squid
root     23358  0.0  0.0  14224   976 pts/0    S+   09:12   0:00 grep --color=auto squid
root     25482  0.0  0.1 109272  6416 ?        Ss   Mar15   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf
proxy    25484  0.0  0.7 166684 45184 ?        S    Mar15   2:27 (squid-1) -YC -f /etc/squid/squid.conf
proxy    25485  0.0  0.0  13280  1648 ?        S    Mar15   0:03 (logfile-daemon) /var/log/squid/access.log
root@paproxy:/#


root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#



root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#


root@paproxy:/# lsof -n|egrep "proxy|squid" | more
squid     25482                  root  cwd       DIR                8,1     4096          2 /
squid     25482                  root  rtd       DIR                8,1     4096          2 /
squid     25482                  root  txt       REG                8,1  6430816   41946763 /usr/sbin/squid
squid     25482                  root  mem       REG                8,1    47648   10093387 /lib/x86_64-linux-gnu/libnss_nis-2.23.
so
squid     25482                  root  mem       REG                8,1    93128   10093399 /lib/x86_64-linux-gnu/http://libnsl-2.23.so
squid     25482                  root  mem       REG                8,1    35688   10093400 /lib/x86_64-linux-gnu/libnss_compat-2.
23.so
squid     25482                  root  mem       REG                8,1    47600   10093403 /lib/x86_64-linux-gnu/libnss_files-2.2
3.so
squid     25482                  root  mem       REG                8,1 25913104   41944015 /usr/lib/x86_64-linux-gnu/libicudata.s
o.55.1
squid     25482                  root  mem       REG                8,1    22520   10093098 /lib/x86_64-linux-gnu/libmnl.so.0.1.0
squid     25482                  root  mem       REG                8,1    26248   41948589 /usr/lib/x86_64-linux-gnu/libnfnetlink
.so.0.2.0


Thank you once again

CM



On Thu, Mar 23, 2017 at 1:59 PM, Eliezer Croitoru <mailto:[hidden email]> wrote:
There is another option!
The log rotate script is doing something nasty or the systemd service file start up squid in a weird way.
The output of:
$ systemctl status squid
$ top -n1 -b
$ ps aux
$ df -h
$ netstat -ntulp
$ lsof -n|egrep "proxy|squid"

How many clients this system has?
Is the system facing the Internet directly or behind some nat(aws or another provider)?

The above are the basic required data to understand the situation.

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: tel:%2B972-5-28704261
Email: mailto:[hidden email]


From: squid-users [mailto:mailto:[hidden email]] On Behalf Of Chee M Gui
Sent: Wednesday, March 22, 2017 5:18 PM
To: mailto:[hidden email]
Subject: [squid-users] Squid stopped working after cache.log and access.log rotation


Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui
_______________________________________________
squid-users mailing list
mailto:[hidden email]
http://lists.squid-cache.org/listinfo/squid-users




--
Chee Meng Gui
Function Engineering
650-833-0660

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Chee M Gui
Hi Eliezer

Thank you for the fast reply.   
Squid is listening on 3128 on the server.    See netstat output below.
We would like to fix the logrotate script (if this is being used?) rather that wait for Ubuntu to fix the Squid package (which may take a while).
There is no cron job under root or proxy or any other users on the server.
The logrotate file in  /etc/cron.daily is also a very old one.

Thank you once again

CMG


root@paproxy:/etc/logrotate.d# netstat -an | more

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN
tcp    13382      0 192.168.5.244:3128      192.168.5.103:49953     CLOSE_WAIT
tcp       90      0 198.27.136.41:52652     198.252.206.25:443      CLOSE_WAIT
tcp      232      0 192.168.5.244:3128      192.168.5.126:63442     CLOSE_WAIT
tcp      440      0 192.168.5.244:3128      192.168.5.126:63423     CLOSE_WAIT
tcp      221      0 192.168.5.244:3128      192.168.5.103:54521     CLOSE_WAIT


root@paproxy:/etc/logrotate.d# crontab -l
no crontab for root



root@paproxy:/etc/logrotate.d# crontab -u proxy -l
no crontab for proxy


root@paproxy:/etc/logrotate.d# vi squid
#
#       Logrotate fragment for squid.
#
/var/log/squid/*.log {
        daily
        compress
        delaycompress
        rotate 2
        missingok
        nocreate
        sharedscripts
        prerotate
                test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports
        endscript
        postrotate
                test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid || /usr/sbin/squid -k rotate
        endscript
}




root@paproxy:/etc/logrotate.d# dpkg -s squid
Package: squid
Status: install ok installed
Priority: optional
Section: web
Installed-Size: 7464
Maintainer: Ubuntu Developers <[hidden email]>
Architecture: amd64
Source: squid3
Version: 3.5.12-1ubuntu7.3
Replaces: squid3 (<< 3.5.12-1ubuntu1~)
Depends: libc6 (>= 2.15), libcap2 (>= 1:2.10), libcomerr2 (>= 1.01), libdb5.3, libecap3 (>= 1.0.1), libexpat1 (>= 2.0.1), libgcc1 (>= 1:3.0), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.10+dfsg~), libldap-2.4-2 (>= 2.4.7), libltdl7 (>= 2.4.6), libnetfilter-conntrack3, libnettle6, libpam0g (>= 0.99.7.1), libsasl2-2, libstdc++6 (>= 5.2), libxml2 (>= 2.7.4), netbase, logrotate (>= 3.5.4-1), squid-common (= 3.5.12-1ubuntu7.3), lsb-base, ssl-cert, init-system-helpers (>> 1.22ubuntu5)
Pre-Depends: adduser
Suggests: squidclient, squid-cgi, squid-purge, resolvconf (>= 0.40), smbclient, ufw, winbindd, apparmor
Breaks: squid3 (<< 3.5.12-1ubuntu1~), ufw (<< 0.35-0ubuntu2~)
Conffiles:
 /etc/apparmor.d/usr.sbin.squid 08e05266f0ef7a9a4ac2c62be29a3ef2
 /etc/init.d/squid f67c63ce21e0ac57a4d16e90909b3e34
 /etc/logrotate.d/squid 2be386088ead3641de5401a9c73a7a57
 /etc/resolvconf/update-libc.d/squid 9968dc6f2fcde9f38a6faea7dfe95dd1
 /etc/squid/errorpage.css 7f1cc06116c222d49d641f0e830ff615
 /etc/squid/squid.conf e73b82ed9d76b47c8b5963175f0ada1e
 /etc/ufw/applications.d/squid 710e7b8ded49bbcd41eb072a0fe1691f
Description: Full featured Web Proxy cache (HTTP proxy)
 Squid is a high-performance proxy caching server for web clients, supporting
 FTP, gopher, ICY and HTTP data objects.
 .
 Squid version 3 is a major rewrite of Squid in C++ and introduces a number of
 new features including ICAP and ESI support.
Original-Maintainer: Luigi Gangitano <[hidden email]>
root@paproxy:/etc/logrotate.d#




root@paproxy:/etc/logrotate.d# ls -al /etc/cron.daily
total 56
drwxr-xr-x  2 root root 4096 Mar 13 16:30 .
drwxr-xr-x 91 root root 4096 Apr  3 13:43 ..
-rwxr-xr-x  1 root root  376 Mar 31  2016 apport
-rwxr-xr-x  1 root root 1474 Oct 31 07:31 apt-compat
-rwxr-xr-x  1 root root  355 May 22  2012 bsdmainutils
-rwxr-xr-x  1 root root 1597 Nov 26  2015 dpkg
-rwxr-xr-x  1 root root  372 May  5  2015 logrotate
-rwxr-xr-x  1 root root 1293 Nov  6  2015 man-db
-rwxr-xr-x  1 root root  539 Jul 16  2014 mdadm
-rwxr-xr-x  1 root root  435 Nov 17  2014 mlocate
-rwxr-xr-x  1 root root  249 Nov 12  2015 passwd
-rw-r--r--  1 root root  102 Apr  5  2016 .placeholder
-rwxr-xr-x  1 root root 3449 Feb 26  2016 popularity-contest
-rwxr-xr-x  1 root root  214 May 24  2016 update-notifier-common








On Thu, Apr 20, 2017 at 12:24 PM, Eliezer Croitoru <[hidden email]> wrote:
Hey CM,

From the output I understand that there wasn't a change in squid packaging for 16.04 and they still use rc\init.d startup scripts.
Also they probably use the same logrotate scripts from very long ago.
We first must understand if squid is running and it seems that systemd sees it as running.
I do not see in lsof output any port listening mentioned so I assume this is the reason for the issue in hands.
The first thing I would do is run a crontab that will check if squid is alive using a cache manager info page fetch and check if it's listening using netstat or ss.
(did you tried to see if squid is listening using netstat or ss??)
The next step would be to check your squid roatate script and to verify it's doing what it suppose to do.
After all this I would recommend changing from the rc\init.d startup script to a real system based one and abandon the old rotation scripts of Ubuntu or fix them.

If you are looking for a fix it's one path and if you are looking to get the work done properly by Ubuntu it's a whole new wagon.
I have been working on squid packages for Ubuntu and Debian that uses system scripts but the packages are not perfected yet.

Let me know the path you want to choose and also your approach to things and I will try to help you with which of the options you will choose to resolve the issues.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: <a href="tel:%2B972-5-28704261" value="+972528704261">+972-5-28704261
Email: [hidden email]


From: Chee M Gui [mailto:[hidden email]]
Sent: Thursday, April 20, 2017 7:17 PM
To: Eliezer Croitoru <[hidden email]>
Cc: [hidden email]
Subject: Re: [squid-users] Squid stopped working after cache.log and access.log rotation

Hi Eliezer

Thank you for the response, and sorry for the late reply.

As requested, here are the output of the commands you suggested:
root@paproxy:/# systemctl status squid
● squid.service - LSB: Squid HTTP Proxy version 3.x
   Loaded: loaded (/etc/init.d/squid; bad; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-15 14:47:53 PDT; 1 months 5 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 4
   Memory: 54.4M
      CPU: 2min 38.957s
   CGroup: /system.slice/squid.service
           ├─25482 /usr/sbin/squid -YC -f /etc/squid/squid.conf
           ├─25484 (squid-1) -YC -f /etc/squid/squid.conf
           ├─25485 (logfile-daemon) /var/log/squid/access.log
           └─25486 (unlinkd)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@paproxy:/#



top - 09:11:45 up 54 days, 23:51,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  6043140 total,  4844728 free,   135292 used,  1063120 buff/cache
KiB Swap:  6222844 total,  6222844 free,        0 used.  5556300 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23355 root      20   0   41668   3776   3236 R   6.7  0.1   0:00.01 top
    1 root      20   0   37884   5968   4020 S   0.0  0.1   0:27.20 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.22 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.01 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   1:28.65 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:05.53 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:05.75 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:02.48 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   18 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
   19 root      20   0       0      0      0 S   0.0  0.0   0:01.26 khungtaskd
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback


root@paproxy:/# ps -aux | grep squid
root     23358  0.0  0.0  14224   976 pts/0    S+   09:12   0:00 grep --color=auto squid
root     25482  0.0  0.1 109272  6416 ?        Ss   Mar15   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf
proxy    25484  0.0  0.7 166684 45184 ?        S    Mar15   2:27 (squid-1) -YC -f /etc/squid/squid.conf
proxy    25485  0.0  0.0  13280  1648 ?        S    Mar15   0:03 (logfile-daemon) /var/log/squid/access.log
root@paproxy:/#


root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#



root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#


root@paproxy:/# lsof -n|egrep "proxy|squid" | more
squid     25482                  root  cwd       DIR                8,1     4096          2 /
squid     25482                  root  rtd       DIR                8,1     4096          2 /
squid     25482                  root  txt       REG                8,1  6430816   41946763 /usr/sbin/squid
squid     25482                  root  mem       REG                8,1    47648   10093387 /lib/x86_64-linux-gnu/libnss_nis-2.23.
so
squid     25482                  root  mem       REG                8,1    93128   10093399 /lib/x86_64-linux-gnu/http://libnsl-2.23.so
squid     25482                  root  mem       REG                8,1    35688   10093400 /lib/x86_64-linux-gnu/libnss_compat-2.
23.so
squid     25482                  root  mem       REG                8,1    47600   10093403 /lib/x86_64-linux-gnu/libnss_files-2.2
3.so
squid     25482                  root  mem       REG                8,1 25913104   41944015 /usr/lib/x86_64-linux-gnu/libicudata.s
o.55.1
squid     25482                  root  mem       REG                8,1    22520   10093098 /lib/x86_64-linux-gnu/libmnl.so.0.1.0
squid     25482                  root  mem       REG                8,1    26248   41948589 /usr/lib/x86_64-linux-gnu/libnfnetlink
.so.0.2.0


Thank you once again

CM



On Thu, Mar 23, 2017 at 1:59 PM, Eliezer Croitoru <mailto:[hidden email]> wrote:
There is another option!
The log rotate script is doing something nasty or the systemd service file start up squid in a weird way.
The output of:
$ systemctl status squid
$ top -n1 -b
$ ps aux
$ df -h
$ netstat -ntulp
$ lsof -n|egrep "proxy|squid"

How many clients this system has?
Is the system facing the Internet directly or behind some nat(aws or another provider)?

The above are the basic required data to understand the situation.

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: tel:%2B972-5-28704261
Email: mailto:[hidden email]


From: squid-users [mailto:[hidden email]:[hidden email]] On Behalf Of Chee M Gui
Sent: Wednesday, March 22, 2017 5:18 PM
To: mailto:[hidden email]
Subject: [squid-users] Squid stopped working after cache.log and access.log rotation


Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui
_______________________________________________
squid-users mailing list
mailto:[hidden email]
http://lists.squid-cache.org/listinfo/squid-users




--
Chee Meng Gui
Function Engineering
<a href="tel:650-833-0660" value="+16508330660">650-833-0660




--
Chee Meng Gui
Function Engineering
650-833-0660

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Squid stopped working after cache.log and access.log rotation

Chee M Gui


Some more info:


cache.log is able to rotate.
Only access.log not rotating

root@paproxy:/# ls -alt /var/log/squid
total 15540
drwxrwxr-x 9 root  syslog     4096 Apr 20 14:42 ..
drwxr-xr-x 2 proxy proxy      4096 Apr 20 06:25 .
-rw-r----- 1 proxy proxy        63 Apr 20 06:25 cache.log
-rw-r----- 1 proxy proxy        63 Apr 19 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Apr 18 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy        63 Mar 19 06:25 cache.log.2
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state
root@paproxy:/#


We installed squid on 3/15/2017/



root@paproxy:/etc/logrotate.d# date +%s
1492724638
root@paproxy:/etc/logrotate.d# more /var/log/squid/access.log.1
1489614186.653      6 192.168.5.103 TCP_MISS/503 3992 GET http://ipv6.msftncsi.com/ncsi.txt - HIER_DIRECT/2001:5a8:100::b817:9fae
text/html
1489614186.668     21 192.168.5.103 TCP_MISS/200 280 GET http://www.msftncsi.com/ncsi.txt - HIER_DIRECT/184.23.159.169 text/plain
1489614186.997    214 192.168.5.103 TCP_MISS/200 617 GET http://login.live.com/ppcrlcheck.srf - HIER_DIRECT/131.253.61.66 text/htm
l



1489757088.048  10750 192.168.5.103 TCP_TUNNEL/200 5454 CONNECT p.ebdr2.com:443 - HIER_DIRECT/74.217.250.5 -
1489757099.057  10783 192.168.5.103 TCP_TUNNEL/200 5454 CONNECT p.ebdr2.com:443 - HIER_DIRECT/74.217.250.5 -


First entry in the access.log.1 file
GMT: Wed, 15 Mar 2017 21:43:06 GMT
Your time zone: 3/15/2017, 2:43:06 PM GMT-7:00 DST

Last entry in the access.log.1 file
GMT: Fri, 17 Mar 2017 13:24:59 GMT
Your time zone: 3/17/2017, 6:24:59 AM GMT-7:00 DST


Squid was installed on 3/15/2017 on the server.





On Thu, Apr 20, 2017 at 2:34 PM, Chee M Gui <[hidden email]> wrote:
Hi Eliezer

Thank you for the fast reply.   
Squid is listening on 3128 on the server.    See netstat output below.
We would like to fix the logrotate script (if this is being used?) rather that wait for Ubuntu to fix the Squid package (which may take a while).
There is no cron job under root or proxy or any other users on the server.
The logrotate file in  /etc/cron.daily is also a very old one.

Thank you once again

CMG


root@paproxy:/etc/logrotate.d# netstat -an | more

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN
tcp    13382      0 192.168.5.244:3128      192.168.5.103:49953     CLOSE_WAIT
tcp       90      0 198.27.136.41:52652     198.252.206.25:443      CLOSE_WAIT
tcp      232      0 192.168.5.244:3128      192.168.5.126:63442     CLOSE_WAIT
tcp      440      0 192.168.5.244:3128      192.168.5.126:63423     CLOSE_WAIT
tcp      221      0 192.168.5.244:3128      192.168.5.103:54521     CLOSE_WAIT


root@paproxy:/etc/logrotate.d# crontab -l
no crontab for root



root@paproxy:/etc/logrotate.d# crontab -u proxy -l
no crontab for proxy


root@paproxy:/etc/logrotate.d# vi squid
#
#       Logrotate fragment for squid.
#
/var/log/squid/*.log {
        daily
        compress
        delaycompress
        rotate 2
        missingok
        nocreate
        sharedscripts
        prerotate
                test ! -x /usr/sbin/sarg-reports || /usr/sbin/sarg-reports
        endscript
        postrotate
                test ! -e /var/run/squid.pid || test ! -x /usr/sbin/squid || /usr/sbin/squid -k rotate
        endscript
}




root@paproxy:/etc/logrotate.d# dpkg -s squid
Package: squid
Status: install ok installed
Priority: optional
Section: web
Installed-Size: 7464
Maintainer: Ubuntu Developers <[hidden email]>
Architecture: amd64
Source: squid3
Version: 3.5.12-1ubuntu7.3
Replaces: squid3 (<< 3.5.12-1ubuntu1~)
Depends: libc6 (>= 2.15), libcap2 (>= 1:2.10), libcomerr2 (>= 1.01), libdb5.3, libecap3 (>= 1.0.1), libexpat1 (>= 2.0.1), libgcc1 (>= 1:3.0), libgssapi-krb5-2 (>= 1.10+dfsg~), libkrb5-3 (>= 1.10+dfsg~), libldap-2.4-2 (>= 2.4.7), libltdl7 (>= 2.4.6), libnetfilter-conntrack3, libnettle6, libpam0g (>= 0.99.7.1), libsasl2-2, libstdc++6 (>= 5.2), libxml2 (>= 2.7.4), netbase, logrotate (>= 3.5.4-1), squid-common (= 3.5.12-1ubuntu7.3), lsb-base, ssl-cert, init-system-helpers (>> 1.22ubuntu5)
Pre-Depends: adduser
Suggests: squidclient, squid-cgi, squid-purge, resolvconf (>= 0.40), smbclient, ufw, winbindd, apparmor
Breaks: squid3 (<< 3.5.12-1ubuntu1~), ufw (<< 0.35-0ubuntu2~)
Conffiles:
 /etc/apparmor.d/usr.sbin.squid 08e05266f0ef7a9a4ac2c62be29a3ef2
 /etc/init.d/squid f67c63ce21e0ac57a4d16e90909b3e34
 /etc/logrotate.d/squid 2be386088ead3641de5401a9c73a7a57
 /etc/resolvconf/update-libc.d/squid 9968dc6f2fcde9f38a6faea7dfe95dd1
 /etc/squid/errorpage.css 7f1cc06116c222d49d641f0e830ff615
 /etc/squid/squid.conf e73b82ed9d76b47c8b5963175f0ada1e
 /etc/ufw/applications.d/squid 710e7b8ded49bbcd41eb072a0fe1691f
Description: Full featured Web Proxy cache (HTTP proxy)
 Squid is a high-performance proxy caching server for web clients, supporting
 FTP, gopher, ICY and HTTP data objects.
 .
 Squid version 3 is a major rewrite of Squid in C++ and introduces a number of
 new features including ICAP and ESI support.
Original-Maintainer: Luigi Gangitano <[hidden email]>
root@paproxy:/etc/logrotate.d#




root@paproxy:/etc/logrotate.d# ls -al /etc/cron.daily
total 56
drwxr-xr-x  2 root root 4096 Mar 13 16:30 .
drwxr-xr-x 91 root root 4096 Apr  3 13:43 ..
-rwxr-xr-x  1 root root  376 Mar 31  2016 apport
-rwxr-xr-x  1 root root 1474 Oct 31 07:31 apt-compat
-rwxr-xr-x  1 root root  355 May 22  2012 bsdmainutils
-rwxr-xr-x  1 root root 1597 Nov 26  2015 dpkg
-rwxr-xr-x  1 root root  372 May  5  2015 logrotate
-rwxr-xr-x  1 root root 1293 Nov  6  2015 man-db
-rwxr-xr-x  1 root root  539 Jul 16  2014 mdadm
-rwxr-xr-x  1 root root  435 Nov 17  2014 mlocate
-rwxr-xr-x  1 root root  249 Nov 12  2015 passwd
-rw-r--r--  1 root root  102 Apr  5  2016 .placeholder
-rwxr-xr-x  1 root root 3449 Feb 26  2016 popularity-contest
-rwxr-xr-x  1 root root  214 May 24  2016 update-notifier-common








On Thu, Apr 20, 2017 at 12:24 PM, Eliezer Croitoru <[hidden email]> wrote:
Hey CM,

From the output I understand that there wasn't a change in squid packaging for 16.04 and they still use rc\init.d startup scripts.
Also they probably use the same logrotate scripts from very long ago.
We first must understand if squid is running and it seems that systemd sees it as running.
I do not see in lsof output any port listening mentioned so I assume this is the reason for the issue in hands.
The first thing I would do is run a crontab that will check if squid is alive using a cache manager info page fetch and check if it's listening using netstat or ss.
(did you tried to see if squid is listening using netstat or ss??)
The next step would be to check your squid roatate script and to verify it's doing what it suppose to do.
After all this I would recommend changing from the rc\init.d startup script to a real system based one and abandon the old rotation scripts of Ubuntu or fix them.

If you are looking for a fix it's one path and if you are looking to get the work done properly by Ubuntu it's a whole new wagon.
I have been working on squid packages for Ubuntu and Debian that uses system scripts but the packages are not perfected yet.

Let me know the path you want to choose and also your approach to things and I will try to help you with which of the options you will choose to resolve the issues.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: <a href="tel:%2B972-5-28704261" value="+972528704261" target="_blank">+972-5-28704261
Email: [hidden email]


From: Chee M Gui [mailto:[hidden email]]
Sent: Thursday, April 20, 2017 7:17 PM
To: Eliezer Croitoru <[hidden email]>
Cc: [hidden email]
Subject: Re: [squid-users] Squid stopped working after cache.log and access.log rotation

Hi Eliezer

Thank you for the response, and sorry for the late reply.

As requested, here are the output of the commands you suggested:
root@paproxy:/# systemctl status squid
● squid.service - LSB: Squid HTTP Proxy version 3.x
   Loaded: loaded (/etc/init.d/squid; bad; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-15 14:47:53 PDT; 1 months 5 days ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 4
   Memory: 54.4M
      CPU: 2min 38.957s
   CGroup: /system.slice/squid.service
           ├─25482 /usr/sbin/squid -YC -f /etc/squid/squid.conf
           ├─25484 (squid-1) -YC -f /etc/squid/squid.conf
           ├─25485 (logfile-daemon) /var/log/squid/access.log
           └─25486 (unlinkd)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
root@paproxy:/#



top - 09:11:45 up 54 days, 23:51,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  6043140 total,  4844728 free,   135292 used,  1063120 buff/cache
KiB Swap:  6222844 total,  6222844 free,        0 used.  5556300 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
23355 root      20   0   41668   3776   3236 R   6.7  0.1   0:00.01 top
    1 root      20   0   37884   5968   4020 S   0.0  0.1   0:27.20 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.22 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.01 ksoftirqd/0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S   0.0  0.0   1:28.65 rcu_sched
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/0
   10 root      rt   0       0      0      0 S   0.0  0.0   0:05.53 watchdog/0
   11 root      rt   0       0      0      0 S   0.0  0.0   0:05.75 watchdog/1
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.14 migration/1
   13 root      20   0       0      0      0 S   0.0  0.0   0:02.48 ksoftirqd/1
   15 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   18 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 perf
   19 root      20   0       0      0      0 S   0.0  0.0   0:01.26 khungtaskd
   20 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback


root@paproxy:/# ps -aux | grep squid
root     23358  0.0  0.0  14224   976 pts/0    S+   09:12   0:00 grep --color=auto squid
root     25482  0.0  0.1 109272  6416 ?        Ss   Mar15   0:00 /usr/sbin/squid -YC -f /etc/squid/squid.conf
proxy    25484  0.0  0.7 166684 45184 ?        S    Mar15   2:27 (squid-1) -YC -f /etc/squid/squid.conf
proxy    25485  0.0  0.0  13280  1648 ?        S    Mar15   0:03 (logfile-daemon) /var/log/squid/access.log
root@paproxy:/#


root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#



root@paproxy:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.9G     0  2.9G   0% /dev
tmpfs           591M   60M  531M  11% /run
/dev/sda1       911G  1.9G  863G   1% /
tmpfs           2.9G   12K  2.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.9G     0  2.9G   0% /sys/fs/cgroup
tmpfs           591M     0  591M   0% /run/user/1000
tmpfs           591M     0  591M   0% /run/user/0
root@paproxy:/#


root@paproxy:/# lsof -n|egrep "proxy|squid" | more
squid     25482                  root  cwd       DIR                8,1     4096          2 /
squid     25482                  root  rtd       DIR                8,1     4096          2 /
squid     25482                  root  txt       REG                8,1  6430816   41946763 /usr/sbin/squid
squid     25482                  root  mem       REG                8,1    47648   10093387 /lib/x86_64-linux-gnu/libnss_nis-2.23.
so
squid     25482                  root  mem       REG                8,1    93128   10093399 /lib/x86_64-linux-gnu/http://libnsl-2.23.so
squid     25482                  root  mem       REG                8,1    35688   10093400 /lib/x86_64-linux-gnu/libnss_compat-2.
23.so
squid     25482                  root  mem       REG                8,1    47600   10093403 /lib/x86_64-linux-gnu/libnss_files-2.2
3.so
squid     25482                  root  mem       REG                8,1 25913104   41944015 /usr/lib/x86_64-linux-gnu/libicudata.s
o.55.1
squid     25482                  root  mem       REG                8,1    22520   10093098 /lib/x86_64-linux-gnu/libmnl.so.0.1.0
squid     25482                  root  mem       REG                8,1    26248   41948589 /usr/lib/x86_64-linux-gnu/libnfnetlink
.so.0.2.0


Thank you once again

CM



On Thu, Mar 23, 2017 at 1:59 PM, Eliezer Croitoru <mailto:[hidden email]> wrote:
There is another option!
The log rotate script is doing something nasty or the systemd service file start up squid in a weird way.
The output of:
$ systemctl status squid
$ top -n1 -b
$ ps aux
$ df -h
$ netstat -ntulp
$ lsof -n|egrep "proxy|squid"

How many clients this system has?
Is the system facing the Internet directly or behind some nat(aws or another provider)?

The above are the basic required data to understand the situation.

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: tel:%2B972-5-28704261
Email: mailto:[hidden email]


From: squid-users [mailto:[hidden email]:[hidden email]] On Behalf Of Chee M Gui
Sent: Wednesday, March 22, 2017 5:18 PM
To: mailto:[hidden email]
Subject: [squid-users] Squid stopped working after cache.log and access.log rotation


Hi All

We recently installed Squid 3.5.12-1ubuntu7.3 on Ubuntu 16.04.2 LTS.  It ran fine at first but stopped working after a while.   telnet server 3128 still works, i.e., opens a blank window, but Squid is just not accepting requests.    Then we realized that there is no new access.log file.   The access.log file stopped rotated at 6:24AM on 3/17/2017.    It looks like Squid wasn't able to create a new access.log?  We could not find any error message in syslog or the cache.log.      We haven't rebooted the server because we want to know what went wrong.   It isn't the firewall blocking Squid because Squid was working fine all the while until recently.  Also after it stopped working, we disabled the firewall to see if it would work but it still didn't work.

root@paproxy:/var/log/squid# ls -alt
total 15536
drwxr-xr-x 2 proxy proxy      4096 Mar 21 06:25 .
-rw-r----- 1 proxy proxy        63 Mar 21 06:25 cache.log
drwxrwxr-x 9 root  syslog     4096 Mar 21 06:25 ..
-rw-r----- 1 proxy proxy        63 Mar 20 06:25 cache.log.1
-rw-r----- 1 proxy proxy        83 Mar 19 06:25 cache.log.2.gz
-rw-r----- 1 proxy proxy  15759111 Mar 17 06:24 access.log.1
-rw-r----- 1 proxy proxy    117223 Mar 17 05:52 netdb.state

Any ideas what went wrong?

Thank you very much in anticipation.

Gui
_______________________________________________
squid-users mailing list
mailto:[hidden email]
http://lists.squid-cache.org/listinfo/squid-users




--
Chee Meng Gui
Function Engineering
<a href="tel:650-833-0660" value="+16508330660" target="_blank">650-833-0660




--
Chee Meng Gui
Function Engineering
<a href="tel:(650)%20833-0660" value="+16508330660" target="_blank">650-833-0660



--
Chee Meng Gui
Function Engineering
650-833-0660

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...