Data usage reported in log files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Data usage reported in log files

Yosi Greenfield
Hello all,

I'm analyzing my squid logs with sarg, and I see that the number of
bytes reported as used by any particular user are often nowhere
near the bytes reported by netflow and tcpdump.

I'm trying to trace my users' data usage by site, but I'm unable to
do so from the log files because of this.

Can someone please explain to me what I might be missing? Why does
squid log report one thing and netflow and tcpdump show something
else?

Thanks very much!

Yosi

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Antony Stone
On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:

> Hello all,
>
> I'm analyzing my squid logs with sarg, and I see that the number of
> bytes reported as used by any particular user are often nowhere
> near the bytes reported by netflow and tcpdump.

Which is larger?

> I'm trying to trace my users' data usage by site, but I'm unable to
> do so from the log files because of this.

Well, what is it you really want to know?

netflow / tcpdump will give you accurate numbers for the quantity of data on
your Internet link - I assume this is what you're most interested in?

Squid will show you what quantity of data goes to/from the clients, but is
that really important?

> Can someone please explain to me what I might be missing? Why does
> squid log report one thing and netflow and tcpdump show something
> else?

Data compression?

HTTP responses are often gzipped, so if tcpdump is showing you smaller numbers
of bytes than Squid reports, that's what I'd look at first.


Antony.

--
This sentence contains exacly three erors.

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yosi Greenfield
Thanks!

Netflow is much larger.

I really want to know exactly what site is costing my users data. Many of
our users are on metered connections and are paying for overage, but I can't
tell where that overage is being used. Are they using youtube, webmail,
wetransfer? I see only a fraction of their actual proxy usage in my squid
logs.

Data compression would give the opposite result, so that's not what I'm
seeing.

Any other ideas?


-----Original Message-----
From: squid-users [mailto:[hidden email]] On
Behalf Of Antony Stone
Sent: Friday, March 10, 2017 2:21 PM
To: [hidden email]
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:

> Hello all,
>
> I'm analyzing my squid logs with sarg, and I see that the number of
> bytes reported as used by any particular user are often nowhere near
> the bytes reported by netflow and tcpdump.

Which is larger?

> I'm trying to trace my users' data usage by site, but I'm unable to do
> so from the log files because of this.

Well, what is it you really want to know?

netflow / tcpdump will give you accurate numbers for the quantity of data on
your Internet link - I assume this is what you're most interested in?

Squid will show you what quantity of data goes to/from the clients, but is
that really important?

> Can someone please explain to me what I might be missing? Why does
> squid log report one thing and netflow and tcpdump show something
> else?

Data compression?

HTTP responses are often gzipped, so if tcpdump is showing you smaller
numbers of bytes than Squid reports, that's what I'd look at first.


Antony.

--
This sentence contains exacly three erors.

                                                   Please reply to the list;
                                                         please *don't* CC
me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Marcus Kool


On 10/03/17 16:27, Yosi Greenfield wrote:

> Thanks!
>
> Netflow is much larger.
>
> I really want to know exactly what site is costing my users data. Many of
> our users are on metered connections and are paying for overage, but I can't
> tell where that overage is being used. Are they using youtube, webmail,
> wetransfer? I see only a fraction of their actual proxy usage in my squid
> logs.
>
> Data compression would give the opposite result, so that's not what I'm
> seeing.
>
> Any other ideas?

Is there any traffic that is not directed to Squid?

Do you use ssl-bump in bump mode ?
If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.

Marcus


> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On
> Behalf Of Antony Stone
> Sent: Friday, March 10, 2017 2:21 PM
> To: [hidden email]
> Subject: Re: [squid-users] Data usage reported in log files
>
> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>
>> Hello all,
>>
>> I'm analyzing my squid logs with sarg, and I see that the number of
>> bytes reported as used by any particular user are often nowhere near
>> the bytes reported by netflow and tcpdump.
>
> Which is larger?
>
>> I'm trying to trace my users' data usage by site, but I'm unable to do
>> so from the log files because of this.
>
> Well, what is it you really want to know?
>
> netflow / tcpdump will give you accurate numbers for the quantity of data on
> your Internet link - I assume this is what you're most interested in?
>
> Squid will show you what quantity of data goes to/from the clients, but is
> that really important?
>
>> Can someone please explain to me what I might be missing? Why does
>> squid log report one thing and netflow and tcpdump show something
>> else?
>
> Data compression?
>
> HTTP responses are often gzipped, so if tcpdump is showing you smaller
> numbers of bytes than Squid reports, that's what I'd look at first.
>
>
> Antony.
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yosi Greenfield
Aha! That could be it. I use sslbump, but not for all users. I'll
check that out, although I think that it's a problem even for bumped
users. Even for bumped users we don't bump all sites, so that really
could be it.

Thanks!


-----Original Message-----
From: squid-users [mailto:[hidden email]] On
Behalf Of Marcus Kool
Sent: Friday, March 10, 2017 3:38 PM
To: [hidden email]
Subject: Re: [squid-users] Data usage reported in log files



On 10/03/17 16:27, Yosi Greenfield wrote:

> Thanks!
>
> Netflow is much larger.
>
> I really want to know exactly what site is costing my users data. Many
> of our users are on metered connections and are paying for overage,
> but I can't tell where that overage is being used. Are they using
> youtube, webmail, wetransfer? I see only a fraction of their actual
> proxy usage in my squid logs.
>
> Data compression would give the opposite result, so that's not what
> I'm seeing.
>
> Any other ideas?

Is there any traffic that is not directed to Squid?

Do you use ssl-bump in bump mode ?
If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.

Marcus


> -----Original Message-----
> From: squid-users [mailto:[hidden email]]
> On Behalf Of Antony Stone
> Sent: Friday, March 10, 2017 2:21 PM
> To: [hidden email]
> Subject: Re: [squid-users] Data usage reported in log files
>
> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>
>> Hello all,
>>
>> I'm analyzing my squid logs with sarg, and I see that the number of
>> bytes reported as used by any particular user are often nowhere near
>> the bytes reported by netflow and tcpdump.
>
> Which is larger?
>
>> I'm trying to trace my users' data usage by site, but I'm unable to
>> do so from the log files because of this.
>
> Well, what is it you really want to know?
>
> netflow / tcpdump will give you accurate numbers for the quantity of
> data on your Internet link - I assume this is what you're most interested
in?

>
> Squid will show you what quantity of data goes to/from the clients,
> but is that really important?
>
>> Can someone please explain to me what I might be missing? Why does
>> squid log report one thing and netflow and tcpdump show something
>> else?
>
> Data compression?
>
> HTTP responses are often gzipped, so if tcpdump is showing you smaller
> numbers of bytes than Squid reports, that's what I'd look at first.
>
>
> Antony.
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov
Gentlemen, and it never occurred to you that there are other types of
traffic besides HTTP / HTTPS, right?

DNS, ICMP, other protocols?


11.03.2017 2:44, Yosi Greenfield пишет:

> Aha! That could be it. I use sslbump, but not for all users. I'll
> check that out, although I think that it's a problem even for bumped
> users. Even for bumped users we don't bump all sites, so that really
> could be it.
>
> Thanks!
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On
> Behalf Of Marcus Kool
> Sent: Friday, March 10, 2017 3:38 PM
> To: [hidden email]
> Subject: Re: [squid-users] Data usage reported in log files
>
>
>
> On 10/03/17 16:27, Yosi Greenfield wrote:
>> Thanks!
>>
>> Netflow is much larger.
>>
>> I really want to know exactly what site is costing my users data. Many
>> of our users are on metered connections and are paying for overage,
>> but I can't tell where that overage is being used. Are they using
>> youtube, webmail, wetransfer? I see only a fraction of their actual
>> proxy usage in my squid logs.
>>
>> Data compression would give the opposite result, so that's not what
>> I'm seeing.
>>
>> Any other ideas?
> Is there any traffic that is not directed to Squid?
>
> Do you use ssl-bump in bump mode ?
> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>
> Marcus
>
>
>> -----Original Message-----
>> From: squid-users [mailto:[hidden email]]
>> On Behalf Of Antony Stone
>> Sent: Friday, March 10, 2017 2:21 PM
>> To: [hidden email]
>> Subject: Re: [squid-users] Data usage reported in log files
>>
>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>
>>> Hello all,
>>>
>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>> bytes reported as used by any particular user are often nowhere near
>>> the bytes reported by netflow and tcpdump.
>> Which is larger?
>>
>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>> do so from the log files because of this.
>> Well, what is it you really want to know?
>>
>> netflow / tcpdump will give you accurate numbers for the quantity of
>> data on your Internet link - I assume this is what you're most interested
> in?
>> Squid will show you what quantity of data goes to/from the clients,
>> but is that really important?
>>
>>> Can someone please explain to me what I might be missing? Why does
>>> squid log report one thing and netflow and tcpdump show something
>>> else?
>> Data compression?
>>
>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>> numbers of bytes than Squid reports, that's what I'd look at first.
>>
>>
>> Antony.
>>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Alex Rousskov
In reply to this post by Marcus Kool
On 03/10/2017 01:37 PM, Marcus Kool wrote:
> Squid has no idea how many bytes go through the (HTTPS) tunnels.

Actually, Squid knows the number of raw (encrypted) TCP payload bytes
inside a tunnel and should log that.

Squid also knows and logs the number of HTTP (decrypted) bytes if the
SSL tunnel is bumped. In that case, the logged number is often smaller
but could also be larger than the corresponding TCP payload, depending
on whether SSL uses compression.

In any case, Squid numbers do not contain TCP/IP/Ethernet headers and
control messages. They may also lack HTTP chunked encoding overheads.
Failed Squid-to-server connections are not logged if they were
successfully retried.

There are also logging/accounting bugs because there is currently no
automated system to detect them. For a recent example, see our fix at
http://bazaar.launchpad.net/~squid/squid/trunk/revision/14838

If you use the latest release and see a disparity (between Squid-logged
numbers and other sources of information) that cannot be explained by
known factors, consider reporting it.


Thank you,

Alex.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Antony Stone
In reply to this post by Yuri Voinov
On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:

> Gentlemen, and it never occurred to you that there are other types of
> traffic besides HTTP / HTTPS, right?
>
> DNS, ICMP, other protocols?

I'm assuming Yosi has been measuring only TCP traffic, but even if he's been
measuring everything, I don't think DNS, ICMP and other protocols would add
more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also
totally-non-Squid traffic on the link being measured.


Antony.

> 11.03.2017 2:44, Yosi Greenfield пишет:
> > Aha! That could be it. I use sslbump, but not for all users. I'll
> > check that out, although I think that it's a problem even for bumped
> > users. Even for bumped users we don't bump all sites, so that really
> > could be it.
> >
> > Thanks!
> >
> >
> > -----Original Message-----
> > From: squid-users [mailto:[hidden email]] On
> > Behalf Of Marcus Kool
> > Sent: Friday, March 10, 2017 3:38 PM
> > To: [hidden email]
> > Subject: Re: [squid-users] Data usage reported in log files
> >
> > On 10/03/17 16:27, Yosi Greenfield wrote:
> >> Thanks!
> >>
> >> Netflow is much larger.
> >>
> >> I really want to know exactly what site is costing my users data. Many
> >> of our users are on metered connections and are paying for overage,
> >> but I can't tell where that overage is being used. Are they using
> >> youtube, webmail, wetransfer? I see only a fraction of their actual
> >> proxy usage in my squid logs.
> >>
> >> Data compression would give the opposite result, so that's not what
> >> I'm seeing.
> >>
> >> Any other ideas?
> >
> > Is there any traffic that is not directed to Squid?
> >
> > Do you use ssl-bump in bump mode ?
> > If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
> >
> > Marcus
> >
> >> -----Original Message-----
> >> From: squid-users [mailto:[hidden email]]
> >> On Behalf Of Antony Stone
> >> Sent: Friday, March 10, 2017 2:21 PM
> >> To: [hidden email]
> >> Subject: Re: [squid-users] Data usage reported in log files
> >>
> >> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>> Hello all,
> >>>
> >>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>> bytes reported as used by any particular user are often nowhere near
> >>> the bytes reported by netflow and tcpdump.
> >>
> >> Which is larger?
> >>
> >>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>> do so from the log files because of this.
> >>
> >> Well, what is it you really want to know?
> >>
> >> netflow / tcpdump will give you accurate numbers for the quantity of
> >> data on your Internet link - I assume this is what you're most
> >> interested in?
> >
> >> Squid will show you what quantity of data goes to/from the clients,
> >> but is that really important?
> >>
> >>> Can someone please explain to me what I might be missing? Why does
> >>> squid log report one thing and netflow and tcpdump show something
> >>> else?
> >>
> >> Data compression?
> >>
> >> HTTP responses are often gzipped, so if tcpdump is showing you smaller
> >> numbers of bytes than Squid reports, that's what I'd look at first.
> >>
> >>
> >> Antony.

--
Normal people think "If it ain't broke, don't fix it".
Engineers think "If it ain't broke, it doesn't have enough features yet".

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov


11.03.2017 2:57, Antony Stone пишет:

> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>
>> Gentlemen, and it never occurred to you that there are other types of
>> traffic besides HTTP / HTTPS, right?
>>
>> DNS, ICMP, other protocols?
> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been
> measuring everything, I don't think DNS, ICMP and other protocols would add
> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also
> totally-non-Squid traffic on the link being measured.
Come on, sure? Even in L7? Really? Cool story, bro!

>
>
> Antony.
>
>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>> check that out, although I think that it's a problem even for bumped
>>> users. Even for bumped users we don't bump all sites, so that really
>>> could be it.
>>>
>>> Thanks!
>>>
>>>
>>> -----Original Message-----
>>> From: squid-users [mailto:[hidden email]] On
>>> Behalf Of Marcus Kool
>>> Sent: Friday, March 10, 2017 3:38 PM
>>> To: [hidden email]
>>> Subject: Re: [squid-users] Data usage reported in log files
>>>
>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>> Thanks!
>>>>
>>>> Netflow is much larger.
>>>>
>>>> I really want to know exactly what site is costing my users data. Many
>>>> of our users are on metered connections and are paying for overage,
>>>> but I can't tell where that overage is being used. Are they using
>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>> proxy usage in my squid logs.
>>>>
>>>> Data compression would give the opposite result, so that's not what
>>>> I'm seeing.
>>>>
>>>> Any other ideas?
>>> Is there any traffic that is not directed to Squid?
>>>
>>> Do you use ssl-bump in bump mode ?
>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>
>>> Marcus
>>>
>>>> -----Original Message-----
>>>> From: squid-users [mailto:[hidden email]]
>>>> On Behalf Of Antony Stone
>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>> To: [hidden email]
>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>
>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>> Hello all,
>>>>>
>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>> bytes reported as used by any particular user are often nowhere near
>>>>> the bytes reported by netflow and tcpdump.
>>>> Which is larger?
>>>>
>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>> do so from the log files because of this.
>>>> Well, what is it you really want to know?
>>>>
>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>> data on your Internet link - I assume this is what you're most
>>>> interested in?
>>>> Squid will show you what quantity of data goes to/from the clients,
>>>> but is that really important?
>>>>
>>>>> Can someone please explain to me what I might be missing? Why does
>>>>> squid log report one thing and netflow and tcpdump show something
>>>>> else?
>>>> Data compression?
>>>>
>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>
>>>>
>>>> Antony.
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov
Of course, there is no stream video from security cams, no voice IP, no
SIP, no torrents, no RDP, no other protocol. They simple does not exists
and we're all believe that's all not above over 1% of overall traffic.
Yes. Sure. Really.

Only web-surfing :) Sure :)


11.03.2017 3:19, Yuri Voinov пишет:

>
> 11.03.2017 2:57, Antony Stone пишет:
>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>
>>> Gentlemen, and it never occurred to you that there are other types of
>>> traffic besides HTTP / HTTPS, right?
>>>
>>> DNS, ICMP, other protocols?
>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been
>> measuring everything, I don't think DNS, ICMP and other protocols would add
>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also
>> totally-non-Squid traffic on the link being measured.
> Come on, sure? Even in L7? Really? Cool story, bro!
>>
>> Antony.
>>
>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>> check that out, although I think that it's a problem even for bumped
>>>> users. Even for bumped users we don't bump all sites, so that really
>>>> could be it.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: squid-users [mailto:[hidden email]] On
>>>> Behalf Of Marcus Kool
>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>> To: [hidden email]
>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>
>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>> Thanks!
>>>>>
>>>>> Netflow is much larger.
>>>>>
>>>>> I really want to know exactly what site is costing my users data. Many
>>>>> of our users are on metered connections and are paying for overage,
>>>>> but I can't tell where that overage is being used. Are they using
>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>> proxy usage in my squid logs.
>>>>>
>>>>> Data compression would give the opposite result, so that's not what
>>>>> I'm seeing.
>>>>>
>>>>> Any other ideas?
>>>> Is there any traffic that is not directed to Squid?
>>>>
>>>> Do you use ssl-bump in bump mode ?
>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>
>>>> Marcus
>>>>
>>>>> -----Original Message-----
>>>>> From: squid-users [mailto:[hidden email]]
>>>>> On Behalf Of Antony Stone
>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>
>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>> bytes reported as used by any particular user are often nowhere near
>>>>>> the bytes reported by netflow and tcpdump.
>>>>> Which is larger?
>>>>>
>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>> do so from the log files because of this.
>>>>> Well, what is it you really want to know?
>>>>>
>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>> data on your Internet link - I assume this is what you're most
>>>>> interested in?
>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>> but is that really important?
>>>>>
>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>> else?
>>>>> Data compression?
>>>>>
>>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>>
>>>>>
>>>>> Antony.
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov
Think of one simple thing. Squid does not see and can not see protocols
that do not support. What do you expect from it? Does it work on L1/L2?
No? Then what is the discussion about?


11.03.2017 3:22, Yuri Voinov пишет:

> Of course, there is no stream video from security cams, no voice IP, no
> SIP, no torrents, no RDP, no other protocol. They simple does not exists
> and we're all believe that's all not above over 1% of overall traffic.
> Yes. Sure. Really.
>
> Only web-surfing :) Sure :)
>
>
> 11.03.2017 3:19, Yuri Voinov пишет:
>> 11.03.2017 2:57, Antony Stone пишет:
>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>
>>>> Gentlemen, and it never occurred to you that there are other types of
>>>> traffic besides HTTP / HTTPS, right?
>>>>
>>>> DNS, ICMP, other protocols?
>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been
>>> measuring everything, I don't think DNS, ICMP and other protocols would add
>>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also
>>> totally-non-Squid traffic on the link being measured.
>> Come on, sure? Even in L7? Really? Cool story, bro!
>>> Antony.
>>>
>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>> check that out, although I think that it's a problem even for bumped
>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>> could be it.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: squid-users [mailto:[hidden email]] On
>>>>> Behalf Of Marcus Kool
>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>> To: [hidden email]
>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>
>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>> Thanks!
>>>>>>
>>>>>> Netflow is much larger.
>>>>>>
>>>>>> I really want to know exactly what site is costing my users data. Many
>>>>>> of our users are on metered connections and are paying for overage,
>>>>>> but I can't tell where that overage is being used. Are they using
>>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>>> proxy usage in my squid logs.
>>>>>>
>>>>>> Data compression would give the opposite result, so that's not what
>>>>>> I'm seeing.
>>>>>>
>>>>>> Any other ideas?
>>>>> Is there any traffic that is not directed to Squid?
>>>>>
>>>>> Do you use ssl-bump in bump mode ?
>>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>>
>>>>> Marcus
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: squid-users [mailto:[hidden email]]
>>>>>> On Behalf Of Antony Stone
>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>> To: [hidden email]
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>>> bytes reported as used by any particular user are often nowhere near
>>>>>>> the bytes reported by netflow and tcpdump.
>>>>>> Which is larger?
>>>>>>
>>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>>> do so from the log files because of this.
>>>>>> Well, what is it you really want to know?
>>>>>>
>>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>>> data on your Internet link - I assume this is what you're most
>>>>>> interested in?
>>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>>> but is that really important?
>>>>>>
>>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>>> else?
>>>>>> Data compression?
>>>>>>
>>>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>>>
>>>>>>
>>>>>> Antony.
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Antony Stone
In reply to this post by Yuri Voinov
On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP, no
> SIP, no torrents, no RDP, no other protocol. They simple does not exists
> and we're all believe that's all not above over 1% of overall traffic.
> Yes. Sure. Really.
>
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from the
IP of the Squid server, so as to ignore everything else he knows is happening
on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid server,
other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types of
> >>> traffic besides HTTP / HTTPS, right?
> >>>
> >>> DNS, ICMP, other protocols?
> >>
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
> >> been measuring everything, I don't think DNS, ICMP and other protocols
> >> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
> >> there is also totally-non-Squid traffic on the link being measured.
> >
> > Come on, sure? Even in L7? Really? Cool story, bro!
> >
> >> Antony.
> >>
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll
> >>>> check that out, although I think that it's a problem even for bumped
> >>>> users. Even for bumped users we don't bump all sites, so that really
> >>>> could be it.
> >>>>
> >>>> Thanks!
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: squid-users [mailto:[hidden email]]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>>
> >>>>> Netflow is much larger.
> >>>>>
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for
> >>>>> overage, but I can't tell where that overage is being used. Are they
> >>>>> using youtube, webmail, wetransfer? I see only a fraction of their
> >>>>> actual proxy usage in my squid logs.
> >>>>>
> >>>>> Data compression would give the opposite result, so that's not what
> >>>>> I'm seeing.
> >>>>>
> >>>>> Any other ideas?
> >>>>
> >>>> Is there any traffic that is not directed to Squid?
> >>>>
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS)
> >>>> tunnels.
> >>>>
> >>>> Marcus
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: squid-users [mailto:[hidden email]]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: [hidden email]
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>>
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>>
> >>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>>>>> bytes reported as used by any particular user are often nowhere near
> >>>>>> the bytes reported by netflow and tcpdump.
> >>>>>
> >>>>> Which is larger?
> >>>>>
> >>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>>>>> do so from the log files because of this.
> >>>>>
> >>>>> Well, what is it you really want to know?
> >>>>>
> >>>>> netflow / tcpdump will give you accurate numbers for the quantity of
> >>>>> data on your Internet link - I assume this is what you're most
> >>>>> interested in?
> >>>>> Squid will show you what quantity of data goes to/from the clients,
> >>>>> but is that really important?
> >>>>>
> >>>>>> Can someone please explain to me what I might be missing? Why does
> >>>>>> squid log report one thing and netflow and tcpdump show something
> >>>>>> else?
> >>>>>
> >>>>> Data compression?
> >>>>>
> >>>>> HTTP responses are often gzipped, so if tcpdump is showing you
> >>>>> smaller numbers of bytes than Squid reports, that's what I'd look at
> >>>>> first.
> >>>>>
> >>>>>
> >>>>> Antony.

--
<flopsie> yes, but this is #lbw, we don't do normal

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov
In reply to this post by Yuri Voinov
According to the above, NetFlow will always show much more traffic than
the SQUID. This is obvious and there is nothing to discuss here. If this
is not clear to someone, put a collector that collects statistics at the
data link level and compare the counters. I'm not just talking about
TCP, Alex. There is also the UDP. And there are a lot of protocols that
squid can not see, including for the simple reason that these packets
are never routed to a SQUID.

We have not seen the network topology and the full configuration of
network devices - what are we arguing about and guessing about?


11.03.2017 3:27, Yuri Voinov пишет:

> Think of one simple thing. Squid does not see and can not see protocols
> that do not support. What do you expect from it? Does it work on L1/L2?
> No? Then what is the discussion about?
>
>
> 11.03.2017 3:22, Yuri Voinov пишет:
>> Of course, there is no stream video from security cams, no voice IP, no
>> SIP, no torrents, no RDP, no other protocol. They simple does not exists
>> and we're all believe that's all not above over 1% of overall traffic.
>> Yes. Sure. Really.
>>
>> Only web-surfing :) Sure :)
>>
>>
>> 11.03.2017 3:19, Yuri Voinov пишет:
>>> 11.03.2017 2:57, Antony Stone пишет:
>>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>>
>>>>> Gentlemen, and it never occurred to you that there are other types of
>>>>> traffic besides HTTP / HTTPS, right?
>>>>>
>>>>> DNS, ICMP, other protocols?
>>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's been
>>>> measuring everything, I don't think DNS, ICMP and other protocols would add
>>>> more than 1% on top of HTTP/S, unless (as Marcus suggested) there is also
>>>> totally-non-Squid traffic on the link being measured.
>>> Come on, sure? Even in L7? Really? Cool story, bro!
>>>> Antony.
>>>>
>>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>>> check that out, although I think that it's a problem even for bumped
>>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>>> could be it.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: squid-users [mailto:[hidden email]] On
>>>>>> Behalf Of Marcus Kool
>>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>>> To: [hidden email]
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Netflow is much larger.
>>>>>>>
>>>>>>> I really want to know exactly what site is costing my users data. Many
>>>>>>> of our users are on metered connections and are paying for overage,
>>>>>>> but I can't tell where that overage is being used. Are they using
>>>>>>> youtube, webmail, wetransfer? I see only a fraction of their actual
>>>>>>> proxy usage in my squid logs.
>>>>>>>
>>>>>>> Data compression would give the opposite result, so that's not what
>>>>>>> I'm seeing.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>> Is there any traffic that is not directed to Squid?
>>>>>>
>>>>>> Do you use ssl-bump in bump mode ?
>>>>>> If not, Squid has no idea how many bytes go through the (HTTPS) tunnels.
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: squid-users [mailto:[hidden email]]
>>>>>>> On Behalf Of Antony Stone
>>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>>> To: [hidden email]
>>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>>
>>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>>>> bytes reported as used by any particular user are often nowhere near
>>>>>>>> the bytes reported by netflow and tcpdump.
>>>>>>> Which is larger?
>>>>>>>
>>>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>>>> do so from the log files because of this.
>>>>>>> Well, what is it you really want to know?
>>>>>>>
>>>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>>>> data on your Internet link - I assume this is what you're most
>>>>>>> interested in?
>>>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>>>> but is that really important?
>>>>>>>
>>>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>>>> else?
>>>>>>> Data compression?
>>>>>>>
>>>>>>> HTTP responses are often gzipped, so if tcpdump is showing you smaller
>>>>>>> numbers of bytes than Squid reports, that's what I'd look at first.
>>>>>>>
>>>>>>>
>>>>>>> Antony.
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Antony Stone
On Friday 10 March 2017 at 22:33:44, Yuri Voinov wrote:

> We have not seen the network topology and the full configuration of
> network devices - what are we arguing about and guessing about?

Nobody is arguing, and we are guessing so that we might be helpful to Yosi who
asked the question.

Incidentally, please could you consider putting all of your comments (which
are unrelated to further replies from other people) into a single posting,
instead of sending, for example, four emails to the list, each replying only
to your own previous comment?

That would make things far easier to follow in the conversation.


Thanks,


Antony.

--
I thought of going into banking, until I lost interest.

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yosi Greenfield
In reply to this post by Antony Stone
Gentlemen,

Thanks Antony. Yes, we are accounting for everything else. I'm
talking about port 3128 and 3129 only.

Any other traffic is being tracked both by netflow and tcpdump and
they match. What does not match is 3128/9 and squid log.

I'll report back after the weekend if the discrepancy is all
sslbump traffic.

Thank you all,
Yosi


-----Original Message-----
From: squid-users [mailto:[hidden email]] On
Behalf Of Antony Stone
Sent: Friday, March 10, 2017 4:31 PM
To: [hidden email]
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP,
> no SIP, no torrents, no RDP, no other protocol. They simple does not
> exists and we're all believe that's all not above over 1% of overall
traffic.
> Yes. Sure. Really.
>
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from
the IP of the Squid server, so as to ignore everything else he knows is
happening on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid
server, other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types of
> >>> traffic besides HTTP / HTTPS, right?
> >>>
> >>> DNS, ICMP, other protocols?
> >>
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
> >> been measuring everything, I don't think DNS, ICMP and other protocols
> >> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
> >> there is also totally-non-Squid traffic on the link being measured.
> >
> > Come on, sure? Even in L7? Really? Cool story, bro!
> >
> >> Antony.
> >>
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll
> >>>> check that out, although I think that it's a problem even for bumped
> >>>> users. Even for bumped users we don't bump all sites, so that really
> >>>> could be it.
> >>>>
> >>>> Thanks!
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: squid-users [mailto:[hidden email]]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>>
> >>>>> Netflow is much larger.
> >>>>>
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for
> >>>>> overage, but I can't tell where that overage is being used. Are they
> >>>>> using youtube, webmail, wetransfer? I see only a fraction of their
> >>>>> actual proxy usage in my squid logs.
> >>>>>
> >>>>> Data compression would give the opposite result, so that's not what
> >>>>> I'm seeing.
> >>>>>
> >>>>> Any other ideas?
> >>>>
> >>>> Is there any traffic that is not directed to Squid?
> >>>>
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS)
> >>>> tunnels.
> >>>>
> >>>> Marcus
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: squid-users [mailto:[hidden email]]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: [hidden email]
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>>
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>>
> >>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
> >>>>>> bytes reported as used by any particular user are often nowhere
near

> >>>>>> the bytes reported by netflow and tcpdump.
> >>>>>
> >>>>> Which is larger?
> >>>>>
> >>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
> >>>>>> do so from the log files because of this.
> >>>>>
> >>>>> Well, what is it you really want to know?
> >>>>>
> >>>>> netflow / tcpdump will give you accurate numbers for the quantity of
> >>>>> data on your Internet link - I assume this is what you're most
> >>>>> interested in?
> >>>>> Squid will show you what quantity of data goes to/from the clients,
> >>>>> but is that really important?
> >>>>>
> >>>>>> Can someone please explain to me what I might be missing? Why does
> >>>>>> squid log report one thing and netflow and tcpdump show something
> >>>>>> else?
> >>>>>
> >>>>> Data compression?
> >>>>>
> >>>>> HTTP responses are often gzipped, so if tcpdump is showing you
> >>>>> smaller numbers of bytes than Squid reports, that's what I'd look at
> >>>>> first.
> >>>>>
> >>>>>
> >>>>> Antony.

--
<flopsie> yes, but this is #lbw, we don't do normal

                                                   Please reply to the list;
                                                         please *don't* CC
me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov
In reply to this post by Antony Stone


11.03.2017 3:43, Antony Stone пишет:
> On Friday 10 March 2017 at 22:33:44, Yuri Voinov wrote:
>
>> We have not seen the network topology and the full configuration of
>> network devices - what are we arguing about and guessing about?
> Nobody is arguing, and we are guessing so that we might be helpful to Yosi who
> asked the question.
Guessing can be worse than a lack of response. As they take them away
from the true picture. Especially when you do not have any facts.
>
> Incidentally, please could you consider putting all of your comments (which
> are unrelated to further replies from other people) into a single posting,
> instead of sending, for example, four emails to the list, each replying only
> to your own previous comment?
>
> That would make things far easier to follow in the conversation.
I'll think about it in the future. I usually do not get into the
discussion here, except for very rare cases.
>
>
> Thanks,
>
>
> Antony.
>

--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Yuri Voinov
In reply to this post by Yosi Greenfield


11.03.2017 3:47, Yosi Greenfield пишет:
> Gentlemen,
>
> Thanks Antony. Yes, we are accounting for everything else. I'm
> talking about port 3128 and 3129 only.
>
> Any other traffic is being tracked both by netflow and tcpdump and
> they match. What does not match is 3128/9 and squid log.
It can be also because of tunneled traffic.

>
> I'll report back after the weekend if the discrepancy is all
> sslbump traffic.
>
> Thank you all,
> Yosi
>
>
> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On
> Behalf Of Antony Stone
> Sent: Friday, March 10, 2017 4:31 PM
> To: [hidden email]
> Subject: Re: [squid-users] Data usage reported in log files
>
> On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:
>
>> Of course, there is no stream video from security cams, no voice IP,
>> no SIP, no torrents, no RDP, no other protocol. They simple does not
>> exists and we're all believe that's all not above over 1% of overall
> traffic.
>> Yes. Sure. Really.
>>
>> Only web-surfing :) Sure :)
> Thanks for the standard sarcasm.
>
> Has it occurred to you that Yosi might have been measuring traffic to & from
> the IP of the Squid server, so as to ignore everything else he knows is
> happening on his network, so he can compare like with like?
>
> My "not more than 1%" was for the additional traffic to/from the Squid
> server, other than HTTP/S.
>
>
> Antony.
>
>> 11.03.2017 3:19, Yuri Voinov пишет:
>>> 11.03.2017 2:57, Antony Stone пишет:
>>>> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
>>>>> Gentlemen, and it never occurred to you that there are other types of
>>>>> traffic besides HTTP / HTTPS, right?
>>>>>
>>>>> DNS, ICMP, other protocols?
>>>> I'm assuming Yosi has been measuring only TCP traffic, but even if he's
>>>> been measuring everything, I don't think DNS, ICMP and other protocols
>>>> would add more than 1% on top of HTTP/S, unless (as Marcus suggested)
>>>> there is also totally-non-Squid traffic on the link being measured.
>>> Come on, sure? Even in L7? Really? Cool story, bro!
>>>
>>>> Antony.
>>>>
>>>>> 11.03.2017 2:44, Yosi Greenfield пишет:
>>>>>> Aha! That could be it. I use sslbump, but not for all users. I'll
>>>>>> check that out, although I think that it's a problem even for bumped
>>>>>> users. Even for bumped users we don't bump all sites, so that really
>>>>>> could be it.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: squid-users [mailto:[hidden email]]
>>>>>> On Behalf Of Marcus Kool
>>>>>> Sent: Friday, March 10, 2017 3:38 PM
>>>>>> To: [hidden email]
>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>
>>>>>> On 10/03/17 16:27, Yosi Greenfield wrote:
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Netflow is much larger.
>>>>>>>
>>>>>>> I really want to know exactly what site is costing my users data.
>>>>>>> Many of our users are on metered connections and are paying for
>>>>>>> overage, but I can't tell where that overage is being used. Are they
>>>>>>> using youtube, webmail, wetransfer? I see only a fraction of their
>>>>>>> actual proxy usage in my squid logs.
>>>>>>>
>>>>>>> Data compression would give the opposite result, so that's not what
>>>>>>> I'm seeing.
>>>>>>>
>>>>>>> Any other ideas?
>>>>>> Is there any traffic that is not directed to Squid?
>>>>>>
>>>>>> Do you use ssl-bump in bump mode ?
>>>>>> If not, Squid has no idea how many bytes go through the (HTTPS)
>>>>>> tunnels.
>>>>>>
>>>>>> Marcus
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: squid-users [mailto:[hidden email]]
>>>>>>> On Behalf Of Antony Stone
>>>>>>> Sent: Friday, March 10, 2017 2:21 PM
>>>>>>> To: [hidden email]
>>>>>>> Subject: Re: [squid-users] Data usage reported in log files
>>>>>>>
>>>>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I'm analyzing my squid logs with sarg, and I see that the number of
>>>>>>>> bytes reported as used by any particular user are often nowhere
> near
>>>>>>>> the bytes reported by netflow and tcpdump.
>>>>>>> Which is larger?
>>>>>>>
>>>>>>>> I'm trying to trace my users' data usage by site, but I'm unable to
>>>>>>>> do so from the log files because of this.
>>>>>>> Well, what is it you really want to know?
>>>>>>>
>>>>>>> netflow / tcpdump will give you accurate numbers for the quantity of
>>>>>>> data on your Internet link - I assume this is what you're most
>>>>>>> interested in?
>>>>>>> Squid will show you what quantity of data goes to/from the clients,
>>>>>>> but is that really important?
>>>>>>>
>>>>>>>> Can someone please explain to me what I might be missing? Why does
>>>>>>>> squid log report one thing and netflow and tcpdump show something
>>>>>>>> else?
>>>>>>> Data compression?
>>>>>>>
>>>>>>> HTTP responses are often gzipped, so if tcpdump is showing you
>>>>>>> smaller numbers of bytes than Squid reports, that's what I'd look at
>>>>>>> first.
>>>>>>>
>>>>>>>
>>>>>>> Antony.
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Data usage reported in log files

Eliezer Croitoru
In reply to this post by Yosi Greenfield
Hey Yosi,

Can you see if the differences is on the incoming or outgoing traffic?
Squid will only account for incoming and if you are using some kind of caching with the quick_abort and other partial content prefetch it would make sense that the actual consumption of the bits from the Internet to squid will not match from squid to clients.

If you can send me or share with others your squid.conf we might be able to understand if something there might cause such an issue.

Thanks,
Eliezer

* Feel free to contact me directly by skype or phone
----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Yosi Greenfield
Sent: Friday, March 10, 2017 11:47 PM
To: [hidden email]
Subject: Re: [squid-users] Data usage reported in log files

Gentlemen,

Thanks Antony. Yes, we are accounting for everything else. I'm talking about port 3128 and 3129 only.

Any other traffic is being tracked both by netflow and tcpdump and they match. What does not match is 3128/9 and squid log.

I'll report back after the weekend if the discrepancy is all sslbump traffic.

Thank you all,
Yosi


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Antony Stone
Sent: Friday, March 10, 2017 4:31 PM
To: [hidden email]
Subject: Re: [squid-users] Data usage reported in log files

On Friday 10 March 2017 at 22:22:59, Yuri Voinov wrote:

> Of course, there is no stream video from security cams, no voice IP,
> no SIP, no torrents, no RDP, no other protocol. They simple does not
> exists and we're all believe that's all not above over 1% of overall
traffic.
> Yes. Sure. Really.
>
> Only web-surfing :) Sure :)

Thanks for the standard sarcasm.

Has it occurred to you that Yosi might have been measuring traffic to & from the IP of the Squid server, so as to ignore everything else he knows is happening on his network, so he can compare like with like?

My "not more than 1%" was for the additional traffic to/from the Squid server, other than HTTP/S.


Antony.

> 11.03.2017 3:19, Yuri Voinov пишет:
> > 11.03.2017 2:57, Antony Stone пишет:
> >> On Friday 10 March 2017 at 21:50:19, Yuri Voinov wrote:
> >>> Gentlemen, and it never occurred to you that there are other types
> >>> of traffic besides HTTP / HTTPS, right?
> >>>
> >>> DNS, ICMP, other protocols?
> >>
> >> I'm assuming Yosi has been measuring only TCP traffic, but even if
> >> he's been measuring everything, I don't think DNS, ICMP and other
> >> protocols would add more than 1% on top of HTTP/S, unless (as
> >> Marcus suggested) there is also totally-non-Squid traffic on the link being measured.
> >
> > Come on, sure? Even in L7? Really? Cool story, bro!
> >
> >> Antony.
> >>
> >>> 11.03.2017 2:44, Yosi Greenfield пишет:
> >>>> Aha! That could be it. I use sslbump, but not for all users. I'll
> >>>> check that out, although I think that it's a problem even for
> >>>> bumped users. Even for bumped users we don't bump all sites, so
> >>>> that really could be it.
> >>>>
> >>>> Thanks!
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: squid-users
> >>>> [mailto:[hidden email]]
> >>>> On Behalf Of Marcus Kool
> >>>> Sent: Friday, March 10, 2017 3:38 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>
> >>>> On 10/03/17 16:27, Yosi Greenfield wrote:
> >>>>> Thanks!
> >>>>>
> >>>>> Netflow is much larger.
> >>>>>
> >>>>> I really want to know exactly what site is costing my users data.
> >>>>> Many of our users are on metered connections and are paying for
> >>>>> overage, but I can't tell where that overage is being used. Are
> >>>>> they using youtube, webmail, wetransfer? I see only a fraction
> >>>>> of their actual proxy usage in my squid logs.
> >>>>>
> >>>>> Data compression would give the opposite result, so that's not
> >>>>> what I'm seeing.
> >>>>>
> >>>>> Any other ideas?
> >>>>
> >>>> Is there any traffic that is not directed to Squid?
> >>>>
> >>>> Do you use ssl-bump in bump mode ?
> >>>> If not, Squid has no idea how many bytes go through the (HTTPS)
> >>>> tunnels.
> >>>>
> >>>> Marcus
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: squid-users
> >>>>> [mailto:[hidden email]]
> >>>>> On Behalf Of Antony Stone
> >>>>> Sent: Friday, March 10, 2017 2:21 PM
> >>>>> To: [hidden email]
> >>>>> Subject: Re: [squid-users] Data usage reported in log files
> >>>>>
> >>>>> On Friday 10 March 2017 at 20:14:36, Yosi Greenfield wrote:
> >>>>>> Hello all,
> >>>>>>
> >>>>>> I'm analyzing my squid logs with sarg, and I see that the
> >>>>>> number of bytes reported as used by any particular user are
> >>>>>> often nowhere
near

> >>>>>> the bytes reported by netflow and tcpdump.
> >>>>>
> >>>>> Which is larger?
> >>>>>
> >>>>>> I'm trying to trace my users' data usage by site, but I'm
> >>>>>> unable to do so from the log files because of this.
> >>>>>
> >>>>> Well, what is it you really want to know?
> >>>>>
> >>>>> netflow / tcpdump will give you accurate numbers for the
> >>>>> quantity of data on your Internet link - I assume this is what
> >>>>> you're most interested in?
> >>>>> Squid will show you what quantity of data goes to/from the
> >>>>> clients, but is that really important?
> >>>>>
> >>>>>> Can someone please explain to me what I might be missing? Why
> >>>>>> does squid log report one thing and netflow and tcpdump show
> >>>>>> something else?
> >>>>>
> >>>>> Data compression?
> >>>>>
> >>>>> HTTP responses are often gzipped, so if tcpdump is showing you
> >>>>> smaller numbers of bytes than Squid reports, that's what I'd
> >>>>> look at first.
> >>>>>
> >>>>>
> >>>>> Antony.

--
<flopsie> yes, but this is #lbw, we don't do normal

                                                   Please reply to the list;
                                                         please *don't* CC me.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users