Squid 3.5 https facebook caching

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Squid 3.5 https facebook caching

tester100
Hi guys

i am currently using this setup on my squid 3.5.28 version for https
filtering using ssl certificate

its caching http and https (some specific extensions) on facebook i can
cache images,css, and other javascript files..

aldo when i press play to play the video and try to cache it , it simply
does not play any videos i can only play the live feeds transmission, this
is the squid.conf files and the store-id.pl i am using



# SQUID CONFIGURATION OF CYBERSCIE.COM
#

acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged)
machines
acl localnet src 192.168.1.0/24
acl localnet src 192.168.2.0/24

acl SSL_ports port 443
acl SSL_ports port 5353
acl Safe_ports port 21
acl Safe_ports port 22
acl Safe_ports port 53
acl Safe_ports port 70
acl Safe_ports port 80
acl Safe_ports port 210
acl Safe_ports port 280
acl Safe_ports port 1025-65535
acl Safe_ports port 443
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl Safe_ports port 5353
acl Safe_ports port 18901-18909
acl Safe_ports port 1818
acl Safe_ports port 39190
acl Safe_ports port 40000-40010
acl Safe_ports port 7777
acl Safe_ports port 19101
acl Safe_ports port 27780
acl Safe_ports port 29000
acl Safe_ports port 22100
acl Safe_ports port 5121
acl Safe_ports port 6000-6152
acl Safe_ports port 2001
acl Safe_ports port 9601-9602
acl Safe_ports port 8085
acl Safe_ports port 11011-11041
acl Safe_ports port 13413
acl Safe_ports port 19000
acl Safe_ports port 5105
acl Safe_ports port 10009
acl Safe_ports port 12060-12070
acl Safe_ports port 6000-6001
acl Safe_ports port 29200
acl Safe_ports port 10402
acl Safe_ports port 9600
acl Safe_ports port 15002
acl Safe_ports port 16402-16502
acl Safe_ports port 5126
acl Safe_ports port 3010
acl Safe_ports port 11031  
acl Safe_ports port 11440-11460
acl Safe_ports port 11100-11125
acl Safe_ports port 4300
acl Safe_ports port 12011
acl Safe_ports port 12110
acl Safe_ports port 15001
acl Safe_ports port 15002
acl Safe_ports port 7341
acl Safe_ports port 7451
acl Safe_ports port 7808
acl Safe_ports port 30000
acl Safe_ports port 9001
acl Safe_ports port 9030
acl Safe_ports port 953
acl Safe_ports port 42051-42052
acl Safe_ports port 36567
acl Safe_ports port 8001
acl Safe_ports port 14000-14050
acl Safe_ports port 27019
acl Safe_ports port 28901-28920
acl Safe_ports port 7201-7208
acl Safe_ports port 17001-17002
acl Safe_ports port 14300-14440
acl Safe_ports port 15100-15150
acl Safe_ports port 7770-7790
acl Safe_ports port 16320-16340
acl Safe_ports port 9000-9160
acl Safe_ports port 7200
acl Safe_ports port 7400
acl Safe_ports port 7106
acl Safe_ports port 7999
acl Safe_ports port 47611
acl Safe_ports port 36567
acl Safe_ports port 10087
acl Safe_ports port 27000-27050
acl Safe_ports port 27014-27050
acl Safe_ports port 4380
acl Safe_ports port 3478
acl Safe_ports port 4379
acl Safe_ports port 8890
acl Safe_ports port 9339
acl Safe_ports port 8890
acl Safe_ports port 7200-7210
acl Safe_ports port 7450-7460
acl Safe_ports port 8000
acl Safe_ports port 64990-65010
acl CONNECT method CONNECT

# ACCESS RULES
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access allow localhost
http_access deny all

# LISTENING PORT SQUID

http_port 3128 ssl-bump cert=/etc/squid/ssl_certs/myCA.pem
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB


# CONNECTION HANDLING
qos_flows local-hit=0x30
collapsed_forwarding on
balance_on_multiple_ip on
detect_broken_pconn on
client_persistent_connections off
server_persistent_connections on

# DNS OPTIONS
#dns_packet_max 4096
dns_defnames on
dns_v4_first on
connect_retries 2
negative_dns_ttl 1 second
quick_abort_min 0
quick_abort_max 0
quick_abort_pct 80
range_offset_limit 0
ipcache_low 98
ipcache_high 99
ipcache_size 4096
fqdncache_size 2048
pipeline_prefetch 0

# MISCELEANOUS
memory_pools off
reload_into_ims on
max_filedescriptors 65536

# CACHE MANAGEMENT
cache_mem 512 MB
maximum_object_size_in_memory 128 KB
memory_replacement_policy heap GDSF
cache_effective_group proxy
cache_effective_user proxy
cache_dir aufs /cache/cache 100000 16 256
coredump_dir /cache/cache
cache_mgr cyberscie
visible_hostname [hidden email]
minimum_object_size 0 KB
maximum_object_size 1 GB
read_ahead_gap 64 KB #Amount of data to buffer from server to client
cache_replacement_policy heap LFUDA
store_dir_select_algorithm least-load
cache_swap_low 90
cache_swap_high 95

# LOG FILE OPTIONS
logfile_daemon /usr/lib/squid/log_file_daemon
access_log daemon:/var/log/squid/access.log squid
cache_log /dev/null #cache_log /var/log/squid/cache.log(to enable)
cache_store_log none
logfile_rotate 3
pid_filename /var/run/squid.pid

# FILTERING HTTPS
acl 1 dstdomain .fbcdn.net .akamaihd.net .fbsbx.com
#acl 2a dstdomain .mahadana.com .mql4.com .metaquotes.net
acl 2 url_regex -i ^https?:\/\/attachment\.fbsbx\.com\/.*\?(id=[0-9]*).*
acl 2 url_regex -i
\.fbsbx\.com\/.*\/(.*\.(unity3d|pak|zip|exe|dll|jpg|png|gif|swf)/)$
acl 2 url_regex -i ^https?:\/\/.*\.ytimg\.com(.*\.(webp|jpg|gif))
acl 2 url_regex -i ^https?:\/\/([^\.]*)\.yimg\.com\/(.*)
acl 2 url_regex -i ^https?:\/\/.*\.gstatic\.com\/images\?q=tbn\:(.*)
acl 2 url_regex -i
^https?:\/\/.*\.reverbnation\.com\/.*\/(ec_stream_song|download_song_direct|stream_song)\/([0-9]*).*
acl 2 url_regex -i
^https?:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|.exoclick\.com|interclick.\com|\.googlesyndication\.com|\.auditude\.com|.visiblemeasures\.com|yieldmanager|cpxinteractive)(.*)
acl 2 url_regex -i ^https?:\/\/(.*?)\/(ads)\?(.*?)
acl 2 url_regex -i ^https?:\/\/.*steampowered\.com\/.*\/([0-9]+\/(.*))
acl 3 url_regex -i
^https?:\/\/(.*?)\/speedtest\/.*\.(jpg|txt|png|gif|swf)\?.*
acl 3 url_regex -i speedtest\/.*\.(jpg|txt|png|gif|swf)\?.*
acl 4 url_regex -i reverbnation.*audio_player.*ec_stream_song.*$
acl 5 url_regex -i utm.gif.*
acl 6 url_regex -i c.android.clients.google.com.market.GetBinary.GetBinary.*
acl 7 url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204).*$
acl 7 url_regex -i
\.c\.(youtube|google)\.com\/(get_video|videoplayback|videoplay).*$
acl 7 url_regex -i (youtube|google).*\/videoplayback\?.*
acl 8 http_status 302
acl getmethod method GET

ssl_bump splice localhost
acl 9 at_step SslBump1
acl 10 at_step SslBump2
acl 11 at_step SslBump3
ssl_bump peek 9 all
ssl_bump bump 10 all
ssl_bump bump 11 all

sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
sslcrtd_children 16 startup=1 idle=1
sslproxy_capath /etc/ssl/certs
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER #this line fixing www.gmail.com,
mail.yahoo.com for some errors
always_direct allow all
ssl_unclean_shutdown on

# STORE ID
store_id_program /usr/bin/perl /etc/squid/store-id.pl
store_id_children 10 startup=5 idle=2 concurrency=10
store_id_access allow 1
store_id_access allow 2
store_id_access allow 3
store_id_access allow 4
store_id_access allow 5
store_id_access allow 6
store_id_access allow 7
store_miss deny 7 8
send_hit deny 7 8
store_id_access deny all

# TUNNING CACHE
max_stale 1 years
vary_ignore_expire on
shutdown_lifetime 10 seconds

# REFRESH PATTERN
refresh_pattern -i https?:\/\/.*\.xx\.fbcdn\.net\/.*\.(jpg|png) 43830 99%
259200 override-expire override-lastmod ignore-reload
refresh_pattern static\.(xx|ak)\.fbcdn\.net*\.(jpg|gif|png) 241920 99%
241920 ignore-reload override-expire ignore-no-store
refresh_pattern ^https?\:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 241920
99% 241920 ignore-reload override-expire ignore-no-store
refresh_pattern (akamaihd|fbcdn)\.net 14400 99% 518400  ignore-no-store
ignore-private ignore-reload ignore-must-revalidate store-stale
refresh_pattern (audio|video)\/(webm|mp4) 129600 99% 129600 ignore-reload
override-expire override-lastmod ignore-must-revalidate  ignore-private
ignore-no-store ignore-auth store-stale
refresh_pattern -i \/speedtest\/.*\.(txt|jpg|png|swf)  0  99% 14400
override-expire ignore-reload ignore-private ignore-reload override-lastmod
reload-into-ims
refresh_pattern -i reverbnation.com 1440 99% 14400 override-expire
override-lastmod ignore-no-cache ignore-private ignore-must-revalidate
ignore-reload store-stale
refresh_pattern -i (yimg|twimg)\.com\.* 1440 100% 129600 override-expire
ignore-reload reload-into-ims
refresh_pattern -i (ytimg|ggpht)\.com\.* 1440 80% 129600 override-expire
override-lastmod ignore-auth ignore-reload reload-into-ims
refresh_pattern -i
(get_video\?|videoplayback\?|videodownload\?|\.mp4|\.webm|\.flv|((audio|video)\/(webm|mp4)))
241920 100% 241920 override-expire ignore-reload ignore-private
ignore-no-store ignore-must-revalidate reload-into-ims ignore-auth
store-stale
refresh_pattern -i ^https?\:\/\/.*\.googlevideo\.com\/videoplayback.* 10080
99% 43200 override-lastmod override-expire ignore-reload reload-into-ims
ignore-private reload-into-ims ignore-auth store-stale
refresh_pattern
^\.*(streamate.doublepimp.com.*\.js\?|utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).*
1440 99% 14400 ignore-private override-expire ignore-reload ignore-auth
max-stale=1440
refresh_pattern \.(ico|video-stats) 1440 99% 14400 override-expire
ignore-reload ignore-private ignore-auth override-lastmod
ignore-must-revalidate
refresh_pattern
^http://((cbk|mt|khm|mlt|tbn)[0-9]?)\.google\.co(m|\.uk|\.id) 1440 99% 14400
override-expire override-lastmod ignore-reload ignore-private ignore-auth
ignore-must-revalidate
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 1440 99% 14400
override-expire override-lastmod
refresh_pattern galleries\.video(\?|sz) 1440 99% 14400 override-expire
ignore-reload ignore-must-revalidate ignore-private
refresh_pattern \.wikimapia\.org\/? 1440 99% 14400 override-expire
override-lastmod ignore-reload ignore-private
refresh_pattern -i (livescore.com|goal.com|bobet) 0 50% 60
refresh_pattern
(photobucket|pbsrc|flickr|yimg|ytimg|twimg|gravatar)\.com.*\.(jp(e?g|e|2)|gif|png|tiff?|bmp|swf|mp(4|3))
1440 99% 14400 override-expire ignore-reload ignore-private
refresh_pattern
(zynga|topeleven|ninjasaga|mafiawars|cityville|farmville|crowdstar|spilcdn|agame|popcap)\.com/.*
1440 99% 14400 override-expire ignore-reload ignore-private
refresh_pattern -i
\.(3gp|7z|ace|asx|bin|deb|divx|dvr-ms|ram|rpm|exe|inc|cab|qt) 10080 80%
10080 override-expire override-lastmod reload-into-ims
refresh_pattern -i
\.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar|iop|nzp|pak|mar|msp)
10080 80% 10080 override-expire override-lastmod reload-into-ims
ignore-reload
refresh_pattern -i
\.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll) 10080 80% 10080
override-expire override-lastmod reload-into-ims
refresh_pattern -i
\.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob|webm)
10080 80% 10080 override-expire override-lastmod reload-into-ims
refresh_pattern -i
\.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv)
10080 80% 10080 override-expire override-lastmod reload-into-ims
refresh_pattern -i
\.(3gp|7z|ace|asx|bin|deb|cup|dvr-ms|ram|rpm|exe|inc|cab|qt) 10080 100%
43800 override-expire override-lastmod ignore-reload ignore-no-store
ignore-private ignore-auth ignore-must-revalidate store-stale
refresh_pattern -i
\.(rar|jar|gz|tgz|bz2|iso|m1v|m2(v|p)|mo(d|v)|arj|lha|lzh|zip|tar|pak|cup)
10080 100% 43800 override-expire override-lastmod ignore-reload
ignore-no-store ignore-private ignore-auth ignore-must-revalidate
store-stale
refresh_pattern -i
\.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf|dat|ad|txt|dll) 10080 100% 43800
override-expire override-lastmod ignore-reload ignore-no-store
ignore-private ignore-auth ignore-must-revalidate store-stale
refresh_pattern -i
\.(avi|ac4|mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p)|og(x|v|a|g)|rm|r(a|p)m|snd|vob)
10080 100% 43800 override-expire override-lastmod ignore-reload
ignore-no-store ignore-private ignore-auth ignore-must-revalidate
store-stale
refresh_pattern -i
\.(pp(t?x)|s|t)|pdf|rtf|wax|wm(a|v)|wmx|wpl|cb(r|z|t)|xl(s?x)|do(c?x)|flv|x-flv)
10080 100% 43800 override-expire override-lastmod ignore-reload
ignore-no-store ignore-private ignore-auth ignore-must-revalidate
store-stale
refresh_pattern -i .(html|htm|css|js|xml)$ 1440 75% 40320
refresh_pattern -i .index.(html|htm)$ 0 75% 43800
refresh_pattern -i ^http.*squid\.internal.* 43200 100% 799000
override-expire override-lastmod ignore-reload ignore-no-store
ignore-must-revalidate ignore-private ignore-auth

#KEEP THESE LINES AT BOTTOM OF CONFIGURATION
refresh_pattern ^ftp:  1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern .  0 20% 4320



and the store-id.pl below



#!/usr/bin/perl
#
# storeid.pl with debug opt - based on storeurl.pl
# @ http://www2.fh-lausitz.de/launic/comp/misc/squid/projekt_youtube/
#
# modified by cyberscie.com
 
use IO::File;
$|=1;
STDOUT->autoflush(1);
$debug=0; ## recommended:0
$bypassallrules=0; ## recommended:0
$sucks=""; ## unused
$sucks="sucks" if ($debug>=1);
$timenow="";
$printtimenow=1;   ## print timenow: 0|1
my $logfile = '/tmp/storeid.log';

open my $logfh, '>>', $logfile
    or die "Couldn't open $logfile for appending: $!\n" if $debug;
$logfh->autoflush(1) if $debug;

while (<>) {
$timenow=time()." " if ($printtimenow);
print $logfh "$timenow"."in : $_" if ($debug>=1);
chop;
my $myURL = $_;
@X = split(" ",$myURL);
$a = $X[0]; ## channel id
$b = $X[1]; ## url
$c = $X[2]; ## ip address
$u = $b; ## url

if ($bypassallrules){
 $out="$u"; ## map 1:1

} elsif ($u=~
m/http.*\.(fbcdn|akamaihd)\.net\/h(profile|photos).*[\d\w].*\/([\w]\d+x\d+\/.*\.[\d\w]{3}).*/)
{
        $out="OK store-id=http://fbcdn.net.squid.internal/" . $2 . "/" . $3 ;

} elsif ($u=~
m/^http(.*)static(.*)(akamaihd|fbcdn).net\/rsrc.php\/(.*\/.*\/(.*).(js|css|png|gif))(\?(.*)|$)/)
{
        $out="OK store-id=http://fbcdn.net.squid.internal/static/" . $5 . "." . $6
;

} elsif ($u=~ m/^https?\:\/\/.*utm.gif.*/) {
        $out="OK store-id=http://google-analytics.squid.internal/__utm.gif";
 
} elsif ($u=~ m/^https?\:\/\/.*\/speedtest\/(.*\.(jpg|txt)).*/) {
        $out="OK store-id=http://speedtest.squid.internal/" . $1;
 
} elsif ($u=~ m/^https?\:\/\/.*\/(.*\..*(mp4|3gp|flv))\?.*/) {
        $out="OK store-id=http://video-file.squid.internal/" . $1;

} elsif ($u=~
m/^https?\:\/\/c2lo\.reverbnation\.com\/audio_player\/ec_stream_song\/(.*)\?.*/)
{
        $out="OK store-id=http://reverbnation.squid.internal/" . $1;
 
} elsif ($u=~
m/^https?\:\/\/.*\.c\.android\.clients\.google\.com\/market\/GetBinary\/GetBinary\/(.*\/.*)\?.*/)
{
        $out="OK store-id=http://playstore-android.squid.internal/" . $1;

} elsif ($u =~
m/^http:\/\/([a-z])[0-9]?(\.gstatic\.com.*|\.wikimapia\.org.*)/) {
        $out="OK store-id=http://gstatic.squid.internal/" . $2;

} elsif ($u =~
m/^https?:\/\/.*(googleusercontent.com|blogspot.com)\/(.*)\/([a-z0-9]+)(-[a-z]-[a-z]-[a-z]+)?\/(.*\.(jpg|png))/){
        $out="OK store-id=http://googleusercontent.squid.internal/" . $5;

} elsif ($_ =~
m/^https?:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|.exoclick\.com|interclick.\com|\.googlesyndication\.com|\.auditude\.com|.visiblemeasures\.com|yieldmanager|cpxinteractive)(.*)/){
        $out="OK store-id=http://ads.squid.internal/" . $3;

} elsif ($u=~ m/^http\:\/\/.*\.4shared\.com\/download\/(.*)\/.*/) {
        $out="OK store-id=http://4shared.squid.internal/" . $1;

} elsif ($u =~ m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {
        $out="OK store-id=http://ziddu.squid.internal/" . $1;

} elsif ($u =~ m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/)
{
        $out="OK store-id=http://cdn.yimg.squid.internal/" . $3;

} elsif (($u =~ /filehippo/) &&
(m/^https?:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {
        @y = ($1,$2,$4,$5); $y[0] =~ s/[a-z0-9]{2,5}/cdn./;
        $out="OK store-id=http://filehippo.squid.internal/" . $3;
       
} elsif ($u[1] =~
m/^http:\/\/.*dlink__[23]Fdownload_[23]F([\w\d-]+)_3Ftsid.*/) {
        $1 =~ s/_5F/_/g;
        $out="OK store-id=http://4shared.squid.internal/" . $1;

} elsif ($u=~ m/^https?\:\/\/.*youtube.*ptracking.*/){
        @video_id = m/[&?]video_id\=([^\&\s]*)/;
        @cpn = m/[&?]cpn\=([^\&\s]*)/;
        unless (-e "/tmp/@cpn"){
        open FILE, ">/tmp/@cpn";
        print FILE "@video_id";
        close FILE;
        }
        $out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*stream_204.*/){
        @docid = m/[&?]docid\=([^\&\s]*)/;
        @cpn = m/[&?]cpn\=([^\&\s]*)/;
        unless (-e "/tmp/@cpn"){
        open FILE, ">/tmp/@cpn";
        print FILE "@docid";
        close FILE;
        }
        $out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*youtube.*player_204.*/){
        @v = m/[&?]v\=([^\&\s]*)/;
        @cpn = m/[&?]cpn\=([^\&\s]*)/;
        unless (-e "/tmp/@cpn"){
        open FILE, ">/tmp/@cpn";
        print FILE "@v";
        close FILE;
        }
        $out="ERR";
 
} elsif ($u=~ m/^https?\:\/\/.*(youtube|googlevideo).*videoplayback.*/){
        @itag = m/[&?](itag\=[0-9]*)/;
        @range = m/[&?](range\=[^\&\s]*)/;
        @cpn = m/[&?]cpn\=([^\&\s]*)/;
        @mime = m/[&?](mime\=[^\&\s]*)/;
        @id = m/[&?]id\=([^\&\s]*)/;
 
        if (defined(@cpn[0])){
                if (-e "/tmp/@cpn"){
                open FILE, "/tmp/@cpn";
                @id = <FILE>;
                close FILE;}
        }
        $out="OK store-id=http://video-srv.squid.internal/id=@id@mime@range";

} else {
        $out="ERR";
}
        print $logfh "$timenow"."out: $a $out\n" if ($debug>=1);
        print "$a $out\n";
}
close $logfh if ($debug);





This way i can play the facebook videos, but no caching is done i only get
TCP_TUNNEL/200



http_port 3129 tproxy ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_certs/squid.crt
key=/etc/squid/ssl_certs/squid.key
cipher=ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA128-SHA:AES128-SHA:RC4-SHA:HIGH:!aNULL:!MD5:!ADH




And this way i can cache https but cannot play the videos on the facebook at
all



http_port 3128 ssl-bump cert=/etc/squid/ssl_certs/myCA.pem
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB




Any ideas or hints on what could be wrong? i am kind of lost now.. any tips
will be appreciate



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

tester100
Aldo

when i press refresh or when i clean history on my browser and login to
facebook again i can see this memory hits on the log.. with .mp4 video
extensions but cannot play it at all

<http://squid-web-proxy-cache.1019090.n4.nabble.com/file/t377703/m6ZdWuf.png>



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

Amos Jeffries
Administrator
On 17/04/19 3:11 am, tester100 wrote:
> Aldo
>
> when i press refresh or when i clean history on my browser and login to
> facebook again i can see this memory hits on the log.. with .mp4 video
> extensions but cannot play it at all

Yes those Browser controls are normally the way one forces proxies
awareness of problems to make them fix this type of issue.

However, the proxy admin configured a lot of ignore-* options on the
refresh_patterns as well as a global conversion of reload into IMS
fetches ("reload_into_ims on"). As a result the Browser reload or
revalidate requests get ignored or converted into less effective fetches.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

Amos Jeffries
Administrator
In reply to this post by tester100
On 17/04/19 3:04 am, tester100 wrote:

> Hi guys
>
> i am currently using this setup on my squid 3.5.28 version for https
> filtering using ssl certificate
>
> its caching http and https (some specific extensions) on facebook i can
> cache images,css, and other javascript files..
>
> aldo when i press play to play the video and try to cache it , it simply
> does not play any videos i can only play the live feeds transmission, this
> is the squid.conf files and the store-id.pl i am using
>
>
>
> # SQUID CONFIGURATION OF CYBERSCIE.COM
> #
>
> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
> acl localnet src fc00::/7       # RFC 4193 local private network range
> acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged)
> machines
> acl localnet src 192.168.1.0/24
> acl localnet src 192.168.2.0/24
>
> acl SSL_ports port 443
> acl SSL_ports port 5353
> acl Safe_ports port 21
> acl Safe_ports port 22
> acl Safe_ports port 53
> acl Safe_ports port 70
> acl Safe_ports port 80
> acl Safe_ports port 210
> acl Safe_ports port 280
> acl Safe_ports port 1025-65535

The above line means that any of the *many* entries you have for ports
over 1024 are a pointless waste of memory and CPU cycles.

Please start by running "squid -k parse" on your config and fix all the
issues that get mentioned.
...

> acl CONNECT method CONNECT
>
> # ACCESS RULES
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localnet
> http_access allow localhost
> http_access deny all
>
> # LISTENING PORT SQUID
>
> http_port 3128 ssl-bump cert=/etc/squid/ssl_certs/myCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
>
>
> # CONNECTION HANDLING
> qos_flows local-hit=0x30
> collapsed_forwarding on
> balance_on_multiple_ip on

The above breaks performance of server persistent connections....

> detect_broken_pconn on
> client_persistent_connections off
> server_persistent_connections on

... which is the only type of persistence you have enabled.

>
> # DNS OPTIONS
> #dns_packet_max 4096
> dns_defnames on
> dns_v4_first on
> connect_retries 2
> negative_dns_ttl 1 second
> quick_abort_min 0
> quick_abort_max 0
> quick_abort_pct 80
> range_offset_limit 0

Hm, so all transactions which are started will run until completion even
if no client needs that response.

> ipcache_low 98
> ipcache_high 99
> ipcache_size 4096
> fqdncache_size 2048
> pipeline_prefetch 0
>
> # MISCELEANOUS
> memory_pools off
> reload_into_ims on
> max_filedescriptors 65536
>
> # CACHE MANAGEMENT
> cache_mem 512 MB
> maximum_object_size_in_memory 128 KB
> memory_replacement_policy heap GDSF
> cache_effective_group proxy
> cache_effective_user proxy
> cache_dir aufs /cache/cache 100000 16 256
> coredump_dir /cache/cache
> cache_mgr cyberscie
> visible_hostname [hidden email]

The above looks like an email address. Not an FQDN.

"hostname" is the name of the machine running Squid. To work properly it
should be a FQDN that can be resolved with DNS.


> minimum_object_size 0 KB
> maximum_object_size 1 GB
> read_ahead_gap 64 KB #Amount of data to buffer from server to client
> cache_replacement_policy heap LFUDA
> store_dir_select_algorithm least-load
> cache_swap_low 90
> cache_swap_high 95
>
> # LOG FILE OPTIONS
> logfile_daemon /usr/lib/squid/log_file_daemon
> access_log daemon:/var/log/squid/access.log squid
> cache_log /dev/null #cache_log /var/log/squid/cache.log(to enable)

This is a bad idea. What do you expect will happen to the machine when
Squid renames the /dev/null path to /dev/null.0 and places a file at
that location?


> cache_store_log none

It is not necessary to set things to their default value in Squid-3.

> logfile_rotate 3
> pid_filename /var/run/squid.pid
>
> # FILTERING HTTPS
> acl 1 dstdomain .fbcdn.net .akamaihd.net .fbsbx.com
> #acl 2a dstdomain .mahadana.com .mql4.com .metaquotes.net
> acl 2 url_regex -i ^https?:\/\/attachment\.fbsbx\.com\/.*\?(id=[0-9]*).*
> acl 2 url_regex -i
> \.fbsbx\.com\/.*\/(.*\.(unity3d|pak|zip|exe|dll|jpg|png|gif|swf)/)$


The above "2" lines are pointless. The ACL called "1" has already
matched and allowed Store-ID processing of all the domains this pattern
might match.

> acl 2 url_regex -i ^https?:\/\/.*\.ytimg\.com(.*\.(webp|jpg|gif))

The above pattern does not match what it may seem to match.

Notice that;
  A) there is no path-segment delimiter ('/' or '\?') required. So the
thing that _looks_ like a file extension can match when existing in the
domain name (eg http://hello.ytimg.com.gif-fy.invalid/ will be allowed)
 B)


> acl 2 url_regex -i ^https?:\/\/([^\.]*)\.yimg\.com\/(.*)
> acl 2 url_regex -i ^https?:\/\/.*\.gstatic\.com\/images\?q=tbn\:(.*)



It is pointless to place "(.*)" or ".*" or ".+" at the start or end of a
regex pattern.

Arbitrary suffix is implicit and all this will do is slow the regex
processing down even further trying to match the entire (possibly VERY
long) URL against ".*"
 That goes for all places you use regex patterns.


> acl 2 url_regex -i
> ^https?:\/\/.*\.reverbnation\.com\/.*\/(ec_stream_song|download_song_direct|stream_song)\/([0-9]*).*
> acl 2 url_regex -i
> ^https?:\/\/([a-z0-9.]*)(\.doubleclick\.net|\.quantserve\.com|.exoclick\.com|interclick.\com|\.googlesyndication\.com|\.auditude\.com|.visiblemeasures\.com|yieldmanager|cpxinteractive)(.*)
> acl 2 url_regex -i ^https?:\/\/(.*?)\/(ads)\?(.*?)
> acl 2 url_regex -i ^https?:\/\/.*steampowered\.com\/.*\/([0-9]+\/(.*))
> acl 3 url_regex -i
> ^https?:\/\/(.*?)\/speedtest\/.*\.(jpg|txt|png|gif|swf)\?.*
> acl 3 url_regex -i speedtest\/.*\.(jpg|txt|png|gif|swf)\?.*

Notice that the second line for "3" matches everything the first pattern
does, and a lot more. You can erase the first pattern and save a lot of
CPU without any change in proxy allow/deny permissions.

> acl 4 url_regex -i reverbnation.*audio_player.*ec_stream_song.*$
> acl 5 url_regex -i utm.gif.*
> acl 6 url_regex -i c.android.clients.google.com.market.GetBinary.GetBinary.*
> acl 7 url_regex -i youtube.*(ptracking|stream_204|player_204|gen_204).*$
> acl 7 url_regex -i
> \.c\.(youtube|google)\.com\/(get_video|videoplayback|videoplay).*$
> acl 7 url_regex -i (youtube|google).*\/videoplayback\?.*
> acl 8 http_status 302
> acl getmethod method GET
>
> ssl_bump splice localhost
> acl 9 at_step SslBump1
> acl 10 at_step SslBump2
> acl 11 at_step SslBump3
> ssl_bump peek 9 all
> ssl_bump bump 10 all
> ssl_bump bump 11 all

Why the weird numbers?

The "all" on all the above ssl_bump lines are pointless and may be
confusing you.


>
> sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M 4MB
> sslcrtd_children 16 startup=1 idle=1
> sslproxy_capath /etc/ssl/certs
> sslproxy_cert_error allow all

Above prevents Squid from acting on TLS errors. In fact it can cause
some to be hidden which should be fatal to the transaction.


> sslproxy_flags DONT_VERIFY_PEER #this line fixing www.gmail.com,

Absolutely No! The above line 'fixes' nothing. All it does is tell Squid
not to bother checking TLS security.

Those problems still exist, still cause other side effects, and are
often breaking things for clients. But you cannot see that because it is
being hidden by the above setting.

 ... and as a bonus (for the bad guys) your proxy can now be hijacked by
a malicious HTTPS server without any hints being given about it happening.


I suspect that whatever is going wrong the crypto activity is part of
it. But without those crypto issues being visible nobody can say for
sure. There are also refresh_pattern issues mentionend below.


> mail.yahoo.com for some errors
> always_direct allow all

Please remove this. You do not have any cache_peer lines. This setting
was only ever needed for a single Squid-3.2 release over a 2-week
period. That bug was long, long, long ago fixed.


> ssl_unclean_shutdown on
>
> # STORE ID
> store_id_program /usr/bin/perl /etc/squid/store-id.pl
> store_id_children 10 startup=5 idle=2 concurrency=10
> store_id_access allow 1
> store_id_access allow 2
> store_id_access allow 3
> store_id_access allow 4
> store_id_access allow 5
> store_id_access allow 6

So if all the 1 thru 6 ACLs are just url_regex patterns. Why bother
having them as separate ACLs? you can list all the patterns in one ACL
and save a lot of CPU cycles and time.


> store_id_access allow 7
> store_miss deny 7 8
> send_hit deny 7 8

It would be a lot more performant to switch those around. Check 8 then
7. Like so:

  store_miss deny 8 7
  send_hit deny 8 7


> store_id_access deny all
>

This should really be up with the other store_id_access lines.


> # TUNNING CACHE
> max_stale 1 years
> vary_ignore_expire on
> shutdown_lifetime 10 seconds
>
> # REFRESH PATTERN
> refresh_pattern -i https?:\/\/.*\.xx\.fbcdn\.net\/.*\.(jpg|png) 43830 99%
> 259200 override-expire override-lastmod ignore-reload

Um,

1) override-lastmod is generally a bad idea. It prevents the
Last-Modified header telling Squid that an object as any previous time
since it was updated - this option actively *reduces* the time objects
can be cached.

2) overide-expire should not be used for sites like Facebook which
provide well behaved cacheability headers. Like (1) it actively breaks
caching with often the opposite result to what one wants.

3) ignore-reload is part of why your Browser "refresh" attempts are
failing to do anything at all.

4) override-expire is *shortening* the caching time for these objects.
Facebook actually has pretty good cacheability once you get past the
problem of it all being hidden behind crypto.


> refresh_pattern static\.(xx|ak)\.fbcdn\.net*\.(jpg|gif|png) 241920 99%
> 241920 ignore-reload override-expire ignore-no-store

5) ignore-no-store is a bad idea. This *forces* private details from one
persons FB profile pages to be delivered to other clients. It exists
only because there are some very badly designed sites abusing the
Cache-Control header. Facebook is *not* one of those sites.


> refresh_pattern ^https?\:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 241920
> 99% 241920 ignore-reload override-expire ignore-no-store
> refresh_pattern (akamaihd|fbcdn)\.net 14400 99% 518400  ignore-no-store
> ignore-private ignore-reload ignore-must-revalidate store-stale


6) ignore-private has been made relatively safe in the latest Squid. BUT
the revalidation mechanisms are required for it to be safe at all.
 It is a very bad idea to use with either ignore-reload or
ignore-must-revalidate ... let alone both at once. Security
vulnerabilities will exist as a result of these options used together.

7) store-stale is in a similar position of requiring revalidation /
reload to be possible. But with less severe results - only badly broken
web page display.


These issues caused by (5), (6), and (7) could be at least a part of
what is going wrong. Probably also some other things.


> refresh_pattern (audio|video)\/(webm|mp4) 129600 99% 129600 ignore-reload
> override-expire override-lastmod ignore-must-revalidate  ignore-private
> ignore-no-store ignore-auth store-stale


Notice that if the earlier FB patterns did not match this makes videos
and audio URLs forced to be immediately stale/expired, forced to be
cached anyway, forced all clients/users to get the same objects, and
then also prohibits anything from updating the cache object if a broken
one gets into cache somehow.

You were saying something about video problems?


The remainder of your refresh_patterns show a lot of repeats of these
issues.

Remember: these options are dangerous. Use with great care. And
understand what the options are doing



>
> This way i can play the facebook videos, but no caching is done i only get
> TCP_TUNNEL/200
>
>
>
> http_port 3129 tproxy ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_certs/squid.crt
> key=/etc/squid/ssl_certs/squid.key
> cipher=ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA128-SHA:AES128-SHA:RC4-SHA:HIGH:!aNULL:!MD5:!ADH
>


Do you know what the differences between these two port lines are?

In one the client is aware that the proxy exists and sends details to it
in a CONNECT request.


>
> And this way i can cache https but cannot play the videos on the facebook at
> all
>
>
>
> http_port 3128 ssl-bump cert=/etc/squid/ssl_certs/myCA.pem
> generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
>
>
>
>
> Any ideas or hints on what could be wrong? i am kind of lost now.. any tips
> will be appreciate
>
>
>
> --
> Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

tester100
Amos  

big thxs for all your input

it just shows me that i know nothing about squid that i am complete newbie,
and that i need to spend my time reading all the manual and config examples.

big thanks i will have some guidance on reading and research for the next
couple of days now.



--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

Amos Jeffries
Administrator
On 18/04/19 12:03 pm, tester100 wrote:
> Amos  
>
> big thxs for all your input
>
> it just shows me that i know nothing about squid that i am complete newbie,
> and that i need to spend my time reading all the manual and config examples.
>

I did not mean to imply a lot of reading was needed. Just some in
relation to the items I mentioned as probably leading to your issue. The
rest can be long-term goals to fix up.

FYI: The Squid wiki <http://wiki.squid-cache.org/> and config manual
<http://www.squid-cache.org/Doc/config/> (the v3.5 pages for your Squid
version) are the most accurate information sources behind reading the
code itself. But keep in mind that Squid-3 is also outdated nowdays,
Squid-4 and later have changed some significant feature behaviours.


Most of the things I pointed out were useful at some point (eg Squid-2),
and may still be for some use-cases. But for which Squid behaviour has
changed since how-tos and tutorials advising them were written.


> big thanks i will have some guidance on reading and research for the next
> couple of days now.
>

You are welcome. Any further questions or advice wanted please feel free
to ask. Helping each other use Squid is a what this mailing list is
about - for experts and newbies alike.

Cheers
Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

Eliezer Croitoru-3
Just to add:

Facebook has these headers for many of their videos:

 1.
    Cache-Control:
    max-age=1209600, no-transform


So what happens is that the client browser will save these URLs for a
very long time and it's good.

It takes of burden from the intermediate proxy.

I wrote some code that works for most of the facebook public videos at:

http://gogs.ngtech.co.il/NgTech-LTD/storeid-helpers/raw/master/facebook--video-2019.rb


Hope it helps.

Eliezer


On 4/18/2019 1:45 PM, Amos Jeffries wrote:

> On 18/04/19 12:03 pm, tester100 wrote:
>> Amos
>>
>> big thxs for all your input
>>
>> it just shows me that i know nothing about squid that i am complete newbie,
>> and that i need to spend my time reading all the manual and config examples.
>>
> I did not mean to imply a lot of reading was needed. Just some in
> relation to the items I mentioned as probably leading to your issue. The
> rest can be long-term goals to fix up.
>
> FYI: The Squid wiki <http://wiki.squid-cache.org/> and config manual
> <http://www.squid-cache.org/Doc/config/> (the v3.5 pages for your Squid
> version) are the most accurate information sources behind reading the
> code itself. But keep in mind that Squid-3 is also outdated nowdays,
> Squid-4 and later have changed some significant feature behaviours.
>
>
> Most of the things I pointed out were useful at some point (eg Squid-2),
> and may still be for some use-cases. But for which Squid behaviour has
> changed since how-tos and tutorials advising them were written.
>
>
>> big thanks i will have some guidance on reading and research for the next
>> couple of days now.
>>
> You are welcome. Any further questions or advice wanted please feel free
> to ask. Helping each other use Squid is a what this mailing list is
> about - for experts and newbies alike.
>
> Cheers
> Amos
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--

----

Eliezer Croitoru <http://ngtech.co.il/main-en/>
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email] <mailto:[hidden email]>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

Amos Jeffries
Administrator
On 19/04/19 9:17 am, Eliezer Croitoru wrote:

> Just to add:
>
> Facebook has these headers for many of their videos:
>
> 1.
>    Cache-Control:
>    max-age=1209600, no-transform
>
>
> So what happens is that the client browser will save these URLs for a
> very long time and it's good.


As will Squid unless the admin has configured refresh_pattern options
that force expiry earlier.

Amos
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|

Re: Squid 3.5 https facebook caching

ishtiak
Please guide me how  to setup caching Facebook
I will pay for the services  




--
Sent from: http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users