Quantcast

URL list from a URL

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

URL list from a URL

Jason B. Nance
Hello,

I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates

Which returns a list of URLs, such as:

http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
http://centos.host-engine.com/7.3.1611/updates/x86_64/
http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/

Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.

So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).

I started to go down the external_acl_type but am wondering if I'm missing something obvious.

Regards,

j
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Yuri Voinov
Yes.

Functionality you required is:

http://wiki.squid-cache.org/Features/StoreID


21.03.2017 21:52, Jason B. Nance пишет:

> Hello,
>
> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>
> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>
> Which returns a list of URLs, such as:
>
> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
> http://centos.host-engine.com/7.3.1611/updates/x86_64/
> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>
> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>
> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>
> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>
> Regards,
>
> j
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Jason B. Nance
Hi Yuri,

I should have mentioned that I'm not caching, I'm only using Squid for whitelisting in this case.  Would you still say this is the right path?  It seems that there is a fair amount of hard coding in this method at least based on:




----- Original Message -----
From: "Yuri Voinov" <[hidden email]>
To: [hidden email]
Sent: Tuesday, March 21, 2017 1:19:43 PM
Subject: Re: [squid-users] URL list from a URL

Yes.

Functionality you required is:

http://wiki.squid-cache.org/Features/StoreID


21.03.2017 21:52, Jason B. Nance пишет:

> Hello,
>
> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>
> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>
> Which returns a list of URLs, such as:
>
> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
> http://centos.host-engine.com/7.3.1611/updates/x86_64/
> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>
> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>
> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>
> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>
> Regards,
>
> j
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Jason B. Nance
I'm sorry, this message was sent prematurely. :-\

Completed message follows.


Hi Yuri,

I should have mentioned that I'm not caching, I'm only using Squid for whitelisting in this case.  Would you still say this is the right path?  It seems that there is a fair amount of hard coding in this method at least based on:

http://wiki.squid-cache.org/Features/StoreID/DB

I guess a URL regex could also work given that all the URIs are similar.

Regards,

j


----- Original Message -----
From: "Yuri Voinov" <[hidden email]>
To: [hidden email]
Sent: Tuesday, March 21, 2017 1:19:43 PM
Subject: Re: [squid-users] URL list from a URL

Yes.

Functionality you required is:

http://wiki.squid-cache.org/Features/StoreID


21.03.2017 21:52, Jason B. Nance пишет:

> Hello,
>
> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>
> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>
> Which returns a list of URLs, such as:
>
> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
> http://centos.host-engine.com/7.3.1611/updates/x86_64/
> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>
> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>
> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>
> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>
> Regards,
>
> j
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users

--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Yuri Voinov


22.03.2017 2:32, Jason B. Nance пишет:

> I'm sorry, this message was sent prematurely. :-\
>
> Completed message follows.
>
>
> Hi Yuri,
>
> I should have mentioned that I'm not caching, I'm only using Squid for whitelisting in this case.  Would you still say this is the right path?  It seems that there is a fair amount of hard coding in this method at least based on:
>
> http://wiki.squid-cache.org/Features/StoreID/DB
>
> I guess a URL regex could also work given that all the URIs are similar.
Mmmmmmmm. May be. You can write common regex for all mirrors, yes.

> Regards,
>
> j
>
>
> ----- Original Message -----
> From: "Yuri Voinov" <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, March 21, 2017 1:19:43 PM
> Subject: Re: [squid-users] URL list from a URL
>
> Yes.
>
> Functionality you required is:
>
> http://wiki.squid-cache.org/Features/StoreID
>
>
> 21.03.2017 21:52, Jason B. Nance пишет:
>> Hello,
>>
>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>
>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>
>> Which returns a list of URLs, such as:
>>
>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>
>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>
>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>
>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>>
>> Regards,
>>
>> j
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
--
Bugs to the Future

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

0x613DEC46.asc (2K) Download Attachment
signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Alex Rousskov
In reply to this post by Jason B. Nance
On 03/21/2017 02:30 PM, Jason B. Nance wrote:

> I should have mentioned that I'm not caching, I'm only using Squid
> for whitelisting in this case.  Would you still say this is the right
> path?

No. You probably have two better options:

1. Use a file with list of mirror URLs as an ACL parameter. Write a
script that updates that file and reconfigures Squid as needed. Please
keep in mind that Squid reconfiguration is currently a relatively
heavy/intrusive operation, even if there were not changes except for
that single ACL.

2. Write an external_acl helper that will consult the mirror list. This
will make each HTTP transaction a little slower (because it needs to go
to the helper) but eliminates reconfigurations. The helper itself or
some other script will still need to update the mirror list as needed,
of course.


HTH,

Alex.




>> Hello,
>>
>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>
>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>
>> Which returns a list of URLs, such as:
>>
>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>
>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>
>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>
>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
The current StoreID helper can be converted pretty fast into what he needs.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]


-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
Sent: Tuesday, March 21, 2017 11:43 PM
To: [hidden email]
Subject: Re: [squid-users] URL list from a URL

On 03/21/2017 02:30 PM, Jason B. Nance wrote:

> I should have mentioned that I'm not caching, I'm only using Squid
> for whitelisting in this case.  Would you still say this is the right
> path?

No. You probably have two better options:

1. Use a file with list of mirror URLs as an ACL parameter. Write a
script that updates that file and reconfigures Squid as needed. Please
keep in mind that Squid reconfiguration is currently a relatively
heavy/intrusive operation, even if there were not changes except for
that single ACL.

2. Write an external_acl helper that will consult the mirror list. This
will make each HTTP transaction a little slower (because it needs to go
to the helper) but eliminates reconfigurations. The helper itself or
some other script will still need to update the mirror list as needed,
of course.


HTH,

Alex.




>> Hello,
>>
>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>
>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>
>> Which returns a list of URLs, such as:
>>
>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>
>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>
>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>
>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Alex Rousskov
On 03/21/2017 06:17 PM, Eliezer Croitoru wrote:
> The current StoreID helper can be converted pretty fast into what he needs.

Jason needs to block access. How can a [converted] StoreID helper block
access without becoming an external_acl helper?

Alex.

> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
> Sent: Tuesday, March 21, 2017 11:43 PM
> To: [hidden email]
> Subject: Re: [squid-users] URL list from a URL
>
> On 03/21/2017 02:30 PM, Jason B. Nance wrote:
>
>> I should have mentioned that I'm not caching, I'm only using Squid
>> for whitelisting in this case.  Would you still say this is the right
>> path?
>
> No. You probably have two better options:
>
> 1. Use a file with list of mirror URLs as an ACL parameter. Write a
> script that updates that file and reconfigures Squid as needed. Please
> keep in mind that Squid reconfiguration is currently a relatively
> heavy/intrusive operation, even if there were not changes except for
> that single ACL.
>
> 2. Write an external_acl helper that will consult the mirror list. This
> will make each HTTP transaction a little slower (because it needs to go
> to the helper) but eliminates reconfigurations. The helper itself or
> some other script will still need to update the mirror list as needed,
> of course.
>
>
> HTH,
>
> Alex.
>
>
>
>
>>> Hello,
>>>
>>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>>
>>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>>
>>> Which returns a list of URLs, such as:
>>>
>>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>>
>>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>>
>>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>>
>>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Jason B. Nance
In reply to this post by Alex Rousskov
Thank you.  In regards to the external ACL helper, would I basically say something like:

external_acl_type foo %URL /path/to/my/helper

And then have my helper compare the contents of %URL to the mirror list and spit out "OK" for pass or "ERR" for fail?

Thanks,

j


----- Original Message -----
From: "Alex Rousskov" <[hidden email]>
To: [hidden email]
Cc: "Jason Nance" <[hidden email]>
Sent: Tuesday, March 21, 2017 4:42:33 PM
Subject: Re: [squid-users] URL list from a URL

On 03/21/2017 02:30 PM, Jason B. Nance wrote:

> I should have mentioned that I'm not caching, I'm only using Squid
> for whitelisting in this case.  Would you still say this is the right
> path?

No. You probably have two better options:

1. Use a file with list of mirror URLs as an ACL parameter. Write a
script that updates that file and reconfigures Squid as needed. Please
keep in mind that Squid reconfiguration is currently a relatively
heavy/intrusive operation, even if there were not changes except for
that single ACL.

2. Write an external_acl helper that will consult the mirror list. This
will make each HTTP transaction a little slower (because it needs to go
to the helper) but eliminates reconfigurations. The helper itself or
some other script will still need to update the mirror list as needed,
of course.


HTH,

Alex.




>> Hello,
>>
>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>
>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>
>> Which returns a list of URLs, such as:
>>
>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>
>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>
>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>
>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
In reply to this post by Alex Rousskov
Hey Alex,

I didn't meant by convert the StoreID helper to convert it into an external_acl helper....
It has both OK and ERR and a "checklist" which would be a match or not.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: Alex Rousskov [mailto:[hidden email]]
Sent: Wednesday, March 22, 2017 3:51 PM
To: [hidden email]
Cc: Eliezer Croitoru <[hidden email]>
Subject: Re: [squid-users] URL list from a URL

On 03/21/2017 06:17 PM, Eliezer Croitoru wrote:
> The current StoreID helper can be converted pretty fast into what he needs.

Jason needs to block access. How can a [converted] StoreID helper block
access without becoming an external_acl helper?

Alex.

> -----Original Message-----
> From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
> Sent: Tuesday, March 21, 2017 11:43 PM
> To: [hidden email]
> Subject: Re: [squid-users] URL list from a URL
>
> On 03/21/2017 02:30 PM, Jason B. Nance wrote:
>
>> I should have mentioned that I'm not caching, I'm only using Squid
>> for whitelisting in this case.  Would you still say this is the right
>> path?
>
> No. You probably have two better options:
>
> 1. Use a file with list of mirror URLs as an ACL parameter. Write a
> script that updates that file and reconfigures Squid as needed. Please
> keep in mind that Squid reconfiguration is currently a relatively
> heavy/intrusive operation, even if there were not changes except for
> that single ACL.
>
> 2. Write an external_acl helper that will consult the mirror list. This
> will make each HTTP transaction a little slower (because it needs to go
> to the helper) but eliminates reconfigurations. The helper itself or
> some other script will still need to update the mirror list as needed,
> of course.
>
>
> HTH,
>
> Alex.
>
>
>
>
>>> Hello,
>>>
>>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>>
>>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>>
>>> Which returns a list of URLs, such as:
>>>
>>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>>
>>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>>
>>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>>
>>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>
> _______________________________________________
> squid-users mailing list
> [hidden email]
> http://lists.squid-cache.org/listinfo/squid-users
>


_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Alex Rousskov
On 03/22/2017 09:25 AM, Eliezer Croitoru wrote:

> I didn't meant by convert the StoreID helper to convert it into an external_acl helper....
> It has both OK and ERR and a "checklist" which would be a match or not.

Sigh. The combination of your answers does not make any sense to me.
Squid does not use StoreID helpers to block access, regardless of what a
StoreID helper returns. It is certainly possible to take some StoreID
helper code and make an external_acl helper out of it, but that falls
under my option #2.

Perhaps what you meant to say is something like "Use StoreID helper X
available at Y to implement option #2 -- that X code has everything you
need!"?

Alex.


> -----Original Message-----
> From: Alex Rousskov [mailto:[hidden email]]
> Sent: Wednesday, March 22, 2017 3:51 PM
> To: [hidden email]
> Cc: Eliezer Croitoru <[hidden email]>
> Subject: Re: [squid-users] URL list from a URL
>
> On 03/21/2017 06:17 PM, Eliezer Croitoru wrote:
>> The current StoreID helper can be converted pretty fast into what he needs.
>
> Jason needs to block access. How can a [converted] StoreID helper block
> access without becoming an external_acl helper?
>
> Alex.
>
>> -----Original Message-----
>> From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
>> Sent: Tuesday, March 21, 2017 11:43 PM
>> To: [hidden email]
>> Subject: Re: [squid-users] URL list from a URL
>>
>> On 03/21/2017 02:30 PM, Jason B. Nance wrote:
>>
>>> I should have mentioned that I'm not caching, I'm only using Squid
>>> for whitelisting in this case.  Would you still say this is the right
>>> path?
>>
>> No. You probably have two better options:
>>
>> 1. Use a file with list of mirror URLs as an ACL parameter. Write a
>> script that updates that file and reconfigures Squid as needed. Please
>> keep in mind that Squid reconfiguration is currently a relatively
>> heavy/intrusive operation, even if there were not changes except for
>> that single ACL.
>>
>> 2. Write an external_acl helper that will consult the mirror list. This
>> will make each HTTP transaction a little slower (because it needs to go
>> to the helper) but eliminates reconfigurations. The helper itself or
>> some other script will still need to update the mirror list as needed,
>> of course.
>>
>>
>> HTH,
>>
>> Alex.
>>
>>
>>
>>
>>>> Hello,
>>>>
>>>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>>>
>>>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>>>
>>>> Which returns a list of URLs, such as:
>>>>
>>>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>>>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>>>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>>>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>>>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>>>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>>>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>>>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>>>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>>>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>>>
>>>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>>>
>>>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>>>
>>>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
Almost, What I was talking about and meant was to take the current official StoreID helper written in perl and remove the parts of the ID replacement while leaving the regex matching intact.
Then pass to the StoreID helper the url and the source IP and let it match the url to the regex from the list in the "DB file".
If it matches either return OK or ERR depends on the intention of the helper.
...The X code have almost everything needed so just convert it.
I am offering to convert it if there is some interest in it.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Alex Rousskov
Sent: Wednesday, March 22, 2017 6:10 PM
To: [hidden email]
Subject: Re: [squid-users] URL list from a URL

On 03/22/2017 09:25 AM, Eliezer Croitoru wrote:

> I didn't meant by convert the StoreID helper to convert it into an external_acl helper....
> It has both OK and ERR and a "checklist" which would be a match or not.

Sigh. The combination of your answers does not make any sense to me.
Squid does not use StoreID helpers to block access, regardless of what a StoreID helper returns. It is certainly possible to take some StoreID helper code and make an external_acl helper out of it, but that falls under my option #2.

Perhaps what you meant to say is something like "Use StoreID helper X available at Y to implement option #2 -- that X code has everything you need!"?

Alex.


> -----Original Message-----
> From: Alex Rousskov [mailto:[hidden email]]
> Sent: Wednesday, March 22, 2017 3:51 PM
> To: [hidden email]
> Cc: Eliezer Croitoru <[hidden email]>
> Subject: Re: [squid-users] URL list from a URL
>
> On 03/21/2017 06:17 PM, Eliezer Croitoru wrote:
>> The current StoreID helper can be converted pretty fast into what he needs.
>
> Jason needs to block access. How can a [converted] StoreID helper
> block access without becoming an external_acl helper?
>
> Alex.
>
>> -----Original Message-----
>> From: squid-users [mailto:[hidden email]]
>> On Behalf Of Alex Rousskov
>> Sent: Tuesday, March 21, 2017 11:43 PM
>> To: [hidden email]
>> Subject: Re: [squid-users] URL list from a URL
>>
>> On 03/21/2017 02:30 PM, Jason B. Nance wrote:
>>
>>> I should have mentioned that I'm not caching, I'm only using Squid
>>> for whitelisting in this case.  Would you still say this is the
>>> right path?
>>
>> No. You probably have two better options:
>>
>> 1. Use a file with list of mirror URLs as an ACL parameter. Write a
>> script that updates that file and reconfigures Squid as needed.
>> Please keep in mind that Squid reconfiguration is currently a
>> relatively heavy/intrusive operation, even if there were not changes
>> except for that single ACL.
>>
>> 2. Write an external_acl helper that will consult the mirror list.
>> This will make each HTTP transaction a little slower (because it
>> needs to go to the helper) but eliminates reconfigurations. The
>> helper itself or some other script will still need to update the
>> mirror list as needed, of course.
>>
>>
>> HTH,
>>
>> Alex.
>>
>>
>>
>>
>>>> Hello,
>>>>
>>>> I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:
>>>>
>>>> http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates
>>>>
>>>> Which returns a list of URLs, such as:
>>>>
>>>> http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
>>>> http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
>>>> http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
>>>> http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
>>>> http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
>>>> http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
>>>> http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
>>>> http://centos.host-engine.com/7.3.1611/updates/x86_64/
>>>> http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
>>>> http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/
>>>>
>>>> Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.
>>>>
>>>> So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).
>>>>
>>>> I started to go down the external_acl_type but am wondering if I'm missing something obvious.
>>
>> _______________________________________________
>> squid-users mailing list
>> [hidden email]
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
In reply to this post by Jason B. Nance
My suggestion is to run every minute a curl request into a tmp file and then compare to the existing one, then sort and uniq and replace the existing DB
In the external acl helper script use some kind of version testing by the last date updated.
You can use a simple comparison to do that.
So two scripts:
1 - update by a crontab
2 - external_acl helper

Will update later.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Jason B. Nance
Sent: Tuesday, March 21, 2017 5:53 PM
To: [hidden email]
Subject: [squid-users] URL list from a URL

Hello,

I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates

Which returns a list of URLs, such as:

http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
http://centos.host-engine.com/7.3.1611/updates/x86_64/
http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/

Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.

So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).

I started to go down the external_acl_type but am wondering if I'm missing something obvious.

Regards,

j
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
In reply to this post by Jason B. Nance
OK so I wrote these helpers that can help to allow the updates despite the changes in urls on gist:
https://gist.github.com/elico/dc1af72344231d6d49af4eacecfae8df

The squid.conf should be something like:
external_acl_type centos_mirror_check ipv4 concurrency=200 ttl=15 %URI %SRC %METHOD /opt/bin/centos-extacl.rb /etc/squid/centosrepos.txt
acl centos_mirror_check_acl external centos_mirror_check
htttp_access allow !CONNECT centos_mirror_check_acl


Let me know if this is not enough or there is a need for some tweaks.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Jason B. Nance
Sent: Tuesday, March 21, 2017 5:53 PM
To: [hidden email]
Subject: [squid-users] URL list from a URL

Hello,

I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates

Which returns a list of URLs, such as:

http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
http://centos.host-engine.com/7.3.1611/updates/x86_64/
http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/

Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.

So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).

I started to go down the external_acl_type but am wondering if I'm missing something obvious.

Regards,

j
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
And if you want the full list of mirrors being updated instead of using curl you can use the next script to parse the full mirror list page:
http://mirror-status.centos.org/

Using the script:
https://gist.github.com/elico/dc1af72344231d6d49af4eacecfae8df#file-update-mirrors-list-sh

Let me know if it fills the gaps.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Eliezer Croitoru
Sent: Friday, March 24, 2017 4:33 PM
To: 'Jason B. Nance' <[hidden email]>; [hidden email]
Subject: Re: [squid-users] URL list from a URL

OK so I wrote these helpers that can help to allow the updates despite the changes in urls on gist:
https://gist.github.com/elico/dc1af72344231d6d49af4eacecfae8df

The squid.conf should be something like:
external_acl_type centos_mirror_check ipv4 concurrency=200 ttl=15 %URI %SRC %METHOD /opt/bin/centos-extacl.rb /etc/squid/centosrepos.txt
acl centos_mirror_check_acl external centos_mirror_check
htttp_access allow !CONNECT centos_mirror_check_acl


Let me know if this is not enough or there is a need for some tweaks.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Jason B. Nance
Sent: Tuesday, March 21, 2017 5:53 PM
To: [hidden email]
Subject: [squid-users] URL list from a URL

Hello,

I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates

Which returns a list of URLs, such as:

http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
http://centos.host-engine.com/7.3.1611/updates/x86_64/
http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/

Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.

So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).

I started to go down the external_acl_type but am wondering if I'm missing something obvious.

Regards,

j
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: URL list from a URL

Eliezer Croitoru
Sorry a type with the link so:
And if you want the full list of mirrors being updated instead of using curl you can use the next script to parse the full mirror list page:
http://mirror-status.centos.org/

Using the script:
https://gist.github.com/elico/dc1af72344231d6d49af4eacecfae8df#file-get-full-mirrolist-rb

Eliezer


----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Eliezer Croitoru
Sent: Friday, March 24, 2017 5:59 PM
To: 'Jason B. Nance' <[hidden email]>; [hidden email]
Subject: Re: [squid-users] URL list from a URL

And if you want the full list of mirrors being updated instead of using curl you can use the next script to parse the full mirror list page:
http://mirror-status.centos.org/

Using the script:
https://gist.github.com/elico/dc1af72344231d6d49af4eacecfae8df#file-update-mirrors-list-sh

Let me know if it fills the gaps.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Eliezer Croitoru
Sent: Friday, March 24, 2017 4:33 PM
To: 'Jason B. Nance' <[hidden email]>; [hidden email]
Subject: Re: [squid-users] URL list from a URL

OK so I wrote these helpers that can help to allow the updates despite the changes in urls on gist:
https://gist.github.com/elico/dc1af72344231d6d49af4eacecfae8df

The squid.conf should be something like:
external_acl_type centos_mirror_check ipv4 concurrency=200 ttl=15 %URI %SRC %METHOD /opt/bin/centos-extacl.rb /etc/squid/centosrepos.txt acl centos_mirror_check_acl external centos_mirror_check htttp_access allow !CONNECT centos_mirror_check_acl


Let me know if this is not enough or there is a need for some tweaks.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: [hidden email]



-----Original Message-----
From: squid-users [mailto:[hidden email]] On Behalf Of Jason B. Nance
Sent: Tuesday, March 21, 2017 5:53 PM
To: [hidden email]
Subject: [squid-users] URL list from a URL

Hello,

I'm using Squid 3.5.20 and wonder if it is possible to define an ACL which retrieves the list of URLs from another URL (similar to pointing to a file).  In this specific use case it is to allow a Foreman server to sync Yum content from the CentOS mirrors.  I tell Foreman to use the following URL:

http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=updates

Which returns a list of URLs, such as:

http://repo1.dal.innoscale.net/centos/7.3.1611/updates/x86_64/
http://linux.mirrors.es.net/centos/7.3.1611/updates/x86_64/
http://reflector.westga.edu/repos/CentOS/7.3.1611/updates/x86_64/
http://mirror.jax.hugeserver.com/centos/7.3.1611/updates/x86_64/
http://ftp.linux.ncsu.edu/pub/CentOS/7.3.1611/updates/x86_64/
http://mirror.nexcess.net/CentOS/7.3.1611/updates/x86_64/
http://mirror.web-ster.com/centos/7.3.1611/updates/x86_64/
http://centos.host-engine.com/7.3.1611/updates/x86_64/
http://mirror.raystedman.net/centos/7.3.1611/updates/x86_64/
http://mirror.linux.duke.edu/pub/centos/7.3.1611/updates/x86_64/

Foreman then starts a new HTTP connection (not a redirect) to attempt to connect to those in turn until it works.

So I would like to configure Squid to allow the Foreman server access to any of those URLs (the list changes somewhat often).

I started to go down the external_acl_type but am wondering if I'm missing something obvious.

Regards,

j
_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
[hidden email]
http://lists.squid-cache.org/listinfo/squid-users
Loading...