ah, OK - thanks for the info and the outlook ;)
ideally, my rough idea/wish would be to have a local site variable
pointing to a cache
MYCACHE="httpscache.foo.baz"
and ask users to use
${MYCACHE}/http(s)://actual.url
when fanning out jobs, i.e., as an 'on-demand cache' additional to all
the automatically cached/staged things (I don't know if I would like to
send all a node's HTTP(S) traffic through a cache, which would probably
easier to setup/control than relying on users using the 'right' URL...)
But tbh I don't know, if there is some demand for something like that
but I would worry that at some point a hub or so would crumble under
user tasks with thousands of jobs pulling all the same containers (user
container jobs just as an example - if they actually take off)
Cheers and thanks,
Thomas
On 29/11/2018 19.40, Yang, Wei wrote:
> Hi Thomas,
>
> Xcache can talk to client (docker on your laptop) via HTTP(s) if correctly configured. But Xcache currently has difficulty talking to HTTP(s) data sources. We have a prototype for Xcache to talk to the HTTP data source but it is still a prototype.
>
> On the other hand, we may be accomplish your request by another type of Xrootd cache, the FRM cache. I say _maybe_ because I am not 100% sure how to handle the second https:// in your url. Will get it back to you soon.
>
> regards,
> --
> Wei Yang | [log in to unmask] | 650-926-3338 (O)
>
>
> On 11/29/18, 6:53 AM, "[log in to unmask] on behalf of Thomas Hartmann" <[log in to unmask] on behalf of [log in to unmask]> wrote:
>
> Hi all,
>
> maybe a naive question: but could I setup/use an XCache to cache container layers?
>
> let's say, I have users who pull regularly containers on a number of batch modes
> ... pull docker://SOMEHUB/foo/thingy
> --> i. e., gets over https
>
> I would like to avoid to fiddle around with (reverse) proxies and faking certificates on the way. So instead I am looking for something like magically caching secure connections.
> E.g., putting explicitly an intermediate XCache in the way
> ... pull http://MyXcache.baz/https://SOMEHUB/foo/thingy
> and magically resolve & cache all requests for the layer files.
>
> My hope would be to move a bit of the load on hubs or CIs to a local cache.
>
> Cheers,
> Thomas
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
