Hi Thomas,
As I am still awake, we do have two possible solutions for you. The first
is using Xcache (the simpler solution) which may or may not work,
depending on how dockerhub works visavi get requests. The Xcache approach
is easy to try and will tell is pretty quickly if it's workable. If not,
the FRM solution is the next step but we need to find out if that can
support multiple sources. So, bear with us!
Andy
P.S. We are totally supportive and if we need to extend one or the other
solutions to work with dockerhub, we can do it -- though that will take
time (hopefully within your timeframe).
On Fri, 30 Nov 2018, Thomas Hartmann wrote:
> ah, OK - thanks for the info and the outlook ;)
>
>
> ideally, my rough idea/wish would be to have a local site variable
> pointing to a cache
> MYCACHE="httpscache.foo.baz"
> and ask users to use
> ${MYCACHE}/http(s)://actual.url
> when fanning out jobs, i.e., as an 'on-demand cache' additional to all
> the automatically cached/staged things (I don't know if I would like to
> send all a node's HTTP(S) traffic through a cache, which would probably
> easier to setup/control than relying on users using the 'right' URL...)
>
> But tbh I don't know, if there is some demand for something like that
> but I would worry that at some point a hub or so would crumble under
> user tasks with thousands of jobs pulling all the same containers (user
> container jobs just as an example - if they actually take off)
>
> Cheers and thanks,
> Thomas
>
> On 29/11/2018 19.40, Yang, Wei wrote:
>> Hi Thomas,
>>
>> Xcache can talk to client (docker on your laptop) via HTTP(s) if correctly configured. But Xcache currently has difficulty talking to HTTP(s) data sources. We have a prototype for Xcache to talk to the HTTP data source but it is still a prototype.
>>
>> On the other hand, we may be accomplish your request by another type of Xrootd cache, the FRM cache. I say _maybe_ because I am not 100% sure how to handle the second https:// in your url. Will get it back to you soon.
>>
>> regards,
>> --
>> Wei Yang | [log in to unmask] | 650-926-3338 (O)
>>
>>
>> ˙˙On 11/29/18, 6:53 AM, "[log in to unmask] on behalf of Thomas Hartmann" <[log in to unmask] on behalf of [log in to unmask]> wrote:
>>
>> Hi all,
>>
>> maybe a naive question: but could I setup/use an XCache to cache container layers?
>>
>> let's say, I have users who pull regularly containers on a number of batch modes
>> ... pull docker://SOMEHUB/foo/thingy
>> --> i. e., gets over https
>>
>> I would like to avoid to fiddle around with (reverse) proxies and faking certificates on the way. So instead I am looking for something like magically caching secure connections.
>> E.g., putting explicitly an intermediate XCache in the way
>> ... pull http://MyXcache.baz/https://SOMEHUB/foo/thingy
>> and magically resolve & cache all requests for the layer files.
>>
>> My hope would be to move a bit of the load on hubs or CIs to a local cache.
>>
>> Cheers,
>> Thomas
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
