XROOTD-L Archives

Support use of xrootd by HEP experiments

XROOTD-L@LISTSERV.SLAC.STANFORD.EDU

Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Thomas Hartmann <[log in to unmask]>
Reply To:
Support use of xrootd by HEP experiments <[log in to unmask]>
Date:
Fri, 30 Nov 2018 10:23:59 +0100
Content-Type:
text/plain
Parts/Attachments:
text/plain (67 lines)
ah, OK - thanks for the info and the outlook ;)


ideally, my rough idea/wish would be to have a local site variable
pointing to a cache
  MYCACHE="httpscache.foo.baz"
and ask users to use
  ${MYCACHE}/http(s)://actual.url
when fanning out jobs, i.e., as an 'on-demand cache' additional to all
the automatically cached/staged things (I don't know if I would like to
send all a node's HTTP(S) traffic through a cache, which would probably
easier to setup/control than relying on users using the 'right' URL...)

But tbh I don't know, if there is some demand for something like that
but I would worry that at some point a hub or so would crumble under
user tasks with thousands of jobs pulling all the same containers (user
container jobs just as an example - if they actually take off)

Cheers and thanks,
  Thomas

On 29/11/2018 19.40, Yang, Wei wrote:
> Hi Thomas,
> 
> Xcache can talk to client (docker on your laptop) via HTTP(s) if correctly configured. But Xcache currently has difficulty talking to HTTP(s) data sources. We have a prototype for Xcache to talk to the HTTP data source but it is still a prototype.
> 
> On the other hand, we may be accomplish your request by another type of Xrootd cache, the FRM cache. I say _maybe_ because I am not 100% sure how to handle the second https:// in your url. Will get it back to you soon.
> 
> regards,
> --
> Wei Yang  |  [log in to unmask]  |  650-926-3338 (O)
>  
> 
> ´╗┐On 11/29/18, 6:53 AM, "[log in to unmask] on behalf of Thomas Hartmann" <[log in to unmask] on behalf of [log in to unmask]> wrote:
> 
>     Hi all,
>     
>     maybe a naive question: but could I setup/use an XCache to cache container layers?
>     
>     let's say, I have users who pull regularly containers on a number of batch modes
>         ... pull docker://SOMEHUB/foo/thingy
>     --> i. e., gets over https
>     
>     I would like to avoid to fiddle around with (reverse) proxies and faking certificates on the way. So instead I am looking for something like magically caching secure connections. 
>     E.g., putting explicitly an intermediate XCache in the way
>        ... pull http://MyXcache.baz/https://SOMEHUB/foo/thingy
>     and magically resolve & cache all requests for the layer files. 
>     
>     My hope would be to move a bit of the load on hubs or CIs to a local cache. 
>     
>     Cheers,
>       Thomas 
>     
>     ########################################################################
>     Use REPLY-ALL to reply to list
>     
>     To unsubscribe from the XROOTD-L list, click the following link:
>     https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>     
> 

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

ATOM RSS1 RSS2