Hi Brian,
The whole protocol is fully documented on the XRootD web site (I am
suprised Albert didn't mention it as he had a hand in writing it).
https://xrootd.slac.stanford.edu/doc/dev49/tpc_protocol.htm
A pdf version is also available.
It is technically impossible to use a TPC rendezvous token as anything
other than an opaque key. It is generated as a random value so as to be
relatively unique and, as the name implies, passed around to achieve a
rendezvous between the source and destination on a particular file. The
TPC rendezvous processing actually displaces authorization since the
rendezvous is limited in scope (i.e. access to a single specific file) and
we are able to identify the parties in that rendezvous once it occurs.
The rendezvous does not supplant authentication. But having the
destination using the client's ztn token when logging into the source is
not at all horrible. In the way rendezvous TPC work the ztn token will
never be used for anything other than validation (which is all that
ztn cares about anyway).
As far as I am concerned the problem is really solved once you realize
what all the pieces are actually needed and used or not used at all but
are there to merely satisfy technical requirements.
Andy
On Fri, 1 Apr 2022, Bockelman, Brian wrote:
> Yes -- the client needs a valid token to talk with both source and destination.
>
> Isn't the protocol something like:
>
> client -> source:
> login
> auth > token1 presented here
> open > rendezvous key presented here
>
> client -> destination:
> login
> auth > token2 presented here
> open > rendezvous key presented here
>
> destination -> source:
> login
> auth > here, I'm proposing to present the rendezvous key that the destination has and source already recognizes.
> open > rendezvous key
>
> (Again, any high-level writeup of the XRootD TPC sequence would be really helpful... is there a CHEP paper or something?)
>
> Brian
>
> On Apr 1, 2022, at 8:37 AM, Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>> wrote:
>
> Well I don't know how this is going to work, though.
>
>
> Here's the sequence of communications:
>
> protocol
> login
> auth > ZTN token presented here
> open > Rendezvous key present here (like the JWT token)
>
> I don't think the tpc.key is generated before open.
>
> ________________________________________________
> Albert L. Rossi
> Senior Software Developer
> Scientific Computing Division, Scientific Data Services, Distributed Data Development
> FCC 229A
> Mail Station 369 (FCC 2W)
> Fermi National Accelerator Laboratory
> Batavia, IL 60510
> (630) 840-3023
>
> ________________________________
> From: Bockelman, Brian <[log in to unmask]<mailto:[log in to unmask]>>
> Sent: Friday, April 1, 2022 8:33 AM
> To: Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>>
> Cc: Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>>; xrootd-dev <[log in to unmask]<mailto:[log in to unmask]>>
> Subject: Re: ZTN and TPC
>
> Ah - that's fine too.
>
> The point is that if the source server needs to present a recognized token to do the ZTN authentication, the rendezvous key appears to be a perfectly fine one to use. Seems that it's quite a jump in logic to assume the destination server's ZTN session token is recognized by the source (in fact, I would strongly discourage that approach -- I *really* like the fact the only commonality needed is the client and that you don't need the two servers to have compatible authorization schemes).
>
> Brian
>
> On Apr 1, 2022, at 8:26 AM, Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>> wrote:
>
> Actually, if I'm not mistaken, it is the client that generates the rendezvous key and hands it off to both source and destination.
>
> Al
>
> ________________________________________________
> Albert L. Rossi
> Senior Software Developer
> Scientific Computing Division, Scientific Data Services, Distributed Data Development
> FCC 229A
> Mail Station 369 (FCC 2W)
> Fermi National Accelerator Laboratory
> Batavia, IL 60510
> (630) 840-3023
>
> ________________________________
> From: Bockelman, Brian <[log in to unmask]<mailto:[log in to unmask]>>
> Sent: Friday, April 1, 2022 8:22 AM
> To: Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>>
> Cc: Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>>; xrootd-dev <[log in to unmask]<mailto:[log in to unmask]>>
> Subject: Re: ZTN and TPC
>
> Hi all,
>
> Still not sure I'm following all the conversation here because I don't know exactly how the XRootD TPC protocol works.
>
> *That said*, I think the closest equivalent to what's done for HTTP is to utilize the rendezvous token as a valid token for ZTN. Who says that the ZTN authentication mechanism is limited to a single token type?
>
> My understanding is that the source server is the one that generates the rendezvous token -- hence, shouldn't it be a token the source server recognizes?
>
> The problem with using the session token given to the destination server is that it might only be valid at the destination server.
>
> Brian
>
>> On Mar 29, 2022, at 3:34 PM, Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>> wrote:
>>
>> Hi Albert,
>>
>> On Tue, 29 Mar 2022, Albert Rossi wrote:
>>
>>> So essentially, even though we (both) can do TPC using the rendezvous key strategy, once you require ZTN all bets are off on TPC.
>> Almost. I can see a glimmer of hope here but only because of the way TPC is implemented in xrootd. If the incomming client on the destination server has used ZTN then it would be very easy to promote that credential to the process that is doing the actual third party transfer. This is what we already do for delegated x509. Then if the source server asks for a token we will simply hand over the token the client gave us. That should make the source server happy and allow us to present the rendezvous token. There are a lot of assumptions here so it's not a sure bet but clearly not zero either.
>>
>> Andy
>>
>>
>>> Thanks for confirming this.
>>>
>>> Cheers, Al
>>>
>>> ________________________________________________
>>> Albert L. Rossi
>>> Senior Software Developer
>>> Scientific Computing Division, Scientific Data Services, Distributed Data Development
>>> FCC 229A
>>> Mail Station 369 (FCC 2W)
>>> Fermi National Accelerator Laboratory
>>> Batavia, IL 60510
>>> (630) 840-3023
>>>
>>> ________________________________
>>> From: Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>>
>>> Sent: Tuesday, March 29, 2022 3:22 PM
>>> To: Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>>
>>> Cc: xrootd-dev <[log in to unmask]<mailto:[log in to unmask]>>
>>> Subject: Re: ZTN and TPC
>>>
>>>
>>>
>>> On Tue, 29 Mar 2022, Albert Rossi wrote:
>>>
>>>> When they do accept this strategy, will dCache have to turn on ZTN to allow it?
>>> If they switch that by default to omit specifying the the token on each
>>> path because they are relying on the default token, then yes you will need
>>> to turn on ZTN. I don't see any other possible outcome in this scenario.
>>> However, that would mean one of two possible outcomes: a) everyone has
>>> switched to using token based authorization, or b) normal clients have
>>> switched to token based authorization but servers and some special clients
>>> still use x509 or some other scheme.
>>>
>>>> If so, how does dCache make an exception with regard to third-party
>>>> clients, since I am assuming the third-party connection will not have a
>>>> ZTN available to it?
>>> The third party connection may very well have ZTN available to it, the
>>> problem is that it may not have a token (actually servers likely will not
>>> have tokens). Will the server still be able to authenticate using x509 or
>>> some other non-token based mechanism? If so, then that would be one
>>> possible avenue (as I indicated above). Otherwise, I don't see how this
>>> could be done with what we have now without eliminating all authentication
>>> or carving out exceptions on a server to server bases which would not
>>> scale.
>>>
>>>> How is this handled in xrootd?
>>> The answer is that it does not. The situation is identical to that of
>>> dCache. Now, recall that all of the token-based TPC is being done in the
>>> context of https which has no defined authentication to speak of. So,
>>> this problem never comes up in that context. So it's not suprising no one
>>> is considering the pain it is causing in session based protocols.
>>>
>>> Andy
>>>
>>>
>>>> That was, and still is, my principal question.
>>>>
>>>> Al
>>>> ________________________________________________
>>>> Albert L. Rossi
>>>> Senior Software Developer
>>>> Scientific Computing Division, Scientific Data Services, Distributed Data Development
>>>> FCC 229A
>>>> Mail Station 369 (FCC 2W)
>>>> Fermi National Accelerator Laboratory
>>>> Batavia, IL 60510
>>>> (630) 840-3023
>>>>
>>>> ________________________________
>>>> From: Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>>
>>>> Sent: Tuesday, March 29, 2022 2:59 PM
>>>> To: Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>>
>>>> Cc: xrootd-dev <[log in to unmask]<mailto:[log in to unmask]>>
>>>> Subject: Re: ZTN and TPC
>>>>
>>>> Hi Albert,
>>>>
>>>> You are not hallucinating. OSG has not yet made the change to the SciToken
>>>> library to use the ztn token in the absence of the token in the cgi. I
>>>> will talk to them on Thursday about that. So, everything you showed me is
>>>> consistent.
>>>>
>>>> Andy
>>>>
>>>>
>>>> On Tue, 29 Mar 2022, Albert Rossi wrote:
>>>>
>>>>> Me again.
>>>>>
>>>>> I just did an experiment with a simple read using a bearer token.
>>>>>
>>>>> built xrootd tip of master from yesterday:
>>>>>
>>>>> commit b5f279d616c49b65664ca06b98c52ec13fcc26fe (HEAD -> master, origin/master, origin/HEAD)
>>>>> Author: Michal Simon <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Date: Mon Mar 28 14:52:46 2022 +0200
>>>>>
>>>>> [XrdCl] xrdcp --server: report IP stack to stderr.
>>>>>
>>>>> * IPv4: "!-!IPv4"
>>>>> * IPv6: "!-!IPv6"
>>>>>
>>>>>
>>>>> Get the Cilogon/fermilab token:
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ htgettoken -a fermicloud543.fnal.gov<http://fermicloud543.fnal.gov/> -i fermilab
>>>>> Attempting to get token from https://fermicloud543.fnal.gov:8200<https://fermicloud543.fnal.gov:8200/> ... succeeded
>>>>> Storing bearer token in /run/user/8773/bt_u8773
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ httokendecode
>>>>> {
>>>>> "wlcg.ver": "1.0",
>>>>> "aud": "https://wlcg.cern.ch/jwt/v1/any",
>>>>> "sub": "[log in to unmask]<mailto:[log in to unmask]>",
>>>>> "nbf": 1648568096,
>>>>> "scope": "storage.create:/fermilab/users/arossi compute.create compute.read compute.cancel compute.modify storage.read:/fermilab/users/arossi",
>>>>> "iss": "https://urldefense.proofpoint.com/v2/url?u=https-3A__cilogon.org_fermilab&d=DwIDAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=p0VegQeZyTJZXWtjmRu7FNgdUAzuLyQDFfeLlTgTWmELXQ8yqvZsRANfgFeXPaEF&s=g_NSvV0H310zXlnDz_QjheigW6MaAWdN_aD6qS7H67U&e= ",
>>>>> "exp": 1648578901,
>>>>> "iat": 1648568101,
>>>>> "wlcg.groups": [
>>>>> "/fermilab"
>>>>> ],
>>>>> "jti": "https://urldefense.proofpoint.com/v2/url?u=https-3A__cilogon.org_oauth2_14fc3800d21ae572deed19b2df2a3f4-3Ftype-3DaccessToken-26ts-3D1648568101030-26version-3Dv2.0-26lifetime-3D10800000&d=DwIDAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=p0VegQeZyTJZXWtjmRu7FNgdUAzuLyQDFfeLlTgTWmELXQ8yqvZsRANfgFeXPaEF&s=SUrViCdvONgTs59-IGghFJU0FlhLsUne7BRR8RhFsx4&e= "
>>>>> }
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ cat $XDG_RUNTIME_DIR/bt_u8773
>>>>> eyJ0eXAiOiJKV1QiLCJraWQiOiJCODYzNDk1MUZEMUUzMTVEQUY3NUM5NEFFQ0YwMzY2OCIsImFsZyI6IlJTMjU2In0.eyJ3bGNnLnZlciI6IjEuMCIsImF1ZCI6Imh0dHBzOi8vd2xjZy5jZXJuLmNoL2p3dC92MS9hbnkiLCJzdWIiOiJhcm9zc2lAZm5hbC5nb3YiLCJuYmYiOjE2NDg1NjgwOTYsInNjb3BlIjoic3RvcmFnZS5jcmVhdGU6L2Zlcm1pbGFiL3VzZXJzL2Fyb3NzaSBjb21wdXRlLmNyZWF0ZSBjb21wdXRlLnJlYWQgY29tcHV0ZS5jYW5jZWwgY29tcHV0ZS5tb2RpZnkgc3RvcmFnZS5yZWFkOi9mZXJtaWxhYi91c2Vycy9hcm9zc2kiLCJpc3MiOiJodHRwczovL2NpbG9nb24ub3JnL2Zlcm1pbGFiIiwiZXhwIjoxNjQ4NTc4OTAxLCJpYXQiOjE2NDg1NjgxMDEsIndsY2cuZ3JvdXBzIjpbIi9mZXJtaWxhYiJdLCJqdGkiOiJodHRwczovL2NpbG9nb24ub3JnL29hdXRoMi8xNGZjMzgwMGQyMWFlNTcyZGVlZDE5YjJkZjJhM2Y0P3R5cGU9YWNjZXNzVG9rZW4mdHM9MTY0ODU2ODEwMTAzMCZ2ZXJzaW9uPXYyLjAmbGlmZXRpbWU9MTA4MDAwMDAifQ.MwW7gI65R6-tEEoJNqTokE0LHQo6q4FalWvtPxkITflFXqFrCl9D5bO4n_WstuewbkY-uG2ofhJ-6qe38SoGwASyZ35ZmqMictbrXyF9cBPxlFSR7XIsih2VyRMYp8W6LKFmHA1QA8yAxtHgsPm-got2r6RvtkD84F92ZuQ5qlJ-fsg4sR1HA5uc5gjKJWNED1d1iOcgatFT5UGGaIk713631TZGJ51nlFjS4LKwJ03Jqy9YlqCXotBY97rUXgPN_yDKI
mu
>> q
>>> G
>>>> tizBNY3yozBAg_grCry2Q-Gn1QAFJ3KootLDctU3aNFDnm8TGF2k4lcrvLE4FUsna7vzL0ehsveig
>>>>>
>>>>>
>>>>> 1. ZTN OFF:
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ xrdcp5x -f xroots://fndcatemp1.fnal.gov:1094//fermilab/users/arossi/data_1b /dev/null
>>>>> [0B/0B][100%][==================================================][0B/s]
>>>>> Run: [ERROR] Server responded with an error: [3010] Unable to open /fermilab/users/arossi/data_1b; permission denied (source)
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ xrdcp5x -f xroots://fndcatemp1.fnal.gov:1094//fermilab/users/arossi/data_1b?authz=Bearer%20`cat$XDG_RUNTIME_DIR/bt_u8773` /dev/null
>>>>> [1B/1B][100%][==================================================][1B/s]
>>>>>
>>>>>
>>>>> 1. ZTN ON:
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ xrdcp5x -f xroots://fndcatemp1.fnal.gov:1094//fermilab/users/arossi/data_1b /dev/null
>>>>> [0B/0B][100%][==================================================][0B/s]
>>>>> Run: [ERROR] Server responded with an error: [3010] Unable to open /fermilab/users/arossi/data_1b; permission denied (source)
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ xrdcp5x -f xroots://fndcatemp1.fnal.gov:1094//fermilab/users/arossi/data_1b?authz=Bearer%20`cat$XDG_RUNTIME_DIR/bt_u8773` /dev/null
>>>>> [1B/1B][100%][==================================================][1B/s]
>>>>>
>>>>>
>>>>> Sanity check: ZTN is being enforced, and the client is using the token in the known location. If I remove the runtime dir env variable and put the token in an unknown location, authentication fails:
>>>>>
>>>>>
>>>>> [arossi@fndcatemp1 ~]$ mv /run/user/8773/bt_u8773 .
>>>>> [arossi@fndcatemp1 ~]$ export XDG_RUNTIME_DIR=
>>>>> [arossi@fndcatemp1 ~]$ xrdcp5x -f xroots://fndcatemp1.fnal.gov:1094//fermilab/users/arossi/data_1b?authz=Bearer%20`cat bt_u8773` /dev/null
>>>>> [0B/0B][100%][==================================================][0B/s]
>>>>> Run: [FATAL] Auth failed: No protocols left to try (source)
>>>>>
>>>>>
>>>>>
>>>>> So there is no difference between the two scenarios in terms of downstream authorization on open (i.e., the server is not falling back to the ZTN token for authorization).
>>>>>
>>>>> Is there a special switch that needs to be flipped to make this happen?
>>>>>
>>>>> Or did I totally misunderstand what has been going on concerning fallback?
>>>>>
>>>>>
>>>>> Cheers, Al
>>>>>
>>>>> ________________________________________________
>>>>> Albert L. Rossi
>>>>> Senior Software Developer
>>>>> Scientific Computing Division, Scientific Data Services, Distributed Data Development
>>>>> FCC 229A
>>>>> Mail Station 369 (FCC 2W)
>>>>> Fermi National Accelerator Laboratory
>>>>> Batavia, IL 60510
>>>>> (630) 840-3023
>>>>>
>>>>> ________________________________
>>>>> From: Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Sent: Tuesday, March 29, 2022 7:25 AM
>>>>> To: Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Cc: xrootd-dev <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Subject: Re: ZTN and TPC
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> I understand the points you are making below. So it is my fault for not making my concerns clearer.
>>>>>
>>>>> dCache can indeed accept scitokens for authorization without authentication. That is what I fixed. But that is not the problem.
>>>>>
>>>>> I was just trying to look at this from the standpoint of the client, and what Brian was originally worried about -- unless I've misunderstood that.
>>>>>
>>>>> So, let me re-ask the question in another way.
>>>>>
>>>>> I am an xrootd client. I want to do TPC.
>>>>>
>>>>> But my user does not want to expose a token on the path query as a CGI element. After the changes you have made/are making, authorization can fall back to the ZTN token, provided that token has expressed subject and claims as well as issuer and audience.
>>>>>
>>>>> But the server I am talking to does not have ZTN turned on; it only has enabled scitoken authorization.
>>>>>
>>>>> There is a token at XDG_RUNTIME_DIR in the env in which I initiate the transfer.
>>>>>
>>>>> Am I going to succeed in getting authorized?
>>>>>
>>>>> Al
>>>>> ________________________________________________
>>>>> Albert L. Rossi
>>>>> Senior Software Developer
>>>>> Scientific Computing Division, Scientific Data Services, Distributed Data Development
>>>>> FCC 229A
>>>>> Mail Station 369 (FCC 2W)
>>>>> Fermi National Accelerator Laboratory
>>>>> Batavia, IL 60510
>>>>> (630) 840-3023
>>>>>
>>>>> ________________________________
>>>>> From: Andrew Hanushevsky <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Sent: Tuesday, March 29, 2022 1:17 AM
>>>>> To: Albert Rossi <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Cc: xrootd-dev <[log in to unmask]<mailto:[log in to unmask]>>
>>>>> Subject: Re: ZTN and TPC
>>>>>
>>>>> Hi Albert,
>>>>>
>>>>> Lots of questions here. OK, so let me frst say what is going on here.
>>>>>
>>>>> a) When enabled, ztn simply asks the connecting client (irrespective of
>>>>> what it is going to do) to provide a valid token. This token may or may
>>>>> not be used for future authorization purposes. That decision is totally
>>>>> independent. Why? Because at the point we ask for a ztn token all we want
>>>>> to know is that the client has or can obtain a valid token. That is all we
>>>>> care about. So, the following points must keep this in mind.
>>>>>
>>>>> On Fri, 25 Mar 2022, Albert Rossi wrote:
>>>>>
>>>>>> However, that is not the question I have. What I am writing about here
>>>>>> has to do with ZTN in this equation. If your ZTN module is loaded, how
>>>>>> does it know to allow the third-party client to get a "pass", since that
>>>>>> client does not have any JWT token?
>>>>> Totally independent decision points. Consider this:
>>>>> a) Client logs in and at that point we have no idea what the purpose is
>>>>> but we ask for a ztn token.
>>>>> b) Client supplies one and we check if it is valid, if so, client is
>>>>> allowed to proceed.
>>>>> c) Client wants to do a TPC. Perfectly acceptable. Does the ztn token have
>>>>> anything to do with that? Not necessarily. In fact, we really don't know
>>>>> and wish Brian would weigh on this but alas no word.
>>>>>
>>>>>> Or does it still get the ZTN token even though it does not provide a
>>>>>> token for authorization to the source server?
>>>>> Again, please reread the beginning statement. When we ask for a zrtn token
>>>>> we only wish to ascertain the client's ability to get one. It has nothing
>>>>> to do with any future autrhorization. To the extend that the authorization
>>>>> module may use the zn token is up to that module. We really don't care.
>>>>>
>>>>>> Or do you have to turn ZTN off with TPC? >
>>>>> Absolutely not! That is not what the primary purpose of ztn is. I think
>>>>> you are really confusing the purpose of ztn and what redezvous TPC is
>>>>> trying to accomplish. The problem here is that unlike xrootd where we can
>>>>> easily suppress authorization when we know this is a rendezvous TPC dCache
>>>>> appears not to be able to do this. So, it needs some kind of token in
>>>>> addition to the rendezvous token to move forward. The question is which
>>>>> token are we talking about. My asnwer is without Brian's weigh in I don't
>>>>> know and there is silence at his end. So we are both out of luck.
>>>>>
>>>>>> I am asking these questions because I have not figured out, for dCache,
>>>>>> how to (a) specify ZTN as an authentication protocol, but (b) allow a
>>>>>> specifically third-party connection not to have to present a ZTN token.
>>>>>> At authentication time, it does not seem to me the server knows enough
>>>>>> about the client to be able to distinguish what it is.
>>>>>> Or does it?
>>>>> No it does not matter. The problem here is that any server may ask for a
>>>>> ztn token and you better be prepared to supply one or have another
>>>>> authentication protocol you can fall back to. I would suggest the simplest
>>>>> approach here. In the presence of a TPC should the target server ask for a
>>>>> token then you supply the ztn token should you have it. If you do not then
>>>>> you will need to fallback on some other authentication protocol that the
>>>>> target server supports. If there is no other protcol then whole thing
>>>>> simply fails and you live with that.
>>>>>
>>>>> Andy
>>>>>
>>>>>> Some guidance here
>>>>> would be very helpful, > > Thanks, Al
>>>>>>
>>>>>> ________________________________________________
>>>>>> Albert L. Rossi
>>>>>> Senior Software Developer
>>>>>> Scientific Computing Division, Scientific Data Services, Distributed Data Development
>>>>>> FCC 229A
>>>>>> Mail Station 369 (FCC 2W)
>>>>>> Fermi National Accelerator Laboratory
>>>>>> Batavia, IL 60510
>>>>>> (630) 840-3023
>>>>>>
>>>>>>
>>>>>> ########################################################################
>>>>>> Use REPLY-ALL to reply to list
>>>>>>
>>>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DDEV-26A-3D1&d=DwIBAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=9aahSq-wsU59ZSWZ00xk_zy5ZFU6hyg63E0HPoGzJQ8F6TWVj47l3nCukhbHNHEw&s=_VQWTuyj544srHCnttohyT1-ZjVHIbgM2r3_V-1H_Oo&e=
>>>>>>
>>>>>
>>>>
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-DEV list, click the following link:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DDEV-26A-3D1&d=DwIFAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=zXFEgJXnliB42aWDvIWnW6T6negNGOaguaVLe3FHpG0uzQ2MITOTpXXrERmz7YfW&s=wHotMwXapMoWXe0UGdYIavmZcmBWb_pjSvJkHqA8Jho&e=
>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|
