 |
 |
This is a nice catch! Thanks, to everyone looking into this. Cheers, Justas On Wed, 16 Feb 2022 at 00:45, Petr Vokac <[log in to unmask]> wrote: > Proxy generated with 1024 key size seems to be mainly OSG problem, > because this size is used by default only by voms-proxy-init2. Those who > rely on voms-proxy-init3 > 3.3.2 (released in October 2020) or arcproxy > >= 6.4.0 (released in November 2019) gets by default 2048 keys. > > Be aware there might still be software which "accidentally" generate > proxies with smaller than default size, e.g. FTS < 3.11.0 (latest > version released in September 2021) used fixed key size 1024 > (https://its.cern.ch/jira/browse/FTS-1700). > > Saying that it would be useful to have more clear error messages. > > Petr > > On 2/16/22 08:18, Matevz Tadel wrote: > > Andy and I did some deep debugging of where the cert init fails ... and > it > > turned out the problem was 1024-bit key of the proxy cert on the client > machine > > --- openssl-1.1.1k (which comes with cos-8) apparently requires the key > to be at > > least 2048 bits when calling SSL_CTX_use_certificate_file(). > > > > After we replaced it with my 2048-bit key cert it worked no problem. > > > > On fedora-34 the default key length generated by voms-proxy-init is > already > > 2048, on centos-7.9 it is still 1024. One can get longer keys with -bits > option: > > voms-proxy-init -bits 2048 -voms cms > > > > Andy will try to get more informative error messages from openssl ... > but this > > doesn't seem entirely trivial as the error gets set at the top-level > function > > and we had to go three levels down to see the actual error code that we > were > > able to google. > > > > Matevz > > > > On 2/15/22 19:46, Justas Balcas wrote: > >> That is correct, Matevz > >> > >> On Tue, Feb 15, 2022, 18:39 Matevz Tadel <[log in to unmask] > >> <mailto:[log in to unmask]>> wrote: > >> > >> Thank you Andy! > >> > >> The error is permanent, I think. Justas, can you please confirm? > >> > >> Matevz > >> > >> On 2/15/22 17:49, Andrew Hanushevsky wrote: > >> > Here are the common problems when the client issues that message: > >> > > >> > 1) The certificate's chain of trust is broken (could be because > the root CA > >> > can't be verified, or the root/intermediate certificate has > expired). This > >> would > >> > cause sproradic failures (note that if the certificate is > improperly installed > >> > then the failure would always occur which is not the case here). > >> > > >> > 2) The certificate must be renewed before the expiry of the > certificate to > >> avoid > >> > any conflict arising out of time violation. Ensure that the > date/time is set > >> > correctly on your computer since that might be used to assess > the validity > >> > period of the SSL certificate of the website. This would cause > repated > >> failures > >> > on some sites but not others depending on how close the cert was > to expiring. > >> > > >> > 3) The certificate structure is broken, or the certificate's > signature > >> can't be > >> > checked. This is the hardest to solve but one reason could me > random memory > >> > corruption. I would suggest runningthe client with valgrind and > try to > >> match the > >> > valgring log with the invalid cert message. > >> > > >> > 4) Firewall issues may cause problems with they have been > enabled for > >> > "encrypted/SSL scanning or checking." > >> > > >> > 5) If tghe server is using only SHA-1 encryption then those cert > are > >> flagged as > >> > insecure and need to update their security certificates. > However, would most > >> > likely cause permanent errors. > >> > > >> > Andy > >> > > >> > > >> > > >> > On Tue, 15 Feb 2022, Matevz Tadel wrote: > >> > > >> >> Andy, > >> >> > >> >> What's with the invalid certificate in the client log? > >> >> > >> >> [2022-02-15 09:56:19.549139 -0800][Debug ][XRootDTransport ] > >> >> [transfer-9.ultralight.org > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e= > >:1095.0] > >> Sending out kXR_login request, username: > >> >> root, cgi: > >> >> ?xrd.cc=us&xrd.tz > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.tz&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=-Rc4_LmnigwN9z03K7gZrf-B9w2_b0g2EkivTF0FV9U&e= > >=-8&xrd.appname=xrdcp&xrd.info > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.info&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=j_3eK4Ghw7zsEUpD0XXrLaG5_Maj8wMSMXpXUC-Up5Y&e= > >=&xrd.hostname=xrd-cache-3.ultralight.org > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dcache-2D3.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=PYd32OF7sFBQVcQsXBH9TGn-f4DSZENRYSAySzIRVUE&e= > >&xrd.rn=v5.2.0, > >> >> dual-stack: true, private IPv4: false, private IPv6: false > >> >> [2022-02-15 09:56:19.549209 -0800][Debug ][AsyncSock ] > >> >> [transfer-9.ultralight.org > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e= > >:1095.0] > >> TLS hand-shake exchange. > >> >> > >> >> ===> HERE: > >> >> [2022-02-15 09:56:19.551762 -0800][Error ][TlsMsg ] > >> [TLS_Context:] > >> >> Unable to create TLS context; invalid certificate. > >> >> > >> >> [2022-02-15 09:56:19.551903 -0800][Error ][AsyncSock ] > >> >> [transfer-9.ultralight.org > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e= > >:1095.0] > >> Socket error while handshaking: [FATAL] TLS > >> >> error > >> >> [2022-02-15 09:56:19.551920 -0800][Debug ][AsyncSock ] > >> >> [transfer-9.ultralight.org > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e= > >:1095.0] > >> Closing the socket > >> >> > >> >> Can I run in gdb to get more info? What is good place to start > poking? > >> >> > >> >> I was assuming it's the server cert that client does not like > ... but it does > >> >> look ok to me :) > >> >> > >> >> Matevz > >> >> > >> >> > >> >> On 2/15/22 14:04, Andrew Hanushevsky wrote: > >> >>> Hi Bockjoo, > >> >>> > >> >>> Unfortunately, that's not the way it works. While gsi doesn't > need to > >> use TLS > >> >>> ztn does. Since the erver doesn't know which protocol the > client will > >> eventually > >> >>> settle on, the connection has to use TLS right from the start. > That > >> means you > >> >>> cannot use ztn with incapable clients. > >> >>> > >> >>> Andy > >> >>> > >> >>> > >> >>> On Tue, 15 Feb 2022, Bockjoo Kim wrote: > >> >>> > >> >>>> Hi Andy, > >> >>>> > >> >>>> There are two sec.protocols: gsi and ztn. > >> >>>> > >> >>>> Doesn't the interaction go through gsi and if it fails, will > it go > >> through ztn? > >> >>>> > >> >>>> For incapable clients, the gsi can succeed, no? > >> >>>> > >> >>>> Thanks, > >> >>>> > >> >>>> Bockjoo > >> >>>> > >> >>>> On 2/15/22 14:52, Andrew Hanushevsky wrote: > >> >>>>> Hi Justas, > >> >>>>> > >> >>>>> If you look into the log you will notice a warning that > tells you that TLS > >> >>>>> will always be on regardless of the "capable" setting because > >> authentication > >> >>>>> protocol ztn requires tls. So, this may be the source of the > problem, > >> >>>>> certainly it will be for incapable clients. > >> >>>>> > >> >>>>> Andy > >> >>>>> > >> >>>>> > >> >>>>> On Tue, 15 Feb 2022, Justas Balcas wrote: > >> >>>>> > >> >>>>>> Hi, > >> >>>>>> > >> >>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I > took out 1 > >> server > >> >>>>>> from prod and played with full debug mode, on/off tls. > >> >>>>>> > >> >>>>>> Logs from server/client are available here: > >> >>>>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e= > > > >> >>>>>> > >> >>>>>> > >> >>>>>> To turn TLS, I added this config: > >> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem > >> >>>>>> /etc/grid-security/xrootd/xrootdkey.pem > >> >>>>>> xrd.tlsca certdir /etc/grid-security/certificates > >> >>>>>> xrootd.tls capable all > >> >>>>>> sec.protocol /usr/lib64 ztn > >> >>>>>> > >> >>>>>> And with TLS on - I always get: > >> >>>>>> TLS hand-shake exchange. > >> >>>>>> Socket error while handshaking: [FATAL] TLS error > >> >>>>>> Closing the socket > >> >>>>>> > >> >>>>>> > >> >>>>>> If it helps, here is full config: > >> >>>>>> > >> >>>>>> all.export /tmp stage > >> >>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN > >> >>>>>> all.adminpath /var/spool/xrootd > >> >>>>>> all.pidpath /var/run/xrootd > >> >>>>>> > >> >>>>>> # XrootD Security > >> >>>>>> # --------------------------------------- > >> >>>>>> xrootd.seclib /usr/lib64/libXrdSec.so > >> >>>>>> sec.protocol /usr/lib64 gsi > -certdir:/etc/grid-security/certificates > >> >>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem > >> >>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3 > >> >>>>>> -authzfun:libXrdLcmaps.so -authzto:900 > >> >>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 > -gmapto:0 > >> >>>>>> acc.authdb /etc/xrootd/auth_file_stageout > >> >>>>>> ofs.authorize > >> >>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret > >> >>>>>> ofs.authlib ++ libXrdMacaroons.so > >> >>>>>> ofs.authlib ++ libXrdAccSciTokens.so > >> >>>>>> # -------------------------------------- > >> >>>>>> # XrootD Monitoring > >> >>>>>> # -------------------------------------- > >> >>>>>> # Monitoring for AAA Dashboard : > >> >>>>>> xrd.report 169.228.130.91:9931 > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9931&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=AW4Xegl5wxbeERkKViynNkyDzu4pC5VhL7QR8sFGF44&e= > > > >> every 30s all sync > >> >>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn > ops xfr 5 dest > >> >>>>>> files io info user 169.228.130.91:9930 > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=9w5jNESVzo8Kp4KzAUI0Ymeo-wcerH5BqhqE0Rfk6oU&e= > > > >> dest fstat info user > >> >>>>>> xrd-mon.osgstorage.org:9930 > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dmon.osgstorage.org-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=lMHFksodly_4DGpPCSpc0-Wb9UGjl1_8yif-C2TNQrI&e= > > > >> >>>>>> all.sitename T2_US_Caltech > >> >>>>>> # ------------------------------------- > >> >>>>>> # Configure redirector/server > >> >>>>>> # ------------------------------------- > >> >>>>>> set xrdr = xrootd-redir-stageout.ultralight.org > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__xrootd-2Dredir-2Dstageout.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=HTNBiz85qLXQGkB75MafMeY4xc6vwkMUMuatd5_g0mo&e= > > > >> >>>>>> xrd.port 1095 > >> >>>>>> all.manager $(xrdr):1213 > >> >>>>>> > >> >>>>>> if $(xrdr) > >> >>>>>> # It's role is manager > >> >>>>>> all.role manager > >> >>>>>> # Redirect all lookup calls to original data servers. > Redirector > >> does not > >> >>>>>> have visibility of FS > >> >>>>>> cms.dfs lookup distrib mdhold 20m redirect immed > >> >>>>>> else > >> >>>>>> # Role is server > >> >>>>>> all.role server > >> >>>>>> # The known managers (local redirector) > >> >>>>>> all.manager meta $(xrdr):1213 > >> >>>>>> > >> >>>>>> # Enable xrootd checksum calculation "on-the-fly" using > multiuser plugin > >> >>>>>> # This makes XRootD to write the files with the > >> >>>>>> # ownership of the user that authenticated to the server > and not as the > >> >>>>>> # 'xrootd' user > >> >>>>>> ofs.osslib ++ libXrdMultiuser.so > >> >>>>>> # Enable the checksum wrapper > >> >>>>>> ofs.ckslib * libXrdMultiuser.so > >> >>>>>> # Control of checksum > >> >>>>>> xrootd.chksum max 10 adler32 > >> >>>>>> multiuser.checksumonwrite on > >> >>>>>> multiuser.umask 0002 > >> >>>>>> > >> >>>>>> fi > >> >>>>>> # ------------------------------------- > >> >>>>>> # Allow only specific path, checksum config > >> >>>>>> # ------------------------------------- > >> >>>>>> # Allow any path to be exported; this is further refined in > the authfile. > >> >>>>>> all.export / > >> >>>>>> > >> >>>>>> # Hosts allowed to use this xrootd cluster > >> >>>>>> cms.allow host * > >> >>>>>> > >> >>>>>> # Enable xrootd debugging > >> >>>>>> xrootd.trace emsg login stall redirect > >> >>>>>> cms.trace defer files forward redirect > >> >>>>>> > >> >>>>>> # Disable async. Related issue: > >> >>>>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e= > > > >> >>>>>> > >> >>>>>> xrootd.async off > >> >>>>>> > >> >>>>>> # ------------------------------------- > >> >>>>>> # Integrate with CMS Namespaces > >> >>>>>> # It will see files under /store/... > >> >>>>>> # ------------------------------------- > >> >>>>>> oss.localroot /storage/cms > >> >>>>>> # ------------------------------------- > >> >>>>>> # Configure davs/https for TPC > >> >>>>>> # ------------------------------------- > >> >>>>>> # Enable https over XrootD > >> >>>>>> if exec xrootd > >> >>>>>> xrd.protocol http:1095 /usr/lib64/libXrdHttp.so > >> >>>>>> http.cadir /etc/grid-security/certificates > >> >>>>>> http.cert /etc/grid-security/xrootd/xrootdcert.pem > >> >>>>>> http.key /etc/grid-security/xrootd/xrootdkey.pem > >> >>>>>> http.secxtractor /usr/lib64/libXrdLcmaps.so > >> >>>>>> http.secretkey XXXXXXX > >> >>>>>> # Enable third-party-copy > >> >>>>>> http.exthandler xrdtpc libXrdHttpTPC.so > >> >>>>>> # Pass the bearer token to the Xrootd authorization > framework. > >> >>>>>> http.header2cgi Authorization authz > >> >>>>>> http.listingdeny yes > >> >>>>>> http.desthttps yes > >> >>>>>> http.selfhttps2http no > >> >>>>>> http.staticpreload > >> >>>>>> > >> > https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= > > > >> >>>>>> > >> >>>>>> /etc/xrootd/robots.txt > >> >>>>>> http.exthandler xrdmacaroons libXrdMacaroons.so > >> >>>>>> fi > >> >>>>>> > >> >>>>>> > >> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem > >> >>>>>> /etc/grid-security/xrootd/xrootdkey.pem > >> >>>>>> xrd.tlsca certdir /etc/grid-security/certificates > >> >>>>>> xrootd.tls capable all > >> >>>>>> sec.protocol /usr/lib64 ztn > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> xrootd.trace all > >> >>>>>> xrd.trace all > >> >>>>>> ofs.trace all > >> >>>>>> pfc.trace all > >> >>>>>> cms.trace all > >> >>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 > Error, 2 > >> >>>>>> Warning, 1 Info) > >> >>>>>> pss.setopt DebugLevel 4 > >> >>>>>> > >> >>>>>> > ######################################################################## > >> >>>>>> Use REPLY-ALL to reply to list > >> >>>>>> > >> >>>>>> To unsubscribe from the XROOTD-L list, click the following > link: > >> >>>>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > > > >> >>>>>> > >> >>>>>> > >> >>>>> > >> >>>>> > ######################################################################## > >> >>>>> Use REPLY-ALL to reply to list > >> >>>>> > >> >>>>> To unsubscribe from the XROOTD-L list, click the following > link: > >> >>>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > > > >> >>>>> > >> >>>> > >> >>>> > >> >>>> > ######################################################################## > >> >>>> Use REPLY-ALL to reply to list > >> >>>> > >> >>>> To unsubscribe from the XROOTD-L list, click the following > link: > >> >>>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > > > >> >>>> > >> >>>> > >> >>> > >> >>> > ######################################################################## > >> >>> Use REPLY-ALL to reply to list > >> >>> > >> >>> To unsubscribe from the XROOTD-L list, click the following > link: > >> >>> > >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > >> < > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > > > >> >>> > >> >>> > >> >> > >> > > ######################################################################## > > Use REPLY-ALL to reply to list > > > > To unsubscribe from the XROOTD-L list, click the following link: > > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1