Proxy generated with 1024 key size seems to be mainly OSG problem, because this size is used by default only by voms-proxy-init2. Those who rely on voms-proxy-init3 > 3.3.2 (released in October 2020) or arcproxy >= 6.4.0 (released in November 2019) gets by default 2048 keys. Be aware there might still be software which "accidentally" generate proxies with smaller than default size, e.g. FTS < 3.11.0 (latest version released in September 2021) used fixed key size 1024 (https://its.cern.ch/jira/browse/FTS-1700). Saying that it would be useful to have more clear error messages. Petr On 2/16/22 08:18, Matevz Tadel wrote: > Andy and I did some deep debugging of where the cert init fails ... and it > turned out the problem was 1024-bit key of the proxy cert on the client machine > --- openssl-1.1.1k (which comes with cos-8) apparently requires the key to be at > least 2048 bits when calling SSL_CTX_use_certificate_file(). > > After we replaced it with my 2048-bit key cert it worked no problem. > > On fedora-34 the default key length generated by voms-proxy-init is already > 2048, on centos-7.9 it is still 1024. One can get longer keys with -bits option: > voms-proxy-init -bits 2048 -voms cms > > Andy will try to get more informative error messages from openssl ... but this > doesn't seem entirely trivial as the error gets set at the top-level function > and we had to go three levels down to see the actual error code that we were > able to google. > > Matevz > > On 2/15/22 19:46, Justas Balcas wrote: >> That is correct, Matevz >> >> On Tue, Feb 15, 2022, 18:39 Matevz Tadel <[log in to unmask] >> <mailto:[log in to unmask]>> wrote: >> >> Thank you Andy! >> >> The error is permanent, I think. Justas, can you please confirm? >> >> Matevz >> >> On 2/15/22 17:49, Andrew Hanushevsky wrote: >> > Here are the common problems when the client issues that message: >> > >> > 1) The certificate's chain of trust is broken (could be because the root CA >> > can't be verified, or the root/intermediate certificate has expired). This >> would >> > cause sproradic failures (note that if the certificate is improperly installed >> > then the failure would always occur which is not the case here). >> > >> > 2) The certificate must be renewed before the expiry of the certificate to >> avoid >> > any conflict arising out of time violation. Ensure that the date/time is set >> > correctly on your computer since that might be used to assess the validity >> > period of the SSL certificate of the website. This would cause repated >> failures >> > on some sites but not others depending on how close the cert was to expiring. >> > >> > 3) The certificate structure is broken, or the certificate's signature >> can't be >> > checked. This is the hardest to solve but one reason could me random memory >> > corruption. I would suggest runningthe client with valgrind and try to >> match the >> > valgring log with the invalid cert message. >> > >> > 4) Firewall issues may cause problems with they have been enabled for >> > "encrypted/SSL scanning or checking." >> > >> > 5) If tghe server is using only SHA-1 encryption then those cert are >> flagged as >> > insecure and need to update their security certificates. However, would most >> > likely cause permanent errors. >> > >> > Andy >> > >> > >> > >> > On Tue, 15 Feb 2022, Matevz Tadel wrote: >> > >> >> Andy, >> >> >> >> What's with the invalid certificate in the client log? >> >> >> >> [2022-02-15 09:56:19.549139 -0800][Debug ][XRootDTransport ] >> >> [transfer-9.ultralight.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] >> Sending out kXR_login request, username: >> >> root, cgi: >> >> ?xrd.cc=us&xrd.tz >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.tz&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=-Rc4_LmnigwN9z03K7gZrf-B9w2_b0g2EkivTF0FV9U&e=>=-8&xrd.appname=xrdcp&xrd.info >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.info&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=j_3eK4Ghw7zsEUpD0XXrLaG5_Maj8wMSMXpXUC-Up5Y&e=>=&xrd.hostname=xrd-cache-3.ultralight.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dcache-2D3.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=PYd32OF7sFBQVcQsXBH9TGn-f4DSZENRYSAySzIRVUE&e=>&xrd.rn=v5.2.0, >> >> dual-stack: true, private IPv4: false, private IPv6: false >> >> [2022-02-15 09:56:19.549209 -0800][Debug ][AsyncSock ] >> >> [transfer-9.ultralight.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] >> TLS hand-shake exchange. >> >> >> >> ===> HERE: >> >> [2022-02-15 09:56:19.551762 -0800][Error ][TlsMsg ] >> [TLS_Context:] >> >> Unable to create TLS context; invalid certificate. >> >> >> >> [2022-02-15 09:56:19.551903 -0800][Error ][AsyncSock ] >> >> [transfer-9.ultralight.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] >> Socket error while handshaking: [FATAL] TLS >> >> error >> >> [2022-02-15 09:56:19.551920 -0800][Debug ][AsyncSock ] >> >> [transfer-9.ultralight.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] >> Closing the socket >> >> >> >> Can I run in gdb to get more info? What is good place to start poking? >> >> >> >> I was assuming it's the server cert that client does not like ... but it does >> >> look ok to me :) >> >> >> >> Matevz >> >> >> >> >> >> On 2/15/22 14:04, Andrew Hanushevsky wrote: >> >>> Hi Bockjoo, >> >>> >> >>> Unfortunately, that's not the way it works. While gsi doesn't need to >> use TLS >> >>> ztn does. Since the erver doesn't know which protocol the client will >> eventually >> >>> settle on, the connection has to use TLS right from the start. That >> means you >> >>> cannot use ztn with incapable clients. >> >>> >> >>> Andy >> >>> >> >>> >> >>> On Tue, 15 Feb 2022, Bockjoo Kim wrote: >> >>> >> >>>> Hi Andy, >> >>>> >> >>>> There are two sec.protocols: gsi and ztn. >> >>>> >> >>>> Doesn't the interaction go through gsi and if it fails, will it go >> through ztn? >> >>>> >> >>>> For incapable clients, the gsi can succeed, no? >> >>>> >> >>>> Thanks, >> >>>> >> >>>> Bockjoo >> >>>> >> >>>> On 2/15/22 14:52, Andrew Hanushevsky wrote: >> >>>>> Hi Justas, >> >>>>> >> >>>>> If you look into the log you will notice a warning that tells you that TLS >> >>>>> will always be on regardless of the "capable" setting because >> authentication >> >>>>> protocol ztn requires tls. So, this may be the source of the problem, >> >>>>> certainly it will be for incapable clients. >> >>>>> >> >>>>> Andy >> >>>>> >> >>>>> >> >>>>> On Tue, 15 Feb 2022, Justas Balcas wrote: >> >>>>> >> >>>>>> Hi, >> >>>>>> >> >>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I took out 1 >> server >> >>>>>> from prod and played with full debug mode, on/off tls. >> >>>>>> >> >>>>>> Logs from server/client are available here: >> >>>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e= >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e=> >> >>>>>> >> >>>>>> >> >>>>>> To turn TLS, I added this config: >> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem >> >>>>>> /etc/grid-security/xrootd/xrootdkey.pem >> >>>>>> xrd.tlsca certdir /etc/grid-security/certificates >> >>>>>> xrootd.tls capable all >> >>>>>> sec.protocol /usr/lib64 ztn >> >>>>>> >> >>>>>> And with TLS on - I always get: >> >>>>>> TLS hand-shake exchange. >> >>>>>> Socket error while handshaking: [FATAL] TLS error >> >>>>>> Closing the socket >> >>>>>> >> >>>>>> >> >>>>>> If it helps, here is full config: >> >>>>>> >> >>>>>> all.export /tmp stage >> >>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN >> >>>>>> all.adminpath /var/spool/xrootd >> >>>>>> all.pidpath /var/run/xrootd >> >>>>>> >> >>>>>> # XrootD Security >> >>>>>> # --------------------------------------- >> >>>>>> xrootd.seclib /usr/lib64/libXrdSec.so >> >>>>>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates >> >>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem >> >>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3 >> >>>>>> -authzfun:libXrdLcmaps.so -authzto:900 >> >>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 -gmapto:0 >> >>>>>> acc.authdb /etc/xrootd/auth_file_stageout >> >>>>>> ofs.authorize >> >>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret >> >>>>>> ofs.authlib ++ libXrdMacaroons.so >> >>>>>> ofs.authlib ++ libXrdAccSciTokens.so >> >>>>>> # -------------------------------------- >> >>>>>> # XrootD Monitoring >> >>>>>> # -------------------------------------- >> >>>>>> # Monitoring for AAA Dashboard : >> >>>>>> xrd.report 169.228.130.91:9931 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9931&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=AW4Xegl5wxbeERkKViynNkyDzu4pC5VhL7QR8sFGF44&e=> >> every 30s all sync >> >>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest >> >>>>>> files io info user 169.228.130.91:9930 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=9w5jNESVzo8Kp4KzAUI0Ymeo-wcerH5BqhqE0Rfk6oU&e=> >> dest fstat info user >> >>>>>> xrd-mon.osgstorage.org:9930 >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dmon.osgstorage.org-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=lMHFksodly_4DGpPCSpc0-Wb9UGjl1_8yif-C2TNQrI&e=> >> >>>>>> all.sitename T2_US_Caltech >> >>>>>> # ------------------------------------- >> >>>>>> # Configure redirector/server >> >>>>>> # ------------------------------------- >> >>>>>> set xrdr = xrootd-redir-stageout.ultralight.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrootd-2Dredir-2Dstageout.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=HTNBiz85qLXQGkB75MafMeY4xc6vwkMUMuatd5_g0mo&e=> >> >>>>>> xrd.port 1095 >> >>>>>> all.manager $(xrdr):1213 >> >>>>>> >> >>>>>> if $(xrdr) >> >>>>>> # It's role is manager >> >>>>>> all.role manager >> >>>>>> # Redirect all lookup calls to original data servers. Redirector >> does not >> >>>>>> have visibility of FS >> >>>>>> cms.dfs lookup distrib mdhold 20m redirect immed >> >>>>>> else >> >>>>>> # Role is server >> >>>>>> all.role server >> >>>>>> # The known managers (local redirector) >> >>>>>> all.manager meta $(xrdr):1213 >> >>>>>> >> >>>>>> # Enable xrootd checksum calculation "on-the-fly" using multiuser plugin >> >>>>>> # This makes XRootD to write the files with the >> >>>>>> # ownership of the user that authenticated to the server and not as the >> >>>>>> # 'xrootd' user >> >>>>>> ofs.osslib ++ libXrdMultiuser.so >> >>>>>> # Enable the checksum wrapper >> >>>>>> ofs.ckslib * libXrdMultiuser.so >> >>>>>> # Control of checksum >> >>>>>> xrootd.chksum max 10 adler32 >> >>>>>> multiuser.checksumonwrite on >> >>>>>> multiuser.umask 0002 >> >>>>>> >> >>>>>> fi >> >>>>>> # ------------------------------------- >> >>>>>> # Allow only specific path, checksum config >> >>>>>> # ------------------------------------- >> >>>>>> # Allow any path to be exported; this is further refined in the authfile. >> >>>>>> all.export / >> >>>>>> >> >>>>>> # Hosts allowed to use this xrootd cluster >> >>>>>> cms.allow host * >> >>>>>> >> >>>>>> # Enable xrootd debugging >> >>>>>> xrootd.trace emsg login stall redirect >> >>>>>> cms.trace defer files forward redirect >> >>>>>> >> >>>>>> # Disable async. Related issue: >> >>>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e= >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e=> >> >>>>>> >> >>>>>> xrootd.async off >> >>>>>> >> >>>>>> # ------------------------------------- >> >>>>>> # Integrate with CMS Namespaces >> >>>>>> # It will see files under /store/... >> >>>>>> # ------------------------------------- >> >>>>>> oss.localroot /storage/cms >> >>>>>> # ------------------------------------- >> >>>>>> # Configure davs/https for TPC >> >>>>>> # ------------------------------------- >> >>>>>> # Enable https over XrootD >> >>>>>> if exec xrootd >> >>>>>> xrd.protocol http:1095 /usr/lib64/libXrdHttp.so >> >>>>>> http.cadir /etc/grid-security/certificates >> >>>>>> http.cert /etc/grid-security/xrootd/xrootdcert.pem >> >>>>>> http.key /etc/grid-security/xrootd/xrootdkey.pem >> >>>>>> http.secxtractor /usr/lib64/libXrdLcmaps.so >> >>>>>> http.secretkey XXXXXXX >> >>>>>> # Enable third-party-copy >> >>>>>> http.exthandler xrdtpc libXrdHttpTPC.so >> >>>>>> # Pass the bearer token to the Xrootd authorization framework. >> >>>>>> http.header2cgi Authorization authz >> >>>>>> http.listingdeny yes >> >>>>>> http.desthttps yes >> >>>>>> http.selfhttps2http no >> >>>>>> http.staticpreload >> >>>>>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e=> >> >>>>>> >> >>>>>> /etc/xrootd/robots.txt >> >>>>>> http.exthandler xrdmacaroons libXrdMacaroons.so >> >>>>>> fi >> >>>>>> >> >>>>>> >> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem >> >>>>>> /etc/grid-security/xrootd/xrootdkey.pem >> >>>>>> xrd.tlsca certdir /etc/grid-security/certificates >> >>>>>> xrootd.tls capable all >> >>>>>> sec.protocol /usr/lib64 ztn >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> xrootd.trace all >> >>>>>> xrd.trace all >> >>>>>> ofs.trace all >> >>>>>> pfc.trace all >> >>>>>> cms.trace all >> >>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2 >> >>>>>> Warning, 1 Info) >> >>>>>> pss.setopt DebugLevel 4 >> >>>>>> >> >>>>>> ######################################################################## >> >>>>>> Use REPLY-ALL to reply to list >> >>>>>> >> >>>>>> To unsubscribe from the XROOTD-L list, click the following link: >> >>>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> ######################################################################## >> >>>>> Use REPLY-ALL to reply to list >> >>>>> >> >>>>> To unsubscribe from the XROOTD-L list, click the following link: >> >>>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> >> >>>>> >> >>>> >> >>>> >> >>>> ######################################################################## >> >>>> Use REPLY-ALL to reply to list >> >>>> >> >>>> To unsubscribe from the XROOTD-L list, click the following link: >> >>>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> >> >>>> >> >>>> >> >>> >> >>> ######################################################################## >> >>> Use REPLY-ALL to reply to list >> >>> >> >>> To unsubscribe from the XROOTD-L list, click the following link: >> >>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> >> >>> >> >>> >> >> >> > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1