 |
 |
Andy and I did some deep debugging of where the cert init fails ... and it turned out the problem was 1024-bit key of the proxy cert on the client machine --- openssl-1.1.1k (which comes with cos-8) apparently requires the key to be at least 2048 bits when calling SSL_CTX_use_certificate_file(). After we replaced it with my 2048-bit key cert it worked no problem. On fedora-34 the default key length generated by voms-proxy-init is already 2048, on centos-7.9 it is still 1024. One can get longer keys with -bits option: voms-proxy-init -bits 2048 -voms cms Andy will try to get more informative error messages from openssl ... but this doesn't seem entirely trivial as the error gets set at the top-level function and we had to go three levels down to see the actual error code that we were able to google. Matevz On 2/15/22 19:46, Justas Balcas wrote: > That is correct, Matevz > > On Tue, Feb 15, 2022, 18:39 Matevz Tadel <[log in to unmask] > <mailto:[log in to unmask]>> wrote: > > Thank you Andy! > > The error is permanent, I think. Justas, can you please confirm? > > Matevz > > On 2/15/22 17:49, Andrew Hanushevsky wrote: > > Here are the common problems when the client issues that message: > > > > 1) The certificate's chain of trust is broken (could be because the root CA > > can't be verified, or the root/intermediate certificate has expired). This > would > > cause sproradic failures (note that if the certificate is improperly installed > > then the failure would always occur which is not the case here). > > > > 2) The certificate must be renewed before the expiry of the certificate to > avoid > > any conflict arising out of time violation. Ensure that the date/time is set > > correctly on your computer since that might be used to assess the validity > > period of the SSL certificate of the website. This would cause repated > failures > > on some sites but not others depending on how close the cert was to expiring. > > > > 3) The certificate structure is broken, or the certificate's signature > can't be > > checked. This is the hardest to solve but one reason could me random memory > > corruption. I would suggest runningthe client with valgrind and try to > match the > > valgring log with the invalid cert message. > > > > 4) Firewall issues may cause problems with they have been enabled for > > "encrypted/SSL scanning or checking." > > > > 5) If tghe server is using only SHA-1 encryption then those cert are > flagged as > > insecure and need to update their security certificates. However, would most > > likely cause permanent errors. > > > > Andy > > > > > > > > On Tue, 15 Feb 2022, Matevz Tadel wrote: > > > >> Andy, > >> > >> What's with the invalid certificate in the client log? > >> > >> [2022-02-15 09:56:19.549139 -0800][Debug ][XRootDTransport ] > >> [transfer-9.ultralight.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] > Sending out kXR_login request, username: > >> root, cgi: > >> ?xrd.cc=us&xrd.tz > <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.tz&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=-Rc4_LmnigwN9z03K7gZrf-B9w2_b0g2EkivTF0FV9U&e=>=-8&xrd.appname=xrdcp&xrd.info > <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.info&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=j_3eK4Ghw7zsEUpD0XXrLaG5_Maj8wMSMXpXUC-Up5Y&e=>=&xrd.hostname=xrd-cache-3.ultralight.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dcache-2D3.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=PYd32OF7sFBQVcQsXBH9TGn-f4DSZENRYSAySzIRVUE&e=>&xrd.rn=v5.2.0, > >> dual-stack: true, private IPv4: false, private IPv6: false > >> [2022-02-15 09:56:19.549209 -0800][Debug ][AsyncSock ] > >> [transfer-9.ultralight.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] > TLS hand-shake exchange. > >> > >> ===> HERE: > >> [2022-02-15 09:56:19.551762 -0800][Error ][TlsMsg ] > [TLS_Context:] > >> Unable to create TLS context; invalid certificate. > >> > >> [2022-02-15 09:56:19.551903 -0800][Error ][AsyncSock ] > >> [transfer-9.ultralight.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] > Socket error while handshaking: [FATAL] TLS > >> error > >> [2022-02-15 09:56:19.551920 -0800][Debug ][AsyncSock ] > >> [transfer-9.ultralight.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0] > Closing the socket > >> > >> Can I run in gdb to get more info? What is good place to start poking? > >> > >> I was assuming it's the server cert that client does not like ... but it does > >> look ok to me :) > >> > >> Matevz > >> > >> > >> On 2/15/22 14:04, Andrew Hanushevsky wrote: > >>> Hi Bockjoo, > >>> > >>> Unfortunately, that's not the way it works. While gsi doesn't need to > use TLS > >>> ztn does. Since the erver doesn't know which protocol the client will > eventually > >>> settle on, the connection has to use TLS right from the start. That > means you > >>> cannot use ztn with incapable clients. > >>> > >>> Andy > >>> > >>> > >>> On Tue, 15 Feb 2022, Bockjoo Kim wrote: > >>> > >>>> Hi Andy, > >>>> > >>>> There are two sec.protocols: gsi and ztn. > >>>> > >>>> Doesn't the interaction go through gsi and if it fails, will it go > through ztn? > >>>> > >>>> For incapable clients, the gsi can succeed, no? > >>>> > >>>> Thanks, > >>>> > >>>> Bockjoo > >>>> > >>>> On 2/15/22 14:52, Andrew Hanushevsky wrote: > >>>>> Hi Justas, > >>>>> > >>>>> If you look into the log you will notice a warning that tells you that TLS > >>>>> will always be on regardless of the "capable" setting because > authentication > >>>>> protocol ztn requires tls. So, this may be the source of the problem, > >>>>> certainly it will be for incapable clients. > >>>>> > >>>>> Andy > >>>>> > >>>>> > >>>>> On Tue, 15 Feb 2022, Justas Balcas wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I took out 1 > server > >>>>>> from prod and played with full debug mode, on/off tls. > >>>>>> > >>>>>> Logs from server/client are available here: > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e=> > >>>>>> > >>>>>> > >>>>>> To turn TLS, I added this config: > >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem > >>>>>> /etc/grid-security/xrootd/xrootdkey.pem > >>>>>> xrd.tlsca certdir /etc/grid-security/certificates > >>>>>> xrootd.tls capable all > >>>>>> sec.protocol /usr/lib64 ztn > >>>>>> > >>>>>> And with TLS on - I always get: > >>>>>> TLS hand-shake exchange. > >>>>>> Socket error while handshaking: [FATAL] TLS error > >>>>>> Closing the socket > >>>>>> > >>>>>> > >>>>>> If it helps, here is full config: > >>>>>> > >>>>>> all.export /tmp stage > >>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN > >>>>>> all.adminpath /var/spool/xrootd > >>>>>> all.pidpath /var/run/xrootd > >>>>>> > >>>>>> # XrootD Security > >>>>>> # --------------------------------------- > >>>>>> xrootd.seclib /usr/lib64/libXrdSec.so > >>>>>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates > >>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem > >>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3 > >>>>>> -authzfun:libXrdLcmaps.so -authzto:900 > >>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 -gmapto:0 > >>>>>> acc.authdb /etc/xrootd/auth_file_stageout > >>>>>> ofs.authorize > >>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret > >>>>>> ofs.authlib ++ libXrdMacaroons.so > >>>>>> ofs.authlib ++ libXrdAccSciTokens.so > >>>>>> # -------------------------------------- > >>>>>> # XrootD Monitoring > >>>>>> # -------------------------------------- > >>>>>> # Monitoring for AAA Dashboard : > >>>>>> xrd.report 169.228.130.91:9931 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9931&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=AW4Xegl5wxbeERkKViynNkyDzu4pC5VhL7QR8sFGF44&e=> > every 30s all sync > >>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest > >>>>>> files io info user 169.228.130.91:9930 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=9w5jNESVzo8Kp4KzAUI0Ymeo-wcerH5BqhqE0Rfk6oU&e=> > dest fstat info user > >>>>>> xrd-mon.osgstorage.org:9930 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dmon.osgstorage.org-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=lMHFksodly_4DGpPCSpc0-Wb9UGjl1_8yif-C2TNQrI&e=> > >>>>>> all.sitename T2_US_Caltech > >>>>>> # ------------------------------------- > >>>>>> # Configure redirector/server > >>>>>> # ------------------------------------- > >>>>>> set xrdr = xrootd-redir-stageout.ultralight.org > <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrootd-2Dredir-2Dstageout.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=HTNBiz85qLXQGkB75MafMeY4xc6vwkMUMuatd5_g0mo&e=> > >>>>>> xrd.port 1095 > >>>>>> all.manager $(xrdr):1213 > >>>>>> > >>>>>> if $(xrdr) > >>>>>> # It's role is manager > >>>>>> all.role manager > >>>>>> # Redirect all lookup calls to original data servers. Redirector > does not > >>>>>> have visibility of FS > >>>>>> cms.dfs lookup distrib mdhold 20m redirect immed > >>>>>> else > >>>>>> # Role is server > >>>>>> all.role server > >>>>>> # The known managers (local redirector) > >>>>>> all.manager meta $(xrdr):1213 > >>>>>> > >>>>>> # Enable xrootd checksum calculation "on-the-fly" using multiuser plugin > >>>>>> # This makes XRootD to write the files with the > >>>>>> # ownership of the user that authenticated to the server and not as the > >>>>>> # 'xrootd' user > >>>>>> ofs.osslib ++ libXrdMultiuser.so > >>>>>> # Enable the checksum wrapper > >>>>>> ofs.ckslib * libXrdMultiuser.so > >>>>>> # Control of checksum > >>>>>> xrootd.chksum max 10 adler32 > >>>>>> multiuser.checksumonwrite on > >>>>>> multiuser.umask 0002 > >>>>>> > >>>>>> fi > >>>>>> # ------------------------------------- > >>>>>> # Allow only specific path, checksum config > >>>>>> # ------------------------------------- > >>>>>> # Allow any path to be exported; this is further refined in the authfile. > >>>>>> all.export / > >>>>>> > >>>>>> # Hosts allowed to use this xrootd cluster > >>>>>> cms.allow host * > >>>>>> > >>>>>> # Enable xrootd debugging > >>>>>> xrootd.trace emsg login stall redirect > >>>>>> cms.trace defer files forward redirect > >>>>>> > >>>>>> # Disable async. Related issue: > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e=> > >>>>>> > >>>>>> xrootd.async off > >>>>>> > >>>>>> # ------------------------------------- > >>>>>> # Integrate with CMS Namespaces > >>>>>> # It will see files under /store/... > >>>>>> # ------------------------------------- > >>>>>> oss.localroot /storage/cms > >>>>>> # ------------------------------------- > >>>>>> # Configure davs/https for TPC > >>>>>> # ------------------------------------- > >>>>>> # Enable https over XrootD > >>>>>> if exec xrootd > >>>>>> xrd.protocol http:1095 /usr/lib64/libXrdHttp.so > >>>>>> http.cadir /etc/grid-security/certificates > >>>>>> http.cert /etc/grid-security/xrootd/xrootdcert.pem > >>>>>> http.key /etc/grid-security/xrootd/xrootdkey.pem > >>>>>> http.secxtractor /usr/lib64/libXrdLcmaps.so > >>>>>> http.secretkey XXXXXXX > >>>>>> # Enable third-party-copy > >>>>>> http.exthandler xrdtpc libXrdHttpTPC.so > >>>>>> # Pass the bearer token to the Xrootd authorization framework. > >>>>>> http.header2cgi Authorization authz > >>>>>> http.listingdeny yes > >>>>>> http.desthttps yes > >>>>>> http.selfhttps2http no > >>>>>> http.staticpreload > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e= > <https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e=> > >>>>>> > >>>>>> /etc/xrootd/robots.txt > >>>>>> http.exthandler xrdmacaroons libXrdMacaroons.so > >>>>>> fi > >>>>>> > >>>>>> > >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem > >>>>>> /etc/grid-security/xrootd/xrootdkey.pem > >>>>>> xrd.tlsca certdir /etc/grid-security/certificates > >>>>>> xrootd.tls capable all > >>>>>> sec.protocol /usr/lib64 ztn > >>>>>> > >>>>>> > >>>>>> > >>>>>> xrootd.trace all > >>>>>> xrd.trace all > >>>>>> ofs.trace all > >>>>>> pfc.trace all > >>>>>> cms.trace all > >>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2 > >>>>>> Warning, 1 Info) > >>>>>> pss.setopt DebugLevel 4 > >>>>>> > >>>>>> ######################################################################## > >>>>>> Use REPLY-ALL to reply to list > >>>>>> > >>>>>> To unsubscribe from the XROOTD-L list, click the following link: > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> > >>>>>> > >>>>>> > >>>>> > >>>>> ######################################################################## > >>>>> Use REPLY-ALL to reply to list > >>>>> > >>>>> To unsubscribe from the XROOTD-L list, click the following link: > >>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> > >>>>> > >>>> > >>>> > >>>> ######################################################################## > >>>> Use REPLY-ALL to reply to list > >>>> > >>>> To unsubscribe from the XROOTD-L list, click the following link: > >>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> > >>>> > >>>> > >>> > >>> ######################################################################## > >>> Use REPLY-ALL to reply to list > >>> > >>> To unsubscribe from the XROOTD-L list, click the following link: > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e= > <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=> > >>> > >>> > >> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1