LISTSERV 16.5 - XROOTD-L Archives

On Wed, 16 Feb 2022 at 00:45, Petr Vokac <[log in to unmask]> wrote:

Proxy generated with 1024 key size seems to be mainly OSG problem,
because this size is used by default only by voms-proxy-init2. Those who
rely on voms-proxy-init3 > 3.3.2 (released in October 2020) or arcproxy
>= 6.4.0 (released in November 2019) gets by default 2048 keys.

Be aware there might still be software which "accidentally" generate
proxies with smaller than default size, e.g. FTS < 3.11.0 (latest
version released in September 2021) used fixed key size 1024
(https://its.cern.ch/jira/browse/FTS-1700).

Saying that it would be useful to have more clear error messages.

Petr

On 2/16/22 08:18, Matevz Tadel wrote:
> Andy and I did some deep debugging of where the cert init fails ... and it
> turned out the problem was 1024-bit key of the proxy cert on the client machine
> --- openssl-1.1.1k (which comes with cos-8) apparently requires the key to be at
> least 2048 bits when calling SSL_CTX_use_certificate_file().
>
> After we replaced it with my 2048-bit key cert it worked no problem.
>
> On fedora-34 the default key length generated by voms-proxy-init is already
> 2048, on centos-7.9 it is still 1024. One can get longer keys with -bits option:
> voms-proxy-init -bits 2048 -voms cms
>
> Andy will try to get more informative error messages from openssl ... but this
> doesn't seem entirely trivial as the error gets set at the top-level function
> and we had to go three levels down to see the actual error code that we were
> able to google.
>
> Matevz
>
> On 2/15/22 19:46, Justas Balcas wrote:
>> That is correct, Matevz
>>
>> On Tue, Feb 15, 2022, 18:39 Matevz Tadel <[log in to unmask]
>> <mailto:[log in to unmask]>> wrote:
>>
>> Thank you Andy!
>>
>> The error is permanent, I think. Justas, can you please confirm?
>>
>> Matevz
>>
>> On 2/15/22 17:49, Andrew Hanushevsky wrote:
>> > Here are the common problems when the client issues that message:
>> >
>> > 1) The certificate's chain of trust is broken (could be because the root CA
>> > can't be verified, or the root/intermediate certificate has expired). This
>> would
>> > cause sproradic failures (note that if the certificate is improperly installed
>> > then the failure would always occur which is not the case here).
>> >
>> > 2) The certificate must be renewed before the expiry of the certificate to
>> avoid
>> > any conflict arising out of time violation. Ensure that the date/time is set
>> > correctly on your computer since that might be used to assess the validity
>> > period of the SSL certificate of the website. This would cause repated
>> failures
>> > on some sites but not others depending on how close the cert was to expiring.
>> >
>> > 3) The certificate structure is broken, or the certificate's signature
>> can't be
>> > checked. This is the hardest to solve but one reason could me random memory
>> > corruption. I would suggest runningthe client with valgrind and try to
>> match the
>> > valgring log with the invalid cert message.
>> >
>> > 4) Firewall issues may cause problems with they have been enabled for
>> > "encrypted/SSL scanning or checking."
>> >
>> > 5) If tghe server is using only SHA-1 encryption then those cert are
>> flagged as
>> > insecure and need to update their security certificates. However, would most
>> > likely cause permanent errors.
>> >
>> > Andy
>> >
>> >
>> >
>> > On Tue, 15 Feb 2022, Matevz Tadel wrote:
>> >
>> >> Andy,
>> >>
>> >> What's with the invalid certificate in the client log?
>> >>
>> >> [2022-02-15 09:56:19.549139 -0800][Debug ][XRootDTransport   ]
>> >> [transfer-9.ultralight.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0]
>> Sending out kXR_login request, username:
>> >> root, cgi:
>> >> ?xrd.cc=us&xrd.tz
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.tz&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=-Rc4_LmnigwN9z03K7gZrf-B9w2_b0g2EkivTF0FV9U&e=>=-8&xrd.appname=xrdcp&xrd.info
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd.info&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=j_3eK4Ghw7zsEUpD0XXrLaG5_Maj8wMSMXpXUC-Up5Y&e=>=&xrd.hostname=xrd-cache-3.ultralight.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dcache-2D3.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=PYd32OF7sFBQVcQsXBH9TGn-f4DSZENRYSAySzIRVUE&e=>&xrd.rn=v5.2.0,
>> >> dual-stack: true, private IPv4: false, private IPv6: false
>> >> [2022-02-15 09:56:19.549209 -0800][Debug ][AsyncSock         ]
>> >> [transfer-9.ultralight.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0]
>> TLS hand-shake exchange.
>> >>
>> >> ===> HERE:
>> >> [2022-02-15 09:56:19.551762 -0800][Error ][TlsMsg            ]
>> [TLS_Context:]
>> >> Unable to create TLS context; invalid certificate.
>> >>
>> >> [2022-02-15 09:56:19.551903 -0800][Error ][AsyncSock         ]
>> >> [transfer-9.ultralight.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0]
>> Socket error while handshaking: [FATAL] TLS
>> >> error
>> >> [2022-02-15 09:56:19.551920 -0800][Debug ][AsyncSock         ]
>> >> [transfer-9.ultralight.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__transfer-2D9.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=nlq02y75DI9NFCQLEryIqIoDoyClnOqIZlWhu5rJJMc&e=>:1095.0]
>> Closing the socket
>> >>
>> >> Can I run in gdb to get more info? What is good place to start poking?
>> >>
>> >> I was assuming it's the server cert that client does not like ... but it does
>> >> look ok to me :)
>> >>
>> >> Matevz
>> >>
>> >>
>> >> On 2/15/22 14:04, Andrew Hanushevsky wrote:
>> >>> Hi Bockjoo,
>> >>>
>> >>> Unfortunately, that's not the way it works. While gsi doesn't need to
>> use TLS
>> >>> ztn does. Since the erver doesn't know which protocol the client will
>> eventually
>> >>> settle on, the connection has to use TLS right from the start. That
>> means you
>> >>> cannot use ztn with incapable clients.
>> >>>
>> >>> Andy
>> >>>
>> >>>
>> >>> On Tue, 15 Feb 2022, Bockjoo Kim wrote:
>> >>>
>> >>>> Hi Andy,
>> >>>>
>> >>>> There are two sec.protocols: gsi and ztn.
>> >>>>
>> >>>> Doesn't the interaction go through gsi and if it fails, will it go
>> through ztn?
>> >>>>
>> >>>> For incapable clients, the gsi can succeed, no?
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Bockjoo
>> >>>>
>> >>>> On 2/15/22 14:52, Andrew Hanushevsky wrote:
>> >>>>> Hi Justas,
>> >>>>>
>> >>>>> If you look into the log you will notice a warning that tells you that TLS
>> >>>>> will always be on regardless of the "capable" setting because
>> authentication
>> >>>>> protocol ztn requires tls. So, this may be the source of the problem,
>> >>>>> certainly it will be for incapable clients.
>> >>>>>
>> >>>>> Andy
>> >>>>>
>> >>>>>
>> >>>>> On Tue, 15 Feb 2022, Justas Balcas wrote:
>> >>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> Server's/Clients are running > 5.3.X release (not 5.4). I took out 1
>> server
>> >>>>>> from prod and played with full debug mode, on/off tls.
>> >>>>>>
>> >>>>>> Logs from server/client are available here:
>> >>>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e=
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__login-2D1.hep.caltech.edu_-7Ejbalcas_tls_&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=tns3GlZokUnAN9f_4ZHOR1kQXE0DrfqOgDK3bos6dS4&e=>
>> >>>>>>
>> >>>>>>
>> >>>>>> To turn TLS, I added this config:
>> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem
>> >>>>>> /etc/grid-security/xrootd/xrootdkey.pem
>> >>>>>> xrd.tlsca certdir /etc/grid-security/certificates
>> >>>>>> xrootd.tls capable all
>> >>>>>> sec.protocol /usr/lib64 ztn
>> >>>>>>
>> >>>>>> And with TLS on - I always get:
>> >>>>>> TLS hand-shake exchange.
>> >>>>>> Socket error while handshaking: [FATAL] TLS error
>> >>>>>> Closing the socket
>> >>>>>>
>> >>>>>>
>> >>>>>> If it helps, here is full config:
>> >>>>>>
>> >>>>>> all.export /tmp stage
>> >>>>>> frm.xfr.copycmd /bin/cp /dev/null $PFN
>> >>>>>> all.adminpath /var/spool/xrootd
>> >>>>>> all.pidpath /var/run/xrootd
>> >>>>>>
>> >>>>>> # XrootD Security
>> >>>>>> # ---------------------------------------
>> >>>>>> xrootd.seclib /usr/lib64/libXrdSec.so
>> >>>>>> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates
>> >>>>>> -cert:/etc/grid-security/xrootd/xrootdcert.pem
>> >>>>>> -key:/etc/grid-security/xrootd/xrootdkey.pem -crl:3
>> >>>>>> -authzfun:libXrdLcmaps.so -authzto:900
>> >>>>>> -authzfunparms:lcmapscfg=/etc/xrootd/lcmaps.cfg -gmapopt:10 -gmapto:0
>> >>>>>> acc.authdb /etc/xrootd/auth_file_stageout
>> >>>>>> ofs.authorize
>> >>>>>> macaroons.secretkey /etc/xrootd/macaroon-secret
>> >>>>>> ofs.authlib ++ libXrdMacaroons.so
>> >>>>>> ofs.authlib ++ libXrdAccSciTokens.so
>> >>>>>> # --------------------------------------
>> >>>>>> # XrootD Monitoring
>> >>>>>> # --------------------------------------
>> >>>>>> # Monitoring for AAA Dashboard :
>> >>>>>> xrd.report 169.228.130.91:9931
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9931&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=AW4Xegl5wxbeERkKViynNkyDzu4pC5VhL7QR8sFGF44&e=>
>> every 30s all sync
>> >>>>>> xrootd.monitor all auth flush 30s window 5s fstat 60 lfn ops xfr 5 dest
>> >>>>>> files io info user 169.228.130.91:9930
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__169.228.130.91-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=9w5jNESVzo8Kp4KzAUI0Ymeo-wcerH5BqhqE0Rfk6oU&e=>
>> dest fstat info user
>> >>>>>> xrd-mon.osgstorage.org:9930
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrd-2Dmon.osgstorage.org-3A9930&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=lMHFksodly_4DGpPCSpc0-Wb9UGjl1_8yif-C2TNQrI&e=>
>> >>>>>> all.sitename T2_US_Caltech
>> >>>>>> # -------------------------------------
>> >>>>>> # Configure redirector/server
>> >>>>>> # -------------------------------------
>> >>>>>> set xrdr = xrootd-redir-stageout.ultralight.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__xrootd-2Dredir-2Dstageout.ultralight.org&d=DwMFaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=9Kg_R4c2zF6HHZSpAb3h3VrMr_pFJQ260VSKabOPtkulYwNWdPSP-sd5LVhMjPpw&s=HTNBiz85qLXQGkB75MafMeY4xc6vwkMUMuatd5_g0mo&e=>
>> >>>>>> xrd.port 1095
>> >>>>>> all.manager $(xrdr):1213
>> >>>>>>
>> >>>>>> if $(xrdr)
>> >>>>>> # It's role is manager
>> >>>>>> all.role manager
>> >>>>>> # Redirect all lookup calls to original data servers. Redirector
>> does not
>> >>>>>> have visibility of FS
>> >>>>>> cms.dfs lookup distrib mdhold 20m redirect immed
>> >>>>>> else
>> >>>>>> # Role is server
>> >>>>>> all.role server
>> >>>>>> # The known managers (local redirector)
>> >>>>>> all.manager meta $(xrdr):1213
>> >>>>>>
>> >>>>>> # Enable xrootd checksum calculation "on-the-fly" using multiuser plugin
>> >>>>>> # This makes XRootD to write the files with the
>> >>>>>> # ownership of the user that authenticated to the server and not as the
>> >>>>>> # 'xrootd' user
>> >>>>>> ofs.osslib ++ libXrdMultiuser.so
>> >>>>>> # Enable the checksum wrapper
>> >>>>>> ofs.ckslib * libXrdMultiuser.so
>> >>>>>> # Control of checksum
>> >>>>>> xrootd.chksum max 10 adler32
>> >>>>>> multiuser.checksumonwrite on
>> >>>>>> multiuser.umask 0002
>> >>>>>>
>> >>>>>> fi
>> >>>>>> # -------------------------------------
>> >>>>>> # Allow only specific path, checksum config
>> >>>>>> # -------------------------------------
>> >>>>>> # Allow any path to be exported; this is further refined in the authfile.
>> >>>>>> all.export /
>> >>>>>>
>> >>>>>> # Hosts allowed to use this xrootd cluster
>> >>>>>> cms.allow host *
>> >>>>>>
>> >>>>>> # Enable xrootd debugging
>> >>>>>> xrootd.trace emsg login stall redirect
>> >>>>>> cms.trace defer files forward redirect
>> >>>>>>
>> >>>>>> # Disable async. Related issue:
>> >>>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e=
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_xrootd_xrootd_issues_1113&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=L2XeYWMRRqstD75CgZb1yCHO9dgWL2K6Uqmqto5rx_Q&e=>
>> >>>>>>
>> >>>>>> xrootd.async off
>> >>>>>>
>> >>>>>> # -------------------------------------
>> >>>>>> # Integrate with CMS Namespaces
>> >>>>>> # It will see files under /store/...
>> >>>>>> # -------------------------------------
>> >>>>>> oss.localroot /storage/cms
>> >>>>>> # -------------------------------------
>> >>>>>> # Configure davs/https for TPC
>> >>>>>> # -------------------------------------
>> >>>>>> # Enable https over XrootD
>> >>>>>> if exec xrootd
>> >>>>>> xrd.protocol http:1095 /usr/lib64/libXrdHttp.so
>> >>>>>> http.cadir /etc/grid-security/certificates
>> >>>>>> http.cert /etc/grid-security/xrootd/xrootdcert.pem
>> >>>>>> http.key /etc/grid-security/xrootd/xrootdkey.pem
>> >>>>>> http.secxtractor /usr/lib64/libXrdLcmaps.so
>> >>>>>> http.secretkey XXXXXXX
>> >>>>>> # Enable third-party-copy
>> >>>>>> http.exthandler xrdtpc libXrdHttpTPC.so
>> >>>>>> # Pass the bearer token to the Xrootd authorization framework.
>> >>>>>> http.header2cgi Authorization authz
>> >>>>>> http.listingdeny yes
>> >>>>>> http.desthttps yes
>> >>>>>> http.selfhttps2http no
>> >>>>>> http.staticpreload
>> >>>>>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e=
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__static_robots.txt&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=Laz4x7NvD_fTheDJu5m_cEES6_pePEidCLIAkrYNvPs&e=>
>> >>>>>>
>> >>>>>> /etc/xrootd/robots.txt
>> >>>>>> http.exthandler xrdmacaroons libXrdMacaroons.so
>> >>>>>> fi
>> >>>>>>
>> >>>>>>
>> >>>>>> xrd.tls /etc/grid-security/xrootd/xrootdcert.pem
>> >>>>>> /etc/grid-security/xrootd/xrootdkey.pem
>> >>>>>> xrd.tlsca certdir /etc/grid-security/certificates
>> >>>>>> xrootd.tls capable all
>> >>>>>> sec.protocol /usr/lib64 ztn
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> xrootd.trace all
>> >>>>>> xrd.trace    all
>> >>>>>> ofs.trace    all
>> >>>>>> pfc.trace    all
>> >>>>>> cms.trace    all
>> >>>>>> # To debug connections to the fedration (5 Dump, 4 Debug, 3 Error, 2
>> >>>>>> Warning, 1 Info)
>> >>>>>> pss.setopt   DebugLevel 4
>> >>>>>>
>> >>>>>> ########################################################################
>> >>>>>> Use REPLY-ALL to reply to list
>> >>>>>>
>> >>>>>> To unsubscribe from the XROOTD-L list, click the following link:
>> >>>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> ########################################################################
>> >>>>> Use REPLY-ALL to reply to list
>> >>>>>
>> >>>>> To unsubscribe from the XROOTD-L list, click the following link:
>> >>>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> ########################################################################
>> >>>> Use REPLY-ALL to reply to list
>> >>>>
>> >>>> To unsubscribe from the XROOTD-L list, click the following link:
>> >>>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=>
>> >>>>
>> >>>>
>> >>>
>> >>> ########################################################################
>> >>> Use REPLY-ALL to reply to list
>> >>>
>> >>> To unsubscribe from the XROOTD-L list, click the following link:
>> >>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__listserv.slac.stanford.edu_cgi-2Dbin_wa-3FSUBED1-3DXROOTD-2DL-26A-3D1&d=DwIDaQ&c=-35OiAkTchMrZOngvJPOeA&r=f2PhPg2_OoVvPAKGXfp4WG1YcdhQC9qsy2uMHw3Z_6k&m=cxGPTdnshuISdao1V4PsW-krXJuMxpk57T8cXetcP32UCNNa9pgqm--1NTcRuVNW&s=nmhVko_mWVLPvjSwEKtKkm17GDZSKRYlu7FW5xSiWAg&e=>
>> >>>
>> >>>
>> >>
>>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1