Hello!
Failures in file transfer cause job abortions. To understand what is
happening I conducted an experiment were I tested the command
xrdcp -d 1 -f <file> /dev/null
101 times for 2 files, in an attempt to identify servers guilty of
transfer failures.
The results are as follows:
File 1. Transfer failed in 4 cases, always associated with host
se01.indiacms.res.in
File 2. Transfer failed in 9 cases, always associated with host
pool0x.ifca.es
The full results are shown in the attachments.
I suppose the problem identified is related to our previous discussion,
where we identified incompatible xrootd versions as being the cause.
Viesturs
A 2015-08-23 01:33, Andrew Hanushevsky escreveu:
> The only practical solution is to contact the administrator. They are
> ultimately reponsible for what happens at the site and we really can't
> do
> much from the outside.
>
> Andy
>
> On Thu, 20 Aug 2015, Viesturs Veckalns wrote:
>
>> What could be a practical solution to this problem?
>> Viesturs
>>
>> A 2015-08-20 16:21, xrootd-dev escreveu:
>>> Dear Viesturs,
>>>
>>> I am referring to the end servers, for example maite.iihe.ac.be .
>>> I guess the only way is to contact the administrator of such a
> server.
>>>
>>> The version installed on your internal machines look fine.
>>>
>>> Gerardo GANIS
>>> CERN, PH Dept, SFT group
>>> [log in to unmask]
>>> tel/cell: (+41) 22 767 6439 / 75 411 1128
>>>
>>> On 20 Aug 2015, at 17:09, Viesturs Veckalns
> <[log in to unmask]>
>>> wrote:
>>>
>>>> Dear Gerardo,
>>>> The packages installed on our internal xroot servers are the
>>> following:
>>>> [root@xroot ~]# rpm -qa|egrep '(xroot|openssl)'|sort
>>>> cms-xrootd-1.2-9.osg32.el6.noarch
>>>> globus-gsi-openssl-error-3.5-1.osg32.el6.x86_64
>>>> globus-openssl-module-4.6-1.osg32.el6.x86_64
>>>> openssl098e-0.9.8e-18.el6_5.2.x86_64
>>>> openssl-1.0.1e-30.el6.11.x86_64
>>>> xrootd-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-client-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-client-devel-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-client-libs-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-cmstfc-1.5.1-10.osg32.el6.x86_64
>>>> xrootd-compat-3.3.6-1.el6.x86_64
>>>> xrootd-compat-client-libs-3.3.6-1.el6.x86_64
>>>> xrootd-compat-libs-3.3.6-1.el6.x86_64
>>>> xrootd-compat-server-libs-3.3.6-1.el6.x86_64
>>>> xrootd-devel-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-lcmaps-0.0.7-11.osg32.el6.x86_64
>>>> xrootd-libs-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-selinux-4.2.1-1.osg32.el6.noarch
>>>> xrootd-server-4.2.1-1.osg32.el6.x86_64
>>>> xrootd-server-libs-4.2.1-1.osg32.el6.x86_64
>>>>
>>>> and on the user interface:
>>>> [root@ui6-cms01 ~]# rpm -qa|egrep '(xroot|openssl)'|sort
>>>> globus-gsi-openssl-error-2.1-10.el6.i686
>>>> globus-gsi-openssl-error-2.1-10.el6.x86_64
>>>> globus-openssl-module-3.3-2.el6.i686
>>>> globus-openssl-module-3.3-2.el6.x86_64
>>>> nordugrid-arc-plugins-xrootd-4.2.0-1.el6.x86_64
>>>> openssl098e-0.9.8e-18.el6_5.2.i686
>>>> openssl098e-0.9.8e-18.el6_5.2.x86_64
>>>> openssl-1.0.1e-30.el6.11.i686
>>>> openssl-1.0.1e-30.el6.11.x86_64
>>>> xmlsec1-openssl-1.2.20-4.el6.x86_64
>>>> xrootd-client-libs-4.1.1-1.el6.x86_64
>>>> xrootd-compat-client-libs-3.3.6-1.el6.x86_64
>>>> xrootd-compat-libs-3.3.6-1.el6.x86_64
>>>> xrootd-libs-4.1.1-1.el6.x86_64
>>>>
>>>> Viesturs
>>>>
>>>> A 2015-08-20 10:13, xrootd-dev escreveu:
>>>>> Dear Viesturs,
>>>>>
>>>>> The second server (maite.iihe.ac.be) seems to be running a very
>>> old
>>>>> version of XRootD, older than 4 years.
>>>>> While we try to be backward compatible, on such a long periods
> and
>>>>> version ranges is very difficult. In particular,
>>>>> the version run is before the devs to openssl 1.x, which
>>> introduced a
>>>>> different hash technology.
>>>>>
>>>>> Could you find out which version is exactly run on the servers
>>> that
>>>>> give the problem?
>>>>>
>>>>> Gerardo GANIS
>>>>> CERN, PH Dept, SFT group
>>>>> [log in to unmask]
>>>>> tel/cell: (+41) 22 767 6439 / 75 411 1128
>>>>>
>>>>> On 19 Aug 2015, at 17:46, Viesturs Veckalns
>>> <[log in to unmask]>
>>>>> wrote:
>>>>>
>>>>>> Dear Gerardo,
>>>>>> I ran export XrdSecDEBUG=2
>>>>>>
>>>>>> and subsequently got 2 different results:
>>>>>> 1. I don't know if the process ran well, but the file apparently
>>> was
>>>>>
>>>>>> read in the end - see testfile.txt.
>>>>>> 2. The same error as before - Authentication with gsi failed:
>>> Your
>>>>>> certificate's issuer is not trusted, that ended with Run:
> [FATAL]
>>>>>> Redirect limit has been reached - see testfile2.txt
>>>>>>
>>>>>> I think LIP CA certificate is in the directory
>>>>>> /etc/grid-security/certificates because it contains a series of
>>>>> files:
>>>>>>
>>>>>> LIPCA.crl_url
>>>>>> LIPCA.info
>>>>>> LIPCA.namespaces
>>>>>> LIPCA.pem
>>>>>> LIPCA.signing_policy
>>>>>>
>>>>>> Cheers,
>>>>>> Viesturs
>>>>>>
>>>>>>
>>>>>> A 2015-08-19 16:04, xrootd-dev escreveu:
>>>>>>> Dear Viesturs,
>>>>>>>
>>>>>>> Can you set
>>>>>>>
>>>>>>> $ export XrdSecDEBUG=2
>>>>>>>
>>>>>>> and run it again and post the result?
>>>>>>>
>>>>>>> Is the LIP CA certificate in the relevant CA directory, e.g.
>>>>>>> /etc/grid-security/certificates ?
>>>>>>>
>>>>>>> Gerardo GANIS
>>>>>>> CERN, PH Dept, SFT group
>>>>>>> [log in to unmask]
>>>>>>> tel/cell: (+41) 22 767 6439 / 75 411 1128
>>>>>>>
>>>>>>> On 19 Aug 2015, at 15:24, Viesturs Veckalns
>>>>> <[log in to unmask]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello!
>>>>>>>> I have a grid certificate issued by LIP CA, that is correctly
>>>>> mapped
>>>>>>> with CERN SiteDB.
>>>>>>>> Unfortunately, it is impossible to receive files over grid
>>>>>>> (certificate's issuer is not trusted being cited as a reason).
>>> A
>>>>>>> complete debug output is copied below.
>>>>>>>>
>>>>>>>> Our IT service thinks it is a problem with the xrootd service.
>>>>>>>> Could you help?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Viesturs
>>>>>>>>
>>>>>>>> xrdcp -d 1 -f
>>>>>>>
>>>>>
>>>
> root://cms-xrd-global.cern.ch//store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root
>>>>>>> /dev/null
>>>>>>>> [2015-08-19 14:13:13.772928 +0100][Error ][XRootDTransport ]
>>>>>>> [maite.iihe.ac.be:1094 #0.0] Authentication with gsi failed:
>>> Your
>>>>>>> certificate's issuer is not trusted.
>>>>>>>> [2015-08-19 14:13:13.773148 +0100][Error ][XRootDTransport ]
>>>>>>> [maite.iihe.ac.be:1094 #0.0] No protocols left to try
>>>>>>>> [2015-08-19 14:13:13.773181 +0100][Error ][AsyncSock ]
>>>>>>> [maite.iihe.ac.be:1094 #0.0] Socket error while handshaking:
>>>>> [FATAL]
>>>>>>> Auth failed
>>>>>>>> [2015-08-19 14:13:13.773273 +0100][Error ][PostMaster ]
>>>>>>> [maite.iihe.ac.be:1094 #0] Unable to recover: [FATAL] Auth
>>> failed.
>>>>>>>> [2015-08-19 14:13:13.773299 +0100][Error ][XRootD ]
>>>>>>> [maite.iihe.ac.be:1094] Impossible to send message kXR_open
>>> (file:
>>>>>>>
>>>>>
>>>
> /store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root,
>>>>>>> mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying
>>> to
>>>>>>> recover.
>>>>>>>> [2015-08-19 14:13:13.773327 +0100][Error ][XRootD ]
>>>>>>> [maite.iihe.ac.be:1094] Handling error while processing
>>> kXR_open
>>>>>>> (file:
>>>>>>>
>>>>>
>>>
> /store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root,
>>>>>>> mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL]
>>>>> Auth
>>>>>>> failed.
>>>>>>>> [2015-08-19 14:13:14.111618 +0100][Error ][XRootD ]
>>>>>>> [maite.iihe.ac.be:1094] Handling error while processing
>>> kXR_open
>>>>>>> (file:
>>>>>>>
>>>>>
>>>
> /store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root?tried=maite.iihe.ac.be,maite.iihe.ac.be,xrootd-redic.pi.infn.it,
>>>>>>> mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL]
>>>>> Auth
>>>>>>> failed.
>>>>>>>> [2015-08-19 14:13:14.949391 +0100][Error ][XRootDTransport ]
>>>>>>> [dcache-cms-xrootd.desy.de:1094 #0.0] Authentication with gsi
>>>>> failed:
>>>>>>> Your certificate's issuer is not trusted.
>>>>>>>> [2015-08-19 14:13:14.949536 +0100][Error ][XRootDTransport ]
>>>>>>> [dcache-cms-xrootd.desy.de:1094 #0.0] No protocols left to try
>>>>>>>> [2015-08-19 14:13:14.949565 +0100][Error ][AsyncSock ]
>>>>>>> [dcache-cms-xrootd.desy.de:1094 #0.0] Socket error while
>>>>> handshaking:
>>>>>>> [FATAL] Auth failed
>>>>>>>> [2015-08-19 14:13:14.949644 +0100][Error ][PostMaster ]
>>>>>>> [dcache-cms-xrootd.desy.de:1094 #0] Unable to recover: [FATAL]
>>>>> Auth
>>>>>>> failed.
>>>>>>>> [2015-08-19 14:13:14.949662 +0100][Error ][XRootD ]
>>>>>>> [dcache-cms-xrootd.desy.de:1094] Impossible to send message
>>>>> kXR_open
>>>>>>> (file:
>>>>>>>
>>>>>
>>>
> /store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root?tried=maite.iihe.ac.be,maite.iihe.ac.be,xrootd-redic.pi.infn.it,maite.iihe.ac.be,maite.iihe.ac.be,xrootd.ba.infn.it,
>>>>>>> mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying
>>> to
>>>>>>> recover.
>>>>>>>> [2015-08-19 14:13:14.949688 +0100][Error ][XRootD ]
>>>>>>> [dcache-cms-xrootd.desy.de:1094] Handling error while
>>> processing
>>>>>>> kXR_open (file:
>>>>>>>
>>>>>
>>>
> /store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root?tried=maite.iihe.ac.be,maite.iihe.ac.be,xrootd-redic.pi.infn.it,maite.iihe.ac.be,maite.iihe.ac.be,xrootd.ba.infn.it,
>>>>>>> mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL]
>>>>> Auth
>>>>>>> failed.
>>>>>>>> [2015-08-19 14:13:15.181594 +0100][Error ][XRootD ]
>>>>>>> [maite.iihe.ac.be:1094] Handling error while processing
>>> kXR_open
>>>>>>> (file:
>>>>>>>
>>>>>
>>>
> /store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v4/10000/00D2A247-2910-E511-9F3D-0CC47A4DEDD2.root?tried=+1213xrootd-redic.pi.infn.it,
>>>>>>> mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL]
>>>>> Auth
>>>>>>> failed.
>>>>>>>>
>>>>>>>
>>>>>
>>>
> [0B/0B][100%][==================================================][0B/s]
>>>>>>>
>>>>>>>> Run: [FATAL] Redirect limit has been reached
>>>>>>>>
>>>>>>>> ÿÿ
>>>>>>>> Reply to this email directly or view it on GitHub.
>>>>>>>>
>>>>>>>>
>>>>>>>> Use REPLY-ALL to reply to list
>>>>>>>>
>>>>>>>> To unsubscribe from the XROOTD-DEV list, click the following
>>>>> link:
>>>>>>>>
>>>>>
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Reply to this email directly or view it on GitHub [1].
>>>>>>>
>>>>>>> *
>>>>>>>
>>>>>>> Links:
>>>>>>> ------
>>>>>>> [1]
>>>>>
> https://github.com/xrootd/xrootd/issues/279#issuecomment-132631599
>>>>>> ÿÿ
>>>>>> Reply to this email directly or view it on GitHub.
>>>>>>
>>>>>>
>>>>>> Use REPLY-ALL to reply to list
>>>>>>
>>>>>> To unsubscribe from the XROOTD-DEV list, click the following
>>> link:
>>>>>>
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>>>
>>>>>
>>>>> --
>>>>> Reply to this email directly or view it on GitHub [1].
>>>>>
>>>>> *
>>>>>
>>>>> Links:
>>>>> ------
>>>>> [1]
>>> https://github.com/xrootd/xrootd/issues/279#issuecomment-132947363
>>>> ÿÿ
>>>> Reply to this email directly or view it on GitHub.
>>>>
>>>>
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>
>>>
>>> --
>>> Reply to this email directly or view it on GitHub [1].
>>>
>>> *
>>>
>>> Links:
>>> ------
>>> [1]
> https://github.com/xrootd/xrootd/issues/279#issuecomment-133047733
>>
>>
>> ---
>> Reply to this email directly or view it on GitHub:
>> https://github.com/xrootd/xrootd/issues/279#issuecomment-133066775
>
> --
> Reply to this email directly or view it on GitHub [1].
>
> *
>
> Links:
> ------
> [1] https://github.com/xrootd/xrootd/issues/279#issuecomment-133767693
Test command:
xrdcp -d 1 -f root://cms-xrd-global.cern.ch//store/mc/Phys14DR/DYJetsToLL_M-50_13TeV-madgraph-pythia8/MINIAODSIM/PU20bx25_PHYS14_25_V1-v1/00000/A266FB5C-796C-E411-B6EE-0025901D493E.root /dev/null
dataset: /DYJetsToLL_M-50_13TeV-madgraph-pythia8/Phys14DR-PU20bx25_PHYS14_25_V1-v1/MINIAODSIM"
author: Viesturs Veckalns [log in to unmask]
date: October 5, 2015
No. Server Transfer failed True/False
-------------------------------------------------------------------------
0 s17n02.hep.wisc.edu False
1 s17n02.hep.wisc.edu False
2 NONE SHOWN
3 nute.csc.fi False
4 hask.csc.fi False
5 s17n02.hep.wisc.edu False
6 s17n02.hep.wisc.edu False
7 xrootd.rcac.purdue.edu False
8 xrootd.rcac.purdue.edu False
9 s17n02.hep.wisc.edu False
10 xrootd.rcac.purdue.edu False
11 s17n02.hep.wisc.edu False
12 s17n02.hep.wisc.edu False
13 xrootd.rcac.purdue.edu False
14 s17n02.hep.wisc.edu False
15 xrootd.rcac.purdue.edu False
16 xrootd.rcac.purdue.edu False
17 xrootd.rcac.purdue.edu False
18 cmseos.fnal.gov False
19 s17n02.hep.wisc.edu False
20 s17n02.hep.wisc.edu False
21 cmseos.fnal.gov False
22 eoscms.cern.ch False
23 xrootd.rcac.purdue.edu False
24 s17n02.hep.wisc.edu False
25 s17n02.hep.wisc.edu False
26 nute.csc.fi False
27 hask.csc.fi False
28 nute.csc.fi False
29 xrootd.rcac.purdue.edu False
30 s17n02.hep.wisc.edu False
31 xrootd.rcac.purdue.edu False
32 xrootd.rcac.purdue.edu False
33 s17n02.hep.wisc.edu False
34 xrootd.rcac.purdue.edu False
35 se01.indiacms.res.in True
36 xrootd.rcac.purdue.edu False
37 xrootd.rcac.purdue.edu False
38 cmseos.fnal.gov False
39 s17n02.hep.wisc.edu False
40 se01.indiacms.res.in True
41 cmseos.fnal.gov False
42 xrootd.rcac.purdue.edu False
43 se01.indiacms.res.in True
44 xrootd.rcac.purdue.edu False
45 xrootd.rcac.purdue.edu False
46 s17n02.hep.wisc.edu False
47 xrootd.rcac.purdue.edu False
48 s17n02.hep.wisc.edu False
49 s17n02.hep.wisc.edu False
50 xrootd.rcac.purdue.edu False
51 xrootd.rcac.purdue.edu False
52 s17n02.hep.wisc.edu False
53 s17n02.hep.wisc.edu False
54 cmseos.fnal.gov False
55 NONE SHOWN
56 eoscms.cern.ch False
57 se01.indiacms.res.in True
58 NONE SHOWN
59 NONE SHOWN
60 cmseos.fnal.gov False
61 NONE SHOWN
62 NONE SHOWN
63 NONE SHOWN
64 NONE SHOWN
65 NONE SHOWN
66 cmseos.fnal.gov False
67 cmseos.fnal.gov False
68 NONE SHOWN
69 NONE SHOWN
70 s17n02.hep.wisc.edu False
71 s17n02.hep.wisc.edu False
72 s17n02.hep.wisc.edu False
73 s17n02.hep.wisc.edu False
74 cmseos.fnal.gov False
75 NONE SHOWN
76 s17n02.hep.wisc.edu False
77 s17n02.hep.wisc.edu False
78 cmseos.fnal.gov False
79 s17n02.hep.wisc.edu False
80 cmseos.fnal.gov False
81 s17n02.hep.wisc.edu False
82 s17n02.hep.wisc.edu False
83 s17n02.hep.wisc.edu False
84 s17n02.hep.wisc.edu False
85 s17n02.hep.wisc.edu False
86 s17n02.hep.wisc.edu False
87 s17n02.hep.wisc.edu False
88 s17n02.hep.wisc.edu False
89 s17n02.hep.wisc.edu False
90 s17n02.hep.wisc.edu False
91 s17n02.hep.wisc.edu False
92 s17n02.hep.wisc.edu False
93 s17n02.hep.wisc.edu False
94 s17n02.hep.wisc.edu False
95 s17n02.hep.wisc.edu False
96 cmseos.fnal.gov False
97 cmseos.fnal.gov False
98 s17n02.hep.wisc.edu False
99 s17n02.hep.wisc.edu False
100 s17n02.hep.wisc.edu False
Test command
xrdcp -d 1 -f root://cms-xrd-global.cern.ch//store/mc/RunIISpring15DR74/WW_TuneCUETP8M1_13TeV-pythia8/MINIAODSIM/Asympt50ns_MCRUN2_74_V9A-v1/10000/A89CCB89-4106-E511-BB7C-00259073E4A0.root /dev/null
dataset: /WW_TuneCUETP8M1_13TeV-pythia8/RunIISpring15DR74-Asympt50ns_MCRUN2_74_V9A-v1/MINIAODSIM
author: Viesturs Veckalns [log in to unmask]
date: October 5, 2015
No. Server Transfer failed True/False
-------------------------------------------------------------------------
0 pool04.ifca.es False
1 NONE SHOWN
2 NONE SHOWN
3 NONE SHOWN
4 cabinet-7-7-0.t2.ucsd.edu False
5 se1.accre.vanderbilt.edu False
6 NONE SHOWN
7 NONE SHOWN
8 NONE SHOWN
9 NONE SHOWN
10 f01-065-135-e.gridka.de False
11 se1.accre.vanderbilt.edu False
12 se1.accre.vanderbilt.edu False
13 pool06.ifca.es True
14 se1.accre.vanderbilt.edu False
15 NONE SHOWN
16 se1.accre.vanderbilt.edu False
17 NONE SHOWN
18 se1.accre.vanderbilt.edu False
19 se1.accre.vanderbilt.edu False
20 pool03.ifca.es False
21 se1.accre.vanderbilt.edu False
22 pool05.ifca.es False
23 pool03.ifca.es True
24 pool03.ifca.es True
25 pool01.ifca.es True
26 pool01.ifca.es True
27 pool02.ifca.es False
28 se1.accre.vanderbilt.edu False
29 pool01.ifca.es True
30 se1.accre.vanderbilt.edu False
31 pool07.ifca.es True
32 se1.accre.vanderbilt.edu False
33 se1.accre.vanderbilt.edu False
34 pool08.ifca.es False
35 pool02.ifca.es True
36 NONE SHOWN
37 pool01.ifca.es True
38 pool06.ifca.es False
39 NONE SHOWN
40 NONE SHOWN
41 NONE SHOWN
42 NONE SHOWN
43 NONE SHOWN
44 NONE SHOWN
45 NONE SHOWN
46 NONE SHOWN
47 se1.accre.vanderbilt.edu False
48 NONE SHOWN
49 NONE SHOWN
50 se1.accre.vanderbilt.edu False
51 NONE SHOWN
52 NONE SHOWN
53 NONE SHOWN
54 NONE SHOWN
55 NONE SHOWN
56 NONE SHOWN
57 NONE SHOWN
58 NONE SHOWN
59 NONE SHOWN
60 NONE SHOWN
61 NONE SHOWN
62 NONE SHOWN
63 NONE SHOWN
64 NONE SHOWN
65 NONE SHOWN
66 NONE SHOWN
67 NONE SHOWN
68 NONE SHOWN
69 NONE SHOWN
70 NONE SHOWN
71 se1.accre.vanderbilt.edu False
72 NONE SHOWN
73 NONE SHOWN
74 NONE SHOWN
75 NONE SHOWN
76 NONE SHOWN
77 NONE SHOWN
78 NONE SHOWN
79 NONE SHOWN
80 NONE SHOWN
81 NONE SHOWN
82 NONE SHOWN
83 NONE SHOWN
84 NONE SHOWN
85 NONE SHOWN
86 NONE SHOWN
87 NONE SHOWN
88 NONE SHOWN
89 NONE SHOWN
90 NONE SHOWN
91 NONE SHOWN
92 NONE SHOWN
93 NONE SHOWN
94 NONE SHOWN
95 NONE SHOWN
96 NONE SHOWN
97 NONE SHOWN
98 NONE SHOWN
99 NONE SHOWN
100 NONE SHOWN
—
Reply to this email directly or view it on GitHub.
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1