Indeed a core dump is usually the thing we need.
--
Wei Yang | [log in to unmask] | 650-926-3338(O)
On 3/21/20, 12:19 PM, "[log in to unmask] on behalf of Horst Severini" <[log in to unmask] on behalf of [log in to unmask]> wrote:
We're running 4.11.1 here at OU.
Cheers,
Horst
Albert Rossi <[log in to unmask]> wrote:
> Hi Horst,
>
> actually, if you notice, all endpoints failed on today's test. So it was not just OU.
>
> The Stanford developers may want you to run a few commands over the core file from gdb once you have it in hand.
>
> What version of xrootd are you running, just out of curiosity? Is it bleeding-edge, or a stable release?
>
> Cheers, Al
>
> ________________________________________________
> Albert L. Rossi
> Application Developer & Systems Analyst III
> Scientific Computing Division, Data Movement Development
> FCC 229A
> Mail Station 369 (FCC 2W)
> Fermi National Accelerator Laboratory
> Batavia, IL 60510
> (630) 840-3023
> ________________________________
> From: Horst Severini <[log in to unmask]>
> Sent: Saturday, March 21, 2020 1:18 PM
> To: [log in to unmask] <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; Albert Rossi <[log in to unmask]>
> Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT
>
> Hi Al,
>
> thanks, good idea. I'll save the next core file.
>
> I'm pretty sure the authentication failures simply came because
> that partition was full and no new proxies or what not could be
> created, so I wouldn't worry about that.
>
> Thanks,
>
> Horst
>
> Albert Rossi <[log in to unmask]> wrote:
>
> > Hi Horst,
> >
> > I would definitely report this to [log in to unmask]
> >
> > As for why the massive authentication failure, I've seen this before, it might have to do with CA cert issues.
> >
> > Cheers, Al
> >
> > ________________________________________________
> > Albert L. Rossi
> > Application Developer & Systems Analyst III
> > Scientific Computing Division, Data Movement Development
> > FCC 229A
> > Mail Station 369 (FCC 2W)
> > Fermi National Accelerator Laboratory
> > Batavia, IL 60510
> > (630) 840-3023
> > ________________________________
> > From: Horst Severini <[log in to unmask]>
> > Sent: Saturday, March 21, 2020 11:19 AM
> > To: [log in to unmask] <[log in to unmask]>
> > Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT
> >
> > Hi all,
> >
> > our /var/ partition had filled up because of too many xrootd core dumps.
> > I cleared those up and restarted xrootd, and things look better again now.
> >
> > Not sure why we keep getting core dumps, though.
> >
> > Cheers,
> >
> > Horst
> >
> > On 3/21/20 5:01 AM, [log in to unmask] wrote:
> > > XROOTD SMOKE TEST SUMMARY
> > > 2020-03-21 10:01:46 GMT
> > >
> > > Client: bogus6.fnal.gov
> > >
> > > XrootD version: v4.11.2
> > >
> > > Reference server: CERN-TRUNK
> > >
> > > Credential delegation: ON
> > >
> > > Checksum: -C adler32
> > >
> > > Total number of round-trip tests: 21
> > >
> > > --------------------------------SOUND ENDPOINTS---------------------------------
> > >
> > > SCORE ENDPT TYPE UP SRC DST DN
> > > --------------------------------------------------------------------------------
> > >
> > > -----------------------------PROBLEMATIC ENDPOINTS------------------------------
> > >
> > > SCORE ENDPT TYPE UP SRC DST DN
> > > --------------------------------------------------------------------------------
> > > 19 BRUSSELS dCache F - F F 0/4
> > > 19 CERN-EOS EOS F - F F 0/4
> > > 19 CERN-TRUNK DPM - F - F 0/2
> > > 19 DESY-PROM dCache F - F F 0/4
> > > 19 FNAL dCache F - F F 0/4
> > > 19 IN2P3-DOMA xrootd F - F F 0/4
> > > 19 PRAGUE DPM F - F F 0/4
> > > 19 RAL-CEPH CEPH F - F F 0/4
> > > 19 RAL-LCG2 Echo F - F F 0/4
> > > 19 SLAC XrootD F - F F 0/4
> > > 19 TRIUMF dCache F - F F 0/4
> > > 19 UKI-BRUNEL DPM F - F F 0/4
> > > 19 UKI-LANC DPM F - F F 0/4
> > > 19 UKI-MAN1 DPM F - F F 0/4
> > > 19 UKI-MAN2 DPM F - F F 0/4
> > > 19 UNI-BONN CephFS F - F F 0/4
> > > 18 OU XrootD F - F F 0/4
> > > 13 BNL dCache F - F F 0/4
> > > 7 AGLT2 dCache F - F F 0/4
> > > 0 CALTECH HDFS F - F F 0/4
> > > 0 TRIUMF-PROD dCache F - F F 0/4
> > >
> > > --------------------------------ERROR DETAILS-----------------------------------
> > >
> > > [1] BRUSSELS
> > > UPLOAD (round-trip-15-upload): 000186 : [2020-03-21 05:01:06.216990 -0500][Error ][XRootD ] [maite.iihe.ac.be:1094] Unable to send the message kXR_open (file: pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-15-tpc-dst-d): 000195 : [2020-03-21 05:01:06.719178 -0500][Error ][Utility ] Unable to open target root://maite.iihe.ac.be:1094/pnfs/iihe//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=maite.iihe.ac.be:1094&tpc.dlgon=1&tpc.key=247f6a10a6b5a9305e75e5e2&tpc.lfn=pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=maite.iihe.ac.be:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-15-download): 000110 : [2020-03-21 05:01:07.460848 -0500][Error ][XRootD ] [maite.iihe.ac.be:1094] Impossible to send message kXR_open (file: pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [2] CERN-EOS
> > > UPLOAD (round-trip-10-upload): 000218 : Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/opstest/tpc/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04; Operation not permitted
> > > TPC_DST_D (round-trip-10-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3011] Unable to remove /eos/opstest/tpc/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; No such file or directory
> > > DOWNLOAD (round-trip-10-download): 000129 : Run: [ERROR] Server responded with an error: [3011] Unable to open file /eos/opstest/tpc/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04; No such file or directory
> > >
> > > [3] CERN-TRUNK
> > > TPC_SRC_D (round-trip-8-tpc-src-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/cern.ch/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> > > DOWNLOAD (round-trip-8-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > >
> > > [4] DESY-PROM
> > > UPLOAD (round-trip-9-upload): 000186 : [2020-03-21 05:00:47.833821 -0500][Error ][XRootD ] [prometheus.desy.de:1095] Unable to send the message kXR_open (file: /VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-9-tpc-dst-d): 000195 : [2020-03-21 05:00:48.337679 -0500][Error ][Utility ] Unable to open target root://prometheus.desy.de:1095//VOs/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=prometheus.desy.de:1095&tpc.dlgon=1&tpc.key=140c2808a6b5a81d5e75e5d0&tpc.lfn=/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=prometheus.desy.de:1095&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-9-download): 000110 : [2020-03-21 05:00:49.246017 -0500][Error ][XRootD ] [prometheus.desy.de:1095] Impossible to send message kXR_open (file: /VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [5] FNAL
> > > UPLOAD (round-trip-14-upload): 000186 : [2020-03-21 05:01:05.136590 -0500][Error ][XRootD ] [stkendca06a.fnal.gov:1094] Unable to send the message kXR_open (file: /pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-14-tpc-dst-d): 000195 : [2020-03-21 05:01:05.235583 -0500][Error ][Utility ] Unable to open target root://stkendca06a.fnal.gov:1094//pnfs/fnal.gov/VOs/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=stkendca06a.fnal.gov:1094&tpc.dlgon=1&tpc.key=0de3b5b8a6b5a9045e75e5e1&tpc.lfn=/pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=stkendca06a.fnal.gov:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-14-download): 000110 : [2020-03-21 05:01:05.444704 -0500][Error ][XRootD ] [stkendca06a.fnal.gov:1094] Impossible to send message kXR_open (file: /pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [6] IN2P3-DOMA
> > > UPLOAD (round-trip-20-upload): 000186 : [2020-03-21 05:01:42.067389 -0500][Error ][XRootD ] [ccxrootddoma.in2p3.fr:1094] Unable to send the message kXR_open (file: /xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-20-tpc-dst-d): 000195 : [2020-03-21 05:01:42.585936 -0500][Error ][Utility ] Unable to open target root://ccxrootddoma.in2p3.fr:1094//xrootd/dteam/doma/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ccxrootddoma.in2p3.fr:1094&tpc.dlgon=1&tpc.key=1c56a380a6b5aa0c5e75e606&tpc.lfn=/xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ccosvms0124.in2p3.fr:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-20-download): 000110 : [2020-03-21 05:01:43.407723 -0500][Error ][XRootD ] [ccxrootddoma.in2p3.fr:1094] Impossible to send message kXR_open (file: /xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [7] PRAGUE
> > > UPLOAD (round-trip-3-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > > TPC_DST_D (round-trip-3-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/farm.particle.cz/home/dteam/tpc-test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> > > DOWNLOAD (round-trip-3-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > >
> > > [8] RAL-CEPH
> > > UPLOAD (round-trip-2-upload): 000186 : [2020-03-21 05:00:18.701154 -0500][Error ][XRootD ] [ceph-test-gw683.gridpp.rl.ac.uk:1094] Unable to send the message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-2-tpc-dst-d): 000195 : [2020-03-21 05:00:19.190035 -0500][Error ][Utility ] Unable to open target root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ceph-test-gw683.gridpp.rl.ac.uk:1094&tpc.dlgon=1&tpc.key=05805ef8a6b5a6f85e75e5b3&tpc.lfn=/dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ceph-test-gw683.gridpp.rl.ac.uk:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-2-download): 000110 : [2020-03-21 05:00:19.969919 -0500][Error ][XRootD ] [ceph-test-gw683.gridpp.rl.ac.uk:1094] Impossible to send message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [9] RAL-LCG2
> > > UPLOAD (round-trip-1-upload): 000356 : [2020-03-21 05:00:05.333273 -0500][Error ][XRootD ] [echo.stfc.ac.uk:1094] Unable to send the message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-1-tpc-dst-d): 000365 : [2020-03-21 05:00:07.998168 -0500][Error ][Utility ] Unable to open target root://echo.stfc.ac.uk:1094//dteam:test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=echo.stfc.ac.uk:1094&tpc.dlgon=1&tpc.key=35a6b208a6b5a6cb5e75e5a7&tpc.lfn=/dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=echo.stfc.ac.uk:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-1-download): 000280 : [2020-03-21 05:00:13.049330 -0500][Error ][XRootD ] [echo.stfc.ac.uk:1094] Impossible to send message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [10] SLAC
> > > UPLOAD (round-trip-11-upload): 000186 : [2020-03-21 05:00:58.184767 -0500][Error ][XRootD ] [griddev03.slac.stanford.edu:2094] Unable to send the message kXR_open (file: /xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-11-tpc-dst-d): 000195 : [2020-03-21 05:00:58.501336 -0500][Error ][Utility ] Unable to open target root://griddev03.slac.stanford.edu:2094//xrootd/atlas/tpctest/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=griddev03.slac.stanford.edu:2094&tpc.dlgon=1&tpc.key=1ad90c00a6b5a8755e75e5da&tpc.lfn=/xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=griddev03.slac.stanford.edu:2094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-11-download): 000110 : [2020-03-21 05:00:58.987351 -0500][Error ][XRootD ] [griddev03.slac.stanford.edu:2094] Impossible to send message kXR_open (file: /xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [11] TRIUMF
> > > UPLOAD (round-trip-16-upload): 000186 : [2020-03-21 05:01:08.742011 -0500][Error ][XRootD ] [ppshead.lcg.triumf.ca:1094] Unable to send the message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-16-tpc-dst-d): 000195 : [2020-03-21 05:01:09.056302 -0500][Error ][Utility ] Unable to open target root://ppshead.lcg.triumf.ca:1094//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ppshead.lcg.triumf.ca:1094&tpc.dlgon=1&tpc.key=000fa3e8a6b5a95c5e75e5e5&tpc.lfn=/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ppshead.lcg.triumf.ca:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-16-download): 000110 : [2020-03-21 05:01:09.513921 -0500][Error ][XRootD ] [ppshead.lcg.triumf.ca:1094] Impossible to send message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [12] UKI-BRUNEL
> > > UPLOAD (round-trip-4-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > > TPC_DST_D (round-trip-4-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/brunel.ac.uk/home/dteam/wlcg-tpc/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> > > DOWNLOAD (round-trip-4-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > >
> > > [13] UKI-LANC
> > > UPLOAD (round-trip-7-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > > TPC_DST_D (round-trip-7-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/lancs.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> > > DOWNLOAD (round-trip-7-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > >
> > > [14] UKI-MAN1
> > > UPLOAD (round-trip-5-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > > TPC_DST_D (round-trip-5-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/tier2.hep.manchester.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> > > DOWNLOAD (round-trip-5-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > >
> > > [15] UKI-MAN2
> > > UPLOAD (round-trip-6-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > > TPC_DST_D (round-trip-6-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/tier2.hep.manchester.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> > > DOWNLOAD (round-trip-6-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> > >
> > > [16] UNI-BONN
> > > UPLOAD (round-trip-13-upload): 000186 : [2020-03-21 05:01:01.114829 -0500][Error ][XRootD ] [xrootd.physik.uni-bonn.de:1094] Unable to send the message kXR_open (file: /cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-13-tpc-dst-d): 000197 : [2020-03-21 05:01:03.253425 -0500][Error ][Utility ] Unable to open target root://xrootd.physik.uni-bonn.de:1094//cephfs/grid/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.physik.uni-bonn.de:1094&tpc.dlgon=1&tpc.key=1ffebc20a6b5a8cd5e75e5dd&tpc.lfn=/cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.physik.uni-bonn.de:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-13-download): 000110 : [2020-03-21 05:01:04.113281 -0500][Error ][XRootD ] [xrootd.physik.uni-bonn.de:1094] Impossible to send message kXR_open (file: /cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [17] OU
> > > UPLOAD (round-trip-12-upload): 000186 : [2020-03-21 05:00:59.696352 -0500][Error ][XRootD ] [se1.oscer.ou.edu:1094] Unable to send the message kXR_open (file: /xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-12-tpc-dst-d): 000195 : [2020-03-21 05:00:59.855213 -0500][Error ][Utility ] Unable to open target root://se1.oscer.ou.edu:1094//xrd/dteam/doma/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=se1.oscer.ou.edu:1094&tpc.dlgon=1&tpc.key=31996728a6b5a8a15e75e5db&tpc.lfn=/xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=se1.oscer.ou.edu:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-12-download): 000110 : [2020-03-21 05:01:00.140248 -0500][Error ][XRootD ] [se1.oscer.ou.edu:1094] Impossible to send message kXR_open (file: /xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [18] BNL
> > > UPLOAD (round-trip-19-upload): 000186 : [2020-03-21 05:01:39.336785 -0500][Error ][XRootD ] [dcqos001.usatlas.bnl.gov:1096] Unable to send the message kXR_open (file: /pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-19-tpc-dst-d): 000195 : [2020-03-21 05:01:39.772977 -0500][Error ][Utility ] Unable to open target root://dcqos001.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/data/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=dcqos001.usatlas.bnl.gov:1096&tpc.dlgon=1&tpc.key=2dc112e0a6b5a9e05e75e603&tpc.lfn=/pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=dcqos001.usatlas.bnl.gov:1096&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-19-download): 000110 : [2020-03-21 05:01:40.554008 -0500][Error ][XRootD ] [dcqos001.usatlas.bnl.gov:1096] Impossible to send message kXR_open (file: /pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [19] AGLT2
> > > UPLOAD (round-trip-21-upload): 000214 : [2020-03-21 05:01:44.550200 -0500][Error ][XRootD ] [xrootd.aglt2.org:1094] Unable to send the message kXR_open (file: pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-21-tpc-dst-d): 000223 : [2020-03-21 05:01:44.731532 -0500][Error ][Utility ] Unable to open target root://xrootd.aglt2.org:1094/pnfs/aglt2.org/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.aglt2.org:1094&tpc.dlgon=1&tpc.key=2af6e2d8a6b5aa385e75e608&tpc.lfn=pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.aglt2.org:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-21-download): 000138 : [2020-03-21 05:01:45.086137 -0500][Error ][XRootD ] [xrootd.aglt2.org:1094] Impossible to send message kXR_open (file: pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [20] CALTECH
> > > UPLOAD (round-trip-18-upload): 000186 : [2020-03-21 05:01:37.383371 -0500][Error ][XRootD ] [transfer-1.ultralight.org:1094] Unable to send the message kXR_open (file: /store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-18-tpc-dst-d): 000195 : [2020-03-21 05:01:37.711002 -0500][Error ][Utility ] Unable to open target root://transfer-1.ultralight.org:1094//store/user/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=transfer-1.ultralight.org:1094&tpc.dlgon=1&tpc.key=26e7de90a6b5a9b45e75e601&tpc.lfn=/store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=transfer-1.ultralight.org:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-18-download): 000110 : [2020-03-21 05:01:38.345042 -0500][Error ][XRootD ] [transfer-1.ultralight.org:1094] Impossible to send message kXR_open (file: /store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
> > > [21] TRIUMF-PROD
> > > UPLOAD (round-trip-17-upload): 000720 : [2020-03-21 05:01:14.574303 -0500][Error ][XRootD ] [xrootd.lcg.triumf.ca:1094] Unable to send the message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> > > TPC_DST_D (round-trip-17-tpc-dst-d): 000729 : [2020-03-21 05:01:19.059769 -0500][Error ][Utility ] Unable to open target root://xrootd.lcg.triumf.ca:1094//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.lcg.triumf.ca:1094&tpc.dlgon=1&tpc.key=00011558a6b5a9885e75e5ef&tpc.lfn=/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.lcg.triumf.ca:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> > > DOWNLOAD (round-trip-17-download): 000644 : [2020-03-21 05:01:28.033733 -0500][Error ][XRootD ] [xrootd.lcg.triumf.ca:1094] Impossible to send message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> > >
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
