 |
 |
We're running 4.11.1 here at OU. Cheers, Horst Albert Rossi <[log in to unmask]> wrote: > Hi Horst, > > actually, if you notice, all endpoints failed on today's test. So it was not just OU. > > The Stanford developers may want you to run a few commands over the core file from gdb once you have it in hand. > > What version of xrootd are you running, just out of curiosity? Is it bleeding-edge, or a stable release? > > Cheers, Al > > ________________________________________________ > Albert L. Rossi > Application Developer & Systems Analyst III > Scientific Computing Division, Data Movement Development > FCC 229A > Mail Station 369 (FCC 2W) > Fermi National Accelerator Laboratory > Batavia, IL 60510 > (630) 840-3023 > ________________________________ > From: Horst Severini <[log in to unmask]> > Sent: Saturday, March 21, 2020 1:18 PM > To: [log in to unmask] <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; Albert Rossi <[log in to unmask]> > Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT > > Hi Al, > > thanks, good idea. I'll save the next core file. > > I'm pretty sure the authentication failures simply came because > that partition was full and no new proxies or what not could be > created, so I wouldn't worry about that. > > Thanks, > > Horst > > Albert Rossi <[log in to unmask]> wrote: > > > Hi Horst, > > > > I would definitely report this to [log in to unmask] > > > > As for why the massive authentication failure, I've seen this before, it might have to do with CA cert issues. > > > > Cheers, Al > > > > ________________________________________________ > > Albert L. Rossi > > Application Developer & Systems Analyst III > > Scientific Computing Division, Data Movement Development > > FCC 229A > > Mail Station 369 (FCC 2W) > > Fermi National Accelerator Laboratory > > Batavia, IL 60510 > > (630) 840-3023 > > ________________________________ > > From: Horst Severini <[log in to unmask]> > > Sent: Saturday, March 21, 2020 11:19 AM > > To: [log in to unmask] <[log in to unmask]> > > Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT > > > > Hi all, > > > > our /var/ partition had filled up because of too many xrootd core dumps. > > I cleared those up and restarted xrootd, and things look better again now. > > > > Not sure why we keep getting core dumps, though. > > > > Cheers, > > > > Horst > > > > On 3/21/20 5:01 AM, [log in to unmask] wrote: > > > XROOTD SMOKE TEST SUMMARY > > > 2020-03-21 10:01:46 GMT > > > > > > Client: bogus6.fnal.gov > > > > > > XrootD version: v4.11.2 > > > > > > Reference server: CERN-TRUNK > > > > > > Credential delegation: ON > > > > > > Checksum: -C adler32 > > > > > > Total number of round-trip tests: 21 > > > > > > --------------------------------SOUND ENDPOINTS--------------------------------- > > > > > > SCORE ENDPT TYPE UP SRC DST DN > > > -------------------------------------------------------------------------------- > > > > > > -----------------------------PROBLEMATIC ENDPOINTS------------------------------ > > > > > > SCORE ENDPT TYPE UP SRC DST DN > > > -------------------------------------------------------------------------------- > > > 19 BRUSSELS dCache F - F F 0/4 > > > 19 CERN-EOS EOS F - F F 0/4 > > > 19 CERN-TRUNK DPM - F - F 0/2 > > > 19 DESY-PROM dCache F - F F 0/4 > > > 19 FNAL dCache F - F F 0/4 > > > 19 IN2P3-DOMA xrootd F - F F 0/4 > > > 19 PRAGUE DPM F - F F 0/4 > > > 19 RAL-CEPH CEPH F - F F 0/4 > > > 19 RAL-LCG2 Echo F - F F 0/4 > > > 19 SLAC XrootD F - F F 0/4 > > > 19 TRIUMF dCache F - F F 0/4 > > > 19 UKI-BRUNEL DPM F - F F 0/4 > > > 19 UKI-LANC DPM F - F F 0/4 > > > 19 UKI-MAN1 DPM F - F F 0/4 > > > 19 UKI-MAN2 DPM F - F F 0/4 > > > 19 UNI-BONN CephFS F - F F 0/4 > > > 18 OU XrootD F - F F 0/4 > > > 13 BNL dCache F - F F 0/4 > > > 7 AGLT2 dCache F - F F 0/4 > > > 0 CALTECH HDFS F - F F 0/4 > > > 0 TRIUMF-PROD dCache F - F F 0/4 > > > > > > --------------------------------ERROR DETAILS----------------------------------- > > > > > > [1] BRUSSELS > > > UPLOAD (round-trip-15-upload): 000186 : [2020-03-21 05:01:06.216990 -0500][Error ][XRootD ] [maite.iihe.ac.be:1094] Unable to send the message kXR_open (file: pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-15-tpc-dst-d): 000195 : [2020-03-21 05:01:06.719178 -0500][Error ][Utility ] Unable to open target root://maite.iihe.ac.be:1094/pnfs/iihe//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=maite.iihe.ac.be:1094&tpc.dlgon=1&tpc.key=247f6a10a6b5a9305e75e5e2&tpc.lfn=pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=maite.iihe.ac.be:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-15-download): 000110 : [2020-03-21 05:01:07.460848 -0500][Error ][XRootD ] [maite.iihe.ac.be:1094] Impossible to send message kXR_open (file: pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [2] CERN-EOS > > > UPLOAD (round-trip-10-upload): 000218 : Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/opstest/tpc/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04; Operation not permitted > > > TPC_DST_D (round-trip-10-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3011] Unable to remove /eos/opstest/tpc/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; No such file or directory > > > DOWNLOAD (round-trip-10-download): 000129 : Run: [ERROR] Server responded with an error: [3011] Unable to open file /eos/opstest/tpc/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04; No such file or directory > > > > > > [3] CERN-TRUNK > > > TPC_SRC_D (round-trip-8-tpc-src-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/cern.ch/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied > > > DOWNLOAD (round-trip-8-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > > > > [4] DESY-PROM > > > UPLOAD (round-trip-9-upload): 000186 : [2020-03-21 05:00:47.833821 -0500][Error ][XRootD ] [prometheus.desy.de:1095] Unable to send the message kXR_open (file: /VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-9-tpc-dst-d): 000195 : [2020-03-21 05:00:48.337679 -0500][Error ][Utility ] Unable to open target root://prometheus.desy.de:1095//VOs/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=prometheus.desy.de:1095&tpc.dlgon=1&tpc.key=140c2808a6b5a81d5e75e5d0&tpc.lfn=/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=prometheus.desy.de:1095&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-9-download): 000110 : [2020-03-21 05:00:49.246017 -0500][Error ][XRootD ] [prometheus.desy.de:1095] Impossible to send message kXR_open (file: /VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [5] FNAL > > > UPLOAD (round-trip-14-upload): 000186 : [2020-03-21 05:01:05.136590 -0500][Error ][XRootD ] [stkendca06a.fnal.gov:1094] Unable to send the message kXR_open (file: /pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-14-tpc-dst-d): 000195 : [2020-03-21 05:01:05.235583 -0500][Error ][Utility ] Unable to open target root://stkendca06a.fnal.gov:1094//pnfs/fnal.gov/VOs/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=stkendca06a.fnal.gov:1094&tpc.dlgon=1&tpc.key=0de3b5b8a6b5a9045e75e5e1&tpc.lfn=/pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=stkendca06a.fnal.gov:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-14-download): 000110 : [2020-03-21 05:01:05.444704 -0500][Error ][XRootD ] [stkendca06a.fnal.gov:1094] Impossible to send message kXR_open (file: /pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [6] IN2P3-DOMA > > > UPLOAD (round-trip-20-upload): 000186 : [2020-03-21 05:01:42.067389 -0500][Error ][XRootD ] [ccxrootddoma.in2p3.fr:1094] Unable to send the message kXR_open (file: /xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-20-tpc-dst-d): 000195 : [2020-03-21 05:01:42.585936 -0500][Error ][Utility ] Unable to open target root://ccxrootddoma.in2p3.fr:1094//xrootd/dteam/doma/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ccxrootddoma.in2p3.fr:1094&tpc.dlgon=1&tpc.key=1c56a380a6b5aa0c5e75e606&tpc.lfn=/xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ccosvms0124.in2p3.fr:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-20-download): 000110 : [2020-03-21 05:01:43.407723 -0500][Error ][XRootD ] [ccxrootddoma.in2p3.fr:1094] Impossible to send message kXR_open (file: /xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [7] PRAGUE > > > UPLOAD (round-trip-3-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > TPC_DST_D (round-trip-3-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/farm.particle.cz/home/dteam/tpc-test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied > > > DOWNLOAD (round-trip-3-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > > > > [8] RAL-CEPH > > > UPLOAD (round-trip-2-upload): 000186 : [2020-03-21 05:00:18.701154 -0500][Error ][XRootD ] [ceph-test-gw683.gridpp.rl.ac.uk:1094] Unable to send the message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-2-tpc-dst-d): 000195 : [2020-03-21 05:00:19.190035 -0500][Error ][Utility ] Unable to open target root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ceph-test-gw683.gridpp.rl.ac.uk:1094&tpc.dlgon=1&tpc.key=05805ef8a6b5a6f85e75e5b3&tpc.lfn=/dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ceph-test-gw683.gridpp.rl.ac.uk:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-2-download): 000110 : [2020-03-21 05:00:19.969919 -0500][Error ][XRootD ] [ceph-test-gw683.gridpp.rl.ac.uk:1094] Impossible to send message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [9] RAL-LCG2 > > > UPLOAD (round-trip-1-upload): 000356 : [2020-03-21 05:00:05.333273 -0500][Error ][XRootD ] [echo.stfc.ac.uk:1094] Unable to send the message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-1-tpc-dst-d): 000365 : [2020-03-21 05:00:07.998168 -0500][Error ][Utility ] Unable to open target root://echo.stfc.ac.uk:1094//dteam:test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=echo.stfc.ac.uk:1094&tpc.dlgon=1&tpc.key=35a6b208a6b5a6cb5e75e5a7&tpc.lfn=/dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=echo.stfc.ac.uk:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-1-download): 000280 : [2020-03-21 05:00:13.049330 -0500][Error ][XRootD ] [echo.stfc.ac.uk:1094] Impossible to send message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [10] SLAC > > > UPLOAD (round-trip-11-upload): 000186 : [2020-03-21 05:00:58.184767 -0500][Error ][XRootD ] [griddev03.slac.stanford.edu:2094] Unable to send the message kXR_open (file: /xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-11-tpc-dst-d): 000195 : [2020-03-21 05:00:58.501336 -0500][Error ][Utility ] Unable to open target root://griddev03.slac.stanford.edu:2094//xrootd/atlas/tpctest/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=griddev03.slac.stanford.edu:2094&tpc.dlgon=1&tpc.key=1ad90c00a6b5a8755e75e5da&tpc.lfn=/xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=griddev03.slac.stanford.edu:2094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-11-download): 000110 : [2020-03-21 05:00:58.987351 -0500][Error ][XRootD ] [griddev03.slac.stanford.edu:2094] Impossible to send message kXR_open (file: /xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [11] TRIUMF > > > UPLOAD (round-trip-16-upload): 000186 : [2020-03-21 05:01:08.742011 -0500][Error ][XRootD ] [ppshead.lcg.triumf.ca:1094] Unable to send the message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-16-tpc-dst-d): 000195 : [2020-03-21 05:01:09.056302 -0500][Error ][Utility ] Unable to open target root://ppshead.lcg.triumf.ca:1094//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ppshead.lcg.triumf.ca:1094&tpc.dlgon=1&tpc.key=000fa3e8a6b5a95c5e75e5e5&tpc.lfn=/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ppshead.lcg.triumf.ca:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-16-download): 000110 : [2020-03-21 05:01:09.513921 -0500][Error ][XRootD ] [ppshead.lcg.triumf.ca:1094] Impossible to send message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [12] UKI-BRUNEL > > > UPLOAD (round-trip-4-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > TPC_DST_D (round-trip-4-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/brunel.ac.uk/home/dteam/wlcg-tpc/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied > > > DOWNLOAD (round-trip-4-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > > > > [13] UKI-LANC > > > UPLOAD (round-trip-7-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > TPC_DST_D (round-trip-7-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/lancs.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied > > > DOWNLOAD (round-trip-7-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > > > > [14] UKI-MAN1 > > > UPLOAD (round-trip-5-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > TPC_DST_D (round-trip-5-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/tier2.hep.manchester.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied > > > DOWNLOAD (round-trip-5-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > > > > [15] UKI-MAN2 > > > UPLOAD (round-trip-6-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > TPC_DST_D (round-trip-6-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/tier2.hep.manchester.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied > > > DOWNLOAD (round-trip-6-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied > > > > > > [16] UNI-BONN > > > UPLOAD (round-trip-13-upload): 000186 : [2020-03-21 05:01:01.114829 -0500][Error ][XRootD ] [xrootd.physik.uni-bonn.de:1094] Unable to send the message kXR_open (file: /cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-13-tpc-dst-d): 000197 : [2020-03-21 05:01:03.253425 -0500][Error ][Utility ] Unable to open target root://xrootd.physik.uni-bonn.de:1094//cephfs/grid/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.physik.uni-bonn.de:1094&tpc.dlgon=1&tpc.key=1ffebc20a6b5a8cd5e75e5dd&tpc.lfn=/cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.physik.uni-bonn.de:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-13-download): 000110 : [2020-03-21 05:01:04.113281 -0500][Error ][XRootD ] [xrootd.physik.uni-bonn.de:1094] Impossible to send message kXR_open (file: /cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [17] OU > > > UPLOAD (round-trip-12-upload): 000186 : [2020-03-21 05:00:59.696352 -0500][Error ][XRootD ] [se1.oscer.ou.edu:1094] Unable to send the message kXR_open (file: /xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-12-tpc-dst-d): 000195 : [2020-03-21 05:00:59.855213 -0500][Error ][Utility ] Unable to open target root://se1.oscer.ou.edu:1094//xrd/dteam/doma/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=se1.oscer.ou.edu:1094&tpc.dlgon=1&tpc.key=31996728a6b5a8a15e75e5db&tpc.lfn=/xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=se1.oscer.ou.edu:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-12-download): 000110 : [2020-03-21 05:01:00.140248 -0500][Error ][XRootD ] [se1.oscer.ou.edu:1094] Impossible to send message kXR_open (file: /xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [18] BNL > > > UPLOAD (round-trip-19-upload): 000186 : [2020-03-21 05:01:39.336785 -0500][Error ][XRootD ] [dcqos001.usatlas.bnl.gov:1096] Unable to send the message kXR_open (file: /pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-19-tpc-dst-d): 000195 : [2020-03-21 05:01:39.772977 -0500][Error ][Utility ] Unable to open target root://dcqos001.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/data/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=dcqos001.usatlas.bnl.gov:1096&tpc.dlgon=1&tpc.key=2dc112e0a6b5a9e05e75e603&tpc.lfn=/pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=dcqos001.usatlas.bnl.gov:1096&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-19-download): 000110 : [2020-03-21 05:01:40.554008 -0500][Error ][XRootD ] [dcqos001.usatlas.bnl.gov:1096] Impossible to send message kXR_open (file: /pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [19] AGLT2 > > > UPLOAD (round-trip-21-upload): 000214 : [2020-03-21 05:01:44.550200 -0500][Error ][XRootD ] [xrootd.aglt2.org:1094] Unable to send the message kXR_open (file: pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-21-tpc-dst-d): 000223 : [2020-03-21 05:01:44.731532 -0500][Error ][Utility ] Unable to open target root://xrootd.aglt2.org:1094/pnfs/aglt2.org/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.aglt2.org:1094&tpc.dlgon=1&tpc.key=2af6e2d8a6b5aa385e75e608&tpc.lfn=pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.aglt2.org:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-21-download): 000138 : [2020-03-21 05:01:45.086137 -0500][Error ][XRootD ] [xrootd.aglt2.org:1094] Impossible to send message kXR_open (file: pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [20] CALTECH > > > UPLOAD (round-trip-18-upload): 000186 : [2020-03-21 05:01:37.383371 -0500][Error ][XRootD ] [transfer-1.ultralight.org:1094] Unable to send the message kXR_open (file: /store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-18-tpc-dst-d): 000195 : [2020-03-21 05:01:37.711002 -0500][Error ][Utility ] Unable to open target root://transfer-1.ultralight.org:1094//store/user/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=transfer-1.ultralight.org:1094&tpc.dlgon=1&tpc.key=26e7de90a6b5a9b45e75e601&tpc.lfn=/store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=transfer-1.ultralight.org:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-18-download): 000110 : [2020-03-21 05:01:38.345042 -0500][Error ][XRootD ] [transfer-1.ultralight.org:1094] Impossible to send message kXR_open (file: /store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > > > > [21] TRIUMF-PROD > > > UPLOAD (round-trip-17-upload): 000720 : [2020-03-21 05:01:14.574303 -0500][Error ][XRootD ] [xrootd.lcg.triumf.ca:1094] Unable to send the message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed > > > TPC_DST_D (round-trip-17-tpc-dst-d): 000729 : [2020-03-21 05:01:19.059769 -0500][Error ][Utility ] Unable to open target root://xrootd.lcg.triumf.ca:1094//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.lcg.triumf.ca:1094&tpc.dlgon=1&tpc.key=00011558a6b5a9885e75e5ef&tpc.lfn=/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.lcg.triumf.ca:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed > > > DOWNLOAD (round-trip-17-download): 000644 : [2020-03-21 05:01:28.033733 -0500][Error ][XRootD ] [xrootd.lcg.triumf.ca:1094] Impossible to send message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover. > > > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1