LISTSERV 16.5 - XROOTD-DEV Archives

That is because the refresh failed to generate it from the hostcert:

Your identity: /DC=org/DC=incommon/C=US/ST=Illinois/L=Batavia/O=Fermi Research Alliance/OU=Fermilab/CN=stkendca06a.fnal.gov

Creating temporary proxy ............................ Done

Contacting voms2.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr] "dteam"

Failed

Error: dteam: User unknown to this VO.

Is the hellasgrid endpoint down? Has it changed?

Thanks, Al

________________________________________________
Albert L. Rossi
Application Developer & Systems Analyst III
Scientific Computing Division, Data Movement Development
FCC 229A
Mail Station 369 (FCC 2W)
Fermi National Accelerator Laboratory
Batavia, IL 60510

(630) 840-3023

From: Albert Rossi <[log in to unmask]>
Sent: Saturday, March 21, 2020 1:39 PM
To: Horst Severini <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; [log in to unmask] <[log in to unmask]>
Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT

Fermi's proxy cert expired.

(630) 840-3023

From: Albert Rossi <[log in to unmask]>
Sent: Saturday, March 21, 2020 1:23 PM
To: Horst Severini <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; [log in to unmask] <[log in to unmask]>
Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT

Hi Horst,

actually, if you notice, all endpoints failed on today's test. So it was not just OU.

The Stanford developers may want you to run a few commands over the core file from gdb once you have it in hand.

What version of xrootd are you running, just out of curiosity? Is it bleeding-edge, or a stable release?

Cheers, Al

(630) 840-3023

From: Horst Severini <[log in to unmask]>
Sent: Saturday, March 21, 2020 1:18 PM
To: [log in to unmask] <[log in to unmask]>; [log in to unmask] <[log in to unmask]>; Albert Rossi <[log in to unmask]>
Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT

Hi Al,

thanks, good idea. I'll save the next core file.

I'm pretty sure the authentication failures simply came because
that partition was full and no new proxies or what not could be
created, so I wouldn't worry about that.

Thanks,

        Horst

Albert Rossi <[log in to unmask]> wrote:

> Hi Horst,
>
> I would definitely report this to [log in to unmask]
>
> As for why the massive authentication failure, I've seen this before, it might have to do with CA cert issues.
>
> Cheers, Al
>
> ________________________________________________
> Albert L. Rossi
> Application Developer & Systems Analyst III
> Scientific Computing Division, Data Movement Development
> FCC 229A
> Mail Station 369 (FCC 2W)
> Fermi National Accelerator Laboratory
> Batavia, IL 60510
> (630) 840-3023
> ________________________________
> From: Horst Severini <[log in to unmask]>
> Sent: Saturday, March 21, 2020 11:19 AM
> To: [log in to unmask] <[log in to unmask]>
> Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT
>
> Hi all,
>
> our /var/ partition had filled up because of too many xrootd core dumps.
> I cleared those up and restarted xrootd, and things look better again now.
>
> Not sure why we keep getting core dumps, though.
>
> Cheers,
>
>         Horst
>
> On 3/21/20 5:01 AM, [log in to unmask] wrote:
> > XROOTD SMOKE TEST SUMMARY
> > 2020-03-21 10:01:46 GMT
> >
> > Client: bogus6.fnal.gov
> >
> > XrootD version: v4.11.2
> >
> > Reference server: CERN-TRUNK
> >
> > Credential delegation: ON
> >
> > Checksum: -C adler32
> >
> > Total number of round-trip tests: 21
> >
> > --------------------------------SOUND ENDPOINTS---------------------------------
> >
> > SCORE   ENDPT               TYPE          UP       SRC       DST        DN
> > --------------------------------------------------------------------------------
> >
> > -----------------------------PROBLEMATIC ENDPOINTS------------------------------
> >
> > SCORE   ENDPT               TYPE          UP       SRC       DST        DN
> > --------------------------------------------------------------------------------
> > 19      BRUSSELS            dCache        F         -         F         F        0/4
> > 19      CERN-EOS            EOS           F         -         F         F        0/4
> > 19      CERN-TRUNK          DPM           -         F         -         F        0/2
> > 19      DESY-PROM           dCache        F         -         F         F        0/4
> > 19      FNAL                dCache        F         -         F         F        0/4
> > 19      IN2P3-DOMA          xrootd        F         -         F         F        0/4
> > 19      PRAGUE              DPM           F         -         F         F        0/4
> > 19      RAL-CEPH            CEPH          F         -         F         F        0/4
> > 19      RAL-LCG2            Echo          F         -         F         F        0/4
> > 19      SLAC                XrootD        F         -         F         F        0/4
> > 19      TRIUMF              dCache        F         -         F         F        0/4
> > 19      UKI-BRUNEL          DPM           F         -         F         F        0/4
> > 19      UKI-LANC            DPM           F         -         F         F        0/4
> > 19      UKI-MAN1            DPM           F         -         F         F        0/4
> > 19      UKI-MAN2            DPM           F         -         F         F        0/4
> > 19      UNI-BONN            CephFS        F         -         F         F        0/4
> > 18      OU                  XrootD        F         -         F         F        0/4
> > 13      BNL                 dCache        F         -         F         F        0/4
> > 7       AGLT2               dCache        F         -         F         F        0/4
> > 0       CALTECH             HDFS          F         -         F         F        0/4
> > 0       TRIUMF-PROD         dCache        F         -         F         F        0/4
> >
> > --------------------------------ERROR DETAILS-----------------------------------
> >
> > [1] BRUSSELS
> >                UPLOAD (round-trip-15-upload): 000186 : [2020-03-21 05:01:06.216990 -0500][Error ][XRootD            ] [maite.iihe.ac.be:1094] Unable to send the message kXR_open (file: pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-15-tpc-dst-d): 000195 : [2020-03-21 05:01:06.719178 -0500][Error ][Utility           ] Unable to open target root://maite.iihe.ac.be:1094/pnfs/iihe//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=maite.iihe.ac.be:1094&tpc.dlgon=1&tpc.key=247f6a10a6b5a9305e75e5e2&tpc.lfn=pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=maite.iihe.ac.be:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-15-download): 000110 : [2020-03-21 05:01:07.460848 -0500][Error ][XRootD            ] [maite.iihe.ac.be:1094] Impossible to send message kXR_open (file: pnfs/iihe//dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [2] CERN-EOS
> >                UPLOAD (round-trip-10-upload): 000218 : Run: [ERROR] Server responded with an error: [3010] Unable to open file /eos/opstest/tpc/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04; Operation not permitted
> >                TPC_DST_D (round-trip-10-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3011] Unable to remove /eos/opstest/tpc/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; No such file or directory
> >                DOWNLOAD (round-trip-10-download): 000129 : Run: [ERROR] Server responded with an error: [3011] Unable to open file /eos/opstest/tpc/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04; No such file or directory
> >
> > [3] CERN-TRUNK
> >                TPC_SRC_D (round-trip-8-tpc-src-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/cern.ch/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> >                DOWNLOAD (round-trip-8-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >
> > [4] DESY-PROM
> >                UPLOAD (round-trip-9-upload): 000186 : [2020-03-21 05:00:47.833821 -0500][Error ][XRootD            ] [prometheus.desy.de:1095] Unable to send the message kXR_open (file: /VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-9-tpc-dst-d): 000195 : [2020-03-21 05:00:48.337679 -0500][Error ][Utility           ] Unable to open target root://prometheus.desy.de:1095//VOs/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=prometheus.desy.de:1095&tpc.dlgon=1&tpc.key=140c2808a6b5a81d5e75e5d0&tpc.lfn=/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=prometheus.desy.de:1095&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-9-download): 000110 : [2020-03-21 05:00:49.246017 -0500][Error ][XRootD            ] [prometheus.desy.de:1095] Impossible to send message kXR_open (file: /VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [5] FNAL
> >                UPLOAD (round-trip-14-upload): 000186 : [2020-03-21 05:01:05.136590 -0500][Error ][XRootD            ] [stkendca06a.fnal.gov:1094] Unable to send the message kXR_open (file: /pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-14-tpc-dst-d): 000195 : [2020-03-21 05:01:05.235583 -0500][Error ][Utility           ] Unable to open target root://stkendca06a.fnal.gov:1094//pnfs/fnal.gov/VOs/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=stkendca06a.fnal.gov:1094&tpc.dlgon=1&tpc.key=0de3b5b8a6b5a9045e75e5e1&tpc.lfn=/pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=stkendca06a.fnal.gov:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-14-download): 000110 : [2020-03-21 05:01:05.444704 -0500][Error ][XRootD            ] [stkendca06a.fnal.gov:1094] Impossible to send message kXR_open (file: /pnfs/fnal.gov/VOs/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [6] IN2P3-DOMA
> >                UPLOAD (round-trip-20-upload): 000186 : [2020-03-21 05:01:42.067389 -0500][Error ][XRootD            ] [ccxrootddoma.in2p3.fr:1094] Unable to send the message kXR_open (file: /xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-20-tpc-dst-d): 000195 : [2020-03-21 05:01:42.585936 -0500][Error ][Utility           ] Unable to open target root://ccxrootddoma.in2p3.fr:1094//xrootd/dteam/doma/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ccxrootddoma.in2p3.fr:1094&tpc.dlgon=1&tpc.key=1c56a380a6b5aa0c5e75e606&tpc.lfn=/xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ccosvms0124.in2p3.fr:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-20-download): 000110 : [2020-03-21 05:01:43.407723 -0500][Error ][XRootD            ] [ccxrootddoma.in2p3.fr:1094] Impossible to send message kXR_open (file: /xrootd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [7] PRAGUE
> >                UPLOAD (round-trip-3-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >                TPC_DST_D (round-trip-3-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/farm.particle.cz/home/dteam/tpc-test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> >                DOWNLOAD (round-trip-3-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >
> > [8] RAL-CEPH
> >                UPLOAD (round-trip-2-upload): 000186 : [2020-03-21 05:00:18.701154 -0500][Error ][XRootD            ] [ceph-test-gw683.gridpp.rl.ac.uk:1094] Unable to send the message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-2-tpc-dst-d): 000195 : [2020-03-21 05:00:19.190035 -0500][Error ][Utility           ] Unable to open target root://ceph-test-gw683.gridpp.rl.ac.uk:1094//dteam:test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ceph-test-gw683.gridpp.rl.ac.uk:1094&tpc.dlgon=1&tpc.key=05805ef8a6b5a6f85e75e5b3&tpc.lfn=/dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ceph-test-gw683.gridpp.rl.ac.uk:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-2-download): 000110 : [2020-03-21 05:00:19.969919 -0500][Error ][XRootD            ] [ceph-test-gw683.gridpp.rl.ac.uk:1094] Impossible to send message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [9] RAL-LCG2
> >                UPLOAD (round-trip-1-upload): 000356 : [2020-03-21 05:00:05.333273 -0500][Error ][XRootD            ] [echo.stfc.ac.uk:1094] Unable to send the message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-1-tpc-dst-d): 000365 : [2020-03-21 05:00:07.998168 -0500][Error ][Utility           ] Unable to open target root://echo.stfc.ac.uk:1094//dteam:test/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=echo.stfc.ac.uk:1094&tpc.dlgon=1&tpc.key=35a6b208a6b5a6cb5e75e5a7&tpc.lfn=/dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=echo.stfc.ac.uk:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-1-download): 000280 : [2020-03-21 05:00:13.049330 -0500][Error ][XRootD            ] [echo.stfc.ac.uk:1094] Impossible to send message kXR_open (file: /dteam:test/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [10] SLAC
> >                UPLOAD (round-trip-11-upload): 000186 : [2020-03-21 05:00:58.184767 -0500][Error ][XRootD            ] [griddev03.slac.stanford.edu:2094] Unable to send the message kXR_open (file: /xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-11-tpc-dst-d): 000195 : [2020-03-21 05:00:58.501336 -0500][Error ][Utility           ] Unable to open target root://griddev03.slac.stanford.edu:2094//xrootd/atlas/tpctest/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=griddev03.slac.stanford.edu:2094&tpc.dlgon=1&tpc.key=1ad90c00a6b5a8755e75e5da&tpc.lfn=/xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=griddev03.slac.stanford.edu:2094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-11-download): 000110 : [2020-03-21 05:00:58.987351 -0500][Error ][XRootD            ] [griddev03.slac.stanford.edu:2094] Impossible to send message kXR_open (file: /xrootd/atlas/tpctest/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [11] TRIUMF
> >                UPLOAD (round-trip-16-upload): 000186 : [2020-03-21 05:01:08.742011 -0500][Error ][XRootD            ] [ppshead.lcg.triumf.ca:1094] Unable to send the message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-16-tpc-dst-d): 000195 : [2020-03-21 05:01:09.056302 -0500][Error ][Utility           ] Unable to open target root://ppshead.lcg.triumf.ca:1094//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=ppshead.lcg.triumf.ca:1094&tpc.dlgon=1&tpc.key=000fa3e8a6b5a95c5e75e5e5&tpc.lfn=/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=ppshead.lcg.triumf.ca:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-16-download): 000110 : [2020-03-21 05:01:09.513921 -0500][Error ][XRootD            ] [ppshead.lcg.triumf.ca:1094] Impossible to send message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [12] UKI-BRUNEL
> >                UPLOAD (round-trip-4-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >                TPC_DST_D (round-trip-4-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/brunel.ac.uk/home/dteam/wlcg-tpc/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> >                DOWNLOAD (round-trip-4-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >
> > [13] UKI-LANC
> >                UPLOAD (round-trip-7-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >                TPC_DST_D (round-trip-7-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/lancs.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> >                DOWNLOAD (round-trip-7-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >
> > [14] UKI-MAN1
> >                UPLOAD (round-trip-5-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >                TPC_DST_D (round-trip-5-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/tier2.hep.manchester.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> >                DOWNLOAD (round-trip-5-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >
> > [15] UKI-MAN2
> >                UPLOAD (round-trip-6-upload): 000214 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >                TPC_DST_D (round-trip-6-tpc-dst-d): 000000 : [ERROR] Server responded with an error: [3010] Unable to remove /dpm/tier2.hep.manchester.ac.uk/home/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D; permission denied
> >                DOWNLOAD (round-trip-6-download): 000127 : Run: [ERROR] Server responded with an error: [3010] Permission denied
> >
> > [16] UNI-BONN
> >                UPLOAD (round-trip-13-upload): 000186 : [2020-03-21 05:01:01.114829 -0500][Error ][XRootD            ] [xrootd.physik.uni-bonn.de:1094] Unable to send the message kXR_open (file: /cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-13-tpc-dst-d): 000197 : [2020-03-21 05:01:03.253425 -0500][Error ][Utility           ] Unable to open target root://xrootd.physik.uni-bonn.de:1094//cephfs/grid/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.physik.uni-bonn.de:1094&tpc.dlgon=1&tpc.key=1ffebc20a6b5a8cd5e75e5dd&tpc.lfn=/cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.physik.uni-bonn.de:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-13-download): 000110 : [2020-03-21 05:01:04.113281 -0500][Error ][XRootD            ] [xrootd.physik.uni-bonn.de:1094] Impossible to send message kXR_open (file: /cephfs/grid/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [17] OU
> >                UPLOAD (round-trip-12-upload): 000186 : [2020-03-21 05:00:59.696352 -0500][Error ][XRootD            ] [se1.oscer.ou.edu:1094] Unable to send the message kXR_open (file: /xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-12-tpc-dst-d): 000195 : [2020-03-21 05:00:59.855213 -0500][Error ][Utility           ] Unable to open target root://se1.oscer.ou.edu:1094//xrd/dteam/doma/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=se1.oscer.ou.edu:1094&tpc.dlgon=1&tpc.key=31996728a6b5a8a15e75e5db&tpc.lfn=/xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=se1.oscer.ou.edu:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-12-download): 000110 : [2020-03-21 05:01:00.140248 -0500][Error ][XRootD            ] [se1.oscer.ou.edu:1094] Impossible to send message kXR_open (file: /xrd/dteam/doma/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [18] BNL
> >                UPLOAD (round-trip-19-upload): 000186 : [2020-03-21 05:01:39.336785 -0500][Error ][XRootD            ] [dcqos001.usatlas.bnl.gov:1096] Unable to send the message kXR_open (file: /pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-19-tpc-dst-d): 000195 : [2020-03-21 05:01:39.772977 -0500][Error ][Utility           ] Unable to open target root://dcqos001.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/data/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=dcqos001.usatlas.bnl.gov:1096&tpc.dlgon=1&tpc.key=2dc112e0a6b5a9e05e75e603&tpc.lfn=/pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=dcqos001.usatlas.bnl.gov:1096&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-19-download): 000110 : [2020-03-21 05:01:40.554008 -0500][Error ][XRootD            ] [dcqos001.usatlas.bnl.gov:1096] Impossible to send message kXR_open (file: /pnfs/usatlas.bnl.gov/data/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [19] AGLT2
> >                UPLOAD (round-trip-21-upload): 000214 : [2020-03-21 05:01:44.550200 -0500][Error ][XRootD            ] [xrootd.aglt2.org:1094] Unable to send the message kXR_open (file: pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-21-tpc-dst-d): 000223 : [2020-03-21 05:01:44.731532 -0500][Error ][Utility           ] Unable to open target root://xrootd.aglt2.org:1094/pnfs/aglt2.org/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.aglt2.org:1094&tpc.dlgon=1&tpc.key=2af6e2d8a6b5aa385e75e608&tpc.lfn=pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.aglt2.org:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-21-download): 000138 : [2020-03-21 05:01:45.086137 -0500][Error ][XRootD            ] [xrootd.aglt2.org:1094] Impossible to send message kXR_open (file: pnfs/aglt2.org/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [20] CALTECH
> >                UPLOAD (round-trip-18-upload): 000186 : [2020-03-21 05:01:37.383371 -0500][Error ][XRootD            ] [transfer-1.ultralight.org:1094] Unable to send the message kXR_open (file: /store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-18-tpc-dst-d): 000195 : [2020-03-21 05:01:37.711002 -0500][Error ][Utility           ] Unable to open target root://transfer-1.ultralight.org:1094//store/user/dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=transfer-1.ultralight.org:1094&tpc.dlgon=1&tpc.key=26e7de90a6b5a9b45e75e601&tpc.lfn=/store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=transfer-1.ultralight.org:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-18-download): 000110 : [2020-03-21 05:01:38.345042 -0500][Error ][XRootD            ] [transfer-1.ultralight.org:1094] Impossible to send message kXR_open (file: /store/user/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >
> > [21] TRIUMF-PROD
> >                UPLOAD (round-trip-17-upload): 000720 : [2020-03-21 05:01:14.574303 -0500][Error ][XRootD            ] [xrootd.lcg.triumf.ca:1094] Unable to send the message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?oss.asize=1048576, mode: 0644, flags: kXR_new kXR_open_updt kXR_async kXR_retstat ): [FATAL] Auth failed
> >                TPC_DST_D (round-trip-17-tpc-dst-d): 000729 : [2020-03-21 05:01:19.059769 -0500][Error ][Utility           ] Unable to open target root://xrootd.lcg.triumf.ca:1094//dteam/from-CERN-TRUNK-21d897f6-51dd-4cc6-9535-58a414108e04-D?oss.asize=0&tpc.dlg=xrootd.lcg.triumf.ca:1094&tpc.dlgon=1&tpc.key=00011558a6b5a9885e75e5ef&tpc.lfn=/dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04&tpc.scgi=tpc.stage=placement&tpc.spr=root&tpc.src=xrootd.lcg.triumf.ca:1094&tpc.stage=copy&tpc.str=1&tpc.tpr=root: [FATAL] Auth failed
> >                DOWNLOAD (round-trip-17-download): 000644 : [2020-03-21 05:01:28.033733 -0500][Error ][XRootD            ] [xrootd.lcg.triumf.ca:1094] Impossible to send message kXR_open (file: /dteam/xrootd-test-data-21d897f6-51dd-4cc6-9535-58a414108e04?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
> >