In the case of a redirector setup, XrdCl will swallow the last error it got from a data server and respond with a generic no servers available. Here's a truncated output: ``` $ BEARER_TOKEN=...TOKEN_OMITTED... xrdcp -d 2 -f root://redirector.osgstorage.org//user/ligo/test_access/access_ligo /dev/null [2023-02-18 11:05:22.910848 -0600][Debug ][Utility ] CopyProcess: 2 jobs to prepare ... snip ... [2023-02-18 11:05:22.912312 -0600][Debug ][Utility ] Opening root://redirector.osgstorage.org:1094//user/ligo/test_access/access_ligo for reading [2023-02-18 11:05:22.912435 -0600][Debug ][File ] [0xca0a80@root://redirector.osgstorage.org:1094//user/ligo/test_access/access_ligo?xrdcl.requuid=c242fdbf-a2e6-4b1b-9ec6-97c48063a2c6] Sending an open command [2023-02-18 11:05:22.912501 -0600][Debug ][ExDbgMsg ] [redirector.osgstorage.org:1094] MsgHandler created: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2023-02-18 11:05:22.912584 -0600][Debug ][PostMaster ] Creating new channel to: root://redirector.osgstorage.org:1094 ... snip ... [2023-02-18 11:05:22.913477 -0600][Debug ][AsyncSock ] [redirector.osgstorage.org:1094.0] Attempting connection to [::ffff:128.104.103.164]:1094 ... snip ... [2023-02-18 11:05:22.961574 -0600][Debug ][XRootDTransport ] [redirector.osgstorage.org:1094.0] Sending out kXR_login request, username: bbockelm, cgi: ?xrd.cc=us&xrd.tz=-6&xrd.appname=xrdcp&xrd.info=&xrd.hostname=example.com&xrd.rn=v5.2.0, dual-stack: true, private IPv4: false, private IPv6: false ... snip ... [2023-02-18 11:05:22.985035 -0600][Debug ][XRootDTransport ] [redirector.osgstorage.org:1094.0] Logged in, session: 3eaf0500b06e0000790000001fba0500 ... snip ... [2023-02-18 11:05:23.063394 -0600][Debug ][ExDbgMsg ] [xrootd-local.unl.edu:1094] Retry at server MsgHandler: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2023-02-18 11:05:23.063460 -0600][Debug ][PostMaster ] Creating new channel to: root://xrootd-local.unl.edu:1094 ... snip ... [2023-02-18 11:05:23.064456 -0600][Debug ][AsyncSock ] [xrootd-local.unl.edu:1094.0] Attempting connection to [2600:900:6:1101:5054:ff:fe65:7c48]:1094 ... snip ... [2023-02-18 11:05:23.066080 -0600][Debug ][XRootDTransport ] [xrootd-local.unl.edu:1094.0] Sending out kXR_login request, username: bbockelm, cgi: ?xrd.cc=us&xrd.tz=-6&xrd.appname=xrdcp&xrd.info=&xrd.hostname=example.com&xrd.rn=v5.2.0, dual-stack: true, private IPv4: false, private IPv6: false ... snip ... [2023-02-18 11:05:23.070075 -0600][Debug ][XRootDTransport ] [xrootd-local.unl.edu:1094.0] Logged in, session: 5b9e020031180000f0000000c9270300 [2023-02-18 11:05:23.070097 -0600][Debug ][XRootDTransport ] [xrootd-local.unl.edu:1094.0] Authentication is required: &P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096: ... snip ... [2023-02-18 11:05:23.089334 -0600][Debug ][XRootDTransport ] [xrootd-local.unl.edu:1094.0] Authenticated with ztn. ... snip ... [2023-02-18 11:05:23.090208 -0600][Debug ][ExDbgMsg ] [red-xfer2.unl.edu:1094] Retry at server MsgHandler: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2023-02-18 11:05:23.090268 -0600][Debug ][PostMaster ] Creating new channel to: root://red-xfer2.unl.edu:1094 ... snip ... [2023-02-18 11:05:23.091116 -0600][Debug ][AsyncSock ] [red-xfer2.unl.edu:1094.0] Attempting connection to [2600:900:6:1101:20f:53ff:fe31:d6a0]:1094 ... snip ... [2023-02-18 11:05:23.092932 -0600][Debug ][XRootDTransport ] [red-xfer2.unl.edu:1094.0] Sending out kXR_login request, username: bbockelm, cgi: ?xrd.cc=us&xrd.tz=-6&xrd.appname=xrdcp&xrd.info=&xrd.hostname=example.com&xrd.rn=v5.2.0, dual-stack: true, private IPv4: false, private IPv6: false ... snip ... [2023-02-18 11:05:23.096605 -0600][Debug ][XRootDTransport ] [red-xfer2.unl.edu:1094.0] Authentication is required: &P=gsi,v:10600,c:ssl,ca:ba240aa8.0|f5f0dfc2.0&P=ztn,0:4096: ... snip ... [2023-02-18 11:05:23.466519 -0600][Debug ][XRootDTransport ] [red-xfer2.unl.edu:1094.0] Authenticated with ztn. ... snip ... [2023-02-18 11:05:23.466623 -0600][Debug ][ExDbgMsg ] [red-xfer2.unl.edu:1094] Moving MsgHandler: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2023-02-18 11:05:23.488628 -0600][Debug ][XRootD ] [red-xfer2.unl.edu:1094] Handling error while processing kXR_open (file: /user/ligo/test_access/access_ligo?, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: permission denied. [2023-02-18 11:05:23.488797 -0600][Debug ][ExDbgMsg ] [redirector.osgstorage.org:1094] Retry at server MsgHandler: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?tried=red-xfer2.unl.edu,xrootd-local.unl.edu, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2023-02-18 11:05:23.489062 -0600][Debug ][ExDbgMsg ] [redirector.osgstorage.org:1094] Moving MsgHandler: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?tried=red-xfer2.unl.edu,xrootd-local.unl.edu, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2023-02-18 11:05:23.530035 -0600][Debug ][XRootD ] [redirector.osgstorage.org:1094] Handling error while processing kXR_open (file: /user/ligo/test_access/access_ligo?tried=red-xfer2.unl.edu,xrootd-local.unl.edu, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2023-02-18 11:05:23.530101 -0600][Debug ][ExDbgMsg ] [redirector.osgstorage.org:1094] Calling MsgHandler: 0xca6850 (message: kXR_open (file: /user/ligo/test_access/access_ligo?tried=red-xfer2.unl.edu,xrootd-local.unl.edu, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2023-02-18 11:05:23.530171 -0600][Debug ][File ] [0xca0a80@root://redirector.osgstorage.org:1094//user/ligo/test_access/access_ligo?xrdcl.requuid=c242fdbf-a2e6-4b1b-9ec6-97c48063a2c6] Open has returned with status [ERROR] Server responded with an error: [3011] No servers are available to read the file. [2023-02-18 11:05:23.530187 -0600][Debug ][File ] [0xca0a80@root://redirector.osgstorage.org:1094//user/ligo/test_access/access_ligo?xrdcl.requuid=c242fdbf-a2e6-4b1b-9ec6-97c48063a2c6] Error while opening at redirector.osgstorage.org:1094: [ERROR] Server responded with an error: [3011] No servers are available to read the file. [2023-02-18 11:05:23.530248 -0600][Debug ][XRootD ] Redirect trace-back: [2023-02-18 11:05:23.530248 -0600][Debug ][XRootD ] 0. Redirected from: root://redirector.osgstorage.org:1094//user/ligo/test_access/access_ligo to: root://xrootd-local.unl.edu:1094/ [2023-02-18 11:05:23.530248 -0600][Debug ][XRootD ] 1. Redirected from: root://xrootd-local.unl.edu:1094/ to: root://red-xfer2.unl.edu:1094/ [2023-02-18 11:05:23.530248 -0600][Debug ][XRootD ] 2. Retrying: root://redirector.osgstorage.org:1094//user/ligo/test_access/access_ligo [2023-02-18 11:05:23.530285 -0600][Debug ][ExDbgMsg ] [redirector.osgstorage.org:1094] Destroying MsgHandler: 0xca6850. [0B/0B][100%][==================================================][0B/s] Run: [ERROR] Server responded with an error: [3011] No servers are available to read the file. (source) .. snip .... ``` To summarize: - Client was redirected twice, ending up at an actual data server. - Data server gave client a "permission denied" - Client returned to the first host (redirector), indicating which servers it had already retried. - Client open resulted in a `[ERROR] Server responded with an error: [3011] No servers are available to read the file.` I would suggest the following logic: 1. If a client connects to a data server and receives a specific error message, and 2. The client returns to the redirector excluding the last data server, and 3. The client gets a "No servers are available" from the redirector, then 4. The client should return the error message from last data server. So the above sequence would result in a: ``` [3010] Permission Denied. ``` Alternately, the client could generate: ``` [3010] Permission Denied. Last redirector response was [3011] No servers are available to read the file. ``` -- Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/1914 You are receiving this because you are subscribed to this thread. Message ID: <[log in to unmask]> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1