Hi George,
Thanks for the clarification. I think the problem is with the command you use for TPC. Since you use /usr/bin/xrdcp in the ofs.tpc directive, the actual copy command xrootd proxy will run is
/usr/bin/xrdcp $1 $2
where $1 is root://eos/file, and $2 is /ECHO_path/file
This obviously won't work because the proxy instance doesn't have acces to /ECHO_path/file. What you can do is to replace /usr/bin/xrdcp by a script, say xrdcp-tpc.sh. In the script, you do
/usr/bin/xrdcp $1 root://ral_ceph/$2
Can you give this a try?
regards,
--
Wei Yang | [log in to unmask] | 650-926-3338 (O)
On 2/14/19, 4:45 AM, "George Patargias - UKRI STFC" <[log in to unmask]> wrote:
Hi Wei,
Thanks, I actually work with Ian on this.
1) There are two xrootd systemd processes running on the Ceph gateway: xroot@proxy and [log in to unmask] So as far as I understand these are two instances. The ofs.tpc directive is placed in the config of [log in to unmask]
2), The source is an eos instance. the ALICE people don't have direct access to that service but they can contance the eos admins.
3) The destination at RAL is a Ceph endpoint and all the errors I see, I pasted in the initial email.
Best,
George
________________________________
From: Yang, Wei <[log in to unmask]>
Sent: 12 February 2019 20:21:08
To: Patargias, George (STFC,RAL,SC); xrootd-l
Cc: Tim Adye; Johnson, Ian (STFC,RAL,SC)
Subject: Re: XRootD TPC error
Hi George,
I am not sure I have a clear picture. I am adding Tim Adye and Ian Johnson who are working with us on the ATLAS experiment side. I will start describing the issue we saw and I will have a few questions on what you see.
The CEPH plugin for Xrootd uses a single slash at the beginning of the path. So for a CEPH endpoint, the URL looks like root://host/path/file (the path/file are object id I guess), while all other xrootd endpoints use double slash like root://host//path/file. Xrootd TPC doesn't like single slash. Skipping the gory details about single vs double slashes, we agree the we will modify CEPH plugin to use double slash.
There is also a workaround by configuring the peer (not your end) to use "ofs.tpc oids ... ". We prefer not to use this in production, and to stick to the above change.
For the error you see, I need a bit more info about the setup and what you did:
1) is the xrootd TPC and ceph plugin run in same instance of Xrootd ? Or are they in separate instances. We know using separate instances (even on the same host) should work. I don't know whether anyone tried to run them in the same instance.
2) If the CEPH endpoint is the destination, do you see any error in TPC?
3) If the CEPH endpoint is the source, do you see any error in TPC.
In both 2) and 3), what is the error message in the CEPH xrootd server and from the client side?
Regarding the config line "ofs.tpc pgm /usr/bin/xrdcp -f --nopbar". I think it is best not to add "--nopbar". Some clients like FTS need this progress bar to determine that a long transfer is alive.
Also, for debugging purpose, /usr/bin/xrdcp can be replaced by a shell script. xrootd will pass the source (in URL format, root://remote-host//path/file) and destination (in /localpath/file) to the script. By using a script, you can log whatever you want to a file.
Also, depend on your setup, if the TPC isn't writing to the local storage, you will have to use a script (we can get to that later when we get answer about 1).
regards,
--
Wei Yang | [log in to unmask] | 650-926-3338 (O)
On 2/12/19, 2:52 AM, "George Patargias - UKRI STFC" <[log in to unmask]> wrote:
Yes, we use the same Ceph back-end plug-in for XRootD
on the dev and production gateways in RAL.
Thanks,
George
________________________________
From: Yang, Wei <[log in to unmask]>
Sent: 11 February 2019 19:31:55
To: Patargias, George (STFC,RAL,SC); xrootd-l
Subject: Re: XRootD TPC error
Does the xrootd gateway use the same plugin for CEPH, like what RAL uses?
--
Wei Yang | [log in to unmask] | 650-926-3338(O)
?-----Original Message-----
From: <[log in to unmask]> on behalf of George Patargias - UKRI STFC <[log in to unmask]>
Date: Monday, February 11, 2019 at 9:04 AM
To: xrootd-l <[log in to unmask]>
Subject: XRootD TPC error
Hello,
We have set up a pool and an xrootd gateway on our Ceph dev cluster for ALICE. Three out four tests (add, get, rm) are green
but the TPC test currently fails.
On the ALICE side the error is
xrdcp (TPC==1) exited with exit code 54: [ERROR] Server responded with an error: [3005] [ERROR] Server responded with an error: [3011] No such file or directory
When I try to copy the file on, I get
Run: [ERROR] Server responded with an error: [3005] [ERROR] Server responded with an error: [3010] Unable to open - tpc origin mismatch /04/36435/05422528-a162-11e8-b5f9-a310691b2def; operation not permitted
I use this directive
ofs.tpc pgm /usr/bin/xrdcp -f --nopbar
Is it in the wrong place in the config file (see attached xrootd-proxy.cfg)? Or is there anything missing from it?
Many thanks in advance,
George
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
