Hi Patrick,
The xrdfs way is the "official" way to determine the space name (i.e.
token). Of course, the way ATLAS works, you also can derive it from the
path which, for ATLAS, is much easier.
You could integrate this with FRM and pass information to the script that
you would write for FRM to invoke to make the transfer using CGI
information. That doesn't help you with mtime. In fact, the only thing you
get here is the queue management and transfer pacing. Sounds like you
already have that worked out so this might not be the most effecient
approach for you. However, if you work it all out using FRM that would
likely be a good addition to the xrootd repo for others to use.
Whatever you decide, if there additional features you would like to see
xrootd have (well, other than just doing all of it for you :-), for
instance, being able to set mtime via xrdcp (e.g. something like a -p
option in cp), please just file an enhancement request in github so we can
track that and make sure it gets into our schedule.
Andy
On Fri, 2 Mar 2018, Patrick McGuigan wrote:
> Thanks Wei,
>
> I will look at implementing a service or the copy script can build a script that I can push to the destination server to a mass update.
>
>
> I did find an awkward way to determine the space name associated with a file:
>
> # xrdfs localhost query xattr /xrd/atlasproddisk/rucio/mc15_13TeV/cc/b9/EVNT.06212746._084813.pool.root.1
> oss.cgroup=ATLASPRODDISK&oss.type=f&oss.used=3045780&oss.mt=1519975567&oss.ct=1519976072&oss.at=1519975567&oss.u=*&oss.g=*&oss.fs=w&ofs.ap=a
>
> I'll have to investigate if using xrdfs is worth the effort.
>
> Thanks again,
>
> Patrick
>
>
>
>
> On 03/02/2018 01:31 AM, Yang, Wei wrote:
>> Hi Patrick,
>>
>> I donšt know the answer of the 1st question. For the 2nd question, there is no way in xrootd to do this. So I use a separate help service on the data servers:
>>
>> So if your copy script do something like "echo filename mtime atime | nc dataserver 5151", then this script will help you set the mtime.
>>
>> #!/bin/sh
>>
>> # xinetd service
>> #service setmtime
>> #{
>> # type = UNLISTED
>> # socket_type = stream
>> # protocol = tcp
>> # wait = no
>> # user = atldq2
>> # server = /u/at/atldq2/bin/setmtime.sh
>> # port = 5151
>> # log_on_failure = HOST ATTEMPT
>> # log_on_success =
>> # disable = no
>> #}
>>
>> exec > /dev/null 2>&1
>> read line
>> set $line
>> [ $# -ne 2 -a $# -ne 3 ] && exit
>> [ ! -f $1 ] && exit
>> if echo $2 | egrep -q '^[0-9]+$'; then # set mtime
>> touch -m -c -d @$2 $1
>> fi
>> [ $# -eq 2 ] && exit
>> if echo $3 | egrep -q '^[0-9]+$'; then # set atime
>> touch -a -c -d @$3 $1
>> fi
>>
>>
>>
>>
>> --
>> Wei Yang | [log in to unmask] | 650-926-3338(O)
>>
>>
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: <[log in to unmask]> on behalf of Patrick McGuigan <[log in to unmask]>
>> Date: Friday, March 2, 2018 at 2:00 AM
>> To: xrootd-l <[log in to unmask]>
>> Subject: Mass data movement between dataservers
>>
>>> Hi,
>>>
>>> I need to move large numbers of files between dataservers. In one case this is
>>> because I want to retire an existing host and, in another case, I need to free
>>> up space on data servers so that overall usage among the dataservers is more
>>> balanced.
>>>
>>> I can do this with existing scripts that I have used in the past, but there are
>>> places I would like to improve things.
>>>
>>> The general operation is to:
>>> 1) create a list of files to be moved and a list of dataservers that will accept
>>> the copies.
>>>
>>> 2) Determine the space name associated with each file, so that the destination
>>> maintains the space name.
>>>
>>> 3) In multiple threads copy the file with space information, verify the local
>>> and remote checksums, delete the local file. I normally kick off one process
>>> per destination host.
>>>
>>>
>>> I utilize a mapping function that looks at the path of a local file to determine
>>> what space name it should belong to. I am wondering if I can ask the dataserver
>>> holding the file, what the space name is. Is this possible, or if I can make an
>>> xrdcp replicate the opaque information.
>>>
>>>
>>> Another place I would like to improve things is to maintain the timestamps
>>> associated with the original file, so that the modtime of the copied file
>>> matches the modtime of the original file. Is there there a way of doing the
>>> equivalent to:
>>>
>>> touch -mt <STAMP> path
>>>
>>> on a file stored in xrootd?
>>>
>>>
>>>
>>> Should I be using FRM to manage all of this?
>>>
>>>
>>> Regards,
>>>
>>> Patrick
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
