Thanks Wei,
I will look at implementing a service or the copy script can build a script that I can push to the destination server to a mass update.
I did find an awkward way to determine the space name associated with a file:
# xrdfs localhost query xattr /xrd/atlasproddisk/rucio/mc15_13TeV/cc/b9/EVNT.06212746._084813.pool.root.1
oss.cgroup=ATLASPRODDISK&oss.type=f&oss.used=3045780&oss.mt=1519975567&oss.ct=1519976072&oss.at=1519975567&oss.u=*&oss.g=*&oss.fs=w&ofs.ap=a
I'll have to investigate if using xrdfs is worth the effort.
Thanks again,
Patrick
On 03/02/2018 01:31 AM, Yang, Wei wrote:
> Hi Patrick,
>
> I donĀ¹t know the answer of the 1st question. For the 2nd question, there is no way in xrootd to do this. So I use a separate help service on the data servers:
>
> So if your copy script do something like "echo filename mtime atime | nc dataserver 5151", then this script will help you set the mtime.
>
> #!/bin/sh
>
> # xinetd service
> #service setmtime
> #{
> # type = UNLISTED
> # socket_type = stream
> # protocol = tcp
> # wait = no
> # user = atldq2
> # server = /u/at/atldq2/bin/setmtime.sh
> # port = 5151
> # log_on_failure = HOST ATTEMPT
> # log_on_success =
> # disable = no
> #}
>
> exec > /dev/null 2>&1
> read line
> set $line
> [ $# -ne 2 -a $# -ne 3 ] && exit
> [ ! -f $1 ] && exit
> if echo $2 | egrep -q '^[0-9]+$'; then # set mtime
> touch -m -c -d @$2 $1
> fi
> [ $# -eq 2 ] && exit
> if echo $3 | egrep -q '^[0-9]+$'; then # set atime
> touch -a -c -d @$3 $1
> fi
>
>
>
>
> --
> Wei Yang | [log in to unmask] | 650-926-3338(O)
>
>
>
>
>
>
>
> -----Original Message-----
> From: <[log in to unmask]> on behalf of Patrick McGuigan <[log in to unmask]>
> Date: Friday, March 2, 2018 at 2:00 AM
> To: xrootd-l <[log in to unmask]>
> Subject: Mass data movement between dataservers
>
>> Hi,
>>
>> I need to move large numbers of files between dataservers. In one case this is
>> because I want to retire an existing host and, in another case, I need to free
>> up space on data servers so that overall usage among the dataservers is more
>> balanced.
>>
>> I can do this with existing scripts that I have used in the past, but there are
>> places I would like to improve things.
>>
>> The general operation is to:
>> 1) create a list of files to be moved and a list of dataservers that will accept
>> the copies.
>>
>> 2) Determine the space name associated with each file, so that the destination
>> maintains the space name.
>>
>> 3) In multiple threads copy the file with space information, verify the local
>> and remote checksums, delete the local file. I normally kick off one process
>> per destination host.
>>
>>
>> I utilize a mapping function that looks at the path of a local file to determine
>> what space name it should belong to. I am wondering if I can ask the dataserver
>> holding the file, what the space name is. Is this possible, or if I can make an
>> xrdcp replicate the opaque information.
>>
>>
>> Another place I would like to improve things is to maintain the timestamps
>> associated with the original file, so that the modtime of the copied file
>> matches the modtime of the original file. Is there there a way of doing the
>> equivalent to:
>>
>> touch -mt <STAMP> path
>>
>> on a file stored in xrootd?
>>
>>
>>
>> Should I be using FRM to manage all of this?
>>
>>
>> Regards,
>>
>> Patrick
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|