 |
 |
Hi all, we will soon have to migrate our data to a new set of file servers, and are pondering how to best approach this. The total transfer volume is in the order of ~5PB, and we will move from several dozen file systems and more partitions (~80 FS + 1 FS รก 16 partitions) to a single FS with only logical partitions (~64). All machines are at least xrootd v4.0.4. Our main issue is that only a fraction of the new storage is actually "new" space for users. We must be able to transfer the bulk of data without users being able to write more than their new share. Is there a recommended approach for this? Anyone have some best practices to share? So far we have considered: - Using xrootd itself to copy-and-delete from old to new servers as a user. + We are reluctant to do this, since it would mean making the entire volume writeable to users. - Using rsync or gpfs tools to transfer the oss localroot/space manually. + Transfer both oss.space and oss.localroot content. I am suspecting that this does not make sense, as the links inside localroot must be changed anyway. Or is there a way to do this? + Transfer only data from the oss.space, and rebuild the namespace. Are there some constraints on how xrootd manages the two that we must preserve? - Our plan is basically to copy all oss.space content from all file systems, then rebuild the namespace from xattr user.XrdFrm.Pfn - Does the allocation of data to oss.space volumes make assumptions? On a 16 partition oss.space, there is a definitive pattern of the paths in each partition, and paths also look like hashes. Do we have to preserve this, or does xrootd take whatever is there? - We expect the namespace to be unique, i.e. files with same logical path are the same. Do we have to check for name collisions when merging partitions? - Our oss.space partitions will only be logical, so load balancing does not matter for us. Can we use one exclusive oss.space "partition" per source server? Does xrootd even need to be aware of these partitions, or is linking to them from the namespace enough? All suggestions and anecdotes are welcome. ;) Cheers, Max ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1