 |
 |
On Oct 5, 2012, at 1:15 PM, Matevz Tadel <[log in to unmask]> wrote: > Hi Brian, > > On 10/05/12 06:48, Brian Bockelman wrote: >> Hi Matevz, >> >> Two thoughts: >> >> 1) Hadoop already records the checksums when the files are written via gridftp. However, as the filesystem doesn't have extended attributes, we shove it into a different file (the checksum for /foo is in /cksums/foo). > > Cool, I see these for our phedex directory ... and we have all of them, cksum, crc32, adler32 and md5 :) So, in our use case, we could use the checksums that are already in our hadoop to crosscheck that the "uncorrupted" file is indeed ok. > Of course, if you use those, you might have to volunteer yourself to getting the checksums calculated for Xrootd-based writes! :) >> 2) I wonder if we could hook the redirector up to the CMS data management system and look these up from that database. > > That would definitely be the authoritative source :) We'd still need to do something for the user and group directories that are not centrally managed. > > Matevz > >> Brian >> >> On Oct 4, 2012, at 9:33 PM, "Andrew Hanushevsky" <[log in to unmask]> wrote: >> >>> Hi Matevz, >>> >>> We of course use extended attributes. If hadoop doesn't naturally have them, you can write your own implementation using XrdSysFAttr as the template like we do for implementing each different file system interface, sigh. >>> >>> Andy >>> >>> -----Original Message----- From: Matevz Tadel >>> Sent: Thursday, October 04, 2012 7:07 PM >>> To: Andrew Hanushevsky >>> Cc: xrootd-dev >>> Subject: Re: Getting checksum from a server >>> >>> Hi Andy, >>> >>> Thanks! I never looked into this part of docs before so I naively expected that >>> checksums can always be obtained / recalculated on demand. I guess I'll live >>> without them for now :) >>> >>> Out of curiosity, how do you store such meta information? It seems problematic >>> for file-systems like hadoop? >>> >>> Matevz >>> >>> On 10/04/12 18:54, Andrew Hanushevsky wrote: >>>> Hi Matevz, >>>> >>>> Please read the documentation on how to configure a server to provide checksums. >>>> The message says it all, the server was not configured to do it. >>>> >>>> a) http://xrootd.org/doc/dev/xrd_config.htm#_Toc336083799 >>>> >>>> b) (optionally) http://xrootd.org/doc/dev/ofs_config.htm#_Toc323059149 >>>> >>>> Andy >>>> >>>> -----Original Message----- From: Matevz Tadel >>>> Sent: Thursday, October 04, 2012 6:45 PM >>>> To: xrootd-dev >>>> Cc: Jeff Dost >>>> Subject: Getting checksum from a server >>>> >>>> Hi, >>>> >>>> How can I get a checksum from a server? I tried using XrdClientAdmin but I get >>>> this printed out when calling admin->GetChecksum(): >>>> >>>> 121004 18:42:10 13529 Xrd: CheckErrorStatus: Server [xrootd.unl.edu] declared: >>>> query chksum is not supported(error code: 3013) >>>> >>>> so it seems this is a dead end. Any other options? >>>> >>>> Matevz >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-DEV list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-DEV list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-DEV list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1