 |
 |
The base encoding comes from the RFC 3230 that describes how the checksum should be returned. The RFC essentially says that legacy checksums need to be returned unencoded while other checksums should be returned encoded base64. It winds up that crc32c is a new checksum while crc32 is a legacy checksum (adler32 was not mentioned). At least, that's the reading of the RFC 3230. On Wed, 8 Mar 2023, Petr Vokac wrote: > Following document doesn't speak about base64 encoding for crc32c > > https://www.iana.org/assignments/http-dig-alg/http-dig-alg.xml > > and Davix checksum extraction magic also don't really expect base64 encoded > crc32c > > https://gitlab.cern.ch/dmc/davix/-/blob/devel/src/utils/checksum_extractor.cpp#L72-L111 > > btw: crc32c is not supported by all our storage implementation (e.g. dCache) > > https://github.com/dCache/dcache/blob/master/modules/dcache/src/main/java/org/dcache/util/Checksums.java#L50-L55 > > Petr > > On 3/8/23 16:00, Doidge, Matt wrote: >> Hello all, >> I've was tinkering after adding crc32c support to one of our xrootd servers >> and saw an odd behaviour with gfal-sum over https when trying to return a >> crc32c checksum - it's returning an encoded version of the checksum. >> gfal-sum over root works as expected. xrdcrc32c performed on the file >> locally returns the expected result. >> >> gfal-sum >> root://stor015.hec.lancs.ac.uk:1095/cephfs/grid/dteam/crc32chttpstest5.txt >> crc32c >> root://stor015.hec.lancs.ac.uk:1095/cephfs/grid/dteam/crc32chttpstest5.txt >> d98fa965 >> >> $gfal-sum >> https://stor015.hec.lancs.ac.uk:1095/cephfs/grid/dteam/crc32chttpstest5.txt >> crc32c >> https://stor015.hec.lancs.ac.uk:1095/cephfs/grid/dteam/crc32chttpstest5.txt >> 2Y+pZQ== >> >> $ echo "2Y+pZQ==" | base64 -d | xxd -ps >> d98fa965 >> >> I tried to deduce if this is a problem within gfal tools, and looking into >> the verbose output (-vvv) I think the "2Y+pZQ==" is what is being returned >> by the server, from a snippet below: >> >> DEBUG Davix: Request sent; retry is 0. >> INFO Davix: < HTTP/1.1 200 OK >> INFO Davix: < Connection: Keep-Alive >> INFO Davix: < Server: XrootD/v5.5.2 >> INFO Davix: < Content-Length: 996 >> INFO Davix: < Digest: crc32c=2Y+pZQ== >> INFO Davix: < >> DEBUG Davix: End of headers. >> >> The file exists on disk with the extended attribute below: >> user.XrdCks.crc32c=0sY3JjMzJjAAAAAAAAAAAAAAAAAABkCJzmAAAAAAAAAATZj6llAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA >> >> (which again if you binary decode you can see the checksum in there). >> >> Note with adler32 we do not see anything out of the ordinary, checksums are >> returned as expected with https:// or root:// Our filesystem xroot is >> writing to is cephfs, the operating system is Rocky8, and xroot version >> 5.5.3. The line used to configure our checksums is just: >> >> xrootd.chksum max 32 adler32 crc32c >> >> I am wondering if this is an xroot bug (then I'll open an issue), a likely >> bug in our setup (then I'd appreciate a pointer to it!) or a problem with >> gfal (where I'll throw them a ticket). >> >> Thanks in advance all, >> Matt >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1