I would dare say this is due to errors in the buffering logic in the TPC
layer. That layer tries to accumulate 1MB of data before sending it off to
the filesystem, ostensibly because anything less may cause issues with
Ceph (I don't know why this is the case but the comments say it is).
That's a tricky affair and when I looked at the code I was mystified by
it's logic. Given the complexity here I would say that is where the
problem occurs and would only affect TPC Pull requests.
This line of thought is further bolstered by corresponding to what you
have observed.
Andy
On Fri, 19 Feb 2021, snafus wrote:
> Dear @esindril ,
> I'm not sure if this is related (and have not fully finished trying to diagnose the problem), but at RAL I am seeing checksum issues in our Test TPC gateway (5.1.0-rc7)
> Our issue, is also with PULL mode.
> It appears that sometimes, at 1MiB points in the file, the preceding n*1024 bytes from the ith MiB are copied into the ith MiB position. It looks like i == n in most cases, but I'm still working through the logic ...
> It is not quite so straightforward, as we get different checksums from repeated tests, but they quite often match each other - i.e. the corruption is happening in similar, but slightly different ways, but apparently on these 1MiB boundaries.
>
> I note these are in async io transfers (I don't recall if we were always doing this for RAL's TPC tests.
> Sorry if it's not relevant, and only a partial diagnosis, but I hope it's useful to add a similar datapoint here.
>
> Cheers,
> James
>
>
>
>
>
> --
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub:
> https://github.com/xrootd/xrootd/issues/1404#issuecomment-782065719
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/xrootd/xrootd/issues/1404#issuecomment-782236886",
"url": "https://github.com/xrootd/xrootd/issues/1404#issuecomment-782236886",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
