 |
 |
Hi Wilko, The protocol document says that when writing a file (including updating) should the data server go down, the client must continue trying to contact the data server until the configurable amount of time has elapsed. At that point, the client must exit. So, it's clear that the client is not following the protocol here. It should never go back to the redirector when a write operation fails. The 9 minute wait is the last desperate attempt by the redirector to save the client. It could probably do better but right now we do not record whether the client actually opened the file for writing (and it shouldn't have to if the client followed the protocol). Now for case two. Since DS0 is not available, the redirector chose DS1. One can argue that was inappropriate but given the resources come and go, that was he only alternative given the current architecture. Yes, it does lead to having a turd file left around -- a problem not easily solved in an automatic way. That said, as the client should not have gone back to the redirector after DS1 became available, the subsequent problem should not have happened. However, given the action, it does point out that the client is treating wites just as reads and inappropriately continuing to write to a new server as if nothing happened. That should not be so. That leaves us with case 3 that you did not pursue. Had you rerun the test at this point with all data servers functioning, either DS1 or DS1 would have been picked. Leaving one corect file and one incorrect file. We have discussed what should happen in this case. We have two solutions: a) The redirector should ask all but the selected data server to remove the file. That's fine for writing but potentially disastrous for updating as the file to be updated might actually be completely lost under certain failure situations. b) Disallow access since this is clearly an erroneous situation. That works most of the time but not all of the time if the data server hosting one of the copies is not currently up. I think we decided that the safer course of action is (b); even though it's not a complete solution. To provide a complete solution would require implementing a transactional system together with a persistent store (i.e., database) to keep track of all such situations. Andy On Wed, 8 Jun 2005, Wilko Kroeger wrote: > > > Hello > > I did some test using xrdcp to write to xrootd and what happens if a data > server goes down. I am using xrootd version 20050525-0946 for the data > servers and redirector and 20050509-2006 for xrdcp. > > The main question is: > Should xrdcp after a data server goes down ever be redirected to a new > data server or should it just exit ? > That's what I assumed but as described in the two tests below the current > behavior is different. > > > First Test: > =========== > > For the first test writing is done via the redirector and once xrdcp > is redirected to the data server and starts transferring data the > xrootd on the data server is stopped. > > I observed the following: > 1) xrdcp goes back to the redirector after the data server is stopped. > 2) the redirector tells the client to wait which ends up to about 9 mins > 3) After waiting for 9 mins the client tries to open the file on the > redirector and > a) crashes with a core > or > b) tries to write, doesn't succeed and stops (no core) > > The log files are available in: > /nfs/objyserv01/objy/databases/wilko/xrootd/problems/xrdcp_writeServerGoesDown > > xrdcp.log : output of xrdcp > rdr_xrdlog.log : xrootd log file of the redirector (datadevsol12) > datadevsol02_xrdlog.log : xrootd log file of the data server > core.3067 : core file from xrootd (the binary is > ~wilko/bbtest/xrootd/20050509-2006/bin/xrdcp on RHEL3) > > > In the case that xrdcp doesn't create a core the log files are: > xrdcp_noCore.log > rdr_xrdlog_noCore.log > datadevsol02_xrdlog_noCore.log > > > In the case of writing, should xrdcp go back to the redirector, or should > it just wait for a certain time, and if the data server isn't available > just quit ? > > > Second Test: > =========== > > For the second test the first test is run twice with a restart of > one of the data servers in between. > Assuming there are two data servers, DS0 and DS1, the following is done: > > 1) xrdcp a new file via the redirector > 2) xrdcp is redirected to DS0, and starts to transfer the file > 3) xrootd on DS0 is stopped, xrdcp waits for 9 mins and then exits > leaving an incomplete copy of the file on DS0 > 4) xrdcp is restarted again with the same file name > 5) xrdcp is redirected to DS1 (DS0 is still down), and starts to transfer > the file > 6) xrootd on DS1 is stopped. xrdcp goes back to the redirector > 7) xrootd on DS0 is restarted > 8) xrdcp is redirected to DS0 and continues to transfer the file. > It doesn't start over writing the file but it continues where it > stopped in step 6, which means the file could be corrupted (e.g.: > in step 2 the first 100MB are transfered and in step 6 it starts > at an offset of 150MB). > > > This question again is: > Shouldn't xrdcp just give up after it failed to write the file. > > If it get redirected after a data server goes down, shouldn't > it start to write the file from the beginning? > > I couldn't test the latest version of xrdcp (session id problem) and I > don't know if the latest version behaves the same way. > > > Cheers, > wilko > > > > > > > > > > > > >