Hi Michael,
Well, we could enhance the endsess request. This is a request that the client sends to make sure that the previous session gets killed; though it is not necessarily always successful but when it’s not, there should not be a pending duplicate open. The original intent was to avoid lock conflicts. I can enhance it to deep six any outstanding messages by simply issues a shutdown on the server’s side. That should take care of the problem. You think that will convince Eric?
Andy
From: murrayc3
Sent: Tuesday, March 20, 2018 4:25 AM
To: xrootd/xrootd
Cc: Andrew Hanushevsky ; Comment
Subject: Re: [xrootd/xrootd] C++ API to XRootD should allow the "open file" retry logic to be turned off (#673)
Hi Andy,
We have discussed a couple of concepts/ideas on our side of the pond and I am going to try to write them down for you here is in this ticket/issue.
Briefly the two concepts/ideas are:
a.. Your server side fix of checking for mutual exclusivity before truncating a file won't work for small files.
b.. We're try to find a TCP/IP option that will guarantee on the client side that a connection to the server is truly "dead" before creating a brand new TCP/IP connection.
Going into more details with your server side fix in XRootD 4.8.2 RC2. The server will now reject a concurrent "open the file and truncate" request when there is an on going file transfer. This is great for relatively long files. Eric has pointed out that very small files will not be protected however. Here is the scenario with XRootD 4.8.2 RC2:
1.. XRootD client sends a "open file and truncate" request to the server.
2.. The corresponding TCP/IP packet gets dropped/lost.
3.. The XRootD client library times out and closes the TCP/IP connection.
4.. The XRootD client library opens a brand new connection to the server.
5.. The XRootD client completes the entire transmission of the small file to the server (open , write, write... and then close).
6.. The lost "open file and truncate" TCP/IP packet gets retransmitted by the TCP/IP stack underneath the XRootD client library.
7.. The server lets the request take the mutual exclusion lock on the file for writing.
8.. The server truncates the file.
9.. The server receives the FIN packet of the close request on the connection made ages ago by the XRootD client library.
We're done, the small file has now been truncated.
Going into more details about making sure the TCP/IP connection is really closed on the client side before creating a new one. We believe the first request to "open and truncate" the file is being replayed by the TCP/IP stack underneath the XRootD client library, even after the library has called close() on the connection's file descriptor. Eric has looked at some TCP/IP socket options and thinks there is a way to guarantee the connection is really closed before opening a new one. Eric sent the following e-mail to us here at CERN:
Maybe we could play with this (man 7 socket):
SO_LINGER
Sets or gets the SO_LINGER option. The argument is a linger
structure.
struct linger {
int l_onoff; /* linger active */
int l_linger; /* how many seconds to linger for */
};
When enabled, a close(2) or shutdown(2) will not return until
all queued messages for the socket have been successfully sent
or the linger timeout has been reached. Otherwise, the call
returns immediately and the closing is done in the background.
When the socket is closed as part of exit(2), it always
lingers in the background.
Cheers,
Eric
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/xrootd/xrootd","title":"xrootd/xrootd","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/xrootd/xrootd"}},"updates":{"snippets":[{"icon":"PERSON","message":"@abh3 in #673: Hi Michael,\n\nWell, we could enhance the endsess request. This is a request that the client sends to make sure that the previous session gets killed; though it is not necessarily always successful but when it’s not, there should not be a pending duplicate open. The original intent was to avoid lock conflicts. I can enhance it to deep six any outstanding messages by simply issues a shutdown on the server’s side. That should take care of the problem. You think that will convince Eric?\n\nAndy\n\nFrom: murrayc3 \nSent: Tuesday, March 20, 2018 4:25 AM\nTo: xrootd/xrootd \nCc: Andrew Hanushevsky ; Comment \nSubject: Re: [xrootd/xrootd] C++ API to XRootD should allow the \"open file\" retry logic to be turned off (#673)\n\nHi Andy,\n\nWe have discussed a couple of concepts/ideas on our side of the pond and I am going to try to write them down for you here is in this ticket/issue.\n\nBriefly the two concepts/ideas are:\n\n a.. Your server side fix of checking for mutual exclusivity before truncating a file won't work for small files. \n b.. We're try to find a TCP/IP option that will guarantee on the client side that a connection to the server is truly \"dead\" before creating a brand new TCP/IP connection. \nGoing into more details with your server side fix in XRootD 4.8.2 RC2. The server will now reject a concurrent \"open the file and truncate\" request when there is an on going file transfer. This is great for relatively long files. Eric has pointed out that very small files will not be protected however. Here is the scenario with XRootD 4.8.2 RC2:\n\n 1.. XRootD client sends a \"open file and truncate\" request to the server. \n 2.. The corresponding TCP/IP packet gets dropped/lost. \n 3.. The XRootD client library times out and closes the TCP/IP connection. \n 4.. The XRootD client library opens a brand new connection to the server. \n 5.. The XRootD client completes the entire transmission of the small file to the server (open , write, write... and then close). \n 6.. The lost \"open file and truncate\" TCP/IP packet gets retransmitted by the TCP/IP stack underneath the XRootD client library. \n 7.. The server lets the request take the mutual exclusion lock on the file for writing. \n 8.. The server truncates the file. \n 9.. The server receives the FIN packet of the close request on the connection made ages ago by the XRootD client library.\n We're done, the small file has now been truncated. \nGoing into more details about making sure the TCP/IP connection is really closed on the client side before creating a new one. We believe the first request to \"open and truncate\" the file is being replayed by the TCP/IP stack underneath the XRootD client library, even after the library has called close() on the connection's file descriptor. Eric has looked at some TCP/IP socket options and thinks there is a way to guarantee the connection is really closed before opening a new one. Eric sent the following e-mail to us here at CERN:\n\nMaybe we could play with this (man 7 socket):\n \n \n SO_LINGER\n Sets or gets the SO_LINGER option. The argument is a linger\n structure.\n \n struct linger {\n int l_onoff; /* linger active */\n int l_linger; /* how many seconds to linger for */\n };\n \n When enabled, a close(2) or shutdown(2) will not return until\n all queued messages for the socket have been successfully sent\n or the linger timeout has been reached. Otherwise, the call\n returns immediately and the closing is done in the background.\n When the socket is closed as part of exit(2), it always\n lingers in the background.\n \n \nCheers,\nEric\n—\nYou are receiving this because you commented.\nReply to this email directly, view it on GitHub, or mute the thread.\n"}],"action":{"name":"View Issue","url":"https://github.com/xrootd/xrootd/issues/673#issuecomment-374756732"}}}
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
