Print

---

Hi Brian,

This seems to be a different problem than the original one.

Previously, it was the FileStateHandler object that got deleted twice:

1) There was a read timeout for a request that was in the inQueue 
   (waiting for a response), this triggered the CloseHandler callback,
   which in turn triggered your routine DelayedClose, which deletes the
   FileStateHandler.

2) There was a second read timeout for the same request that was handled in 1).
   So the same things happend and we got a double delete. The bottom line is
   the request must have been reinserted into the inQueue.

Since we have seen something similar I pointed you to #483


Now, in this case, we see a double delete of the incoming response:

1) A new response comes in and a new Message object gets allocated:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClAsyncSocketHandler.cc#L585

   The AsyncSocketHandler holds a pointer to this message: pIncoming.
   Then the Message is being passed to the Stream::OnIncoming(), and in here 
   it is being passed further to the XRootDTransport::MessageReceived().

   In the MessageReceived we check if the SID of the message is in the Timed-Out SIDs set,
   and if yes we remove it from the set and delete the message:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClXRootDTransport.cc#L1128

   According to the memcheck output that's what happend.

   The MessageReceived routine returns DigestMsg, so the Stream::OnIncoming knows that the
   message was deleted and it returns:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClStream.cc#L465


2) Then from the memcheck output we now that the same message object has been deleted for 
   a second time in XRootDMsgHandler::Process method, the callstack looks like that:

   JobManager::RunJobs() -> HandleIncMsgJob::Run() -> XRootDMsgHandler::Process

   Now the really peculiar thing is that we are creating the HandleIncMsgJob only in one place:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClStream.cc#L509

   The HandleIncMsgJob is being created (it holds pointer to the message) and submitted 
   to the thread-pool.

   Note that this happens also in the Stream::OnIncoming, and it happens after the return 
   statement we reached in 1), so this means the message was not submitted to the 
   thread-pool in 1). It seems that the Stream::OnIncoming has been invoked for a second
   time with the same message. But the Stream::OnIncoming is only being called from the
   AsyncSocketHandler, and the AsyncSocketHandler resets the pointer to pIncoming after
   calling Stream::OnIncoming:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClAsyncSocketHandler.cc#L665

   So right now I cannot see why this happened and I'm really puzzled, 
   I need more time to investigate. 

Cheers,
Mcihal




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/xrootd/xrootd/issues/488#issuecomment-292227790

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1