Hi Brian,

This seems to be a different problem than the original one.

Previously, it was the FileStateHandler object that got deleted twice:

1. There was a read timeout for a request that was in the inQueue
   (waiting for a response), this triggered the CloseHandler callback,
   which in turn triggered your routine DelayedClose, which deletes the
   FileStateHandler.

2. There was a second read timeout for the same request that was handled in 1).
   So the same things happend and we got a double delete. The bottom line is
   the request must have been reinserted into the inQueue.

Since we have seen something similar I pointed you to #483

Now, in this case, we see a double delete of the incoming response:

1. A new response comes in and a new Message object gets allocated:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClAsyncSocketHandler.cc#L585

The AsyncSocketHandler holds a pointer to this message: pIncoming.
Then the Message is being passed to the Stream::OnIncoming(), and in here
it is being passed further to the XRootDTransport::MessageReceived().

In the MessageReceived we check if the SID of the message is in the Timed-Out SIDs set,
and if yes we remove it from the set and delete the message:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClXRootDTransport.cc#L1128

According to the memcheck output that's what happend.

The MessageReceived routine returns DigestMsg, so the Stream::OnIncoming knows that the
message was deleted and it returns:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClStream.cc#L465

2. Then from the memcheck output we now that the same message object has been deleted for
   a second time in XRootDMsgHandler::Process method, the callstack looks like that:

   JobManager::RunJobs() -> HandleIncMsgJob::Run() -> XRootDMsgHandler::Process

   Now the really peculiar thing is that we are creating the HandleIncMsgJob only in one place:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClStream.cc#L509

The HandleIncMsgJob is being created (it holds pointer to the message) and submitted
to the thread-pool.

Note that this happens also in the Stream::OnIncoming, and it happens after the return
statement we reached in 1), so this means the message was not submitted to the
thread-pool in 1). It seems that the Stream::OnIncoming has been invoked for a second
time with the same message. But the Stream::OnIncoming is only being called from the
AsyncSocketHandler, and the AsyncSocketHandler resets the pointer to pIncoming after
calling Stream::OnIncoming:

https://github.com/xrootd/xrootd/blob/master/src/XrdCl/XrdClAsyncSocketHandler.cc#L665

So right now I cannot see why this happened and I'm really puzzled,
I need more time to investigate.

Cheers,
Mcihal

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1