Yes, that would be the other suspect here – invalid handling of the SID.
From: Matevž Tadel
Sent: Monday, March 25, 2019 4:41 PM
To: xrootd/xrootd
Cc: Andrew Hanushevsky ; Mention
Subject: Re: [xrootd/xrootd] [XrdCl] MsgHandler double delete (#943)
Yay, I've got something from valgrind ... it's not in MessageHandler but has to do with Message. Let me know if this makes sense, I'll keep pounding at it for a while :)
[tcpkill was not good enough, finding socket fd with netstat and then calling close(fd) in gdb did the trick).]
[2019-03-25 16:35:00.782184 -0700][Error ][XRootD ] [[log in to unmask]:1094] MsgHandler created: 53fab20 (message: kXR_read (handle: 0x00000000, offset: 36175872, size: 524288) ).
[2019-03-25 16:35:00.783144 -0700][Error ][XRootD ] [[log in to unmask]:1094] Moving MsgHandler: 53fab20 (message: kXR_read (handle: 0x00000000, offset: 36175872, size: 524288) ) from out-queu to in-queue.
[2019-03-25 16:35:00.811473 -0700][Error ][XRootD ] [[log in to unmask]:1094] Calling MsgHandler: 53fab20 (message: kXR_read (handle: 0x00000000, offset: 36175872, size: 524288) ) with status: [SUCCESS] .
[2019-03-25 16:35:00.812114 -0700][Error ][XRootD ] [[log in to unmask]:1094] Destroying MsgHandler: 53fab20.
[2019-03-25 16:35:16.526807 -0700][Error ][XRootD ] [[log in to unmask]:1094] MsgHandler created: 5454260 (message: kXR_read (handle: 0x00000000, offset: 36700160, size: 524288) ).
[2019-03-25 16:35:16.537508 -0700][Error ][Poller ] <[::ffff:169.228.130.108]:38338><--><[::ffff:169.228.130.61]:1094> Unable to enable write notifications: modifying poll events
[2019-03-25 16:35:16.538760 -0700][Error ][XRootD ] [[log in to unmask]:1094] Unable to send the message kXR_read (handle: 0x00000000, offset: 36700160, size: 524288): [FATAL] Poller error
==119570== Thread 87:
==119570== Invalid read of size 2
==119570== at 0x6547A4E: XrdCl::SIDManager::ReleaseSID(unsigned char*) (XrdClSIDManager.cc:61)
==119570== by 0x655F97A: XrdCl::MessageUtils::SendMessage(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams const&, XrdCl::LocalFileHandler*) (XrdClMessageUtils.cc:122)
==119570== by 0x6566096: XrdCl::FileStateHandler::IssueRequest(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams&) (XrdClFileStateHandler.cc:1956)
==119570== by 0x656BC9C: XrdCl::FileStateHandler::SendOrQueue(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams&) (XrdClFileStateHandler.cc:1588)
==119570== by 0x656F1F3: XrdCl::FileStateHandler::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFileStateHandler.cc:692)
==119570== by 0x6563129: XrdCl::File::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFile.cc:194)
==119570== by 0x62D8088: XrdPosixFile::Read(XrdOucCacheIOCB&, char*, long long, int) (XrdPosixFile.cc:409)
==119570== by 0x6B0EB2F: XrdFileCache::File::ProcessBlockRequests(std::list<XrdFileCache::Block*, std::allocator<XrdFileCache::Block*> >&, bool) (XrdFileCacheFile.cc:466)
==119570== by 0x6B128A1: XrdFileCache::File::Read(XrdFileCache::IO*, char*, long long, int) (XrdFileCacheFile.cc:627)
==119570== by 0x6B1AC23: XrdFileCache::IOEntireFile::Read(char*, long long, int) (XrdFileCacheIOEntireFile.cc:174)
==119570== by 0x62DBF41: XrdPosixXrootd::Pread(int, void*, unsigned long, long) (XrdPosixXrootd.cc:725)
==119570== by 0x60BC33F: XrdPssFile::Read(void*, long, unsigned long) (XrdPss.cc:811)
==119570== Address 0x54b1ce0 is 0 bytes inside a block of size 24 free'd
==119570== at 0x4A06C20: realloc (vg_replace_malloc.c:662)
==119570== by 0x653F0D9: XrdCl::XRootDTransport::MultiplexSubStream(XrdCl::Message*, unsigned short, XrdCl::AnyObject&, XrdCl::PathID*) (XrdClBuffer.hh:90)
==119570== by 0x6539E37: XrdCl::Stream::Send(XrdCl::Message*, XrdCl::OutgoingMsgHandler*, bool, long) (XrdClStream.cc:300)
==119570== by 0x653777D: XrdCl::Channel::Send(XrdCl::Message*, XrdCl::OutgoingMsgHandler*, bool, long) (XrdClChannel.cc:312)
==119570== by 0x6536867: XrdCl::PostMaster::Send(XrdCl::URL const&, XrdCl::Message*, XrdCl::OutgoingMsgHandler*, bool, long) (XrdClPostMaster.cc:191)
==119570== by 0x655F8E3: XrdCl::MessageUtils::SendMessage(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams const&, XrdCl::LocalFileHandler*) (XrdClMessageUtils.cc:113)
==119570== by 0x6566096: XrdCl::FileStateHandler::IssueRequest(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams&) (XrdClFileStateHandler.cc:1956)
==119570== by 0x656BC9C: XrdCl::FileStateHandler::SendOrQueue(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams&) (XrdClFileStateHandler.cc:1588)
==119570== by 0x656F1F3: XrdCl::FileStateHandler::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFileStateHandler.cc:692)
==119570== by 0x6563129: XrdCl::File::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFile.cc:194)
==119570== by 0x62D8088: XrdPosixFile::Read(XrdOucCacheIOCB&, char*, long long, int) (XrdPosixFile.cc:409)
==119570== by 0x6B0EB2F: XrdFileCache::File::ProcessBlockRequests(std::list<XrdFileCache::Block*, std::allocator<XrdFileCache::Block*> >&, bool) (XrdFileCacheFile.cc:466)
==119570==
==119570== (action on error) vgdb me ...
==119570== Continuing ...
[2019-03-25 16:39:31.869607 -0700][Error ][XRootD ] [[log in to unmask]:1094] Destroying MsgHandler: 5454260.
[2019-03-25 16:39:33.684680 -0700][Error ][XRootD ] [[log in to unmask]:1094] MsgHandler created: 30e1e600 (message: kXR_read (handle: 0x00000000, offset: 37224448, size: 524288) ).
[2019-03-25 16:39:46.888592 -0700][Error ][Poller ] <[::ffff:169.228.130.108]:38338><--><[::ffff:169.228.130.61]:1094> Unable to disable write notifications: modifying poll events
[2019-03-25 16:39:46.892323 -0700][Error ][XRootD ] [[log in to unmask]:1094] Impossible to send message kXR_read (handle: 0x00000000, offset: 37224448, size: 524288). Trying to recover.
[2019-03-25 16:39:46.893646 -0700][Error ][XRootD ] [[log in to unmask]:1094] Unable to get the response to request kXR_read (handle: 0x00000000, offset: 37224448, size: 524288)
[2019-03-25 16:39:46.894128 -0700][Error ][XRootD ] [[log in to unmask]:1094] Passing to the thread-pool MsgHandler: 30e1e600 (message: kXR_read (handle: 0x00000000, offset: 37224448, size: 524288) ).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/xrootd/xrootd","title":"xrootd/xrootd","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/xrootd/xrootd"}},"updates":{"snippets":[{"icon":"PERSON","message":"@abh3 in #943: Yes, that would be the other suspect here – invalid handling of the SID.\n\nFrom: Matevž Tadel \nSent: Monday, March 25, 2019 4:41 PM\nTo: xrootd/xrootd \nCc: Andrew Hanushevsky ; Mention \nSubject: Re: [xrootd/xrootd] [XrdCl] MsgHandler double delete (#943)\n\nYay, I've got something from valgrind ... it's not in MessageHandler but has to do with Message. Let me know if this makes sense, I'll keep pounding at it for a while :)\n\n[tcpkill was not good enough, finding socket fd with netstat and then calling close(fd) in gdb did the trick).]\n\n[2019-03-25 16:35:00.782184 -0700][Error ][XRootD ] [[log in to unmask]:1094] MsgHandler created: 53fab20 (message: kXR_read (handle: 0x00000000, offset: 36175872, size: 524288) ).\n[2019-03-25 16:35:00.783144 -0700][Error ][XRootD ] [[log in to unmask]:1094] Moving MsgHandler: 53fab20 (message: kXR_read (handle: 0x00000000, offset: 36175872, size: 524288) ) from out-queu to in-queue.\n[2019-03-25 16:35:00.811473 -0700][Error ][XRootD ] [[log in to unmask]:1094] Calling MsgHandler: 53fab20 (message: kXR_read (handle: 0x00000000, offset: 36175872, size: 524288) ) with status: [SUCCESS] .\n[2019-03-25 16:35:00.812114 -0700][Error ][XRootD ] [[log in to unmask]:1094] Destroying MsgHandler: 53fab20.\n[2019-03-25 16:35:16.526807 -0700][Error ][XRootD ] [[log in to unmask]:1094] MsgHandler created: 5454260 (message: kXR_read (handle: 0x00000000, offset: 36700160, size: 524288) ).\n[2019-03-25 16:35:16.537508 -0700][Error ][Poller ] \u003c[::ffff:169.228.130.108]:38338\u003e\u003c--\u003e\u003c[::ffff:169.228.130.61]:1094\u003e Unable to enable write notifications: modifying poll events\n[2019-03-25 16:35:16.538760 -0700][Error ][XRootD ] [[log in to unmask]:1094] Unable to send the message kXR_read (handle: 0x00000000, offset: 36700160, size: 524288): [FATAL] Poller error\n==119570== Thread 87:\n==119570== Invalid read of size 2\n==119570== at 0x6547A4E: XrdCl::SIDManager::ReleaseSID(unsigned char*) (XrdClSIDManager.cc:61)\n==119570== by 0x655F97A: XrdCl::MessageUtils::SendMessage(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams const\u0026, XrdCl::LocalFileHandler*) (XrdClMessageUtils.cc:122)\n==119570== by 0x6566096: XrdCl::FileStateHandler::IssueRequest(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams\u0026) (XrdClFileStateHandler.cc:1956)\n==119570== by 0x656BC9C: XrdCl::FileStateHandler::SendOrQueue(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams\u0026) (XrdClFileStateHandler.cc:1588)\n==119570== by 0x656F1F3: XrdCl::FileStateHandler::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFileStateHandler.cc:692)\n==119570== by 0x6563129: XrdCl::File::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFile.cc:194)\n==119570== by 0x62D8088: XrdPosixFile::Read(XrdOucCacheIOCB\u0026, char*, long long, int) (XrdPosixFile.cc:409)\n==119570== by 0x6B0EB2F: XrdFileCache::File::ProcessBlockRequests(std::list\u003cXrdFileCache::Block*, std::allocator\u003cXrdFileCache::Block*\u003e \u003e\u0026, bool) (XrdFileCacheFile.cc:466)\n==119570== by 0x6B128A1: XrdFileCache::File::Read(XrdFileCache::IO*, char*, long long, int) (XrdFileCacheFile.cc:627)\n==119570== by 0x6B1AC23: XrdFileCache::IOEntireFile::Read(char*, long long, int) (XrdFileCacheIOEntireFile.cc:174)\n==119570== by 0x62DBF41: XrdPosixXrootd::Pread(int, void*, unsigned long, long) (XrdPosixXrootd.cc:725)\n==119570== by 0x60BC33F: XrdPssFile::Read(void*, long, unsigned long) (XrdPss.cc:811)\n==119570== Address 0x54b1ce0 is 0 bytes inside a block of size 24 free'd\n==119570== at 0x4A06C20: realloc (vg_replace_malloc.c:662)\n==119570== by 0x653F0D9: XrdCl::XRootDTransport::MultiplexSubStream(XrdCl::Message*, unsigned short, XrdCl::AnyObject\u0026, XrdCl::PathID*) (XrdClBuffer.hh:90)\n==119570== by 0x6539E37: XrdCl::Stream::Send(XrdCl::Message*, XrdCl::OutgoingMsgHandler*, bool, long) (XrdClStream.cc:300)\n==119570== by 0x653777D: XrdCl::Channel::Send(XrdCl::Message*, XrdCl::OutgoingMsgHandler*, bool, long) (XrdClChannel.cc:312)\n==119570== by 0x6536867: XrdCl::PostMaster::Send(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::OutgoingMsgHandler*, bool, long) (XrdClPostMaster.cc:191)\n==119570== by 0x655F8E3: XrdCl::MessageUtils::SendMessage(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams const\u0026, XrdCl::LocalFileHandler*) (XrdClMessageUtils.cc:113)\n==119570== by 0x6566096: XrdCl::FileStateHandler::IssueRequest(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams\u0026) (XrdClFileStateHandler.cc:1956)\n==119570== by 0x656BC9C: XrdCl::FileStateHandler::SendOrQueue(XrdCl::URL const\u0026, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams\u0026) (XrdClFileStateHandler.cc:1588)\n==119570== by 0x656F1F3: XrdCl::FileStateHandler::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFileStateHandler.cc:692)\n==119570== by 0x6563129: XrdCl::File::Read(unsigned long, unsigned int, void*, XrdCl::ResponseHandler*, unsigned short) (XrdClFile.cc:194)\n==119570== by 0x62D8088: XrdPosixFile::Read(XrdOucCacheIOCB\u0026, char*, long long, int) (XrdPosixFile.cc:409)\n==119570== by 0x6B0EB2F: XrdFileCache::File::ProcessBlockRequests(std::list\u003cXrdFileCache::Block*, std::allocator\u003cXrdFileCache::Block*\u003e \u003e\u0026, bool) (XrdFileCacheFile.cc:466)\n==119570==\n==119570== (action on error) vgdb me ...\n==119570== Continuing ...\n[2019-03-25 16:39:31.869607 -0700][Error ][XRootD ] [[log in to unmask]:1094] Destroying MsgHandler: 5454260.\n[2019-03-25 16:39:33.684680 -0700][Error ][XRootD ] [[log in to unmask]:1094] MsgHandler created: 30e1e600 (message: kXR_read (handle: 0x00000000, offset: 37224448, size: 524288) ).\n[2019-03-25 16:39:46.888592 -0700][Error ][Poller ] \u003c[::ffff:169.228.130.108]:38338\u003e\u003c--\u003e\u003c[::ffff:169.228.130.61]:1094\u003e Unable to disable write notifications: modifying poll events\n[2019-03-25 16:39:46.892323 -0700][Error ][XRootD ] [[log in to unmask]:1094] Impossible to send message kXR_read (handle: 0x00000000, offset: 37224448, size: 524288). Trying to recover.\n[2019-03-25 16:39:46.893646 -0700][Error ][XRootD ] [[log in to unmask]:1094] Unable to get the response to request kXR_read (handle: 0x00000000, offset: 37224448, size: 524288)\n[2019-03-25 16:39:46.894128 -0700][Error ][XRootD ] [[log in to unmask]:1094] Passing to the thread-pool MsgHandler: 30e1e600 (message: kXR_read (handle: 0x00000000, offset: 37224448, size: 524288) ).\n\n—\nYou are receiving this because you were mentioned.\nReply to this email directly, view it on GitHub, or mute the thread.\n"}],"action":{"name":"View Issue","url":"https://github.com/xrootd/xrootd/issues/943#issuecomment-476419683"}}}
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/xrootd/xrootd/issues/943#issuecomment-476419683",
"url": "https://github.com/xrootd/xrootd/issues/943#issuecomment-476419683",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
