LISTSERV 16.5 - QSERV-L Archives

-------- Forwarded Message --------
Subject:	Fwd: [QSERV-L] cmsd server crash on openstack/docker machine
Date:	Tue, 10 Nov 2015 16:32:35 -0800
From:	Fabrice Jammes <[log in to unmask]>
To:	Andrew Hanushevsky <[log in to unmask]>
-------- Forwarded Message --------
Subject:	[QSERV-L] cmsd server crash on openstack/docker machine
Date:	Tue, 10 Nov 2015 10:43:58 -0800
From:	Fabrice Jammes <[log in to unmask]>
To:	qserv-l <[log in to unmask]>
Hello,

Next command crash on NCSA openstack cluster:

qserv@docker-2:~$ 
/qserv/stack/Linux64/xrootd/lsst-dev-gf55037e37d/bin/cmsd -c 
/qserv/run/etc/lsp.cf  -n worker -I v4

No DNS is available on openstack cluster, so I hacked /etc/hosts files 
on both host and containers.

# ping xrootd manager:
qserv@docker-2:~$ ping docker-1
PING docker-1 (172.16.1.238): 56 data bytes
64 bytes from 172.16.1.238: icmp_seq=0 ttl=63 time=1.349 ms

qserv@docker-2:~$ hostname
docker-2
qserv@docker-2:~$ hostname --fqdn
docker-2

Here's the core dump analysis, below.

Thanks

qserv@docker-2:~$ gdb 
/qserv/stack/Linux64/xrootd/lsst-dev-gf55037e37d/bin/cmsd 
/tmp/cores/core.cmsd.2365.docker-2.1447180620
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from 
/qserv/stack/Linux64/xrootd/lsst-dev-gf55037e37d/bin/cmsd...done.
[New LWP 2365]
[New LWP 2371]
[New LWP 2370]
[New LWP 2369]
[New LWP 2368]
[New LWP 2367]
[New LWP 2366]
[New LWP 2372]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by 
`/qserv/stack/Linux64/xrootd/lsst-dev-gf55037e37d/bin/cmsd -c 
/qserv/run/etc/lsp'.
#0  0x0000000000442c1a in XrdOucHash<char>::Add (this=0x0, 
KeyVal=0x7f794b71917a "XrdOucName2NameVec*", KeyData=0x6a28a0 
"68286a0000000000", LifeTime=0,
    opt=(Hash_replace | Hash_dofree))
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/./XrdOuc/XrdOucHash.icc:73
73 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/./XrdOuc/XrdOucHash.icc: 
No such file or directory.
(gdb) bt
#0  0x0000000000442c1a in XrdOucHash<char>::Add (this=0x0, 
KeyVal=0x7f794b71917a "XrdOucName2NameVec*", KeyData=0x6a28a0 
"68286a0000000000", LifeTime=0,
    opt=(Hash_replace | Hash_dofree))
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/./XrdOuc/XrdOucHash.icc:73
#1  0x00007f794b9ba481 in XrdOucHash<char>::Rep (this=0x0, 
KeyVal=0x7f794b71917a "XrdOucName2NameVec*", KeyData=0x6a28a0 
"68286a0000000000", LifeTime=0,
    opt=Hash_dofree) at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/./XrdOuc/XrdOucHash.hh:169
#2  0x00007f794b6d222e in XrdOucEnv::PutPtr (this=0x0, 
varname=0x7f794b71917a "XrdOucName2NameVec*", value=0x6a2868)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdOuc/XrdOucEnv.cc:243
#3  0x00007f794b6d850a in XrdOucN2NLoader::Load (this=0x7fff5154dcd0, 
libName=0x0, urVer=..., envP=0x0)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdOuc/XrdOucN2NLoader.cc:60
#4  0x000000000042f27b in XrdCmsConfig::ConfigN2N (this=0x65f780 
<XrdCms::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdCms/XrdCmsConfig.cc:749
#5  0x000000000042defa in XrdCmsConfig::Configure2 (this=0x65f780 
<XrdCms::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdCms/XrdCmsConfig.cc:409
#6  0x0000000000443a94 in XrdgetProtocol (pname=0x6a16c0 "cmsd", 
parms=0x0, pi=0x65a500 <XrdMain::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdCms/XrdCmsProtocol.cc:126
#7  0x000000000041e0ee in XrdProtLoad::getProtocol (lname=0x0, 
pname=0x6a16c0 "cmsd", parms=0x0, pi=0x65a500 <XrdMain::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdProtLoad.cc:247
#8  0x000000000041db7b in XrdProtLoad::Load (lname=0x0, pname=0x6a16c0 
"cmsd", parms=0x0, pi=0x65a500 <XrdMain::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdProtLoad.cc:101
#9  0x000000000041a79d in XrdConfig::Setup (this=0x65a500 
<XrdMain::Config>, dfltp=0x7fff5154efb0 "cmsd")
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdConfig.cc:1038
#10 0x00000000004189e0 in XrdConfig::Configure (this=0x65a500 
<XrdMain::Config>, argc=7, argv=0x7fff5154e938)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdConfig.cc:519
#11 0x000000000041f3fd in main (argc=7, argv=0x7fff5154e938)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdMain.cc:179
(gdb) where
#0  0x0000000000442c1a in XrdOucHash<char>::Add (this=0x0, 
KeyVal=0x7f794b71917a "XrdOucName2NameVec*", KeyData=0x6a28a0 
"68286a0000000000", LifeTime=0,
    opt=(Hash_replace | Hash_dofree))
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/./XrdOuc/XrdOucHash.icc:73
#1  0x00007f794b9ba481 in XrdOucHash<char>::Rep (this=0x0, 
KeyVal=0x7f794b71917a "XrdOucName2NameVec*", KeyData=0x6a28a0 
"68286a0000000000", LifeTime=0,
    opt=Hash_dofree) at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/./XrdOuc/XrdOucHash.hh:169
#2  0x00007f794b6d222e in XrdOucEnv::PutPtr (this=0x0, 
varname=0x7f794b71917a "XrdOucName2NameVec*", value=0x6a2868)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdOuc/XrdOucEnv.cc:243
#3  0x00007f794b6d850a in XrdOucN2NLoader::Load (this=0x7fff5154dcd0, 
libName=0x0, urVer=..., envP=0x0)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdOuc/XrdOucN2NLoader.cc:60
#4  0x000000000042f27b in XrdCmsConfig::ConfigN2N (this=0x65f780 
<XrdCms::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdCms/XrdCmsConfig.cc:749
#5  0x000000000042defa in XrdCmsConfig::Configure2 (this=0x65f780 
<XrdCms::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdCms/XrdCmsConfig.cc:409
#6  0x0000000000443a94 in XrdgetProtocol (pname=0x6a16c0 "cmsd", 
parms=0x0, pi=0x65a500 <XrdMain::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/XrdCms/XrdCmsProtocol.cc:126
#7  0x000000000041e0ee in XrdProtLoad::getProtocol (lname=0x0, 
pname=0x6a16c0 "cmsd", parms=0x0, pi=0x65a500 <XrdMain::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdProtLoad.cc:247
#8  0x000000000041db7b in XrdProtLoad::Load (lname=0x0, pname=0x6a16c0 
"cmsd", parms=0x0, pi=0x65a500 <XrdMain::Config>)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdProtLoad.cc:101
#9  0x000000000041a79d in XrdConfig::Setup (this=0x65a500 
<XrdMain::Config>, dfltp=0x7fff5154efb0 "cmsd")
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdConfig.cc:1038
#10 0x00000000004189e0 in XrdConfig::Configure (this=0x65a500 
<XrdMain::Config>, argc=7, argv=0x7fff5154e938)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdConfig.cc:519
#11 0x000000000041f3fd in main (argc=7, argv=0x7fff5154e938)
    at 
/qserv/stack/EupsBuildDir/Linux64/xrootd-lsst-dev-gf55037e37d/xrootd-lsst-dev-gf55037e37d/src/Xrd/XrdMain.cc:179
(gdb) quit

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1