Niiice!! On Mar 13, 2015 3:01 PM, Fabrice Jammes <[log in to unmask]> wrote: Thanks to both Andy, and Daniel, Qserv worker is now running on in2p3 cluster. whew, it's a good thing ;-) On 03/13/2015 02:40 PM, Salnikov, Andrei A. wrote: > Hi Fabrice, > > thanks! Check my other email which I sent right before noon and see > if it fixes crash. > > Cheers, > Andy > > Fabrice Jammes wrote on 2015-03-13: >> Hi Andy, >> >> I just checked your email. I met this problem but only on cmsd worker. >> With Daniel help, I have fixed it in u/fjammes/DM-213. See last commits >> : >> https://github.com/lsst/qserv/commit/d5abe798cfceadbcafd0b555de5c986d4a3c >> 7 605 >> https://github.com/lsst/qserv/commit/a784bce2f5bb9f1f1a4631250054838084c9 >> 5 619 >> >> I'll try what you propose for log pb. Thanks! >> >> >> >> On 03/13/2015 10:56 AM, Salnikov, Andrei A. wrote: >>> Hi Fabrice, >>> >>> is master (git master) supposed to work? I get weird message >>> from cmsd when I try to start Qserv (both worker and master): >>> >>> ------ [log in to unmask] phase 1 server >>> initialization completed. ++++++ [log in to unmask] >>> phase 2 server initialization started. Plugin >>> /usr/local/home/salnikov/dm-2294/lib/libxrdoss.so: undefined symbol: >>> _ZN6google8protobuf8internal12kEmptyStringE osslib libxrdoss.so ------ >>> [log in to unmask] phase 2 server initialization >>> failed. 150313 12:30:02 15616 XrdProtocol: Protocol cmsd could not be >>> loaded ------ cmsd [log in to unmask]:1094 >>> initialization failed. >>> >>> if I do 'ldd -r lib/libxrdoss.so' I indeed see few unresolved >>> symbols in it. >>> >>> Cheers, >>> Andy >>> >>> Fabrice Jammes wrote on 2015-03-13: >>>> The way is to start a Qserv master and then a Qserv worker. cmsd will >>>> then fails on the worker. There's a draft script which does that for >>>> in2p3 cluster here: https://github.com/lsst/qserv/blob/u/fjammes/DM- >>>> 213/admin/tools/cluster/cc-in2p3/install-puppet-node.sh You can cut and >>>> paste the configuration part from this one (but not the install part >>>> which is based on in2p3 shared fs) >>>> >>>> You also have to install/setup xrootd built in debug mode if you want >>>> to debug the produced core dump: eups distrib install xrootd xssi-1.0.0 >>>> >>>> Please let me know. Vaikunth is installing multi-node Qserv at NCSA, >>>> so he may have additional informations. >>>> >>>> I hope your in2p3 account will be ready soon. The Qserv setup is >>>> already done here. >>>> >>>> Thanks, >>>> >>>> On 03/13/2015 07:04 AM, Salnikov, Andrei A. wrote: >>>>> Hi Fabrice, >>>>> >>>>> I'd like to reproduce it to understand exactly what happens, >>>>> is there a way to repeat it on NCSA machines? >>>>> >>>>> Cheers, >>>>> Andy >>>>> >>>>> >>>>> Fabrice Jammes wrote on 2015-03-13: >>>>>> Hi Andy, >>>>>> >>>>>> cmsd crash happen when starting worker node. I can easily test the >>>>>> fix if needed. It is blocking for in2p3 cluster install. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> On 03/12/2015 11:04 PM, Salnikov, Andrei A. wrote: >>>>>>> Fabrice, can you tell me what did you do to make it crash? >>>>>>> >>>>>>> Cheers, >>>>>>> Andy >>>>>>> >>>>>>> >>>>>>> Andrew Hanushevsky wrote on 2015-03-12: >>>>>>>> Hi Andy, >>>>>>>> >>>>>>>> Just run what Fabrice ran and it will core right away since >>>>>>>> XrdSysError wasn't told about XrdSysLogger and eventually the >>>>>>>> message is routed to that object. >>>>>>>> >>>>>>>> Andy >>>>>>>> >>>>>>>> On Fri, 13 Mar 2015, Salnikov, Andrei A. wrote: >>>>>>>> >>>>>>>>> Hi Andy, >>>>>>>>> >>>>>>>>> I'm confused, the little piece of code below is the fix, right? >>>>>>>>> Is it possible for me to reproduce crash? I'll try to remember >>>>>>>>> tomorrow what I did for logging, maybe I could see it better. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Andy >>>>>>>>> >>>>>>>>> >>>>>>>>> Andrew Hanushevsky wrote on 2015-03-12: >>>>>>>>>> Hi Andy, >>>>>>>>>> >>>>>>>>>> I don't think there is a fix for this. Could you fix it? I >>>>>>>>>> already have a Jira entry for rewriting that portion of the code >>>>>>>>>> once I get a handle on how to deal with the repo. >>>>>>>>>> >>>>>>>>>> Andy >>>>>>>>>> >>>>>>>>>> On Thu, 12 Mar 2015, Salnikov, Andrei A. wrote: >>>>>>>>>> >>>>>>>>>>> Hi Andy, >>>>>>>>>>> >>>>>>>>>>> sorry for the mess I created. I do not remember exactly what is >>>>>>>>>>> involved there, but if you say that it fixes it then I'm happy. >>>>>>>>>>> Are you going to make that fix or do you want me to do it? Is >>>>>>>>>>> there a ticket for it already? >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Andy >>>>>>>>>>> >>>>>>>>>>> Andrew Hanushevsky wrote on 2015-03-12: >>>>>>>>>>>> Hi Andy, >>>>>>>>>>>> >>>>>>>>>>>> Daniel told me that you did the log4cxx integration for this >>>>>>>>>>>> part of qserv. Unfortunately, the underlying xrootd object >>>>>>>>>>>> that send messages to the log was not initialized when the >>>>>>>>>>>> XrdOssGetSS(Logger, ...) is called. This results in a SEGV >>>>>>>>>>>> when a message is issued. I know this is old code and now >>>>>>>>>>>> there is a better way of doing all of this (and I will do that >>>>>>>>>>>> later) for now the way to fix this is simply insert this code >>>>>>>>>>>> snippedin the initalization section: >>>>>>>>>>>> >>>>>>>>>>>> #include "XrdSys/XrdSysError.hh" >>>>>>>>>>>> >>>>>>>>>>>> namespace XrdSsi >>>>>>>>>>>> { >>>>>>>>>>>> extern XrdSysError Log; >>>>>>>>>>>> }; >>>>>>>>>>>> >>>>>>>>>>>> XrdSsi::Log.logger(Logger); // Logger is the arg in >>>>>>>>>>>> XrdOssGetSS() >>>>>>>>>>>> >>>>>>>>>>>> OK? Like I said this will become far simpler on the next >>>>>>>>>>>> iteration. >>>>>>>>>>>> >>>>>>>>>>>> Andy >>>>>>>>>>>> >>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>> Date: Thu, 12 Mar 2015 14:14:12 -0700 >>>>>>>>>>>> From: Fabrice Jammes <[log in to unmask]> >>>>>>>>>>>> To: Andrew Hanushevsky <[log in to unmask]> >>>>>>>>>>>> Subject: cmsd core log >>>>>>>>>>>> >>>>>>>>>>>> Hi Andy, >>>>>>>>>>>> >>>>>>>>>>>> We introduced logging in this module a few months ago but our >>>>>>>>>>>> mono-node integration test doesn't launch cmsd, so it was >>>>>>>>>>>> never tested. It may be our logging library (lsst::log which >>>>>>>>>>>> relies on log4cxx) which is faulty? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Fabrice >>>>>>>>>>>> >>>>>>>>>>>> Program terminated with signal 11, Segmentation fault. #0 >>>>>>>>>>>> 0x00007fdb86a9fc10 in pthread_mutex_lock () from >>>>>>>>>>>> /lib64/libpthread.so.0 Missing separate debuginfos, use: >>>>>>>>>>>> debuginfo-install expat-2.1.0- 8.el7.x86_64 >>>>>>>>>>>> glibc-2.17-55.el7_0.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 >>>>>>>>>>>> krb5-libs-1.11.3-49.el7.x86_64 libcom_err-1.42.9-4.el7.x86_64 >>>>>>>>>>>> libgcc-4.8.2-16.2.el7_0.x86_64 libicu-50.1.2-11.el7.x86_64 >>>>>>>>>>>> libselinux-2.2.2-6.el7.x86_64 >>>>>>>>>>>> libstdc++-4.8.2-16.2.el7_0.x86_64 >>>>>>>>>>>> nss-softokn-freebl-3.16.2.3-1.el7_0.x86_64 >>>>>>>>>>>> openssl-libs-1.0.1e-34.el7_0.7.x86_64 pcre-8.32-12.el7.x86_64 >>>>>>>>>>>> xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64 (gdb) >>>>>>>>>>>> where #0 0x00007fdb86a9fc10 in pthread_mutex_lock () from >>>>>>>>>>>> /lib64/libpthread.so.0 #1 0x000000000041ed54 in >>>>>>>>>>>> XrdSysMutex::Lock (this=0x10) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/./XrdSys/XrdSysPthread.hh:149 #2 >>>>>>>>>>>> 0x00007fdb86ceb2e6 in XrdSysLogger::Put (this=0x0, iovcnt=2, >>>>>>>>>>>> iov=0x7fff589f1800) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/XrdSys/XrdSysLogger.cc:270 #3 >>>>>>>>>>>> 0x00007fdb86cea817 in XrdSysError::Say (this=0x7fdb7f3aa7e0 >>>>>>>>>>>> <XrdSsi::Log>, txt1=0x1f1efb8 "0312 20:09:50.451 >>>>>>>>>>>> [0x7fdb87464740] INFO QservOss (build/xrdoss/QservOss.cc:260) >>>>>>>>>>>> - QservOss Init", txt2=0x0, >>>>>>>>>>>> txt3=0x0, txt4=0x0, txt5=0x0, txt6=txt6@entry=0x0) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/XrdSys/XrdSysError.cc:163 #4 >>>>>>>>>>>> 0x00007fdb7f19e638 in XrdSsiLogger::Msg (pfx=<optimized out>, >>>>>>>>>>>> txt1=<optimized out>, txt2=<optimized out>, txt3=<optimized >>>>>>>>>>>> out>) >>>>>>>>>>>> at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-4.0.0rc3- >>>>>>>>>> qsClient2.lsst1/xrootd- >>>>>>>>>>>> 4.0.0rc3-qsClient2.lsst1/src/XrdSsi/XrdSsiLogger.cc:66 #5 >>>>>>>>>>>> 0x00007fdb84f38a0d in lsst::qserv::wlog::XrootdAppender::append >>>>>>>>>>>> (this=0x1f2b9e0, event=..., p=...) at >>>>>> build/wlog/XrootdAppender.cc:57 >>>>>>>>>> #6 >>>>>>>>>>>> 0x00007fdb841ebd1e in log4cxx::AppenderSkeleton::doAppend >>>>>>>>>>>> (this=0x1f2b9e0, event=..., pool1=...) at >>>>>>>>>>>> appenderskeleton.cpp:142 #7 0x00007fdb841e9df4 in >>>>>>>>>>>> log4cxx::helpers::AppenderAttachableImpl::appendLoopOnAppenders >>>>>>>>>>>> (this=0x1f2bd20, event=..., p=...) at >>>>>>>>>>>> appenderattachableimpl.cpp:69 #8 0x00007fdb8422f6d5 in >>>>>>>>>>>> log4cxx::Logger::callAppenders (this=this@entry=0x1f2bd90, >>>>>>>>>>>> event=..., p=...) at logger.cpp:93 #9 0x00007fdb8422fa1f in >>>>>>>>>>>> log4cxx::Logger::forcedLog (this=0x1f2bd90, level1=..., >>>>>>>>>>>> message="QservOss Init", location=...) at logger.cpp:124 #10 >>>>>>>>>>>> 0x00007fdb84500412 in lsst::log::Log::vlog (logger=..., >>>>>>>>>>>> level=..., filename="build/xrdoss/QservOss.cc", >>>>>>>>>>>> funcname="virtual int >>>>>>>>>>>> lsst::qserv::xrdoss::QservOss::Init(XrdSysLogger*, const >>>>>> char*)", >>>>>>>>>>>> lineno=lineno@entry=260, fmt=fmt@entry=0x7fdb84f5a7bb >>>>>> "QservOss >>>>>>>>>>>> Init", args=args@entry=0x7fff589f2018) at src/Log.cc:357 #11 >>>>>>>>>>>> 0x00007fdb84500734 in lsst::log::Log::log (logger=..., >>>>>>>>>>>> level=..., filename="build/xrdoss/QservOss.cc", >>>>>>>>>>>> funcname="virtual int >>>>>>>>>>>> lsst::qserv::xrdoss::QservOss::Init(XrdSysLogger*, const >>>>>>>>>>>> char*)", lineno=260, >>>>>>>>>>>> fmt=0x7fdb84f5a7bb "QservOss Init") at src/Log.cc:386 >>>>>>>>>>>> #12 0x00007fdb84f4e4ac in lsst::qserv::xrdoss::QservOss::Init >>>>>>>>>>>> (this=0x1f1fab0, log=0x0, cfgFn=0x0) at >>>>>>>>>>>> build/xrdoss/QservOss.cc:260 #13 0x00007fdb84f4d4f7 in >>>>>>>>>>>> lsst::qserv::xrdoss::QservOss::QservOss รง(this=0x1f1fab0) at >>>>>>>>>>>> build/xrdoss/QservOss.cc:135 #14 0x00007fdb84f4d2f4 in >>>>>>>>>>>> lsst::qserv::xrdoss::QservOss::getInstance () at >>>>>>>>>>>> build/xrdoss/QservOss.cc:101 #15 0x00007fdb84f4e8f8 in >>>>>>>>>>>> XrdOssGetStorageSystem (native_oss=0x7fdb87253100 >>>>>>>>>>>> <XrdOssGetSS(XrdSysLogger*, char const*, char const*, char >>>>>>>>>>>> const*, XrdOucEnv*, XrdVersionInfo&)::myOssSys>, >>>>>>>>>>>> Logger=0x65b3b8 <XrdMain::Config+248>, >> config_fn=0x1efec40 >>>>>>>>>>>> "/qserv/qserv-run/etc/lsp.cf", parms=0x0) at >>>>>>>>>>>> build/xrdoss/QservOss.cc:293 #16 0x00007fdb86ff484d in >>>>>>>>>>>> XrdOssGetSS (Logger=0x65b3b8 <XrdMain::Config+248>, >>>>>>>>>>>> config_fn=0x1efec40 "/qserv/qserv-run/etc/lsp.cf", >>>>>>>>>>>> OssLib=0x1efef70 "libxrdoss.so", OssParms=0x0, envP=0x0, >>>>>>>>>>>> urVer=...) >>>>>>>>>>>> at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/XrdOss/XrdOssApi.cc:135 #17 0x000000000042f08d >>>>>>>>>>>> in XrdCmsConfig::ConfigOSS (this=0x65d340 <XrdCms::Config>) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/XrdCms/XrdCmsConfig.cc:777 #18 >>>>>>>>>>>> 0x000000000042dbc8 in XrdCmsConfig::Configure2 (this=0x65d340 >>>>>>>>>>>> <XrdCms::Config>) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/XrdCms/XrdCmsConfig.cc:408 #19 >>>>>>>>>>>> 0x0000000000443754 in XrdgetProtocol (pname=0x1eef920 "cmsd", >>>>>>>>>>>> parms=0x0, pi=0x65b2c0 <XrdMain::Config>) >>>>>>>>>>>> at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/XrdCms/XrdCmsProtocol.cc:121 #20 >>>>>>>>>>>> 0x000000000041de24 in XrdProtLoad::getProtocol (lname=0x0, >>>>>>>>>>>> pname=0x1eef920 "cmsd", parms=0x0, pi=0x65b2c0 >>>>>>>>>>>> <XrdMain::Config>) >>>>>>>>>>>> at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/Xrd/XrdProtLoad.cc:247 #21 0x000000000041d8b4 >>>>>>>>>>>> in XrdProtLoad::Load (lname=0x0, pname=0x1eef920 "cmsd", >>>>>>>>>>>> parms=0x0, pi=0x65b2c0 <XrdMain::Config>) >>>>>>>>>>>> at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/Xrd/XrdProtLoad.cc:101 #22 0x000000000041a5b7 >>>>>>>>>>>> in XrdConfig::Setup (this=0x65b2c0 <XrdMain::Config>, >>>>>>>>>>>> dfltp=0x7fff589f3d4e "cmsd") at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/Xrd/XrdConfig.cc:1028 #23 0x00000000004187de >>>>>>>>>>>> in XrdConfig::Configure (this=0x65b2c0 <XrdMain::Config>, >>>>>>>>>>>> argc=9, argv=0x7fff589f3318) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/Xrd/XrdConfig.cc:509 #24 0x000000000041f13b in >>>>>>>>>>>> main (argc=9, argv=0x7fff589f3318) at >>>>>>>>>>>> /qserv/stack/EupsBuildDir/Linux64/xrootd-xssi-1.0.0/xrootd- >>>>>>>>>>>> xssi- 1.0.0/src/Xrd/XrdMain.cc:179 >>>>> >>>>> ###################################################################### >>>>> ## Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the QSERV-L list, click the following link: >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1