Here it is! I copied the core file and the corresponding log file (from ccqserv124) to /nfs/lsst/db1/xrdFromIN2P3 at slac. The stack trace: Core was generated by `/afs/in2p3.fr/home/j/jgates/stack/Linux64/xrootd/afed79c5a2-dirty/bin/xrootd -c'. Program terminated with signal 11, Segmentation fault. #0 0x00007f6791f10c10 in pthread_mutex_lock () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install expat-2.1.0-8.el7.x86_64 glibc-2.17-78.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.12.2-14.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libgcc-4.8.3-9.el7.x86_64 libicu-50.1.2-11.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libstdc++-4.8.3-9.el7.x86_64 nss-softokn-freebl-3.16.2.3-9.el7.x86_64 openssl-libs-1.0.1e-42.el7_1.9.x86_64 pcre-8.32-14.el7.x86_64 xz-libs-5.1.2-9alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64 (gdb) where #0 0x00007f6791f10c10 in pthread_mutex_lock () from /lib64/libpthread.so.0 #1 0x00007f678ba68b1e in XrdSsiMutex::Lock (this=0x39382e31343a3652) at /afs/in2p3.fr/home/j/jgates/stack/Linux64/xrootd/master-ged07831eea-dirty/include/xrootd/XrdSsi/XrdSsiAtomics.hh:114 #2 0x00007f678ba6d9f8 in XrdSsiResponder::SetResponse (this=0x7f677c048ea8, strmP=0x7f676401c4f0) at /afs/in2p3.fr/home/j/jgates/stack/Linux64/xrootd/master-ged07831eea-dirty/include/xrootd/XrdSsi/XrdSsiResponder.hh:223 #3 0x00007f678ba6d649 in lsst::qserv::xrdsvc::SsiSession::ReplyChannel::_initStream (this=0x7f674801ff68) at build/xrdsvc/SsiSession_ReplyChannel.cc:99 #4 0x00007f678ba6d579 in lsst::qserv::xrdsvc::SsiSession::ReplyChannel::sendStream (this=0x7f674801ff68, buf=0x7f676401b2b8 "(\r\002", bufLen=256, last=false) at build/xrdsvc/SsiSession_ReplyChannel.cc:87 #5 0x00007f678ba31c39 in lsst::qserv::wdb::QueryAction::Impl::_transmitHeader (this=0x7f676401a380, msg="\b\000\020\001\032\035\n\033\n\tQS1_COUNT\020\000\"\nBIGINT(21)(\b2\005\n\001\060\020") at build/wdb/QueryAction.cc:320 #6 0x00007f678ba31672 in lsst::qserv::wdb::QueryAction::Impl::_transmit (this=0x7f676401a380, last=true) at build/wdb/QueryAction.cc:299 #7 0x00007f678ba31f5b in lsst::qserv::wdb::QueryAction::Impl::_dispatchChannel (this=0x7f676401a380) at build/wdb/QueryAction.cc:399 #8 0x00007f678ba305ab in lsst::qserv::wdb::QueryAction::Impl::act (this=0x7f676401a380) at build/wdb/QueryAction.cc:187 #9 0x00007f678ba33084 in lsst::qserv::wdb::QueryAction::operator() (this=0x7f6764000a78) at build/wdb/QueryAction.cc:450 #10 0x00007f678ba15f46 in lsst::qserv::wcontrol::ForemanImpl::Runner::operator() (this=0x7f674c097db0) > On Aug 7, 2015, at 11:51 PM, Andrew Hanushevsky <[log in to unmask]> wrote: > > This may not be accurate but I know we have run across this problem before. Here is the link: > > http://en.linuxreviews.org/HOWTO_enable_core-dumps > > Andy > > On Fri, 7 Aug 2015, Serge Monkewitz wrote: > >> I˙˙ve never heard of Linux treating daemon˙˙s specially with regards to core dumping - if you set the ulimit in the appropriate init.d script, you should get a core dump as usual. Can you provide a link? >> >> Serge >> >>> On Aug 7, 2015, at 7:02 PM, Andrew Hanushevsky <[log in to unmask]> wrote: >>> >>> Oh yes, if the thing runs as a daemon, Linux will still suppress teh core file. Does it? >>> >>> Andy >> >>> On Fri, 7 Aug 2015, Fritz Mueller wrote: >>> >>>> I'd vote yes on this, thanks. >>>> >>>> On 08/07/2015 06:52 PM, Becla, Jacek wrote: >>>>> ok, I am running with unlimited now. >>>>> The question is: do we want to add that to all our init scripts? >>>>> Ill create a story and will do it >>>>> Jacek >>>>>> On Aug 7, 2015, at 4:25 PM, Serge Monkewitz <[log in to unmask] <mailto:[log in to unmask]>> wrote: >>>>>> ulimit -c unlimited >>>>> ------------------------------------------------------------------------ >>>>> Use REPLY-ALL to reply to list >>>>> To unsubscribe from the QSERV-L list, click the following link: >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >>>> >>>> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1