URL:
<http://savannah.cern.ch/support/?130984>
Summary: xrootd server v3.2.2 segfaults at statistics
generation
Project: XROOTD
Submitted by: jhorky
Submitted on: 2012-08-06 20:34
Category: None
Priority: 5 - Normal
Severity: 4 - Important
Status: None
Privacy: Public
Assigned to: None
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Operating System: GNU/Linux
_______________________________________________________
Details:
Hi,
we are seeing quite regular xrootd server segfaults after upgrading to
v3.2.2. It seems like the crash is somehow related to the load of the server
- the more requests are made the more likely is the crash. Here is the
backtrace of one such fail:
[Switching to Thread 0x472ad940 (LWP 16582)]
0x0000003161264c53 in std::_Rb_tree_increment(std::_Rb_tree_node_base*) ()
from /usr/lib64/libstdc++.so.6
(gdb) bt full
#0 0x0000003161264c53 in std::_Rb_tree_increment(std::_Rb_tree_node_base*)
() from /usr/lib64/libstdc++.so.6
No symbol table info available.
#1 0x000000000041c56b in operator++ (this=0xf70f500,
buff=0x2aaab402a24e "<stats
id=\"xrootd\"><num>1225</num><ops><open>15037</open><rf>0</rf><rd>27733302</rd><pr>0</pr><wr>464</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>14941</misc></ops><aio><num>0</num><max>0</max"...,
blen=90857, do_sync=<value optimized out>) at
/usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:190
No locals.
#2 XrdXrootdStats::Stats (this=0xf70f500,
buff=0x2aaab402a24e "<stats
id=\"xrootd\"><num>1225</num><ops><open>15037</open><rf>0</rf><rd>27733302</rd><pr>0</pr><wr>464</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>14941</misc></ops><aio><num>0</num><max>0</max"...,
blen=90857, do_sync=<value optimized out>) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/XrdXrootd/XrdXrootdStats.cc:87
len = 11925
cl_buff = '\000' <repeats 99 times>
statfmt = "<stats
id=\"xrootd\"><num>%d</num><ops><open>%d</open><rf>%d</rf><rd>%lld</rd><pr>%lld</pr><wr>%lld</wr><sync>%d</sync><getf>%d</getf><putf>%d</putf><misc>%d</misc></ops><aio><num>%lld</num><max>%d</max"...
client_info = "<stats id=\"client\"><%s>%lld</%s></stats>"
#3 0x00002b830214571e in XrdProtLoad::Statistics (
buff=0x2aaab402a24e "<stats
id=\"xrootd\"><num>1225</num><ops><open>15037</open><rf>0</rf><rd>27733302</rd><pr>0</pr><wr>464</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>14941</misc></ops><aio><num>0</num><max>0</max"...,
blen=90857, do_sync=0) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/Xrd/XrdProtLoad.cc:195
i = 1
k = <value optimized out>
totlen = 0
#4 0x00002b8302147f8d in XrdStats::Stats (this=0xf6fcda0, opts=-1073741833)
at /tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/Xrd/XrdStats.cc:196
myTimer = {StopWatch = {tv_sec = 1344270858, tv_usec = 5447},
LastReport = {tv_sec = 1193988228, tv_usec = 259058736}}
bp = 0x2aaab402a24e "<stats
id=\"xrootd\"><num>1225</num><ops><open>15037</open><rf>0</rf><rd>27733302</rd><pr>0</pr><wr>464</wr><sync>0</sync><getf>0</getf><putf>0</putf><misc>14941</misc></ops><aio><num>0</num><max>0</max"...
bl = 90857
sz = 3
do_sync = 3
sgen = 0x2b830214f4d8 "<stats
id=\"sgen\"><as>%d</as><et>%lu</et><toe>%ld</toe></stats>"
tail = 0x2b830214f3e5 "</statistics>"
snul = 0x2b830214f4a8 "<statistics tod=\"0\"
ver=\"v3.2.2\"></statistics>"
ovrhed = 331
#5 0x00002b83021482d3 in XrdStats::Report (this=0xf6fcda0, Dest=0x0,
iVal=<value optimized out>, Opts=<value optimized out>) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/Xrd/XrdStats.cc:122
Data = <value optimized out>
theOpts = -1073741833
Dlen = <value optimized out>
netDest = {0xf6fca80, 0x0}
autoSync = 536870912
repOpts = -9
#6 0x00002b83021484a6 in XrdStatsJob::DoIt (this=0xf715900) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/Xrd/XrdStats.cc:44
No locals.
#7 0x00002b8302146d66 in XrdScheduler::Run (this=0x2b8301ed98c0) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/Xrd/XrdScheduler.cc:287
waiting = 33
jp = 0xf715900
#8 0x00002b8302146ef9 in XrdStartWorking (carg=0xf72d0d0) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/Xrd/XrdScheduler.cc:65
No locals.
#9 0x00002b8302117e7f in XrdSysThread_Xeq (myargs=<value optimized out>) at
/tmp/xrd3-installer-root/xrdbase/xrootd-3.2.2/src/XrdSys/XrdSysPthread.cc:67
ap = 0xf72d0d0
retc = <value optimized out>
#10 0x000000315b60673d in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x000000315aed44bd in clone () from /lib64/libc.so.6
No symbol table info available.
This is the end of the xrd.log:
120806 18:34:23 16710 alicesgm.16506:[log in to unmask] ftsofs_open: 0-644
fn=/15/64218/38f4aea4-cc11-11e1-b3b0-53fab73c02b8
120806 18:34:23 16710 alicesgm.16506:[log in to unmask] oss_Open_ufs: fd=32839
flags=0 mode=644 path=/mnt/fs1/15/64218/38f4aea4-cc11-11e1-b3b0-53fab73c02b8
120806 18:34:23 16606 sec_getParms: 172.16.9.2 sectoken=&P=unix
120806 18:34:24 16681 alicesgm.26757:[log in to unmask] ftsofs_close: use=1
fn=/07/55745/aabddb92-8903-11e1-90a4-3f8b9d20f466
120806 18:34:24 16681 alicesgm.26757:[log in to unmask] ftsofs_close: use=0
fn=dummy
120806 18:34:24 16681 alicesgm.26757:[log in to unmask] ftsofs_close: use=0
fn=dummy
120806 18:34:24 16681 alicesgm.26757:[log in to unmask] ftsofs_vopen:
path=/02/18732/84656f6c-88ab-11e1-b2b5-3f752cabc310 opaque=&authz=-----BEGIN
SEALED CIPHER-----
WiKj2tAOOzxH7RdKVivqF6z28-yHFdcR-qY7oINtHdhzMgzWdyoKMhtjyl2gmuXF-HyoGM3Vfve2
AyoPwB-CI3FqdH0rfKkifoG3p-8WM6clrU7ECwwCpjkZFVcmBd1FF9WuTWZYFAadSxRQM9QsVkk1
Le0zo-JIuVmqjCvgICU=
-----END SEALED CIPHER-----
-----BEGIN SEALED ENVELOPE-----
AAAAgGs22KtHiMHD6551fX+Ub3ZK8vZLe6B+xbWUVtg+z8uP-BVbwDUZQPf-5Kp4rnqyUuHNujuw
QGLVLuK9a39jQLg1UF3MVzwaMNbrjyr8VpLkRL4Y2d+sSAnkEDCD2xSUd8VTwUV84EuQWsu6sdNT
l2QVWjG1Ta0gaItR75gdGcJ4sU6lh62-ONvxL0hXNfF8Tzt2-EIXJLA93za77tE8sJqVpFBFHGdV
NEbiWixeMZ6lv6Tn2Dm8WSPGOg3evkLQU7WgNP1Xqb-X8YNl21E-qfuzATlH8Bi-CMoVMJVNMvmt
O30My9CojNH9pP5YM1nSd31aMTquelQyGcVSVxkHxMisnHmFGCA-BJz6YPvgLDHVbOOyd9x53vF3
y1IGtQtHGI+d9NC4zi2TqsM0Rn4OkSVuy1hhkx-Yn3zaGRWOEzLZ+ktBECK8AElm1vySjzCypyPe
7vq1awqYKWssDFG5c-JJyfwCfMTTHEV8GH4VqtGXqgmISafJ4CIH+6MBFlsFzcczQPxLoK6i-Ugb
qF3ezAgZi+eElrru+f3q7-LrARvfZgXiC6FIei+LmSPZ03s1gE6IqzPdzcxYqDjH7zQndfOtA1GL
GLyCXggNhx3Ma-2gA16oIfQUHNybcuASG3gVp5qJ3jQqSBtCvxXUBAL+Ij7T3HzkDkSnXy16ODax
fOQ1u0cb5D2DuLh4OZeSOI7K1sgeelyyo4+QQsZDt2T-74kwlx9n1fK6QZhg93CNmHJJhdoDFHTb
5YwiTBZPOjMiKxyaWMIHMwxOdiQmnnITf351wqVFdsx0ui8+0hEzX-Sf9UuME3bZdOwyMixJUWH1
Mu8nYdC3HfoJEGSSQBH3d2WcVWlJkMC-a-WxEkJzw2SyO-6LIkmiRHyh0b-zPPzYniVJ+THhK+fU
TGQdTUaO2yHwMuANa0MO0-bI3wHBIDrQT0ow0VhozAUIbIx0UBk5f3hkRd2fMvK7T4aW95ypdy9P
6VFQ8V3ZT+TtTUt9f3UMMWqI64nwr74jDk04Hfh1ZYtEYW4Kn-OVvdHBp+khfjRxnbr7qx06EN9A
ugVrXvrr6d-ARiTdiUOqMAaGAcSB5fQMwjd5UQkwSmP1dZNmj4NsBFuidKJc+sjJZ0LdRVL97fio
DOYwpp7NsCcZqG4e+Q==
-----END SEALED ENVELOPE-----
120806 18:34:24 16681 alicesgm.26757:[log in to unmask] ftsofs_open: 0-644
fn=/02/18732/84656f6c-88ab-11e1-b2b5-3f752cabc310
120806 18:34:24 16681 alicesgm.26757:[log in to unmask] oss_Open_ufs: fd=32821
flags=0 mode=644 path=/mnt/fs1/02/18732/84656f6c-88ab-11e1-b2b5-3f752cabc310
120806 18:34:28 16646 sec_getParms: 172.16.2.11 sectoken=&P=unix
120806 18:34:29 16654 sec_getParms: 172.16.9.2 sectoken=&P=unix
120806 18:34:29 16623 sec_getParms: 172.16.6.2 sectoken=&P=unix
120806 18:34:31 16453 sec_getParms: 172.16.2.4 sectoken=&P=unix
120806 18:34:33 16572 sec_getParms: 172.16.6.42 sectoken=&P=unix
120806 18:34:34 17038 sec_getParms: 172.16.6.21 sectoken=&P=unix
120806 18:34:35 16275 sec_getParms: 172.16.6.19 sectoken=&P=unix
120806 18:34:42 16527 alicesgm.19990:[log in to unmask] ftsofs_vopen:
path=/10/22065/18d504f6-b588-11e1-a578-00266cfd8b68 opaque=&authz=-----BEGIN
SEALED CIPHER-----
dNYG-opBF5kelWJmZ4CA5w7TVRU-fogCJAeeZlNTZLugqe28fRiceAUTpMMF0c1Xn3T4QrJYYYJT
The server is running as ALICE SE.
Please let me know if you need more information.
Jiri Horky
_______________________________________________________
Reply to this item at:
<http://savannah.cern.ch/support/?130984>
_______________________________________________
Message sent via/by LCG Savannah
http://savannah.cern.ch/
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
