URL:
<http://savannah.cern.ch/bugs/?93797>
Summary: xrdcpy failure in extreme copy mode
Project: XROOTD
Submitted by: bdouglas
Submitted on: 2012-04-18 06:27
Severity: 3 - Normal
Priority: 5 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Fixed by commit(s):
_______________________________________________________
Details:
Hello,
Andy asked me to fetch from git head the xrdcpy to test xrdcp in extreme
copy mode. Unfortunately, during my testing the copy has hung. I used gcore
to get a core file but could not load the debug symbols though I told the
system to fetch them and build them initially.
Here is how I built the executable and libraries used for the test:
[root@atlas70 build]# cmake /xrootd-githead/xrootd
-DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_CXX_COMPILER=/usr/bin/g++
-DENABLE_PERL=FALSE -DENABLE_KRB5=FALSE -DCMAKE_BUILD_TYPE=RelWithDebInfo
-- Found READLINE: /usr/lib64/libreadline.so
-- Found OpenSSL: /usr/lib64/libssl.so;/usr/lib64/libcrypto.so
-- ----------------------------------------
-- Installation path: /usr/local
-- C Compiler: /usr/bin/gcc
-- C++ Compiler: /usr/bin/g++
-- Build type: RelWithDebInfo
--
-- Readline support: yes
-- Fuse support: yes
-- Crypto support: yes
-- Kerberos5 support: disabled
-- Perl support: disabled
-- ----------------------------------------
-- Configuring done
-- Generating done
-- Build files have been written to: /xrootd-githead/build
I have attached the screen capture of the build process.
The command that is wedged is -
30001 20510 0.0 0.0 268540 17928 pts/1 Sl 00:20 0:00 xrdcpy -f -v
-np -y 4 -x --debug 1
root://xrd-central.uchicago.edu//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523746._000001.root.1
root://atlas70.hep.anl.gov//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523746._000001.root.1
using netstat I see
tcp 0 0 atlas70.hep.anl.gov:51702 manage.aglt2.org:rootd
ESTABLISHED 20835/xrdcpy
tcp 0 0 atlas70.hep.anl.gov:45870 atlasld1.hep.anl.gov:ldap
ESTABLISHED 20835/xrdcpy
tcp 0 0 atlas70.hep.anl.gov:50702 uct2-grid5.uchicago.e:rootd
ESTABLISHED 20835/xrdcpy
tcp 0 0 atlas70.hep.anl.gov:43864 atlas70.hep.anl.gov:rootd
ESTABLISHED 20835/xrdcpy
tcp 0 0 atlas70.hep.anl.gov:56125 xrd-central.uchicago.:rootd
ESTABLISHED 20835/xrdcpy
tcp 0 0 atlas70.hep.anl.gov:52771 uct2-s12.uchicago.edu:rootd
ESTABLISHED 20835/xrdcpy
Note atlas70 is the machine that I am using andatlasld1 is the ldap server at
ANL.
Here is what I see in gdb
[dbenjamin@atlas70 xrootd-federation]$ gdb -c core.20510
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
[New Thread 20511]
[New Thread 20512]
[New Thread 20516]
[New Thread 20517]
[New Thread 20518]
[New Thread 20519]
[New Thread 20520]
[New Thread 20521]
[New Thread 20522]
[New Thread 20525]
[New Thread 20526]
[New Thread 20510]
warning: no loadable sections found in added symbol-file system-supplied DSO
at 0x7fff864cf000
Core was generated by `xrdcpy -f -v -np -y 4 -x --debug 1
root://xrd-central.uchicago.edu//atlas/dq2/us'.
#0 0x0000003cf5a99031 in ?? ()
(gdb) shared
Reading symbols from /usr/local/lib64/libXrdClient.so.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/local/lib64/libXrdClient.so.1
Reading symbols from /usr/local/lib64/libXrdServer.so.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/local/lib64/libXrdServer.so.1
Reading symbols from /usr/local/lib64/libXrdUtils.so.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/local/lib64/libXrdUtils.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libnss_ldap.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_ldap.so.2
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/local/lib64/libXrdSec.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/local/lib64/libXrdSec.so
Reading symbols from /usr/local/lib64/libXrdSecgsi.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/local/lib64/libXrdSecgsi.so
Reading symbols from /usr/local/lib64/libXrdCryptossl.so.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/local/lib64/libXrdCryptossl.so.1
Reading symbols from /usr/local/lib64/libXrdCrypto.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/local/lib64/libXrdCrypto.so.0
Reading symbols from /lib64/libssl.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from
/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc5-gcc43-opt/17.2.0/LCGCMT/LCGCMT_61c/InstallArea/x86_64-slc5-gcc43-opt/lib/libz.so.1...(no
debugging symbols found)...done.
Loaded symbols for
/cvmfs/atlas.cern.ch/repo/sw/software/x86_64-slc5-gcc43-opt/17.2.0/LCGCMT/LCGCMT_61c/InstallArea/x86_64-slc5-gcc43-opt/lib/libz.so.1
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /usr/local/lib64/libXrdSecunix.so...(no debugging
symbols found)...done.
Loaded symbols for /usr/local/lib64/libXrdSecunix.so
(gdb) thread apply all bt
Thread 12 (Thread 0x2b76af2533c0 (LWP 20510)):
#0 0x0000003cf620b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002b76aefddcae in XrdSysCondVar::Wait(int) () from
/usr/local/lib64/libXrdUtils.so.1
#2 0x0000000000417ffe in ?? ()
Thread 11 (Thread 0x473e9940 (LWP 20526)):
#0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6
#1 0x00002b76aeaed1bf in XrdClientSock::RecvRaw(void*, int, int, int*) ()
from /usr/local/lib64/libXrdClient.so.1
#2 0x00002b76aeb0d401 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb11666 in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from /usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aeb0cbd7 in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from /usr/local/lib64/libXrdClient.so.1
#5 0x00002b76aeb1104e in SocketReaderThread(void*, XrdClientThread*) () from
/usr/local/lib64/libXrdClient.so.1
#6 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#7 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#8 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x47dea940 (LWP 20525)):
#0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6
#1 0x00002b76aeaed1bf in XrdClientSock::RecvRaw(void*, int, int, int*) ()
from /usr/local/lib64/libXrdClient.so.1
#2 0x00002b76aeb0d401 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb11666 in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from /usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aeb0cbd7 in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from /usr/local/lib64/libXrdClient.so.1
#5 0x00002b76aeb1104e in SocketReaderThread(void*, XrdClientThread*) () from
/usr/local/lib64/libXrdClient.so.1
#6 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#7 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#8 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x469e8940 (LWP 20522)):
#0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6
#1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6
#2 0x0000000000409965 in ?? ()
Thread 8 (Thread 0x45fe7940 (LWP 20521)):
#0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6
#2 0x0000000000409965 in ?? ()
Thread 7 (Thread 0x455e6940 (LWP 20520)):
#0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6
#1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6
#2 0x0000000000409965 in ?? ()
Thread 6 (Thread 0x44be5940 (LWP 20519)):
#0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6
#1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6
#2 0x0000000000409965 in ?? ()
Thread 5 (Thread 0x441e4940 (LWP 20518)):
#0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6
#1 0x00002b76aeaed1bf in XrdClientSock::RecvRaw(void*, int, int, int*) ()
from /usr/local/lib64/libXrdClient.so.1
#2 0x00002b76aeb0d401 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb11666 in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from /usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aeb0cbd7 in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from /usr/local/lib64/libXrdClient.so.1
#5 0x00002b76aeb1104e in SocketReaderThread(void*, XrdClientThread*) () from
/usr/local/lib64/libXrdClient.so.1
#6 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#7 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#8 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x423e1940 (LWP 20517)):
#0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6
#1 0x00002b76aeaed1bf in XrdClientSock::RecvRaw(void*, int, int, int*) ()
from /usr/local/lib64/libXrdClient.so.1
#2 0x00002b76aeb0d401 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb11666 in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from /usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aeb0cbd7 in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from /usr/local/lib64/libXrdClient.so.1
#5 0x00002b76aeb1104e in SocketReaderThread(void*, XrdClientThread*) () from
/usr/local/lib64/libXrdClient.so.1
#6 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#7 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#8 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x437e3940 (LWP 20516)):
#0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6
#1 0x00002b76aeaed1bf in XrdClientSock::RecvRaw(void*, int, int, int*) ()
from /usr/local/lib64/libXrdClient.so.1
#2 0x00002b76aeb0d401 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb11666 in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from /usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aeb0cbd7 in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from /usr/local/lib64/libXrdClient.so.1
#5 0x00002b76aeb1104e in SocketReaderThread(void*, XrdClientThread*) () from
/usr/local/lib64/libXrdClient.so.1
#6 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#7 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#8 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x419e0940 (LWP 20512)):
#0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6
#1 0x00002b76aeaed1bf in XrdClientSock::RecvRaw(void*, int, int, int*) ()
from /usr/local/lib64/libXrdClient.so.1
#2 0x00002b76aeb0d401 in XrdClientPhyConnection::ReadRaw(void*, int, int,
int*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb11666 in XrdClientMessage::ReadRaw(XrdClientPhyConnection*)
() from /usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aeb0cbd7 in XrdClientPhyConnection::BuildMessage(bool, bool) ()
from /usr/local/lib64/libXrdClient.so.1
#5 0x00002b76aeb1104e in SocketReaderThread(void*, XrdClientThread*) () from
/usr/local/lib64/libXrdClient.so.1
#6 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#7 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#8 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x40fdf940 (LWP 20511)):
#0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6
#1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6
#2 0x00002b76aeb05bbf in GarbageCollectorThread(void*, XrdClientThread*) ()
from /usr/local/lib64/libXrdClient.so.1
#3 0x00002b76aeb1ea8a in XrdClientThreadDispatcher(void*) () from
/usr/local/lib64/libXrdClient.so.1
#4 0x00002b76aefddd93 in XrdSysThread_Xeq () from
/usr/local/lib64/libXrdUtils.so.1
#5 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6
(gdb)
_______________________________________________________
File Attachments:
-------------------------------------------------------
Date: 2012-04-18 06:27 Name: build-info.txt Size: 55kB By: bdouglas
<http://savannah.cern.ch/bugs/download.php?file_id=25303>
_______________________________________________________
Reply to this item at:
<http://savannah.cern.ch/bugs/?93797>
_______________________________________________
Message sent via/by LCG Savannah
http://savannah.cern.ch/
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
