Follow-up Comment #5, bug #93797 (project xrootd): Hi I repeated the test twice now using xrdcp -x mode In the first test I copied 1000 small files from the US ATLAS regional redirector without incident. In the second case. I have managed to get the a xrdcp -x hang Here is the command used and the output to sysout: cmd: xrdcp -x -np -v -f -d 1 root://xrd-central.uchicago.edu//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physi cs_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 root://atlas70.hep.an l.gov//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZ d3pdExample_NTUP_SMWZ.523740._000001.root.1 Set debug level 1 Extreme Copy enabled. 120421 05:22:58 16657 Xrd: main: (C) 2004-2011 by the XRootD collaboration. Version: v20120412-f837b66 120421 05:22:58 16657 Xrd: Create: (C) 2004-2010 by the Xrootd group. XrdClient $Revision$ - Xrootd version: v3.2.0-rc1 120421 05:22:58 16657 Xrd: ShowUrls: The converted URLs count is 1 120421 05:22:58 16657 Xrd: ShowUrls: URL n.1: root://xrd-central.uchicago.edu:1094//atlas/dq2/user/bdouglas/physics_Egam ma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 . 120421 05:22:58 16657 Xrd: ShowUrls: The converted URLs count is 1 120421 05:22:58 16657 Xrd: ShowUrls: URL n.1: root://xrd-central.uchicago.edu:1094//atlas/dq2/user/bdouglas/physics_Egam ma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 . 120421 05:22:58 16657 Xrd: Open: Access to server granted. 120421 05:22:58 16657 Xrd: Open: Opening the remote file /atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_E gamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 120421 05:22:58 16657 Xrd: Open: File open in progress. 120421 05:22:58 16660 Xrd: HandleServerError: Received redirection to [tier2-03.ochep.ou.edu:1094]. Token=[]]. Opaque=[]. 120421 05:22:58 16660 Xrd: DumpPhyConn: Phyconn entry, [log in to unmask]:1094', LogCnt=0 Valid sec_Client: protocol request for host tier2-03.ochep.ou.edu token='&P=gsi,v:10300,c:ssl,ca:1c3f2ca8.0' sec_PM: Loading gsi protocol object from libXrdSecgsi.so 120421 05:22:58 16660 secgsi_Init: option CACheck: 1 120421 05:22:58 16660 secgsi_Init: testing CA dir(s): /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates 120421 05:22:58 16660 secgsi_Init: using CA dir(s): /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates/ 120421 05:22:58 16660 secgsi_Init: option CRLCheck: 1 ('use-if-available'; download? no) 120421 05:22:58 16660 secgsi_Init: testing CRL dir(s): /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates 120421 05:22:58 16660 secgsi_Init: using CRL dir(s): /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates/ 120421 05:22:58 16660 secgsi_Init: CRL information refreshed every 86400 secs 120421 05:22:58 16660 sut_Cache::Init: cache allocated for 100 entries 120421 05:22:58 16660 sut_Cache::Rehash: Hash table updated (found 0 active entries) 120421 05:22:58 16660 sut_Cache::Init: cache allocated for 2 entries 120421 05:22:58 16660 sut_Cache::Rehash: Hash table updated (found 0 active entries) 120421 05:22:58 16660 secgsi_InitOpts: *** ------------------------------------------------------------ *** 120421 05:22:58 16660 secgsi_InitOpts: Mode: client 120421 05:22:58 16660 secgsi_InitOpts: Debug: 1 120421 05:22:58 16660 secgsi_InitOpts: CA dir: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates 120421 05:22:58 16660 secgsi_InitOpts: CA verification level: 1 120421 05:22:58 16660 secgsi_InitOpts: CRL dir: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates 120421 05:22:58 16660 secgsi_InitOpts: CRL extension: .r0 120421 05:22:58 16660 secgsi_InitOpts: CRL check level: 1 120421 05:22:58 16660 secgsi_InitOpts: CRL refresh time: 86400 120421 05:22:58 16660 secgsi_InitOpts: Certificate: /users/dbenjamin/.globus/usercert.pem 120421 05:22:58 16660 secgsi_InitOpts: Key: /users/dbenjamin/.globus/userkey.pem 120421 05:22:58 16660 secgsi_InitOpts: Proxy file: /tmp/x509up_u30001 120421 05:22:58 16660 secgsi_InitOpts: Proxy validity: 12:00 120421 05:22:58 16660 secgsi_InitOpts: Proxy dep length: 0 120421 05:22:58 16660 secgsi_InitOpts: Proxy bits: 512 120421 05:22:58 16660 secgsi_InitOpts: Proxy sign option: 1 120421 05:22:58 16660 secgsi_InitOpts: Proxy delegation option: 0 : 120421 05:22:58 16660 secgsi_InitOpts: Allowed server names: [*/]<target host name>[/*] 120421 05:22:58 16660 secgsi_InitOpts: Crypto modules: ssl 120421 05:22:58 16660 secgsi_InitOpts: Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc 120421 05:22:58 16660 secgsi_InitOpts: MDigests: sha1:md5 120421 05:22:58 16660 secgsi_InitOpts: *** ------------------------------------------------------------ *** sec_PM: Using gsi protocol, args='v:10300,c:ssl,ca:1c3f2ca8.0' 120421 05:22:58 16660 secgsi_XrdSecProtocolgsi: constructing: host: tier2-03.ochep.ou.edu 120421 05:22:58 16660 secgsi_XrdSecProtocolgsi: p: gsi, plen: 4 120421 05:22:58 16660 secgsi_XrdSecProtocolgsi: mode: client 120421 05:22:58 16660 secgsi_XrdSecProtocolgsi: object created: v.. 120421 05:22:58 16660 secgsi_ParseCrypto: parsing list: ssl 120421 05:22:58 16660 secgsi_ParseCrypto: found module: ssl 120421 05:22:58 16660 crypto_Factory::GetCryptoFactory: loading ssl crypto factory object from libXrdCrypto.so 120421 05:22:58 16660 crypto_Factory::GetCryptoFactory: loading ssl crypto factory object from libXrdCryptossl.so 120421 05:22:58 16660 sut_Rndm::GetBuffer: enter: len: 32 120421 05:22:58 16660 sut_Rndm::Init: taking seed from /dev/urandom 120421 05:22:58 16660 secgsi_ParseCAlist: parsing list: 1c3f2ca8.0 120421 05:22:58 16660 secgsi_GetCA: Querying cache for tag: 1c3f2ca8.0:1 (timestamp:1335003778, refresh fq:86400) 120421 05:22:58 16660 secgsi_GetCA: trying to load CA certificate from /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates/1c3f2ca8.0 120421 05:22:58 16660 cryptossl_X509::IsCA: certificate has 7 extensions 120421 05:22:58 16660 secgsi_VerifyCA: Warning: CA certificate not self-signed: integrity not checked, assuming OK (1c3f2ca8.0) 120421 05:22:58 16660 secgsi_LoadCRL: target file: /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates/1c3f2ca8.r0 120421 05:22:58 16660 sut_Cache::Init: cache allocated for 355 entries 120421 05:22:58 16660 sut_Cache::Rehash: Hash table updated (found 0 active entries) 120421 05:22:58 16660 sut_Cache::Rehash: Hash table updated (found 355 active entries) 120421 05:22:58 16660 secgsi_LoadCRL: CA signing certificate file = /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/etc/grid-security/certificates/1c3f2ca8.0 120421 05:22:58 16660 cryptossl_X509::IsCA: certificate has 7 extensions 120421 05:22:58 16660 sut_Cache::Rehash: Hash table updated (found 1 active entries) 120421 05:22:58 16660 cryptossl_X509::IsCA: certificate has 3 extensions 120421 05:22:58 16660 cryptossl_X509::IsCA: certificate has 6 extensions 120421 05:22:58 16660 sut_Cache::Rehash: Hash table updated (found 1 active entries) 120421 05:22:58 16660 secgsi_getCredentials: version run by server: 10300 120421 05:22:58 16660 secgsi_CheckRtag: Nothing to check 120421 05:22:58 16660 secgsi_getCredentials: Client issuer hash: 1c3f2ca8.0 120421 05:22:58 16660 sut_Rndm::GetString: enter: len: 8 (type: Crypt) 120421 05:22:58 16660 sut_Rndm::GetString: got: ON.5CV1z 120421 05:22:58 16660 secgsi_getCredentials: returned 101 bytes of credentials 120421 05:22:58 16660 crypto_X509Chain::EECname: EEC not found in chain 120421 05:22:58 16660 crypto_X509Chain::EEChash: EEC not found in chain 120421 05:22:58 16660 cryptossl_X509::IsCA: certificate has 6 extensions 120421 05:22:58 16660 secgsi_getCredentials: version run by server: 10300 120421 05:22:58 16660 secgsi_CheckRtag: Random tag successfully checked 120421 05:22:58 16660 sut_Rndm::GetString: enter: len: 8 (type: Crypt) 120421 05:22:58 16660 sut_Rndm::GetString: got: GoCQgFk7 120421 05:22:58 16660 secgsi_getCredentials: returned 5349 bytes of credentials 120421 05:22:58 16657 Xrd: ShowUrls: The converted URLs count is 1 120421 05:22:58 16657 Xrd: ShowUrls: URL n.1: root://atlas70.hep.anl.gov:1094//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1?oss.asize=11653647. 120421 05:22:58 16657 Xrd: ShowUrls: The converted URLs count is 1 120421 05:22:58 16657 Xrd: ShowUrls: URL n.1: root://atlas70.hep.anl.gov:1094//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1?oss.asize=11653647. 120421 05:22:58 16657 Xrd: DumpPhyConn: Phyconn entry, [log in to unmask]:1094', LogCnt=1 Valid 120421 05:22:58 16657 Xrd: DumpPhyConn: Phyconn entry, [log in to unmask]:1094', LogCnt=0 Valid sec_Client: protocol request for host atlas70.hep.anl.gov token='&P=unix' sec_PM: Loading unix protocol object from libXrdSecunix.so sec_PM: Using unix protocol, args='' 120421 05:22:58 16657 Xrd: Open: Access to server granted. 120421 05:22:58 16657 Xrd: Open: Opening the remote file /atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1?oss.asize=11653647 120421 05:22:58 16657 Xrd: Open: File opened succesfully. 120421 05:22:58 16657 Xrd: main: root://xrd-central.uchicago.edu//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 --> root://atlas70.hep.anl.gov//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 120421 05:22:58 16657 Xrd: ShowUrls: The converted URLs count is 1 120421 05:22:58 16657 Xrd: ShowUrls: URL n.1: root://xrd-central.uchicago.edu:1094//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1. 120421 05:22:58 16657 Xrd: ShowUrls: The converted URLs count is 1 120421 05:22:58 16657 Xrd: ShowUrls: URL n.1: root://xrd-central.uchicago.edu:1094//atlas/dq2/user/bdouglas/physics_Egamma/user.bdouglas.physics_Egamma.SMWZd3pdExample.NTUP_SMWZ.f406_m991_p716/SMWZd3pdExample_NTUP_SMWZ.523740._000001.root.1 Here are the specifics from the gdb backtrace - Thread 10 (Thread 0x2b044650bc20 (LWP 16657)): #0 0x0000003cf620b1c0 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003209c1c022 in XrdSysCondVar::Wait (this=0x61aa20, sec=3600) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:110 #2 0x0000000000411f72 in ?? () Thread 9 (Thread 0x47410940 (LWP 16671)): #0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6 #1 0x000000320a01f1e6 in XrdClientSock::RecvRaw (this=0x2aaab4005720, buffer=0x13545a20, length=8, substreamid=-1, usedsubstreamid=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientSock.cc:133 #2 0x000000320a03a362 in XrdClientPhyConnection::ReadRaw (this=0x2aaab40039e0, buf=0x13545a20, len=8, substreamid=-1, usedsubstreamid=0x4740fd9c) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:359 #3 0x000000320a03d458 in XrdClientMessage::ReadRaw (this=0x135459e0, phy=0x2aaab40039e0) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientMessage.cc:152 #4 0x000000320a039b1c in XrdClientPhyConnection::BuildMessage (this=0x2aaab40039e0, IgnoreTimeouts=true, Enqueue=true) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:440 #5 0x000000320a03a152 in SocketReaderThread (arg=0x2aaab40039e0, thr=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:57 #6 0x0000003209c1be7f in XrdSysThread_Xeq (myargs=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:67 #7 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0 #8 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x47e11940 (LWP 16670)): #0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6 #1 0x000000320a01f1e6 in XrdClientSock::RecvRaw (this=0x2aaaac0072c0, buffer=0x1353bd80, length=8, substreamid=-1, usedsubstreamid=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientSock.cc:133 #2 0x000000320a03a362 in XrdClientPhyConnection::ReadRaw (this=0x2aaaac005950, buf=0x1353bd80, len=8, substreamid=-1, usedsubstreamid=0x47e10d9c) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:359 #3 0x000000320a03d458 in XrdClientMessage::ReadRaw (this=0x1353bd40, phy=0x2aaaac005950) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientMessage.cc:152 #4 0x000000320a039b1c in XrdClientPhyConnection::BuildMessage (this=0x2aaaac005950, IgnoreTimeouts=true, Enqueue=true) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:440 #5 0x000000320a03a152 in SocketReaderThread (arg=0x2aaaac005950, thr=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:57 #6 0x0000003209c1be7f in XrdSysThread_Xeq (myargs=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:67 #7 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0 #8 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x4600e940 (LWP 16667)): #0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6 #1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6 #2 0x00000000004084ca in ?? () Thread 6 (Thread 0x4560d940 (LWP 16666)): #0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6 #1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6 #2 0x00000000004084ca in ?? () Thread 5 (Thread 0x44c0c940 (LWP 16665)): #0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6 #1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6 #2 0x00000000004084ca in ?? () Thread 4 (Thread 0x4420b940 (LWP 16664)): #0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6 #1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6 #2 0x00000000004084ca in ?? () Thread 3 (Thread 0x42408940 (LWP 16662)): #0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6 #1 0x000000320a01f1e6 in XrdClientSock::RecvRaw (this=0x1353dc00, buffer=0x1353c5c0, length=8, substreamid=-1, usedsubstreamid=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientSock.cc:133 #2 0x000000320a03a362 in XrdClientPhyConnection::ReadRaw (this=0x1353c6b0, buf=0x1353c5c0, len=8, substreamid=-1, usedsubstreamid=0x42407d9c) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:359 #3 0x000000320a03d458 in XrdClientMessage::ReadRaw (this=0x1353c580, phy=0x1353c6b0) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientMessage.cc:152 #4 0x000000320a039b1c in XrdClientPhyConnection::BuildMessage (this=0x1353c6b0, IgnoreTimeouts=true, Enqueue=true) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:440 #5 0x000000320a03a152 in SocketReaderThread (arg=0x1353c6b0, thr=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:57 #6 0x0000003209c1be7f in XrdSysThread_Xeq (myargs=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:67 #7 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0 #8 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x42e09940 (LWP 16661)): #0 0x0000003cf5aca1e6 in poll () from /lib64/libc.so.6 #1 0x000000320a01f1e6 in XrdClientSock::RecvRaw (this=0x2aaaac002ff0, buffer=0x134b2fa0, length=8, substreamid=-1, usedsubstreamid=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientSock.cc:133 #2 0x000000320a03a362 in XrdClientPhyConnection::ReadRaw (this=0x2aaaac001240, buf=0x134b2fa0, len=8, substreamid=-1, usedsubstreamid=0x42e08d9c) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:359 #3 0x000000320a03d458 in XrdClientMessage::ReadRaw (this=0x134b2f60, phy=0x2aaaac001240) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientMessage.cc:152 #4 0x000000320a039b1c in XrdClientPhyConnection::BuildMessage (this=0x2aaaac001240, IgnoreTimeouts=true, Enqueue=true) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:440 #5 0x000000320a03a152 in SocketReaderThread (arg=0x2aaaac001240, thr=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientPhyConnection.cc:57 #6 0x0000003209c1be7f in XrdSysThread_Xeq (myargs=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:67 #7 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0 #8 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x40bb1940 (LWP 16658)): #0 0x0000003cf5a99031 in nanosleep () from /lib64/libc.so.6 #1 0x0000003cf5a98e54 in sleep () from /lib64/libc.so.6 #2 0x000000320a032a9f in GarbageCollectorThread (arg=0x13485d10, thr=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdClient/XrdClientConnMgr.cc:73 #3 0x0000003209c1be7f in XrdSysThread_Xeq (myargs=<value optimized out>) at /usr/src/debug/xrootd/xrootd/src/XrdSys/XrdSysPthread.cc:67 #4 0x0000003cf620677d in start_thread () from /lib64/libpthread.so.0 #5 0x0000003cf5ad30dd in clone () from /lib64/libc.so.6 _______________________________________________________ Reply to this item at: <http://savannah.cern.ch/bugs/?93797> _______________________________________________ Message sent via/by LCG Savannah http://savannah.cern.ch/ ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1