Hi Andy, I'm the one with the offending host. I can confirm that 4.4 and 4.7 clients also suffer from this. There's nothing interesting in the log (from what I can see). I restarted the server to capture all the initial logging info. It's not 10 minutes, but the logs are basically the same redirects over and over. Thanks! ``` [root@xrootd ~]# rm /var/log/xrootd/clustered/xrootd.log ; systemctl restart xrootd@clustered ; sleep 1 ; tail -n 1000 -f /var/log/xrootd/clustered/xrootd.log rm: remove regular file ‘/var/log/xrootd/clustered/xrootd.log’? y 180621 20:43:28 496496 Starting on Linux 3.10.0-693.2.2.el7.x86_64 Copr. 2004-2012 Stanford University, xrd version v4.8.1 ++++++ xrootd [log in to unmask] initialization started. Config using configuration file /etc/xrootd/xrootd-clustered.cfg =====> xrd.port 1094 if exec xrootd =====> xrd.network nodnr =====> all.sitename T2_US_Vanderbilt_TEST =====> xrd.protocol http:1094 /usr/lib64/libXrdHttp-4.so =====> xrd.report xrootd.t2.ucsd.edu:9931 every 30s all sync Config maximum number of connections restricted to 65536 Plugin No such file or directory loading protocol /usr/lib64/libXrdHttp-4-4.so Config Falling back to using /usr/lib64/libXrdHttp-4.so Plugin loaded Copr. 2012 Stanford University, xrootd protocol 3.1.0 version v4.8.1 ++++++ xrootd protocol initialization started. =====> all.export /store =====> xrootd.fslib throttle default =====> xrootd.trace emsg login stall redirect =====> xrootd.seclib /usr/lib64/libXrdSec.so =====> xrootd.monitor all auth flush 30s mbuff 1472 window 5s dest files io info user xrootd.t2.ucsd.edu:9930 Config exporting /store Plugin loaded ++++++ Authentication system initialization started. Plugin loaded 180621 20:43:28 496496 secgsi_InitOpts: *** ------------------------------------------------------------ *** 180621 20:43:28 496496 secgsi_InitOpts: Mode: server 180621 20:43:28 496496 secgsi_InitOpts: Debug: -1 180621 20:43:28 496496 secgsi_InitOpts: CA dir: /etc/grid-security/certificates 180621 20:43:28 496496 secgsi_InitOpts: CA verification level: 1 180621 20:43:28 496496 secgsi_InitOpts: CRL dir: /etc/grid-security/certificates/ 180621 20:43:28 496496 secgsi_InitOpts: CRL extension: .r0 180621 20:43:28 496496 secgsi_InitOpts: CRL check level: 1 180621 20:43:28 496496 secgsi_InitOpts: CRL refresh time: 86400 180621 20:43:28 496496 secgsi_InitOpts: Certificate: /etc/grid-security/hostcert-xrootd.pem 180621 20:43:28 496496 secgsi_InitOpts: Key: /etc/grid-security/hostkey-xrootd.pem 180621 20:43:28 496496 secgsi_InitOpts: Proxy delegation option: 0 180621 20:43:28 496496 secgsi_InitOpts: GRIDmap file: /etc/grid-security/grid-mapfile 180621 20:43:28 496496 secgsi_InitOpts: GRIDmap option: 1 180621 20:43:28 496496 secgsi_InitOpts: GRIDmap cache entries expiration (secs): 600 180621 20:43:28 496496 secgsi_InitOpts: Client proxy availability in XrdSecEntity.endorsement: 0 180621 20:43:28 496496 secgsi_InitOpts: VOMS option: 1 180621 20:43:28 496496 secgsi_InitOpts: MonInfo option: 1 180621 20:43:28 496496 secgsi_InitOpts: Crypto modules: ssl 180621 20:43:28 496496 secgsi_InitOpts: Ciphers: aes-128-cbc:bf-cbc:des-ede3-cbc 180621 20:43:28 496496 secgsi_InitOpts: MDigests: sha1:md5 180621 20:43:28 496496 secgsi_InitOpts: *** ------------------------------------------------------------ *** =====> sec.protocol /usr/lib64 gsi -certdir:/etc/grid-security/certificates -cert:/etc/grid-security/hostcert-xrootd.pem -key:/etc/grid-security/hostkey-xrootd.pem -crl:1 -moninfo Config 1 authentication directives processed in /etc/xrootd/xrootd-clustered.cfg ------ Authentication system initialization completed. ++++++ Protection system initialization started. Config warning: Security level is set to none; request protection disabled! Config Local protection level: none Config Remote protection level: none ------ Protection system initialization completed. Config Routing for 10.0.11.146: local pub4 prv4 Config Route all4: 10.0.11.146 Dest=[::10.0.11.146]:1094 ++++++ File system initialization started. =====> all.role manager Config warning: ignoring invalid trace option 'conn'. =====> ofs.trace conn =====> ofs.authorize ++++++ Storage system initialization started. =====> all.export /store Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration: oss.alloc 0 0 0 oss.cachescan 600 oss.fdlimit 32768 65536 oss.maxsize 0 oss.trace 0 oss.xfr 1 deny 10800 keep 1200 oss.memfile off max 5158240256 oss.defaults r/w nocheck nodread nomig norcreate nopurge nostage xattr oss.path /store r/w nocheck nodread nomig norcreate nopurge nostage xattr ------ Storage system initialization completed. ++++++ Authorization system initialization started. 180621 20:43:28 496496 acc_Config: Authorization system using configuration in /etc/xrootd/xrootd-clustered.cfg =====> acc.authdb /etc/xrootd/Authfile =====> acc.audit deny grant Config 2 authorization directives processed in /etc/xrootd/xrootd-clustered.cfg Config 1 auth entries processed in /etc/xrootd/Authfile ------ Authorization system initialization completed. ++++++ Configuring manager role. . . =====> cms.trace all =====> all.manager xrootd.accre.vanderbilt.edu:1213 =====> all.manager meta all cms-xrd-transit.cern.ch+ 1213 180621 20:43:28 496496 Configure Global System Identification: clustered-m 1213xrootd.accre.vanderbilt.edu Config 1 manager(s) started. Config effective /etc/xrootd/xrootd-clustered.cfg ofs configuration: all.role manager ofs.authorize ofs.maxdelay 60 ofs.persist manual hold 600 logdir /tmp/clustered/.ofs/posc.log ofs.trace 0 ------ File system manager initialization completed. 180621 20:43:28 496523 cms_Finder: Connected to cmsd via /tmp/clustered/.olb/olbd.nimda Plugin loaded Initializing a Throttled file system. (Throttle Config)> throttle.throttle concurrency 20 Config warning: 'xrootd.prepare logdir' not specified; prepare tracking disabled. 180621 20:43:28 496521 cms_ClientMan: Connected to xrootd.accre.vanderbilt.edu v 3 180621 20:43:28 496521 Hookup xrootd.accre.vanderbilt.edu qt=178ms rw=2 ------ xrootd protocol initialization completed. Copr. 2012 CERN IT, an HTTP implementation for the XROOTD framework. ++++++ HTTP protocol initialization started. Config Routing for 10.0.11.146: local pub4 prv4 Config Route all4: 10.0.11.146 Dest=[::10.0.11.146]:1094 Config warning: ignoring unknown directive 'export'. =====> all.export =====> http.cadir /etc/grid-security/certificates =====> http.cert /etc/grid-security/hostcert-xrootd.pem =====> http.key /etc/grid-security/hostkey-xrootd.pem Plugin No such file or directory loading secxtractorlib /usr/lib64/libXrdLcmaps-4.so Config Falling back to using /usr/lib64/libXrdLcmaps.so Plugin loaded =====> http.secxtractor /usr/lib64/libXrdLcmaps.so =====> http.listingdeny yes =====> http.staticpreload http://static/robots.txt /etc/xrootd/robots.txt =====> http.desthttps yes Using TLS 1.2 180621 20:43:28 496496 sysConfig: XRDROLE: manager 180621 20:43:28 496496 sysConfig: Configured as HTTP(s) redirector. ------ HTTP protocol initialization completed. ------ xrootd [log in to unmask]:1094 initialization completed. 180621 20:43:29 496503 cmsgli.264:29@[::131.225.189.179] XrootdProtocol: more auth requested; sz=2151 180621 20:43:29 496503 XrootdXeq: cmsgli.264:29@[::131.225.189.179] pub IPv4 login as e5355189.0 180621 20:43:29 496521 Receive 129.59.197.121 30 bytes on 3071 180621 20:43:29 496521 Decode xrootd redirects cmsgli.264:29@[::131.225.189.179] to se20.accre.vanderbilt.edu:1094 /store/data/Run2016B/JetHT/AOD/23Sep2016-v3/110002/00638260-0298-E611-8DB3-0CC47A4C8E7E.root 180621 20:43:29 496503 cmsgli.264:29@[::131.225.189.179] XrootdProtocol: redirecting to :0 180621 20:43:39 496502 dcms065.1520388:30@[::192.108.46.248] XrootdProtocol: more auth requested; sz=2151 180621 20:43:39 496502 XrootdXeq: dcms065.1520388:30@[::192.108.46.248] pub IPv4 login as c5ec2ee5.0 180621 20:43:39 496521 Receive 129.59.197.121 30 bytes on 4095 180621 20:43:39 496521 Decode xrootd redirects dcms065.1520388:30@[::192.108.46.248] to se20.accre.vanderbilt.edu:1094 /store/mc/RunIIFall17MiniAODv2/WWTo2L2Nu_NNPDF31_TuneCP5_PSweights_13TeV-powheg-pythia8/MINIAODSIM/PU2017_12Apr2018_94X_mc2017_realistic_v14_ext1-v1/40000/3A447BB7-FF41-E811-964D-3417EBE2F445.root 180621 20:43:39 496502 dcms065.1520388:30@[::192.108.46.248] XrootdProtocol: redirecting to :0 180621 20:43:49 496554 XrootdXeq: pxy7.3631:31@[::140.112.104.122] disc 0:00:01 180621 20:43:54 496502 pilcms31.263:32@[::202.122.36.3] XrootdProtocol: more auth requested; sz=2151 180621 20:43:54 496502 XrootdXeq: pilcms31.263:32@[::202.122.36.3] pub IPv4 login as e5355189.0 180621 20:43:54 496521 Receive 129.59.197.121 30 bytes on 5119 180621 20:43:54 496521 Decode xrootd redirects pilcms31.263:32@[::202.122.36.3] to se20.accre.vanderbilt.edu:1094 /store/data/Run2016F/JetHT/AOD/23Sep2016-v1/90000/F674D02D-FA86-E611-892D-848F69FD46E8.root 180621 20:43:54 496502 pilcms31.263:32@[::202.122.36.3] XrootdProtocol: redirecting to :0 180621 20:43:55 496554 XrootdXeq: dcms065.1520388:30@[::192.108.46.248] disc 0:00:17 ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/xrootd/xrootd/issues/752#issuecomment-399294890 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1