 |
 |
Hi Andy, thanks. I just checked, and that's already installed on se1. We probably did that a while ago when we were looking at other issues. We're running 4.11.1 everywhere on those storage nodes. Should we upgrade to 4.11.2, or is there nothing in the .2 update that's applicable for us right now? Thanks, Horst Andrew Hanushevsky <[log in to unmask]> wrote: > Hi Horst, > > Please make sure to install the debug RPM so that we can get actual > statement numbers. It's called xrootd-debuginfo-4.11.2-1.el7.x86_64.rpm > (assuming you are running Cent7). They should be readily available from > OSG (though we have them too). > > Andy > > On Sat, 21 Mar 2020, Horst Severini wrote: > > > Thanks Andy, > > > > will do! > > > > Horst > > > > Andrew Hanushevsky <[log in to unmask]> wrote: > > > >> Hi Horst, > >> > >> Indeed, one should never see a core dump and if one does appear we > >> definitely want to know about it. When you do see one, here is the first > >> dump of information that would be helpful before we start digging deeper: > >> > >> gdb >executable> <corefile> > >> where > >> quit > >> > >> Cut and past the output into a mail file or posting. We may ask for a > >> detailed traceback of every thread. I think I'll put that process on the > >> xroot web page. For now, at least we will know where it went bonkers. > >> > >> Andy > >> > >> > >> On Sat, 21 Mar 2020, Horst Severini wrote: > >> > >>> Hi Wei, > >>> > >>> yes, we have enough space for a few core dumps in /var/. It's just that there > >>> were 5 or 6 in the last week or two, and that filled /var/ up completely. > >>> I'll keep a closer eye on it for now. > >>> > >>> Thanks, > >>> > >>> Horst > >>> > >>> On 3/21/20 4:39 PM, Yang, Wei wrote: > >>>> Also make sure you have enough space to hold a core dump. It can something > >>>> be 10GB+ > >>>> > >>>> -- > >>>> Wei Yang [log in to unmask] | 650-926-3338(O) > >>>> > >>>> ???-----Original Message----- > >>>> From:<[log in to unmask]> on behalf of Horst > >>>> Severini<[log in to unmask]> > >>>> Date: Saturday, March 21, 2020 at 1:30 PM > >>>> To: xrootd-dev<[log in to unmask]>,"[log in to unmask]" > >>>> <[log in to unmask]> > >>>> Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT > >>>> > >>>> Thanks Wei, > >>>> I'll send you the next one I get!:) > >>>> Cheers, > >>>> Horst > >>>> On 3/21/20 2:55 PM, Yang, Wei wrote: > >>>> > Indeed a core dump is usually the thing we need. > >>>> > > >>>> > -- > >>>> > Wei Yang [log in to unmask] | 650-926-3338(O) > >>>> > > >>>> > On 3/21/20, 12:19 PM,"[log in to unmask] on behalf of > >>>> Horst Severini" <[log in to unmask] on behalf of [log in to unmask]> > >>>> wrote: > >>>> > > >>>> > We're running 4.11.1 here at OU. > >>>> > > >>>> > Cheers, > >>>> > > >>>> > Horst > >>>> > > >>>> > Albert Rossi<[log in to unmask]> wrote: > >>>> > > >>>> > > Hi Horst, > >>>> > > > >>>> > > actually, if you notice, all endpoints failed on today's > >>>> test. So it was not just OU. > >>>> > > > >>>> > > The Stanford developers may want you to run a few commands > >>>> over the core file from gdb once you have it in hand. > >>>> > > > >>>> > > What version of xrootd are you running, just out of > >>>> curiosity? Is it bleeding-edge, or a stable release? > >>>> > > > >>>> > > Cheers, Al > >>>> > > > >>>> > > ________________________________________________ > >>>> > > Albert L. Rossi > >>>> > > Application Developer & Systems Analyst III > >>>> > > Scientific Computing Division, Data Movement Development > >>>> > > FCC 229A > >>>> > > Mail Station 369 (FCC 2W) > >>>> > > Fermi National Accelerator Laboratory > >>>> > > Batavia, IL 60510 > >>>> > > (630) 840-3023 > >>>> > > ________________________________ > >>>> > > From: Horst Severini<[log in to unmask]> > >>>> > > Sent: Saturday, March 21, 2020 1:18 PM > >>>> > >To:[log in to unmask] > >>>> <[log in to unmask]>;[log in to unmask]<[log in to unmask]>; > >>>> Albert Rossi<[log in to unmask]> > >>>> > > Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 > >>>> GMT > >>>> > > > >>>> > > Hi Al, > >>>> > > > >>>> > > thanks, good idea. I'll save the next core file. > >>>> > > > >>>> > > I'm pretty sure the authentication failures simply came > >>>> because > >>>> > > that partition was full and no new proxies or what not could > >>>> be > >>>> > > created, so I wouldn't worry about that. > >>>> > > > >>>> > > Thanks, > >>>> > > > >>>> > > Horst > >>>> > > > >>>> > > Albert Rossi<[log in to unmask]> wrote: > >>>> > > > >>>> > >> Hi Horst, > >>>> > >> > >>>> > >> I would definitely report > >>>> [log in to unmask] > >>>> > >> > >>>> > >> As for why the massive authentication failure, I've seen > >>>> this before, it might have to do with CA cert issues. > >>>> > >> > >>>> > >> Cheers, Al > >>>> > >> > >>>> > >> ________________________________________________ > >>>> > >> Albert L. Rossi > >>>> > >> Application Developer & Systems Analyst III > >>>> > >> Scientific Computing Division, Data Movement Development > >>>> > >> FCC 229A > >>>> > >> Mail Station 369 (FCC 2W) > >>>> > >> Fermi National Accelerator Laboratory > >>>> > >> Batavia, IL 60510 > >>>> > >> (630) 840-3023 > >>>> > >> ________________________________ > >>>> > >> From: Horst Severini<[log in to unmask]> > >>>> > >> Sent: Saturday, March 21, 2020 11:19 AM > >>>> > >>To:[log in to unmask] <[log in to unmask]> > >>>> > >> Subject: Re: XrootD smoke test report for 2020-03-21 > >>>> 10:01:46 GMT > >>>> > >> > >>>> > >> Hi all, > >>>> > >> > >>>> > >> our/var/ partition had filled up because of too many > >>>> xrootd core dumps. > >>>> > >> I cleared those up and restarted xrootd, and things look > >>>> better again now. > >>>> > >> > >>>> > >> Not sure why we keep getting core dumps, though. > >>>> > >> > >>>> > >> Cheers, > >>>> > >> > >>>> > >> Horst > >>>> > >> > >>>> > >> On 3/21/20 5:01 AM,[log in to unmask] wrote: > >>>> > >>> XROOTD SMOKE TEST SUMMARY > >>>> > >>> 2020-03-21 10:01:46 GMT > >>>> > >>> > >>>> > >>> Client: bogus6.fnal.gov > >>>> > >>> > >>>> > >>> XrootD version: v4.11.2 > >>>> > >>> > >>>> > >>> Reference server: CERN-TRUNK > >>>> > >>> > >>>> > >>> Credential delegation: ON > >>>> > >>> > >>>> > >>> Checksum: -C adler32 > >>>> > >>> > >>>> > >>> Total number of round-trip tests: 21 > >>>> > >>> > >>>> > >>> --------------------------------SOUND > >>>> ENDPOINTS--------------------------------- > >>>> > >>> > >>>> > >>> SCORE ENDPT TYPE UP SRC > >>>> DST DN > >>>> > >>> > >>>> -------------------------------------------------------------------------------- > >>>> > >>> > >>>> > >>> -----------------------------PROBLEMATIC > >>>> ENDPOINTS------------------------------ > >>>> > >>> > >>>> > >>> SCORE ENDPT TYPE UP SRC > >>>> DST DN > >>>> > >>> > >>>> -------------------------------------------------------------------------------- > >>>> > >>> 19 BRUSSELS dCache F - > >>>> F F 0/4 > >>>> > >>> 19 CERN-EOS EOS F - > >>>> F F 0/4 > >>>> > >>> 19 CERN-TRUNK DPM - F > >>>> - F 0/2 > >>>> > >>> 19 DESY-PROM dCache F - > >>>> F F 0/4 > >>>> > >>> 19 FNAL dCache F - > >>>> F F 0/4 > >>>> > >>> 19 IN2P3-DOMA xrootd F - > >>>> F F 0/4 > >>>> > >>> 19 PRAGUE DPM F - > >>>> F F 0/4 > >>>> > >>> 19 RAL-CEPH CEPH F - > >>>> F F 0/4 > >>>> > >>> 19 RAL-LCG2 Echo F - > >>>> F F 0/4 > >>>> > >>> 19 SLAC XrootD F - > >>>> F F 0/4 > >>>> > >>> 19 TRIUMF dCache F - > >>>> F F 0/4 > >>>> > >>> 19 UKI-BRUNEL DPM F - > >>>> F F 0/4 > >>>> > >>> 19 UKI-LANC DPM F - > >>>> F F 0/4 > >>>> > >>> 19 UKI-MAN1 DPM F - > >>>> F F 0/4 > >>>> > >>> 19 UKI-MAN2 DPM F - > >>>> F F 0/4 > >>>> > >>> 19 UNI-BONN CephFS F - > >>>> F F 0/4 > >>>> > >>> 18 OU XrootD F - > >>>> F F 0/4 > >>>> > >>> 13 BNL dCache F - > >>>> F F 0/4 > >>>> > >>> 7 AGLT2 dCache F - > >>>> F F 0/4 > >>>> > >>> 0 CALTECH HDFS F - > >>>> F F 0/4 > >>>> > >>> 0 TRIUMF-PROD dCache F - > >>>> F F 0/4 > >>> > > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1