 |
 |
Hi Horst, Other thna those core files nothing that would interest you. Once we see what the problem is maybe an upgrade would be a good thing. Andy On Sat, 21 Mar 2020, Horst Severini wrote: > Hi Andy, > > thanks. I just checked, and that's already installed on se1. > We probably did that a while ago when we were looking at other issues. > > We're running 4.11.1 everywhere on those storage nodes. Should we upgrade > to 4.11.2, or is there nothing in the .2 update that's applicable for us > right now? > > Thanks, > > Horst > > Andrew Hanushevsky <[log in to unmask]> wrote: > >> Hi Horst, >> >> Please make sure to install the debug RPM so that we can get actual >> statement numbers. It's called xrootd-debuginfo-4.11.2-1.el7.x86_64.rpm >> (assuming you are running Cent7). They should be readily available from >> OSG (though we have them too). >> >> Andy >> >> On Sat, 21 Mar 2020, Horst Severini wrote: >> >>> Thanks Andy, >>> >>> will do! >>> >>> Horst >>> >>> Andrew Hanushevsky <[log in to unmask]> wrote: >>> >>>> Hi Horst, >>>> >>>> Indeed, one should never see a core dump and if one does appear we >>>> definitely want to know about it. When you do see one, here is the first >>>> dump of information that would be helpful before we start digging deeper: >>>> >>>> gdb >executable> <corefile> >>>> where >>>> quit >>>> >>>> Cut and past the output into a mail file or posting. We may ask for a >>>> detailed traceback of every thread. I think I'll put that process on the >>>> xroot web page. For now, at least we will know where it went bonkers. >>>> >>>> Andy >>>> >>>> >>>> On Sat, 21 Mar 2020, Horst Severini wrote: >>>> >>>>> Hi Wei, >>>>> >>>>> yes, we have enough space for a few core dumps in /var/. It's just that there >>>>> were 5 or 6 in the last week or two, and that filled /var/ up completely. >>>>> I'll keep a closer eye on it for now. >>>>> >>>>> Thanks, >>>>> >>>>> Horst >>>>> >>>>> On 3/21/20 4:39 PM, Yang, Wei wrote: >>>>>> Also make sure you have enough space to hold a core dump. It can something >>>>>> be 10GB+ >>>>>> >>>>>> -- >>>>>> Wei Yang [log in to unmask] | 650-926-3338(O) >>>>>> >>>>>> ???-----Original Message----- >>>>>> From:<[log in to unmask]> on behalf of Horst >>>>>> Severini<[log in to unmask]> >>>>>> Date: Saturday, March 21, 2020 at 1:30 PM >>>>>> To: xrootd-dev<[log in to unmask]>,"[log in to unmask]" >>>>>> <[log in to unmask]> >>>>>> Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 GMT >>>>>> >>>>>> Thanks Wei, >>>>>> I'll send you the next one I get!:) >>>>>> Cheers, >>>>>> Horst >>>>>> On 3/21/20 2:55 PM, Yang, Wei wrote: >>>>>> > Indeed a core dump is usually the thing we need. >>>>>> > >>>>>> > -- >>>>>> > Wei Yang [log in to unmask] | 650-926-3338(O) >>>>>> > >>>>>> > On 3/21/20, 12:19 PM,"[log in to unmask] on behalf of >>>>>> Horst Severini" <[log in to unmask] on behalf of [log in to unmask]> >>>>>> wrote: >>>>>> > >>>>>> > We're running 4.11.1 here at OU. >>>>>> > >>>>>> > Cheers, >>>>>> > >>>>>> > Horst >>>>>> > >>>>>> > Albert Rossi<[log in to unmask]> wrote: >>>>>> > >>>>>> > > Hi Horst, >>>>>> > > >>>>>> > > actually, if you notice, all endpoints failed on today's >>>>>> test. So it was not just OU. >>>>>> > > >>>>>> > > The Stanford developers may want you to run a few commands >>>>>> over the core file from gdb once you have it in hand. >>>>>> > > >>>>>> > > What version of xrootd are you running, just out of >>>>>> curiosity? Is it bleeding-edge, or a stable release? >>>>>> > > >>>>>> > > Cheers, Al >>>>>> > > >>>>>> > > ________________________________________________ >>>>>> > > Albert L. Rossi >>>>>> > > Application Developer & Systems Analyst III >>>>>> > > Scientific Computing Division, Data Movement Development >>>>>> > > FCC 229A >>>>>> > > Mail Station 369 (FCC 2W) >>>>>> > > Fermi National Accelerator Laboratory >>>>>> > > Batavia, IL 60510 >>>>>> > > (630) 840-3023 >>>>>> > > ________________________________ >>>>>> > > From: Horst Severini<[log in to unmask]> >>>>>> > > Sent: Saturday, March 21, 2020 1:18 PM >>>>>> > >To:[log in to unmask] >>>>>> <[log in to unmask]>;[log in to unmask]<[log in to unmask]>; >>>>>> Albert Rossi<[log in to unmask]> >>>>>> > > Subject: Re: XrootD smoke test report for 2020-03-21 10:01:46 >>>>>> GMT >>>>>> > > >>>>>> > > Hi Al, >>>>>> > > >>>>>> > > thanks, good idea. I'll save the next core file. >>>>>> > > >>>>>> > > I'm pretty sure the authentication failures simply came >>>>>> because >>>>>> > > that partition was full and no new proxies or what not could >>>>>> be >>>>>> > > created, so I wouldn't worry about that. >>>>>> > > >>>>>> > > Thanks, >>>>>> > > >>>>>> > > Horst >>>>>> > > >>>>>> > > Albert Rossi<[log in to unmask]> wrote: >>>>>> > > >>>>>> > >> Hi Horst, >>>>>> > >> >>>>>> > >> I would definitely report >>>>>> [log in to unmask] >>>>>> > >> >>>>>> > >> As for why the massive authentication failure, I've seen >>>>>> this before, it might have to do with CA cert issues. >>>>>> > >> >>>>>> > >> Cheers, Al >>>>>> > >> >>>>>> > >> ________________________________________________ >>>>>> > >> Albert L. Rossi >>>>>> > >> Application Developer & Systems Analyst III >>>>>> > >> Scientific Computing Division, Data Movement Development >>>>>> > >> FCC 229A >>>>>> > >> Mail Station 369 (FCC 2W) >>>>>> > >> Fermi National Accelerator Laboratory >>>>>> > >> Batavia, IL 60510 >>>>>> > >> (630) 840-3023 >>>>>> > >> ________________________________ >>>>>> > >> From: Horst Severini<[log in to unmask]> >>>>>> > >> Sent: Saturday, March 21, 2020 11:19 AM >>>>>> > >>To:[log in to unmask] <[log in to unmask]> >>>>>> > >> Subject: Re: XrootD smoke test report for 2020-03-21 >>>>>> 10:01:46 GMT >>>>>> > >> >>>>>> > >> Hi all, >>>>>> > >> >>>>>> > >> our/var/ partition had filled up because of too many >>>>>> xrootd core dumps. >>>>>> > >> I cleared those up and restarted xrootd, and things look >>>>>> better again now. >>>>>> > >> >>>>>> > >> Not sure why we keep getting core dumps, though. >>>>>> > >> >>>>>> > >> Cheers, >>>>>> > >> >>>>>> > >> Horst >>>>>> > >> >>>>>> > >> On 3/21/20 5:01 AM,[log in to unmask] wrote: >>>>>> > >>> XROOTD SMOKE TEST SUMMARY >>>>>> > >>> 2020-03-21 10:01:46 GMT >>>>>> > >>> >>>>>> > >>> Client: bogus6.fnal.gov >>>>>> > >>> >>>>>> > >>> XrootD version: v4.11.2 >>>>>> > >>> >>>>>> > >>> Reference server: CERN-TRUNK >>>>>> > >>> >>>>>> > >>> Credential delegation: ON >>>>>> > >>> >>>>>> > >>> Checksum: -C adler32 >>>>>> > >>> >>>>>> > >>> Total number of round-trip tests: 21 >>>>>> > >>> >>>>>> > >>> --------------------------------SOUND >>>>>> ENDPOINTS--------------------------------- >>>>>> > >>> >>>>>> > >>> SCORE ENDPT TYPE UP SRC >>>>>> DST DN >>>>>> > >>> >>>>>> -------------------------------------------------------------------------------- >>>>>> > >>> >>>>>> > >>> -----------------------------PROBLEMATIC >>>>>> ENDPOINTS------------------------------ >>>>>> > >>> >>>>>> > >>> SCORE ENDPT TYPE UP SRC >>>>>> DST DN >>>>>> > >>> >>>>>> -------------------------------------------------------------------------------- >>>>>> > >>> 19 BRUSSELS dCache F - >>>>>> F F 0/4 >>>>>> > >>> 19 CERN-EOS EOS F - >>>>>> F F 0/4 >>>>>> > >>> 19 CERN-TRUNK DPM - F >>>>>> - F 0/2 >>>>>> > >>> 19 DESY-PROM dCache F - >>>>>> F F 0/4 >>>>>> > >>> 19 FNAL dCache F - >>>>>> F F 0/4 >>>>>> > >>> 19 IN2P3-DOMA xrootd F - >>>>>> F F 0/4 >>>>>> > >>> 19 PRAGUE DPM F - >>>>>> F F 0/4 >>>>>> > >>> 19 RAL-CEPH CEPH F - >>>>>> F F 0/4 >>>>>> > >>> 19 RAL-LCG2 Echo F - >>>>>> F F 0/4 >>>>>> > >>> 19 SLAC XrootD F - >>>>>> F F 0/4 >>>>>> > >>> 19 TRIUMF dCache F - >>>>>> F F 0/4 >>>>>> > >>> 19 UKI-BRUNEL DPM F - >>>>>> F F 0/4 >>>>>> > >>> 19 UKI-LANC DPM F - >>>>>> F F 0/4 >>>>>> > >>> 19 UKI-MAN1 DPM F - >>>>>> F F 0/4 >>>>>> > >>> 19 UKI-MAN2 DPM F - >>>>>> F F 0/4 >>>>>> > >>> 19 UNI-BONN CephFS F - >>>>>> F F 0/4 >>>>>> > >>> 18 OU XrootD F - >>>>>> F F 0/4 >>>>>> > >>> 13 BNL dCache F - >>>>>> F F 0/4 >>>>>> > >>> 7 AGLT2 dCache F - >>>>>> F F 0/4 >>>>>> > >>> 0 CALTECH HDFS F - >>>>>> F F 0/4 >>>>>> > >>> 0 TRIUMF-PROD dCache F - >>>>>> F F 0/4 >>>>> >>> > ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1