 |
 |
Hi Matevz, Yes, all is OK except for the poor showing of single node HDFS performance. Is it that most file system calls are single threaded? I suspect that is the case otherwise I should not see an order of magnitude difference in performance. Something to ask the HDFS guys. Andy -----Original Message----- From: Matevz Tadel Sent: Tuesday, February 04, 2014 6:44 PM To: Andrew Hanushevsky ; xrootd-dev Subject: Re: cms.dfs question Hi Andy, everybody, The scaling issues indeed only showed when doing open requests through the meta-manager ... so all is OK. Right? :) Sorry for all this noise ... I'll pay a round at the Federated Storage workshop :) Matevz On 2/4/14 5:43 PM, Matevz Tadel wrote: > Hi Andy, > > Before we go too far ... let me check with the guy who ran the tests if he > indeed went through site redirectors directly and not through the meta > manager. > > With meta manager in the game --- the results would make sense, right? > > Matevz > > On 2/4/14 5:29 PM, Matevz Tadel wrote: >> Hi Andy, >> >> All versions are for the local redirectors. >> >> UCSD (3.3.3) had "lookup distrib redirect immed" and performs well >> (linear >> scaling up to 300 Hz). >> >> Wisconsin (3.3.3), Nebraska (3.3.1, I think) and Purdue (3.3.2) had >> "lookup >> central redirect immed" and performed poorly (clogged up at 10-20Hz). >> Wisconsin >> and Nebraska already made the change to UCSD settings and are now scaling >> ok, >> too. >> >> Matevz >> >> On 2/4/14 4:58 PM, Andrew Hanushevsky wrote: >>> Hi Matevz, >>> >>> OK, then I am actually confused as well. So, which site has which option >>> and >>> what release is each of them running. >>> >>> Andy >>> >>> -----Original Message----- From: Matevz Tadel >>> Sent: Tuesday, February 04, 2014 4:49 PM >>> To: Andrew Hanushevsky ; xrootd-dev >>> Subject: Re: cms.dfs question >>> >>> Hi Andy, >>> >>> b) was actually the case, no meta-managers involved, just open request >>> on local >>> manager, for a while we know (believe) is available on the site. And >>> both sites >>> had redirect immed. It was lookup distrib vs. central that made the >>> difference >>> which I did not expect (I also thought that redirect immed is the only >>> thing >>> that matters). >>> >>> Matevz >>> >>> On 2/4/14 4:10 PM, Andrew Hanushevsky wrote: >>>> Hi Matevz, >>>> >>>> Ah, I see you are as confused as Wilko (and perhaps me :-). We just >>>> spent some >>>> time understanding what is going on for you (yes, my explanation made >>>> some >>>> assumptions). >>>> >>>> a) I assumed you first talked to a meta-redirector that asked several >>>> hadoop >>>> sites whether they had the file. You then got redirected to some xrootd >>>> and >>>> opened the file there. Indeed the "redirect" setting in this case is >>>> immaterial >>>> as the lookup has already been done. >>>> >>>> b) If (a) was not true, instead you went directly to a particular >>>> Hadoop >>>> cluster >>>> and opened the file on it's local redirector then the "redirect" makes >>>> a big >>>> difference. In this case, immed means no lookup is done and you get >>>> sent to >>>> some >>>> server which will honor or fail your request. >>>> >>>> Andy >>>> >>>> -----Original Message----- From: Matevz Tadel >>>> Sent: Tuesday, February 04, 2014 3:53 PM >>>> To: Andrew Hanushevsky ; xrootd-dev >>>> Subject: Re: cms.dfs question >>>> >>>> Hi Andy, >>>> >>>> Thanks for the explanation! To make sure I understand: >>>> >>>> 1. When redirector does not know if a file exists, it still has to >>>> perform the >>>> lookup, as configured. >>>> >>>> 2. When using "lookup central", we are actually "measuring" the limit >>>> of hdfs >>>> lookup on a single node (the redirector). >>>> >>>> Now I understand that redirector has to perform the lookup when it >>>> doesn't know >>>> if a file exists ... otherwise it can not report to meta-manager(s). >>>> >>>> Would it make sense to have the equivalent of "lookup none" for open >>>> requests. >>>> The client can then deal directly with a data server. It's true that >>>> the >>>> redirector does not "learn" anything useful in this case so it can lead >>>> to more >>>> trouble down the road, especially with mis-behaving users/clients. >>>> >>>> Matevz >>>> >>>> On 2/4/14 3:20 PM, Andrew Hanushevsky wrote: >>>>> Hi Matevz, >>>>> >>>>> You are getting caught in the lookup" phase. Distributed lookup will >>>>> always >>>>> scale better then central lookup, when a lookup *has* to be performed. >>>>> The >>>>> redirect part is what to do when a lookup can be avoided because the >>>>> information >>>>> is already cached. Immed is always the best option is you have a true >>>>> distributed file system underneath. >>>>> >>>>> Anyway, I can't say that I have convinced people that distributed >>>>> normally has >>>>> better scaling, and I have tried. Unfortunately, the majority still >>>>> seems to >>>>> gravitate to centralized vertical design options because they are more >>>>> comforting. >>>>> >>>>> Andy >>>>> >>>>> -----Original Message----- From: Matevz Tadel >>>>> Sent: Tuesday, February 04, 2014 2:46 PM >>>>> To: xrootd-dev >>>>> Subject: cms.dfs question >>>>> >>>>> Hi, >>>>> >>>>> We (AAA) are doing redirection rate scaling tests and noticed a large >>>>> difference >>>>> between *hadoop* sites based on how cms.dfs is setup. >>>>> >>>>> This works great (scaling beyond 300Hz): >>>>> cms.dfs lookup distrib redirect immed >>>>> and this saturates at ~20Hz: >>>>> cms.dfs lookup central redirect immed >>>>> >>>>> I'm puzzled, because I'd expect that "redirect immed" trumps whatever >>>>> lookup >>>>> setting one might choose. We were lucky -- we had two sites that chose >>>>> different >>>>> values for lookup :) >>>>> >>>>> Matevz >>>>> >>>>> ######################################################################## >>>>> Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the XROOTD-DEV list, click the following link: >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >>>> >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-DEV list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-DEV list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-DEV list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-DEV list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1