Hi Matevz,
Not sure if I'm so convinced all is well.
If I use the HDFS client (via FUSE) in a single thread to query the NN as fast as possible, I get a query rate of about 4.7 kHZ. Why is the Xrootd single-client performance 2 orders of magnitude slower than FUSE?
Brian
On Feb 4, 2014, at 8:44 PM, Matevz Tadel <[log in to unmask]> wrote:
> Hi Andy, everybody,
>
> The scaling issues indeed only showed when doing open requests through the meta-manager ... so all is OK. Right? :)
>
> Sorry for all this noise ... I'll pay a round at the Federated Storage workshop :)
>
> Matevz
>
> On 2/4/14 5:43 PM, Matevz Tadel wrote:
>> Hi Andy,
>>
>> Before we go too far ... let me check with the guy who ran the tests if he
>> indeed went through site redirectors directly and not through the meta manager.
>>
>> With meta manager in the game --- the results would make sense, right?
>>
>> Matevz
>>
>> On 2/4/14 5:29 PM, Matevz Tadel wrote:
>>> Hi Andy,
>>>
>>> All versions are for the local redirectors.
>>>
>>> UCSD (3.3.3) had "lookup distrib redirect immed" and performs well (linear
>>> scaling up to 300 Hz).
>>>
>>> Wisconsin (3.3.3), Nebraska (3.3.1, I think) and Purdue (3.3.2) had "lookup
>>> central redirect immed" and performed poorly (clogged up at 10-20Hz). Wisconsin
>>> and Nebraska already made the change to UCSD settings and are now scaling ok,
>>> too.
>>>
>>> Matevz
>>>
>>> On 2/4/14 4:58 PM, Andrew Hanushevsky wrote:
>>>> Hi Matevz,
>>>>
>>>> OK, then I am actually confused as well. So, which site has which option and
>>>> what release is each of them running.
>>>>
>>>> Andy
>>>>
>>>> -----Original Message----- From: Matevz Tadel
>>>> Sent: Tuesday, February 04, 2014 4:49 PM
>>>> To: Andrew Hanushevsky ; xrootd-dev
>>>> Subject: Re: cms.dfs question
>>>>
>>>> Hi Andy,
>>>>
>>>> b) was actually the case, no meta-managers involved, just open request on local
>>>> manager, for a while we know (believe) is available on the site. And both sites
>>>> had redirect immed. It was lookup distrib vs. central that made the difference
>>>> which I did not expect (I also thought that redirect immed is the only thing
>>>> that matters).
>>>>
>>>> Matevz
>>>>
>>>> On 2/4/14 4:10 PM, Andrew Hanushevsky wrote:
>>>>> Hi Matevz,
>>>>>
>>>>> Ah, I see you are as confused as Wilko (and perhaps me :-). We just spent some
>>>>> time understanding what is going on for you (yes, my explanation made some
>>>>> assumptions).
>>>>>
>>>>> a) I assumed you first talked to a meta-redirector that asked several hadoop
>>>>> sites whether they had the file. You then got redirected to some xrootd and
>>>>> opened the file there. Indeed the "redirect" setting in this case is immaterial
>>>>> as the lookup has already been done.
>>>>>
>>>>> b) If (a) was not true, instead you went directly to a particular Hadoop
>>>>> cluster
>>>>> and opened the file on it's local redirector then the "redirect" makes a big
>>>>> difference. In this case, immed means no lookup is done and you get sent to
>>>>> some
>>>>> server which will honor or fail your request.
>>>>>
>>>>> Andy
>>>>>
>>>>> -----Original Message----- From: Matevz Tadel
>>>>> Sent: Tuesday, February 04, 2014 3:53 PM
>>>>> To: Andrew Hanushevsky ; xrootd-dev
>>>>> Subject: Re: cms.dfs question
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> Thanks for the explanation! To make sure I understand:
>>>>>
>>>>> 1. When redirector does not know if a file exists, it still has to perform the
>>>>> lookup, as configured.
>>>>>
>>>>> 2. When using "lookup central", we are actually "measuring" the limit of hdfs
>>>>> lookup on a single node (the redirector).
>>>>>
>>>>> Now I understand that redirector has to perform the lookup when it doesn't know
>>>>> if a file exists ... otherwise it can not report to meta-manager(s).
>>>>>
>>>>> Would it make sense to have the equivalent of "lookup none" for open requests.
>>>>> The client can then deal directly with a data server. It's true that the
>>>>> redirector does not "learn" anything useful in this case so it can lead to more
>>>>> trouble down the road, especially with mis-behaving users/clients.
>>>>>
>>>>> Matevz
>>>>>
>>>>> On 2/4/14 3:20 PM, Andrew Hanushevsky wrote:
>>>>>> Hi Matevz,
>>>>>>
>>>>>> You are getting caught in the lookup" phase. Distributed lookup will always
>>>>>> scale better then central lookup, when a lookup *has* to be performed. The
>>>>>> redirect part is what to do when a lookup can be avoided because the
>>>>>> information
>>>>>> is already cached. Immed is always the best option is you have a true
>>>>>> distributed file system underneath.
>>>>>>
>>>>>> Anyway, I can't say that I have convinced people that distributed normally has
>>>>>> better scaling, and I have tried. Unfortunately, the majority still seems to
>>>>>> gravitate to centralized vertical design options because they are more
>>>>>> comforting.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> -----Original Message----- From: Matevz Tadel
>>>>>> Sent: Tuesday, February 04, 2014 2:46 PM
>>>>>> To: xrootd-dev
>>>>>> Subject: cms.dfs question
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We (AAA) are doing redirection rate scaling tests and noticed a large
>>>>>> difference
>>>>>> between *hadoop* sites based on how cms.dfs is setup.
>>>>>>
>>>>>> This works great (scaling beyond 300Hz):
>>>>>> cms.dfs lookup distrib redirect immed
>>>>>> and this saturates at ~20Hz:
>>>>>> cms.dfs lookup central redirect immed
>>>>>>
>>>>>> I'm puzzled, because I'd expect that "redirect immed" trumps whatever lookup
>>>>>> setting one might choose. We were lucky -- we had two sites that chose
>>>>>> different
>>>>>> values for lookup :)
>>>>>>
>>>>>> Matevz
>>>>>>
>>>>>> ########################################################################
>>>>>> Use REPLY-ALL to reply to list
>>>>>>
>>>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>>
>>>>
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>> ########################################################################
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-DEV list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|