Hi Andy,
All versions are for the local redirectors.
UCSD (3.3.3) had "lookup distrib redirect immed" and performs well (linear
scaling up to 300 Hz).
Wisconsin (3.3.3), Nebraska (3.3.1, I think) and Purdue (3.3.2) had "lookup
central redirect immed" and performed poorly (clogged up at 10-20Hz). Wisconsin
and Nebraska already made the change to UCSD settings and are now scaling ok, too.
Matevz
On 2/4/14 4:58 PM, Andrew Hanushevsky wrote:
> Hi Matevz,
>
> OK, then I am actually confused as well. So, which site has which option and
> what release is each of them running.
>
> Andy
>
> -----Original Message----- From: Matevz Tadel
> Sent: Tuesday, February 04, 2014 4:49 PM
> To: Andrew Hanushevsky ; xrootd-dev
> Subject: Re: cms.dfs question
>
> Hi Andy,
>
> b) was actually the case, no meta-managers involved, just open request on local
> manager, for a while we know (believe) is available on the site. And both sites
> had redirect immed. It was lookup distrib vs. central that made the difference
> which I did not expect (I also thought that redirect immed is the only thing
> that matters).
>
> Matevz
>
> On 2/4/14 4:10 PM, Andrew Hanushevsky wrote:
>> Hi Matevz,
>>
>> Ah, I see you are as confused as Wilko (and perhaps me :-). We just spent some
>> time understanding what is going on for you (yes, my explanation made some
>> assumptions).
>>
>> a) I assumed you first talked to a meta-redirector that asked several hadoop
>> sites whether they had the file. You then got redirected to some xrootd and
>> opened the file there. Indeed the "redirect" setting in this case is immaterial
>> as the lookup has already been done.
>>
>> b) If (a) was not true, instead you went directly to a particular Hadoop cluster
>> and opened the file on it's local redirector then the "redirect" makes a big
>> difference. In this case, immed means no lookup is done and you get sent to some
>> server which will honor or fail your request.
>>
>> Andy
>>
>> -----Original Message----- From: Matevz Tadel
>> Sent: Tuesday, February 04, 2014 3:53 PM
>> To: Andrew Hanushevsky ; xrootd-dev
>> Subject: Re: cms.dfs question
>>
>> Hi Andy,
>>
>> Thanks for the explanation! To make sure I understand:
>>
>> 1. When redirector does not know if a file exists, it still has to perform the
>> lookup, as configured.
>>
>> 2. When using "lookup central", we are actually "measuring" the limit of hdfs
>> lookup on a single node (the redirector).
>>
>> Now I understand that redirector has to perform the lookup when it doesn't know
>> if a file exists ... otherwise it can not report to meta-manager(s).
>>
>> Would it make sense to have the equivalent of "lookup none" for open requests.
>> The client can then deal directly with a data server. It's true that the
>> redirector does not "learn" anything useful in this case so it can lead to more
>> trouble down the road, especially with mis-behaving users/clients.
>>
>> Matevz
>>
>> On 2/4/14 3:20 PM, Andrew Hanushevsky wrote:
>>> Hi Matevz,
>>>
>>> You are getting caught in the lookup" phase. Distributed lookup will always
>>> scale better then central lookup, when a lookup *has* to be performed. The
>>> redirect part is what to do when a lookup can be avoided because the information
>>> is already cached. Immed is always the best option is you have a true
>>> distributed file system underneath.
>>>
>>> Anyway, I can't say that I have convinced people that distributed normally has
>>> better scaling, and I have tried. Unfortunately, the majority still seems to
>>> gravitate to centralized vertical design options because they are more
>>> comforting.
>>>
>>> Andy
>>>
>>> -----Original Message----- From: Matevz Tadel
>>> Sent: Tuesday, February 04, 2014 2:46 PM
>>> To: xrootd-dev
>>> Subject: cms.dfs question
>>>
>>> Hi,
>>>
>>> We (AAA) are doing redirection rate scaling tests and noticed a large difference
>>> between *hadoop* sites based on how cms.dfs is setup.
>>>
>>> This works great (scaling beyond 300Hz):
>>> cms.dfs lookup distrib redirect immed
>>> and this saturates at ~20Hz:
>>> cms.dfs lookup central redirect immed
>>>
>>> I'm puzzled, because I'd expect that "redirect immed" trumps whatever lookup
>>> setting one might choose. We were lucky -- we had two sites that chose different
>>> values for lookup :)
>>>
>>> Matevz
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|