Hi Andy, everybody,

The scaling issues indeed only showed when doing open requests through the 
meta-manager ... so all is OK. Right? :)

Sorry for all this noise ... I'll pay a round at the Federated Storage workshop :)

Matevz

On 2/4/14 5:43 PM, Matevz Tadel wrote:
> Hi Andy,
>
> Before we go too far ... let me check with the guy who ran the tests if he
> indeed went through site redirectors directly and not through the meta manager.
>
> With meta manager in the game --- the results would make sense, right?
>
> Matevz
>
> On 2/4/14 5:29 PM, Matevz Tadel wrote:
>> Hi Andy,
>>
>> All versions are for the local redirectors.
>>
>> UCSD (3.3.3) had "lookup distrib redirect immed" and performs well (linear
>> scaling up to 300 Hz).
>>
>> Wisconsin (3.3.3), Nebraska (3.3.1, I think) and Purdue (3.3.2) had "lookup
>> central redirect immed" and performed poorly (clogged up at 10-20Hz). Wisconsin
>> and Nebraska already made the change to UCSD settings and are now scaling ok,
>> too.
>>
>> Matevz
>>
>> On 2/4/14 4:58 PM, Andrew Hanushevsky wrote:
>>> Hi Matevz,
>>>
>>> OK, then I am actually confused as well. So, which site has which option and
>>> what release is each of them running.
>>>
>>> Andy
>>>
>>> -----Original Message----- From: Matevz Tadel
>>> Sent: Tuesday, February 04, 2014 4:49 PM
>>> To: Andrew Hanushevsky ; xrootd-dev
>>> Subject: Re: cms.dfs question
>>>
>>> Hi Andy,
>>>
>>> b) was actually the case, no meta-managers involved, just open request on local
>>> manager, for a while we know (believe) is available on the site. And both sites
>>> had redirect immed. It was lookup distrib vs. central that made the difference
>>> which I did not expect (I also thought that redirect immed is the only thing
>>> that matters).
>>>
>>> Matevz
>>>
>>> On 2/4/14 4:10 PM, Andrew Hanushevsky wrote:
>>>> Hi Matevz,
>>>>
>>>> Ah, I see you are as confused as Wilko (and perhaps me :-). We just spent some
>>>> time understanding what is going on for you (yes, my explanation made some
>>>> assumptions).
>>>>
>>>> a) I assumed you first talked to a meta-redirector that asked several hadoop
>>>> sites whether they had the file. You then got redirected to some xrootd and
>>>> opened the file there. Indeed the "redirect" setting in this case is immaterial
>>>> as the lookup has already been done.
>>>>
>>>> b) If (a) was not true, instead you went directly to a particular Hadoop
>>>> cluster
>>>> and opened the file on it's local redirector then the "redirect" makes a big
>>>> difference. In this case, immed means no lookup is done and you get sent to
>>>> some
>>>> server which will honor or fail your request.
>>>>
>>>> Andy
>>>>
>>>> -----Original Message----- From: Matevz Tadel
>>>> Sent: Tuesday, February 04, 2014 3:53 PM
>>>> To: Andrew Hanushevsky ; xrootd-dev
>>>> Subject: Re: cms.dfs question
>>>>
>>>> Hi Andy,
>>>>
>>>> Thanks for the explanation! To make sure I understand:
>>>>
>>>> 1. When redirector does not know if a file exists, it still has to perform the
>>>> lookup, as configured.
>>>>
>>>> 2. When using "lookup central", we are actually "measuring" the limit of hdfs
>>>> lookup on a single node (the redirector).
>>>>
>>>> Now I understand that redirector has to perform the lookup when it doesn't know
>>>> if a file exists ... otherwise it can not report to meta-manager(s).
>>>>
>>>> Would it make sense to have the equivalent of "lookup none" for open requests.
>>>> The client can then deal directly with a data server. It's true that the
>>>> redirector does not "learn" anything useful in this case so it can lead to more
>>>> trouble down the road, especially with mis-behaving users/clients.
>>>>
>>>> Matevz
>>>>
>>>> On 2/4/14 3:20 PM, Andrew Hanushevsky wrote:
>>>>> Hi Matevz,
>>>>>
>>>>> You are getting caught in the lookup" phase. Distributed lookup will always
>>>>> scale better then central lookup, when a lookup *has* to be performed. The
>>>>> redirect part is what to do when a lookup can be avoided because the
>>>>> information
>>>>> is already cached. Immed is always the best option is you have a true
>>>>> distributed file system underneath.
>>>>>
>>>>> Anyway, I can't say that I have convinced people that distributed normally has
>>>>> better scaling, and I have tried.  Unfortunately, the majority still seems to
>>>>> gravitate to centralized vertical design options because they are more
>>>>> comforting.
>>>>>
>>>>> Andy
>>>>>
>>>>> -----Original Message----- From: Matevz Tadel
>>>>> Sent: Tuesday, February 04, 2014 2:46 PM
>>>>> To: xrootd-dev
>>>>> Subject: cms.dfs question
>>>>>
>>>>> Hi,
>>>>>
>>>>> We (AAA) are doing redirection rate scaling tests and noticed a large
>>>>> difference
>>>>> between *hadoop* sites based on how cms.dfs is setup.
>>>>>
>>>>> This works great (scaling beyond 300Hz):
>>>>>    cms.dfs lookup distrib redirect immed
>>>>> and this saturates at ~20Hz:
>>>>>    cms.dfs lookup central redirect immed
>>>>>
>>>>> I'm puzzled, because I'd expect that "redirect immed" trumps whatever lookup
>>>>> setting one might choose. We were lucky -- we had two sites that chose
>>>>> different
>>>>> values for lookup :)
>>>>>
>>>>> Matevz
>>>>>
>>>>> ########################################################################
>>>>> Use REPLY-ALL to reply to list
>>>>>
>>>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>>
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-DEV list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1