 |
 |
Hi Matevz, Yes, with a meta-manager in the way the results would be expected. Andy -----Original Message----- From: Matevz Tadel Sent: Tuesday, February 04, 2014 5:43 PM To: Andrew Hanushevsky ; xrootd-dev Subject: Re: cms.dfs question Hi Andy, Before we go too far ... let me check with the guy who ran the tests if he indeed went through site redirectors directly and not through the meta manager. With meta manager in the game --- the results would make sense, right? Matevz On 2/4/14 5:29 PM, Matevz Tadel wrote: > Hi Andy, > > All versions are for the local redirectors. > > UCSD (3.3.3) had "lookup distrib redirect immed" and performs well (linear > scaling up to 300 Hz). > > Wisconsin (3.3.3), Nebraska (3.3.1, I think) and Purdue (3.3.2) had > "lookup > central redirect immed" and performed poorly (clogged up at 10-20Hz). > Wisconsin > and Nebraska already made the change to UCSD settings and are now scaling > ok, too. > > Matevz > > On 2/4/14 4:58 PM, Andrew Hanushevsky wrote: >> Hi Matevz, >> >> OK, then I am actually confused as well. So, which site has which option >> and >> what release is each of them running. >> >> Andy >> >> -----Original Message----- From: Matevz Tadel >> Sent: Tuesday, February 04, 2014 4:49 PM >> To: Andrew Hanushevsky ; xrootd-dev >> Subject: Re: cms.dfs question >> >> Hi Andy, >> >> b) was actually the case, no meta-managers involved, just open request on >> local >> manager, for a while we know (believe) is available on the site. And both >> sites >> had redirect immed. It was lookup distrib vs. central that made the >> difference >> which I did not expect (I also thought that redirect immed is the only >> thing >> that matters). >> >> Matevz >> >> On 2/4/14 4:10 PM, Andrew Hanushevsky wrote: >>> Hi Matevz, >>> >>> Ah, I see you are as confused as Wilko (and perhaps me :-). We just >>> spent some >>> time understanding what is going on for you (yes, my explanation made >>> some >>> assumptions). >>> >>> a) I assumed you first talked to a meta-redirector that asked several >>> hadoop >>> sites whether they had the file. You then got redirected to some xrootd >>> and >>> opened the file there. Indeed the "redirect" setting in this case is >>> immaterial >>> as the lookup has already been done. >>> >>> b) If (a) was not true, instead you went directly to a particular Hadoop >>> cluster >>> and opened the file on it's local redirector then the "redirect" makes a >>> big >>> difference. In this case, immed means no lookup is done and you get sent >>> to some >>> server which will honor or fail your request. >>> >>> Andy >>> >>> -----Original Message----- From: Matevz Tadel >>> Sent: Tuesday, February 04, 2014 3:53 PM >>> To: Andrew Hanushevsky ; xrootd-dev >>> Subject: Re: cms.dfs question >>> >>> Hi Andy, >>> >>> Thanks for the explanation! To make sure I understand: >>> >>> 1. When redirector does not know if a file exists, it still has to >>> perform the >>> lookup, as configured. >>> >>> 2. When using "lookup central", we are actually "measuring" the limit of >>> hdfs >>> lookup on a single node (the redirector). >>> >>> Now I understand that redirector has to perform the lookup when it >>> doesn't know >>> if a file exists ... otherwise it can not report to meta-manager(s). >>> >>> Would it make sense to have the equivalent of "lookup none" for open >>> requests. >>> The client can then deal directly with a data server. It's true that the >>> redirector does not "learn" anything useful in this case so it can lead >>> to more >>> trouble down the road, especially with mis-behaving users/clients. >>> >>> Matevz >>> >>> On 2/4/14 3:20 PM, Andrew Hanushevsky wrote: >>>> Hi Matevz, >>>> >>>> You are getting caught in the lookup" phase. Distributed lookup will >>>> always >>>> scale better then central lookup, when a lookup *has* to be performed. >>>> The >>>> redirect part is what to do when a lookup can be avoided because the >>>> information >>>> is already cached. Immed is always the best option is you have a true >>>> distributed file system underneath. >>>> >>>> Anyway, I can't say that I have convinced people that distributed >>>> normally has >>>> better scaling, and I have tried. Unfortunately, the majority still >>>> seems to >>>> gravitate to centralized vertical design options because they are more >>>> comforting. >>>> >>>> Andy >>>> >>>> -----Original Message----- From: Matevz Tadel >>>> Sent: Tuesday, February 04, 2014 2:46 PM >>>> To: xrootd-dev >>>> Subject: cms.dfs question >>>> >>>> Hi, >>>> >>>> We (AAA) are doing redirection rate scaling tests and noticed a large >>>> difference >>>> between *hadoop* sites based on how cms.dfs is setup. >>>> >>>> This works great (scaling beyond 300Hz): >>>> cms.dfs lookup distrib redirect immed >>>> and this saturates at ~20Hz: >>>> cms.dfs lookup central redirect immed >>>> >>>> I'm puzzled, because I'd expect that "redirect immed" trumps whatever >>>> lookup >>>> setting one might choose. We were lucky -- we had two sites that chose >>>> different >>>> values for lookup :) >>>> >>>> Matevz >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-DEV list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >>> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-DEV list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-DEV list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-DEV list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-DEV list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1