Hi Matevz,
OK, then all this probably means is that client a goes after file x and
the proxy is very heavily loaded so it takes a bit of time to actually
open the file at the remote location. While the open is taking place,
client b tries to open the same file. So, client b is delayed until client
a finishes opening the file. Nothing particularly wrong here.
Andy
On Fri, 27 Jun 2014, Matevz Tadel wrote:
> Thanks Andy,
>
> This is actually the standard proxy, RAL was running 4.0.0-rc1 the last time
> we talked about it.
>
> Andrew, have you upgraded to 4.0.0 yet?
>
> Matevz
>
> On 06/27/14 10:01, Andrew Hanushevsky wrote:
>> Hi Matevz,
>>
>> No need to turn on debugging here. This particular stall occurs because a
>> file
>> is being opened and the OFS has found that the file is already open or
>> being
>> opened by another client. So, it tries to piggy-back the new open on that
>> handle
>> to avoid actually doing another physical open. The problem is that the
>> other
>> client has not yet released the handle for use; likely being hung up in the
>> proxy code trying to do the open or perhaps a close. The latter problem I
>> thought was solved by the disk caching proxy by doing the closes in the
>> background to avoid holding on to the handle lock for long periods of time.
>>
>> This is not a fatal problem the client will eventually open the file. The
>> ofs
>> layer uses this as congenstion control when there is a lot of open/close
>> contention for the same file. I suppose you can trace opens and closes to
>> get
>> better feeling of how long this takes:
>>
>> ofs.trace open close
>>
>> Assuming this is a disk caching proxy there may be tracing options for that
>> to
>> see what happens during the open/close sequence.
>>
>> Andy
>>
>> On Fri, 27 Jun 2014, Matevz Tadel wrote:
>>
>>> Hi,
>>>
>>> At RAL, they see the following on their outgoing proxy servers (repeating
>>> for
>>> about a minute before the file-open times-out at the application
>>> side):<<FNORD
>>>
>>> When our xrootd proxy cluster is busy, there are sometimes messages like
>>> this
>>> in the logs:
>>>
>>> 140626 16:53:25 24465 ofs_Stall: Stall 3: File
>>> 2EF5AF84-D65A-E311-AB3F-02163E00A0E1.root is being staged; estimated time
>>> to
>>> completion 3 seconds for
>>> /store/mc/Fall13/QCD_Pt-5to10_Tune4C_13TeV_pythia8/GEN-SIM/POSTLS162_V1_castor-v1/10000/2EF5AF84-D65A-E311-AB3F-02163E00A0E1.root
>>>
>>> 140626 16:53:25 24465 pcms054.6545:147@lcg1353 XrootdProtocol: stalling
>>> client
>>> for 3 sec
>>> 140626 16:53:25 24465 pcms054.6545:147@lcg1353 ofs_close: use=0 fn=dummy
>>>
>>> FNORD
>>>
>>> This probably means that the remote file can not be opened for some reason
>>> (like being delayed by external redirector/server)? Would there be a
>>> special
>>> error if the socket can not be opened (due to fd or firewall limits ... or
>>> some other internal limits)? Note that this only happens when the proxies
>>> are
>>> already under heavy load.
>>>
>>> What options should they set to debug this?
>>>
>>> pss.memcache debug ???
>>> xrd.trace conn
>>> xrootd.trace redirect
>>>
>>> Matevz
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|