Hi Alja,
I spoke to Douglas about this. He will be using an ActiveMQ transport. So,
in theory, he will not be subject to delays as ActiveMQ is a
store-and-forward type of queue. So, even if Rucio is not responsive, the
message simply waits in a queue and his command does not hang.
Andu
-----Original Message-----
From: Alja Mrak-Tadel
Sent: Wednesday, December 10, 2014 10:30 AM
To: Andrew Hanushevsky ; Doug Benjamin
Cc: xrootd-dev
Subject: Re: [xrootd] New XrdFileCache option to invoke script on new file
attach (#180)
Hi Andy,
On 12/09/14 12:48, Andrew Hanushevsky wrote:
> Hi Doug,
>
> The specific is that you need not call anything in the ofs and neither
> does Alja as all that just happens in the course of doing file
> operations. The link that I gave you explains in detail what the script
> gets and whatto put in he configuration file to make that happen. Now
> that I think more about it, it's likely it won't solve the whole issue
> because the caching proxy is rather stealthy and you need to know if
> something gets placed in the cache regardless of the reason. The ofs
> callouts don't completely capture that. So, I guess we are back to
> having a callout from FileCache. But we will also need a callout when a
> file gets deleted because you will need to deregister the file.
>
> Now, speaking of registering and deregistering. I would strongly
> recommend the registration and deregistration happen from a separate
> process. The callout should simply write the action needed to a file
> queue and another program reads the queue and does what is needed. I say
> this because Rucio is essentially the LFC with a different face and its
> performance characteristics are likely the same. Knowing that the LFC
> was not particularly speedy I doubt Rucio will be any faster when it
> comes to registering and deregistering files, especially when dozens of
> sites start doing it. Plus, it's likely to suffer occasional failures
> and you don't want the file cache to grind to a halt just because of a
> Rucio hicup.
>
In my implementation I used XrdOucStream::Exec to invoke a command. You
say the Rucio is heavy loaded and if I understand right this is not a
correct solution.
Maybe using ofs.notify instead of a proxy is still a good option. The
command can be a simple c++ program which opens a client for a given
path. Douglas also requested a checksum. I don't know how exactly this
would be implemented, but I think it is possible.
Alja
>
> On Tue, 9 Dec 2014, Doug Benjamin wrote:
>
>> Hi Andy,
>>
>> Please be more specific when the ofs layer should be called and when
>> it should
>> not be called. I do not understand the statement "open r/o with file
>> placement"
>> What information doe sthe OFS layer actually send to the script?
>>
>> Thanks,
>>
>> Doug
>>
>> On 12/09/2014 02:39 AM, Andrew Hanushevsky wrote:
>>> Hi Alja,
>>>
>>> Yes, there are other events. However, I think you should just remove all
>>> he callouts from your caching proxy and tell Doug to use the event
>>> notfication in the OFS layer. See,
>>>
>>> http://xrootd.org/doc/dev4/ofs_config.htm#_Toc392685004
>>>
>>> This is what this whole thing was meant for. I don't think you need
>>> to do
>>> anything special in the caching layer except, perhaps, a call out for a
>>> stage-in (i.e. open r/o with file placement which the ofs callout issn't
>>> quite suitable).
>>>
>>> Andy
>>>
>>> On Mon, 8 Dec 2014, Alja Mrak-Tadel wrote:
>>>
>>> > @abh3
>>> > I guess I misunderstood Doug's request. I thought he wants to have
>>> a given command invoked only on file open event.
>>> >
>>> > Is there other events too? Like Mkdir, Remdir, and Rename.
>>> >
>>> >
>>> > ---
>>> > Reply to this email directly or view it on GitHub:
>>> > https://github.com/xrootd/xrootd/pull/180#issuecomment-66158178
>>>
>>>
>>> Reply to this email directly or view it on GitHub
>>> <https://github.com/xrootd/xrootd/pull/180#issuecomment-66250104>.
>>>
>>>
>>>
>>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
!
>>>
> -
>>> ---
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-DEV list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>>
>>
>>
>>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|