Hey Andy,
Talking with folks, we decided that we'd rather implement things at
the OFS layer. The main driver is that we want this to be a simple
service that translates from the internal protocols to something that
can be exported. Using the OFS layer means (a) we don't have to have
a large disk cache to act as an "in-between" (making setup and
configuration simpler among other things) and (b) no latency.
With regards to your various concerns:
a) Authorization: handled by the underlying file system
b) cluster management: Don't particularly want this at this point;
we're aiming at a far lower scale
c) Extended NS functions: We don't want to expose these; we want a
read-only file system for authenticated users
d) Persistence: Again, read only.
e) MSS coordination: Definitely don't want to support this.
Luckily, after closely examing the Sfs example, the HDFS
implementation is dead-simple and I have a working version with all my
needs met.
Thanks!
Brian
On Jun 5, 2009, at 7:19 PM, Andrew Hanushevsky wrote:
> Hi Brian,
>
> Adding to what Fabrizio said...
>
> If for some reason you still think that a plug-in is more
> appropriate for HDFS then you should really consider writing an OSS
> plug-in as opposed to an OFS plug-in. If you write a plug-in at the
> OFS layer then you are responsible for implementing all of the
> logical functions performed by that layer. These include, among
> others, authorization, cluster management, extended name space
> functions, safe persistence, and MSS co-ordination. The OSS layer is
> merely responsible for conveying data to/from the underlying storage
> system as well as basic name space operations to support itself. To
> me it sounds like HDFS integration is better suited at the OSS
> layer. Plus there's much less to do at that layer.
>
> Andy
>
> On Fri, 5 Jun 2009, Fabrizio Furano wrote:
>
>> Hi Brian,
>>
>> if I understood correctly it does not seem difficult. Just mount
>> your fs partitions into a server, install xrootd there and make it
>> export those partitions by hiding their prefix through the
>> localroot setting.
>>
>> Imo the easiest way is to use the setup used by alice, pretty
>> generic and bundled. A pointer to the instructions is this one:
>>
>> http://savannah.cern.ch/project/xrootd
>> or
>> http://alien.cern.ch/twiki/bin/view/AliEn/HowToInstallXrootdNew
>>
>> Of course the option to start it manually and accessing the full
>> configuration is always ok. The docs are here:
>>
>> http://xrootd.slac.stanford.edu
>>
>> Fabrizio
>>
>> Brian Bockelman wrote:
>>> Hey all,
>>> I've been asked by a few folks about adding an Xrootd interface to
>>> the file system we use locally, HDFS. HDFS's security mechanism
>>> make it so the only true way to be secure is to only allow access
>>> from within the local cluster; I'd like to be able to securely
>>> export the file system to ROOT-based applications running on the
>>> local campus (but not transfer it across the world). HDFS has a
>>> FUSE interface which implements most of the POSIX API, but
>>> apparently not enough to have xrootd work directly on top of it :)
>>> However, HDFS has a pleasant, simple C interface:
>>> http://svn.apache.org/repos/asf/hadoop/core/trunk/src/c++/libhdfs/hdfs.h
>>> I'm trying to provide feedback to those who want it about how hard
>>> this project would be. Could someone help me determine:
>>> 1) What exactly would need to be implemented? (I'm a bit new to
>>> xrootd; I believe I'm looking at implementing a new subclass of
>>> OFS?)
>>> 2) What would be needed to do a minimal working prototype that I
>>> could show to someone.
>>> 3) Is there a sample "simplest implementation" that I could base a
>>> prototype off of?
>>> 4) What documentation exists to help me along the way?
>>> Thanks!
>>> Brian
>>
|