Hello to everyone,
I would like to summarize the status of deployment of XROOTD at BNL.
It's pleasure for me that I can say that we integrated XROOTD to the our
current framework and it's ready to be used by our users for purpose of
analysis.
Our plan was to replace rootd by xrootd and take advantage of load
balancing, fault-tolerance, mss interface etc. I wrestled with many
problems during installation and consecutive activation of all features.
Please take notes of the changes we needed to make to have the needed
features all working.
1)First deploy was at February this year and I got stuck due to the
limitation of load-balancing to the 64 computers. After the
enlargement to "unlimited" nodes I was able to continue on the
additional deployed phases. Before this alteration was XROOTD almost
unusable for us. Many thanks to Andy for recognizing this issue as a
show stopper for us and providing quick fixes and extension.
* currently we are running xrootd on 380 nodes and everything
seems to me that is great (no server crashes or memory leaks)
2)The other problems was through enabling the mss interface (our MSS is
HPSS) and possibility to have distributed disc dynamically populated
through the HPSS files:
After implementation and testing of the two scripts which are
necessarily to obtain files from HPSS (oss.stagecmd and oss.mssgwcmd), I
got stuck on thing that if I want to have more than one file system to
data storing, we have to use the MPS scripts (I didn't find out this
information from the documentation)
After the configuration of MPS scripts, I bumped into a problem which
relates to absent plugin of LFN - PFN conversion:
we are using in our experiment rootd, it has already some files
stored on distributed disc with some PFNs and it is impossible to
configure XROOTD
our case example:
We have all files stored in distributed disc on path:
/data*/.... (where * goes from 1 to 3)
Each file in HPSS is stored on a path:
/home/starreco/....
Each requested file from the HPSS to local disk is
prefixed by the configuration directive oss.localroot /home/starlib
(because I am running xrootd as other user which don't have permission
to /home/stareco to create a symlink).
This configuration directive causes that path
/home/starlib is added to the each open request and then I don't have
accessible data which we have already placed in distributed disk /data*.
In other words, there are now ways to migrate smoothly from rootd to
Xrootd or even have a transition period where /when both system can
coexist. The solution we did in a code is something like: 'check PFN and
if does not exists, go to LFN', but it is addicted and not expandable
solution .The longer term for us is to have an adaptable LFN to PFN
translation rather than the current 'fixed' path pseudo conversion.
Andy, please can you prepare for us the new directive and all about that
as we concluded at XROOTD+SRM meeting ? Thank you.
3)Next problems had to do with password authentication as a recovery of
rootd authentication mechanism:
TAuthenticate::SetGlobalUser("default");
TAuthenticate::SetGlobalPasswd("*******");
First thing was about using wild card of host and then checking of
right srvpuk in cache. By default the function in class
XrdSutCache::Get had the disabled searching with wild card if you
specify only one argument. I had discussed this with Gerri and he
committed my proposed changes into CVS (I checked it in a new version
and it is already here). Thank you Gerri.
Second thing had to do with it that in pwd authentication isn't
possible to authenticate a user as other user and thus replace rootd
global mechanism.
I proposed to Gerri change in the client class TXNetConn like this.
Change:
if(!gSystem->Getenv("XrdSecUSER")){
gSystem->Setenv("XrdSecUSER",User.Data());
}
Before only:
gSystem->Setenv("XrdSecUSER",User.Data());
i.e. it allows to authenticate user as other user by defining the
environment variable: XrdSecUSER.
Gerri didn't find any problem on that.
Could someone please check if it is available in future release ?
Thank you
4)Next problem was the script for measuring load of data servers called
XrdOlbMonPerf. I have found some bugs and repaired them.
Changes:
I have repaired bug related to network result and I added missing
paging I/O result. Also I made some small changes as paths to unix
command etc.
See the attachment.
I still have some problem with stopping this running command after the
xrootd server is stopped. For some reason, this command is still running
and it doesn't die when xrootd is killed.
Some ideas ? Thank you
5)Monitoring issue:
we are monitoring the xrootd daemon and olbd daemon from ganglia
view (reporting cpu and memory usage exploiting ganglia metrics). See
the attachment.
At the end I would like to thank to everyone who helped me during the
deployment and also say: GREAT AND BRILLIANT work to all XROOTD's
collaborators and contributors.
Now that the basic functionalities are in place and working, I am
looking forward to the next steps and improvements along the line of
Xrootd+SRM technology merging and development.
Best Pavel
