Hello to everyone,
I would like to summarize the status of deployment of XROOTD at BNL.
It's pleasure for me that I can say that we integrated XROOTD to the our
current framework and it's ready to be used by our users for purpose of
analysis.
Our plan was to replace rootd by xrootd and take advantage of load
balancing, fault-tolerance, mss interface etc. I wrestled with many
problems during installation and consecutive activation of all features.
Please take notes of the changes we needed to make to have the needed
features all working.
1)First deploy was at February this year and I got stuck due to the
limitation of load-balancing to the 64 computers. After the
enlargement to "unlimited" nodes I was able to continue on the
additional deployed phases. Before this alteration was XROOTD almost
unusable for us. Many thanks to Andy for recognizing this issue as a
show stopper for us and providing quick fixes and extension.
* currently we are running xrootd on 380 nodes and everything
seems to me that is great (no server crashes or memory leaks)
2)The other problems was through enabling the mss interface (our MSS is
HPSS) and possibility to have distributed disc dynamically populated
through the HPSS files:
After implementation and testing of the two scripts which are
necessarily to obtain files from HPSS (oss.stagecmd and oss.mssgwcmd), I
got stuck on thing that if I want to have more than one file system to
data storing, we have to use the MPS scripts (I didn't find out this
information from the documentation)
After the configuration of MPS scripts, I bumped into a problem which
relates to absent plugin of LFN - PFN conversion:
we are using in our experiment rootd, it has already some files
stored on distributed disc with some PFNs and it is impossible to
configure XROOTD
our case example:
We have all files stored in distributed disc on path:
/data*/.... (where * goes from 1 to 3)
Each file in HPSS is stored on a path:
/home/starreco/....
Each requested file from the HPSS to local disk is
prefixed by the configuration directive oss.localroot /home/starlib
(because I am running xrootd as other user which don't have permission
to /home/stareco to create a symlink).
This configuration directive causes that path
/home/starlib is added to the each open request and then I don't have
accessible data which we have already placed in distributed disk /data*.
In other words, there are now ways to migrate smoothly from rootd to
Xrootd or even have a transition period where /when both system can
coexist. The solution we did in a code is something like: 'check PFN and
if does not exists, go to LFN', but it is addicted and not expandable
solution .The longer term for us is to have an adaptable LFN to PFN
translation rather than the current 'fixed' path pseudo conversion.
Andy, please can you prepare for us the new directive and all about that
as we concluded at XROOTD+SRM meeting ? Thank you.
3)Next problems had to do with password authentication as a recovery of
rootd authentication mechanism:
TAuthenticate::SetGlobalUser("default");
TAuthenticate::SetGlobalPasswd("*******");
First thing was about using wild card of host and then checking of
right srvpuk in cache. By default the function in class
XrdSutCache::Get had the disabled searching with wild card if you
specify only one argument. I had discussed this with Gerri and he
committed my proposed changes into CVS (I checked it in a new version
and it is already here). Thank you Gerri.
Second thing had to do with it that in pwd authentication isn't
possible to authenticate a user as other user and thus replace rootd
global mechanism.
I proposed to Gerri change in the client class TXNetConn like this.
Change:
if(!gSystem->Getenv("XrdSecUSER")){
gSystem->Setenv("XrdSecUSER",User.Data());
}
Before only:
gSystem->Setenv("XrdSecUSER",User.Data());
i.e. it allows to authenticate user as other user by defining the
environment variable: XrdSecUSER.
Gerri didn't find any problem on that.
Could someone please check if it is available in future release ?
Thank you
4)Next problem was the script for measuring load of data servers called
XrdOlbMonPerf. I have found some bugs and repaired them.
Changes:
I have repaired bug related to network result and I added missing
paging I/O result. Also I made some small changes as paths to unix
command etc.
See the attachment.
I still have some problem with stopping this running command after the
xrootd server is stopped. For some reason, this command is still running
and it doesn't die when xrootd is killed.
Some ideas ? Thank you
5)Monitoring issue:
we are monitoring the xrootd daemon and olbd daemon from ganglia
view (reporting cpu and memory usage exploiting ganglia metrics). See
the attachment.
At the end I would like to thank to everyone who helped me during the
deployment and also say: GREAT AND BRILLIANT work to all XROOTD's
collaborators and contributors.
Now that the basic functionalities are in place and working, I am
looking forward to the next steps and improvements along the line of
Xrootd+SRM technology merging and development.
Best Pavel
#!/usr/local/bin/perl
## $Id: XrdOlbMonPerf,v 1.8 2004/09/10 12:01:20 elmer Exp $
## this script collects the informations on the server activity like:
## CPU, network I/O, memory usage ...
## modified to get rid of rperf.
## J-Y Nief - CCIN2P3 - 24/05/04
## modified to accept additional options and drop multi-home uf's.
# Andy Hanushevsky - SLAC - 15/07/04
## modified to get right network I/O result, paging I/O and changes in paths to unix commands etc.
## Pavel Jakl - RCF/BNL - 20/08/05
#
# Usage: XrdOlbMonPerf [-I netintf] [-n netspeed] [-m <shmkey>] [<interval>]
# WARNING: if you are running this script on a Linux RH7.x server,
# the network metric is disabled as the netstat output is screwed,
# this info is useless. no problem for RH9 and RHEL3.
# get the name of the server + platform
undef($shmid);
$shmkey = '';
$rhost = `/bin/uname -n`; chomp($rhost);
$enBps = 131072000;
$systype= $^O; #`/bin/uname`; chomp($systype);
$repval = 0;
$netif = '';
$netmax = 0; # bits / s
$monint = 60;
$| = 1; # unbuffer STDOUT
$DEBUG = 0; # switch to debug script
# Get options
#
while(($argval = shift) && (substr($argval,0,1) eq '-'))
{ if ($argval eq '-I')
{Log("Network interface not specified.") if !($netif=shift);}
elsif ($argval eq '-n')
{Log("Network speed not specified.") if !($netmax=shift);}
elsif ($argval eq '-m')
{Log("Shared segment not specified.") if !($shmkey=shift);
$shmid = shmget($shmkey, 4096, 0744) if (shmkey ne '.');
Log("Unable to attach shared memory; $!") if !defined($shmid);
}
else {Log("Unknown option, $argval, ignored", 1);}
}
# Get the interval
#
if ($argval)
{$monint = $argval;
Log("Invalid interval, '$monint'") if !($monint =~ /^\d+$/);
}
# get the number of CPUs on this server
if ( $systype eq 'linux' ) {
$numCPU = `/bin/grep -c cpu /proc/stat` - 1;
}
if ( $systype eq 'solaris' ) {
$resp = `/bin/uname -X | /bin/grep NumCPU`;
chomp($resp);
@answer = split('= ', $resp);
$numCPU = $answer[1];
}
# get the total memory of the server
if ($systype eq 'linux') {
$resp = `/bin/grep MemTotal /proc/meminfo`;
@answer = split('\s+', $resp);
$TotMem = $answer[1]*1024;
}
if ($systype eq 'solaris' && -x "/etc/prtconf") {
$resp = `/etc/prtconf | /bin/grep Memory`;
($TotMem) = $resp =~ /.+:\s*(\d+)/s;
$TotMem *= 1024*1024;
}
# if it is a Linux, retrieve the distrib version.
# necessary for netstat which does not give the same output on RH 7.x, RH 9,
# RHEL...
# netstat usable on RH 9 and RHEL 3 ==> distrib = "RH_OK", else "RH_WRONG"
if ($systype eq 'linux') {
$distrib = "RH_OK";
$resp = `/bin/cat /etc/redhat-release`;
if ( $resp =~ /release 7./ ) { $distrib = "RH_WRONG"; }
}
# determine the column where the idle time and the free mem
# can be found in the 'vmstat' output.
# depends on the platform and on the 'vmstat' version, sigh...
$ind_idle = 0;
$ind_free = 0;
@output = `/usr/bin/vmstat`;
chomp($output[1]);
@answer = split('\s+', $output[1]);
foreach $type(@answer) {
last if ( $type eq 'id' );
$ind_idle++;
}
foreach $type(@answer) {
last if ( $type eq 'free' );
$ind_free++;
}
# in the netstat output, the number of input and output packets is not in the
# same columns whether it is a Linux or SunOS platform
if ( $systype eq 'solaris' ) {
$ind_net1 = 5;
$ind_net2 = 7;
} else {
$ind_net1 = 3;
$ind_net2 = 7;
}
$indx_net = 0;
# check the number of network interface on the server and their
# capability (ethernet or gigabit)
$mtu = 1500;
if (!$netmax)
{@com = `/bin/netstat -i`;
foreach $line(@com) {
if ( $line !~ /Kernel Interface/ && $line !~ /MTU/ ) {
( $line =~ /eth/ ) && ( $netmax = 1e8 );
( $line =~ /ge/ || $line =~ /ce/ ) && ( $netmax = 1e9 );
}
}
if (!$netmax) {$netmax = 1e8;}
}
# Get it in a sleep $monint time loop netstat result and count network I/O
$ipio_old = 0;
while(true) {
@resp=`/bin/netstat -i`;
# what's the CPU utilization ?
@respCPU = `/usr/bin/vmstat 1 2`;
chomp $respCPU[3];
@resp_vmstat = split('\s+',$respCPU[3]);
if ( ($resp_vmstat[$ind_idle] > 100) || ($resp_vmstat[$ind_idle] < 0) ) {
$cpu = 0;
} else {
$cpu = $resp_vmstat[$ind_idle-2] + $resp_vmstat[$ind_idle-1];
}
# what's the runq load ?
chomp($respLoad = `/usr/bin/uptime`);
@respSplit = split(',',$respLoad);
$load = int($respSplit[4]*25/$numCPU); # Assumes some magic upper number i.e. 1 CPU load 4 = 100%
$load = 100 if $load > 100;
# what's the network I/O activity?
foreach $resp (@resp){
if ( ( $resp !~ /lo/ && $resp !~ /Iface/ && $systype eq 'linux' && $distrib ne 'RH_WRONG' )||
( $systype eq 'solaris' ) ) {
@respSplit = split('\s+',$resp);
print "DEBUG ".join("|",@respSplit)."\n" if ($DEBUG);
if ( $systype eq 'solaris' ) {
$net_activity = ($respSplit[$ind_net1] + $respSplit[$ind_net2]) * $mtu * 8 / $monint;
$ipio = int ( $net_activity / $netmax * 100 );
}else {
$measure[1] = $net[$indx_net] = $respSplit[$ind_net1] + $respSplit[$ind_net2];
print "DEBUG measure is $measure[1] $netmax\n" if ($DEBUG);
if ( $indx_net ) {
$net_activity = ($measure[1] - $measure[0]) * $mtu * 8 / $monint;
$ipio = int ( $net_activity / $netmax * 100 );
$measure[0] = $measure[1];
}
$indx_net = $indx_net ^ 1;
}
$ipio_old = $ipio;
} else {
$ipio = $ipio_old;
}
}
$ipio = 100 if $ipio > 100;
$ipio_old = $ipio;
# what's the memory load ?
$used = $TotMem - $resp_vmstat[$ind_free]*1024;
$mem = int($used/$TotMem*100);
$mem = 100 if $mem > 100;
#
$repval++;
# what's the paging I/O activity ?
# useless as this metric is highly correlated with some of the others above.
# being kept for backward compatibility with the load balancer.
if ( -e "/proc/meminfo"){
@items = split(" ",`/bin/cat /proc/meminfo | /bin/grep Swap:`);
$pgio = int($items[2]/$items[1]*100);
} else {
$pgio = 0;
}
if (!defined($shmid)) {
if ($DEBUG){ print "Load:$load CPU:$cpu MEM:$mem PGIO:$pgio IPIO:$ipio\n";}
else { print "$load $cpu $mem $pgio $ipio\n";}
}
else {
$resp = pack('LLLLL',$load,$cpu,$mem,$pgio,$ipio);
Log("Unable to write into shared memory; $!")
if !shmwrite($shmid, $resp, 0, length($resp));
}
#Sleep some time
sleep $monint;
}
sub Log {
my($msg, $ret) = @_;
print STDERR "XrdOlbMonPerf: $msg\n";
system('/bin/logger', '-p', 'daemon.notice', "XrdOlbMonPerf: $msg");
return 0 if $ret;
exit(1);
}
#!/bin/sh
#
# Simple gmetric script to display the memory and CPU used by xrootd
# authors: Ofer Rind and Jerome Lauret, RCF/BNL
#
MCHAN=`/bin/awk '/mcast_chan/{print $2}' /etc/gmond.conf`
GMETRIC="/usr/bin/gmetric --mcast_ttl=3 -c$MCHAN -d3000"
for proc in xrootd olbd; do
stats=( `/bin/ps --no-headers -o rss,vsz,%cpu -C $proc` )
n=3
while [ ${#stats[*]} -gt $n ] ; do
let m=n%3
stats[$m]=`/bin/echo ${stats[$m]} ${stats[$n]}|/bin/awk '{print $1+$2}'`
let n+=1
done
if [ ${stats} ] || eval "stats=( 0 0 0 )"; then
$GMETRIC -nmem_${proc} -tuint32 -uKB --value ${stats[0]}
$GMETRIC -nvsz_${proc} -tuint32 -uKB --value ${stats[1]}
$GMETRIC -ncpu_${proc} -tfloat -u% --value ${stats[2]}
fi
done
|