Hello to everyone,

I would like to summarize the status of deployment of XROOTD at BNL. 
It's pleasure for me that I can say that we integrated XROOTD to the our 
current framework and it's ready to be used by our users for purpose of 
analysis.

Our plan was to replace rootd by xrootd and take advantage of load 
balancing, fault-tolerance, mss interface etc. I wrestled with many 
problems during installation and consecutive activation of all features. 
Please take notes of the changes we needed to make to have the needed 
features all working.

1)First deploy was at February this year and I got stuck due to the 
limitation of  load-balancing to the 64 computers. After  the 
enlargement to "unlimited" nodes I was able to continue on the 
additional deployed phases. Before this alteration was XROOTD almost 
unusable for us. Many thanks to Andy for recognizing this issue as a 
show stopper for us and providing quick fixes and extension.

         * currently we are running xrootd on 380 nodes and everything
           seems to me that is great (no server crashes or memory leaks)

2)The other problems was through enabling the mss interface (our MSS is 
HPSS) and possibility to have distributed disc dynamically populated 
through the HPSS files:

After implementation and testing of the two scripts which are 
necessarily to obtain files from HPSS (oss.stagecmd and oss.mssgwcmd), I 
got stuck on thing that if I want to have more than one file system to 
data storing, we have to use the MPS scripts  (I didn't find out this 
information from the documentation)

After the configuration of MPS scripts, I bumped into a problem which 
relates to absent plugin of LFN - PFN conversion:
       we are using in our experiment rootd, it has already some files 
stored on distributed disc with some PFNs and it is impossible to 
configure XROOTD
       our case example:                                               
                We have all files stored in distributed disc on path:
                    /data*/....  (where * goes from 1 to 3)              
              
                Each file in HPSS is stored on a path:                   
                                         
                    /home/starreco/....
               Each requested file from the HPSS to local disk is 
prefixed by the configuration directive oss.localroot /home/starlib 
(because I am running xrootd as other user which don't have permission 
to /home/stareco to create a symlink).
               This configuration directive causes that path 
/home/starlib is added to the each open request and then I don't have 
accessible data which we have already placed in distributed disk /data*.

In other words, there are now ways to migrate smoothly from rootd to 
Xrootd or even have a transition period where /when both system can 
coexist. The solution we did in a code is something like: 'check PFN and 
if does not exists, go to LFN', but it is addicted and not expandable 
solution .The longer term for us is to have an adaptable LFN to PFN 
translation rather than the current 'fixed' path pseudo conversion.

Andy, please can you prepare for us the new directive and all about that 
as we concluded at XROOTD+SRM meeting ? Thank you.

3)Next problems had to do with password authentication as a recovery of 
rootd authentication mechanism:                 
       
            TAuthenticate::SetGlobalUser("default");
            TAuthenticate::SetGlobalPasswd("*******");

  First thing was about using wild card of host and then checking of 
right srvpuk in cache.   By default the function in class 
XrdSutCache::Get had the disabled searching with wild card if you 
specify only one argument.   I had discussed this with Gerri and he 
committed my proposed changes into CVS (I checked it in a new version 
and it is already here).   Thank you Gerri.

  Second thing had to do with it that in pwd authentication isn't 
possible to authenticate a user as other user and thus replace rootd 
global mechanism.
  I proposed to Gerri change in the client class TXNetConn like this.
   Change:

    if(!gSystem->Getenv("XrdSecUSER")){
       gSystem->Setenv("XrdSecUSER",User.Data());
     }

    Before only:
    gSystem->Setenv("XrdSecUSER",User.Data());

  i.e. it allows to authenticate user as other user by defining the 
environment variable: XrdSecUSER.
  Gerri didn't find any problem on that.
    
  Could someone please check if it is available in future release ?    
Thank you

4)Next problem was the script for measuring load of data servers called 
XrdOlbMonPerf. I have found some bugs and repaired them.
 Changes:
     I have repaired bug related to network result and I added missing 
paging I/O result. Also I made some small changes as paths to unix 
command etc.

     See the attachment.

 I still have some problem with stopping this running command after the 
xrootd server is stopped. For some reason, this command is still running 
and it doesn't die when xrootd is killed.
 Some ideas ? Thank you
 

5)Monitoring issue:
    we are monitoring the xrootd daemon and olbd daemon from ganglia 
view (reporting cpu and memory usage exploiting ganglia metrics). See 
the attachment.

At the end I would like to thank to everyone who helped me during the 
deployment and also say: GREAT AND BRILLIANT work to all XROOTD's 
collaborators and contributors.
Now that the basic functionalities are in place and working, I am 
looking forward to the next steps and improvements along the line of 
Xrootd+SRM technology merging and development.
Best Pavel





#!/usr/local/bin/perl

## $Id: XrdOlbMonPerf,v 1.8 2004/09/10 12:01:20 elmer Exp $
## this script collects the informations on the server activity like:
## CPU, network I/O, memory usage ...
## modified to get rid of rperf.
## J-Y Nief - CCIN2P3 - 24/05/04
## modified to accept additional options and drop multi-home uf's.
#  Andy Hanushevsky - SLAC - 15/07/04
## modified to get right network I/O result, paging I/O and changes in paths to unix commands etc.
## Pavel Jakl - RCF/BNL - 20/08/05
#
# Usage: XrdOlbMonPerf [-I netintf] [-n netspeed] [-m <shmkey>] [<interval>]
# WARNING: if you are running this script on a Linux RH7.x server, 
#          the network metric is disabled as the netstat output is screwed, 
#          this info is useless. no problem for RH9 and RHEL3.

# get the name of the server + platform


undef($shmid);
$shmkey = '';
$rhost  = `/bin/uname -n`; chomp($rhost);
$enBps  = 131072000;
$systype= $^O;                #`/bin/uname`; chomp($systype);
$repval = 0;
$netif  = '';
$netmax = 0; # bits / s
$monint = 60;
$| = 1;                       # unbuffer STDOUT

$DEBUG  = 0;                  # switch to debug script

# Get options
#
  while(($argval = shift) && (substr($argval,0,1) eq '-'))
       {   if ($argval eq '-I')
              {Log("Network interface not specified.") if !($netif=shift);}
        elsif ($argval eq '-n')
              {Log("Network speed not specified.") if !($netmax=shift);}
        elsif ($argval eq '-m')
              {Log("Shared segment not specified.") if !($shmkey=shift);
               $shmid = shmget($shmkey, 4096, 0744) if (shmkey ne '.');
               Log("Unable to attach shared memory; $!") if !defined($shmid);
              }
       else   {Log("Unknown option, $argval, ignored", 1);}
      }

# Get the interval
#
if ($argval) 
   {$monint = $argval;
    Log("Invalid interval, '$monint'") if !($monint =~ /^\d+$/);
   }

# get the number of CPUs on this server
if ( $systype eq 'linux' ) {
  $numCPU = `/bin/grep -c cpu /proc/stat` - 1;
}

if ( $systype eq 'solaris' ) {
  $resp = `/bin/uname -X | /bin/grep NumCPU`;
  chomp($resp);
  @answer = split('= ', $resp);
  $numCPU = $answer[1];
}

# get the total memory of the server
if ($systype eq 'linux') {
  $resp = `/bin/grep MemTotal /proc/meminfo`;
  @answer = split('\s+', $resp);
  $TotMem = $answer[1]*1024;
}

if ($systype eq 'solaris' && -x "/etc/prtconf") {
  $resp = `/etc/prtconf | /bin/grep Memory`;
  ($TotMem) = $resp =~ /.+:\s*(\d+)/s;
  $TotMem *= 1024*1024;
}

# if it is a Linux, retrieve the distrib version.
# necessary for netstat which does not give the same output on RH 7.x, RH 9, 
# RHEL...
# netstat usable on RH 9 and RHEL 3 ==> distrib = "RH_OK", else "RH_WRONG"
if ($systype eq 'linux') {
  $distrib = "RH_OK";
  $resp = `/bin/cat /etc/redhat-release`;
  if ( $resp =~ /release 7./ ) { $distrib = "RH_WRONG"; }
}

# determine the column where the idle time and the free mem
# can be found in the 'vmstat' output.
# depends on the platform and on the 'vmstat' version, sigh... 
$ind_idle = 0;
$ind_free = 0;
@output = `/usr/bin/vmstat`;
chomp($output[1]);
@answer = split('\s+', $output[1]);
foreach $type(@answer) {
  last if ( $type eq 'id' );
  $ind_idle++;
}
foreach $type(@answer) {
  last if ( $type eq 'free' );
  $ind_free++;
}

# in the netstat output, the number of input and output packets is not in the 
# same columns whether it is a Linux or SunOS platform
if ( $systype eq 'solaris' ) {
  $ind_net1 = 5;
  $ind_net2 = 7;
} else {
  $ind_net1 = 3;
  $ind_net2 = 7;
}
$indx_net = 0;

# check the number of network interface on the server and their 
# capability (ethernet or gigabit)
$mtu = 1500;
if (!$netmax)
   {@com = `/bin/netstat -i`;
    foreach $line(@com) {
       if ( $line !~ /Kernel Interface/ && $line !~ /MTU/ ) {
          ( $line =~ /eth/ ) && ( $netmax = 1e8 );
          ( $line =~ /ge/ || $line =~ /ce/ ) && ( $netmax = 1e9 );
          }
      }
    if (!$netmax) {$netmax = 1e8;}
   }

# Get it in a sleep $monint time loop netstat result and count network I/O
$ipio_old = 0;
while(true) {
  @resp=`/bin/netstat -i`;
  
  # what's the CPU utilization ?
  @respCPU = `/usr/bin/vmstat 1 2`;
  chomp $respCPU[3];
  @resp_vmstat = split('\s+',$respCPU[3]);
  if ( ($resp_vmstat[$ind_idle] > 100) || ($resp_vmstat[$ind_idle] < 0) ) {
    $cpu = 0;
  } else {
    $cpu = $resp_vmstat[$ind_idle-2] + $resp_vmstat[$ind_idle-1];
  }

  # what's the runq load ?
  chomp($respLoad = `/usr/bin/uptime`);
  @respSplit = split(',',$respLoad);
  $load = int($respSplit[4]*25/$numCPU);  # Assumes some magic upper number i.e. 1 CPU load 4 = 100%
  $load = 100 if $load > 100;

  # what's the network I/O activity?
  foreach $resp (@resp){
      if ( ( $resp !~ /lo/ && $resp !~ /Iface/ && $systype eq 'linux' && $distrib ne 'RH_WRONG' )|| 
	   ( $systype eq 'solaris' ) ) {
       	@respSplit = split('\s+',$resp);

	print "DEBUG ".join("|",@respSplit)."\n" if ($DEBUG);

	if ( $systype eq 'solaris' ) {
	    $net_activity = ($respSplit[$ind_net1] + $respSplit[$ind_net2]) * $mtu * 8 / $monint;
	    $ipio = int ( $net_activity / $netmax * 100 );
	}else {
	    $measure[1] = $net[$indx_net] = $respSplit[$ind_net1] + $respSplit[$ind_net2];
	    print "DEBUG measure is $measure[1] $netmax\n" if ($DEBUG);
	    if ( $indx_net ) {
		$net_activity = ($measure[1] - $measure[0]) * $mtu * 8 / $monint;
		$ipio = int ( $net_activity / $netmax * 100 );
		$measure[0] = $measure[1];
	    }
	    $indx_net = $indx_net ^ 1;
	}
	$ipio_old = $ipio;
      } else {
	$ipio = $ipio_old;
      }
     }
  $ipio = 100 if $ipio > 100;
  $ipio_old = $ipio;

  # what's the memory load ?
  $used = $TotMem - $resp_vmstat[$ind_free]*1024;
  $mem = int($used/$TotMem*100);
  $mem = 100 if $mem > 100;
  #
  $repval++;

  # what's the paging I/O activity ?
  # useless as this metric is highly correlated with some of the others above.
  # being kept for backward compatibility with the load balancer.
  if ( -e "/proc/meminfo"){
      @items = split(" ",`/bin/cat /proc/meminfo | /bin/grep Swap:`);
      $pgio  = int($items[2]/$items[1]*100);
  } else {
      $pgio  = 0;
  }
  
  if (!defined($shmid)) {
      if ($DEBUG){ print "Load:$load CPU:$cpu MEM:$mem PGIO:$pgio IPIO:$ipio\n";}
      else       { print "$load $cpu $mem $pgio $ipio\n";}
  }
  else {
    $resp = pack('LLLLL',$load,$cpu,$mem,$pgio,$ipio);
    Log("Unable to write into shared memory; $!")
      if !shmwrite($shmid, $resp, 0, length($resp));
  }
   #Sleep some time
   sleep $monint;
}


sub Log {
  my($msg, $ret) = @_;
  print STDERR "XrdOlbMonPerf: $msg\n";
  system('/bin/logger', '-p', 'daemon.notice', "XrdOlbMonPerf: $msg");
  return 0 if $ret;
  exit(1);
}



#!/bin/sh
# 
# Simple gmetric script to display the memory and CPU used by xrootd
# authors: Ofer Rind and Jerome Lauret, RCF/BNL 
# 

MCHAN=`/bin/awk '/mcast_chan/{print $2}' /etc/gmond.conf`
GMETRIC="/usr/bin/gmetric --mcast_ttl=3 -c$MCHAN -d3000"

for proc in xrootd olbd; do
  stats=( `/bin/ps --no-headers -o rss,vsz,%cpu -C $proc` )
  n=3
  while [ ${#stats[*]} -gt $n ] ; do
    let m=n%3
    stats[$m]=`/bin/echo ${stats[$m]} ${stats[$n]}|/bin/awk '{print $1+$2}'`
    let n+=1
  done
  if [ ${stats} ] || eval "stats=( 0 0 0 )"; then
    $GMETRIC -nmem_${proc} -tuint32 -uKB --value ${stats[0]}
    $GMETRIC -nvsz_${proc} -tuint32 -uKB --value ${stats[1]}
    $GMETRIC -ncpu_${proc} -tfloat  -u%  --value ${stats[2]}
  fi
done