XROOTD-L Archives

Support use of xrootd by HEP experiments

XROOTD-L@LISTSERV.SLAC.STANFORD.EDU

Options: Use Forum View

Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Message: [<< First] [< Prev] [Next >] [Last >>]
Topic: [<< First] [< Prev] [Next >] [Last >>]
Author: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Subject:
From:
Reply To:
Support use of xrootd by HEP experiments <[log in to unmask]>
Date:
Sat, 9 Nov 2013 17:46:57 +0100
Content-Type:
text/plain
Parts/Attachments:
text/plain (220 lines)
Dear Andrew and Yangwei:

   Thanks for the support. Now there is a new problem, xrdcp reports 
that the data server valtical00 is full:

xrdcp: Copy to valtical00.cern.ch failed on close!
xrdcp: Unable to write 
/localdisk/xrootd/users/qing/data12_8TeV/SMDILEP_p1328_p1329/user.qing.data12_8TeV.periodA.physics_Muons.PhysCont.NTUP_SMWZ.grp14_v01_p1328_p1329_2LepSkim_v2/user.qing.001619._00845.skimmed.root; 
no space left on device

But looking at valtical00, we can see there are a lot of free space on 
/localdisk*, seems these disks are not used as extended space for 
/localdisk.

[root@valtical00 localdisk]# df -h | grep localdisk
/dev/sda1             1.8T  196M  1.7T   1% /localdisk7
/dev/sdb1             1.8T  1.2T  569G  68% /localdisk5
/dev/sdd1             1.8T  1.7T     0 100% /localdisk
/dev/sde1             1.8T  1.2T  568G  68% /localdisk2
/dev/sdf1             1.8T  1.2T  568G  68% /localdisk3
/dev/sdg1             1.8T  1.2T  570G  68% /localdisk4
/dev/sdh1             1.8T  1.2T  564G  68% /localdisk6

The following is the local xrootd configuration file, I used oss.space 
and removed the xa option:

[root@valtical00 localdisk]# cat /etc/xrootd/xrootd-clustered.cfg
all.export /localdisk/xrootd
oss.space public /localdisk/xrootd/*
oss.space public /localdisk2/xrootd
oss.space public /localdisk3/xrootd
oss.space public /localdisk4/xrootd
oss.space public /localdisk5/xrootd
oss.space public /localdisk6/xrootd
oss.space public /localdisk7/xrootd
all.export /localdisk/proofbox

set xrdr=valtical.cern.ch
all.manager $(xrdr):1213
if $(xrdr) && named cns
       all.export /data/inventory
       xrd.port 1095
else if $(xrdr)
       all.role manager
       xrd.port 1094
else
       all.role server
       oss.localroot /
       #oss.localroot /local/xrootd
       ofs.notify closew create mkdir mv rm rmdir trunc | 
/usr/bin/XrdCnsd -d -D 2 -i 90 -b root://$(xrdr):2094
       add cms.space if you have less the 11GB
       # cms.space options 
http://xrootd.slac.stanford.edu/doc/dev/cms_config.htm
       cms.space min 2g 5g
fi

 From the cmsd.log I see that all the space are correctly imported into 
xrootd:


[root@valtical00 xrootd]# cat cmsd.log
131109 17:18:09 4725 Scalla is starting. . .
Copr.  2004-2012 Stanford University, xrd version v3.3.1
++++++ cmsd [log in to unmask] initialization started.
Config using configuration file /etc/xrootd/xrootd-clustered.cfg
Config maximum number of connections restricted to 65536
Copr.  2007 Stanford University/SLAC cmsd.
++++++ [log in to unmask] phase 1 initialization started.
=====> all.export /localdisk/xrootd
=====> all.export /localdisk/proofbox
=====> all.manager valtical.cern.ch:1213
=====> all.role server
=====> oss.localroot /
=====> cms.space min 2g 5g
The following paths are available to the redirector:
w  /localdisk/proofbox
w  /localdisk/xrootd

------ [log in to unmask] phase 1 server initialization completed.
++++++ [log in to unmask] phase 2 server initialization started.
Config warning: adminpath resides in /tmp and may be unstable!
++++++ Storage system initialization started.
=====> all.export /localdisk/xrootd
=====> oss.space public /localdisk/xrootd/*
=====> oss.space public /localdisk2/xrootd
=====> oss.space public /localdisk3/xrootd
=====> oss.space public /localdisk4/xrootd
=====> oss.space public /localdisk5/xrootd
=====> oss.space public /localdisk6/xrootd
=====> oss.space public /localdisk7/xrootd
=====> all.export /localdisk/proofbox
=====> oss.localroot /
++++++ Configuring standalone mode . . .
Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
        oss.alloc        0 0 0
        oss.cachescan    600
        oss.fdlimit      32768 65536
        oss.maxsize      0
        oss.localroot /
        oss.trace        0
        oss.xfr          1 deny 10800 keep 1200
        oss.memfile off  max 12593575936
        oss.space public /localdisk/xrootd/test
        oss.space public /localdisk/xrootd/users
        oss.space public /localdisk/xrootd/test4
        oss.space public /localdisk/xrootd/sam
        oss.space public /localdisk2/xrootd
        oss.space public /localdisk3/xrootd
        oss.space public /localdisk4/xrootd
        oss.space public /localdisk5/xrootd
        oss.space public /localdisk6/xrootd
        oss.space public /localdisk7/xrootd
        oss.defaults  r/w  nocheck nodread nomig norcreate nopurge 
nostage xattr
        oss.path /localdisk/proofbox r/w  nocheck nodread nomig 
norcreate nopurge nostage xattr
        oss.path /localdisk/xrootd r/w  nocheck nodread nomig norcreate 
nopurge nostage xattr
------ Storage system initialization completed.
131109 17:18:09 4725 Meter: Found 7 filesystem(s); 12TB total (65% 
util); 4TB free (1TB max)
------ [log in to unmask] phase 2 server initialization completed.
------ cmsd [log in to unmask]:57515 initialization completed.
131109 17:18:09 4738 Start: Waiting for primary server to login.
131109 17:18:19 4740 do_Login:: Primary server 4659 logged in; data port 
is 1094
131109 17:18:19 4728 Config: server service enabled.
131109 17:18:19 4741 State: Status changed to active + nostaging
131109 17:18:19 4729 ManTree: Now connected to 1 root node(s)
131109 17:18:19 4729 Protocol: Logged into valtical

   So any idea where the problem stays?

   Cheers,Gang





On 05/23/2013 08:27 PM, Andrew Hanushevsky wrote:
> Hi Qang,
>
> When you convert oss.cache to oss.space, please drop the "xa" in the 
> statement. That option is not relevant for oss.space as it always uses 
> "xa" mode.
>
> Andy
>
> On Thu, 23 May 2013, Yang, Wei wrote:
>
>> Hi Qang,
>>
>> this is right. you just need to replace oss.cache with oss.space 
>> (actually both work, but ongoing, we will only use oss.space). and 
>> please move to 3.3.1 if possible. there is a few issues with 3.3.0 
>> but i forgot what are they.
>>
>> regards,
>> Wei Yang  |  [log in to unmask]  |  650-926-3338(O)
>>
>>
>> On May 23, 2013, at 6:56 AM, Gang Qin wrote:
>>
>>> Dear Expert:
>>>
>>>  We are migrating xrootd from version 3.0.5 to 3.3.0, in our 
>>> previous configuration file we have:
>>>
>>> all.export /localdisk/xrootd
>>> set osscachepath = /localdisk/xrootd
>>> oss.cache public $(osscachepath)/* xa
>>> oss.cache public /localdisk2/xrootd xa
>>> oss.cache public /localdisk3/xrootd xa
>>> oss.cache public /localdisk4/xrootd xa
>>> oss.cache public /localdisk5/xrootd xa
>>> oss.cache public /localdisk6/xrootd xa
>>>
>>>   However, seems oss.cache is phased out in the new version. so can 
>>> we simply change it to something like:
>>>
>>> all.export /localdisk/xrootd
>>> set osscachepath = /localdisk/xrootd
>>> oss.space public $(osscachepath)/* xa
>>> oss.space public /localdisk2/xrootd xa
>>> oss.space public /localdisk3/xrootd xa
>>> oss.space public /localdisk4/xrootd xa
>>> oss.space public /localdisk5/xrootd xa
>>> oss.space public /localdisk6/xrootd xa
>>>
>>>   Cheers,Gang
>>>
>>>
>>>
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1

ATOM RSS1 RSS2