Dear Andrew and Yangwei:
Thanks for the support. Now there is a new problem, xrdcp reports
that the data server valtical00 is full:
xrdcp: Copy to valtical00.cern.ch failed on close!
xrdcp: Unable to write
/localdisk/xrootd/users/qing/data12_8TeV/SMDILEP_p1328_p1329/user.qing.data12_8TeV.periodA.physics_Muons.PhysCont.NTUP_SMWZ.grp14_v01_p1328_p1329_2LepSkim_v2/user.qing.001619._00845.skimmed.root;
no space left on device
But looking at valtical00, we can see there are a lot of free space on
/localdisk*, seems these disks are not used as extended space for
/localdisk.
[root@valtical00 localdisk]# df -h | grep localdisk
/dev/sda1 1.8T 196M 1.7T 1% /localdisk7
/dev/sdb1 1.8T 1.2T 569G 68% /localdisk5
/dev/sdd1 1.8T 1.7T 0 100% /localdisk
/dev/sde1 1.8T 1.2T 568G 68% /localdisk2
/dev/sdf1 1.8T 1.2T 568G 68% /localdisk3
/dev/sdg1 1.8T 1.2T 570G 68% /localdisk4
/dev/sdh1 1.8T 1.2T 564G 68% /localdisk6
The following is the local xrootd configuration file, I used oss.space
and removed the xa option:
[root@valtical00 localdisk]# cat /etc/xrootd/xrootd-clustered.cfg
all.export /localdisk/xrootd
oss.space public /localdisk/xrootd/*
oss.space public /localdisk2/xrootd
oss.space public /localdisk3/xrootd
oss.space public /localdisk4/xrootd
oss.space public /localdisk5/xrootd
oss.space public /localdisk6/xrootd
oss.space public /localdisk7/xrootd
all.export /localdisk/proofbox
set xrdr=valtical.cern.ch
all.manager $(xrdr):1213
if $(xrdr) && named cns
all.export /data/inventory
xrd.port 1095
else if $(xrdr)
all.role manager
xrd.port 1094
else
all.role server
oss.localroot /
#oss.localroot /local/xrootd
ofs.notify closew create mkdir mv rm rmdir trunc |
/usr/bin/XrdCnsd -d -D 2 -i 90 -b root://$(xrdr):2094
add cms.space if you have less the 11GB
# cms.space options
http://xrootd.slac.stanford.edu/doc/dev/cms_config.htm
cms.space min 2g 5g
fi
From the cmsd.log I see that all the space are correctly imported into
xrootd:
[root@valtical00 xrootd]# cat cmsd.log
131109 17:18:09 4725 Scalla is starting. . .
Copr. 2004-2012 Stanford University, xrd version v3.3.1
++++++ cmsd [log in to unmask] initialization started.
Config using configuration file /etc/xrootd/xrootd-clustered.cfg
Config maximum number of connections restricted to 65536
Copr. 2007 Stanford University/SLAC cmsd.
++++++ [log in to unmask] phase 1 initialization started.
=====> all.export /localdisk/xrootd
=====> all.export /localdisk/proofbox
=====> all.manager valtical.cern.ch:1213
=====> all.role server
=====> oss.localroot /
=====> cms.space min 2g 5g
The following paths are available to the redirector:
w /localdisk/proofbox
w /localdisk/xrootd
------ [log in to unmask] phase 1 server initialization completed.
++++++ [log in to unmask] phase 2 server initialization started.
Config warning: adminpath resides in /tmp and may be unstable!
++++++ Storage system initialization started.
=====> all.export /localdisk/xrootd
=====> oss.space public /localdisk/xrootd/*
=====> oss.space public /localdisk2/xrootd
=====> oss.space public /localdisk3/xrootd
=====> oss.space public /localdisk4/xrootd
=====> oss.space public /localdisk5/xrootd
=====> oss.space public /localdisk6/xrootd
=====> oss.space public /localdisk7/xrootd
=====> all.export /localdisk/proofbox
=====> oss.localroot /
++++++ Configuring standalone mode . . .
Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
oss.alloc 0 0 0
oss.cachescan 600
oss.fdlimit 32768 65536
oss.maxsize 0
oss.localroot /
oss.trace 0
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 12593575936
oss.space public /localdisk/xrootd/test
oss.space public /localdisk/xrootd/users
oss.space public /localdisk/xrootd/test4
oss.space public /localdisk/xrootd/sam
oss.space public /localdisk2/xrootd
oss.space public /localdisk3/xrootd
oss.space public /localdisk4/xrootd
oss.space public /localdisk5/xrootd
oss.space public /localdisk6/xrootd
oss.space public /localdisk7/xrootd
oss.defaults r/w nocheck nodread nomig norcreate nopurge
nostage xattr
oss.path /localdisk/proofbox r/w nocheck nodread nomig
norcreate nopurge nostage xattr
oss.path /localdisk/xrootd r/w nocheck nodread nomig norcreate
nopurge nostage xattr
------ Storage system initialization completed.
131109 17:18:09 4725 Meter: Found 7 filesystem(s); 12TB total (65%
util); 4TB free (1TB max)
------ [log in to unmask] phase 2 server initialization completed.
------ cmsd [log in to unmask]:57515 initialization completed.
131109 17:18:09 4738 Start: Waiting for primary server to login.
131109 17:18:19 4740 do_Login:: Primary server 4659 logged in; data port
is 1094
131109 17:18:19 4728 Config: server service enabled.
131109 17:18:19 4741 State: Status changed to active + nostaging
131109 17:18:19 4729 ManTree: Now connected to 1 root node(s)
131109 17:18:19 4729 Protocol: Logged into valtical
So any idea where the problem stays?
Cheers,Gang
On 05/23/2013 08:27 PM, Andrew Hanushevsky wrote:
> Hi Qang,
>
> When you convert oss.cache to oss.space, please drop the "xa" in the
> statement. That option is not relevant for oss.space as it always uses
> "xa" mode.
>
> Andy
>
> On Thu, 23 May 2013, Yang, Wei wrote:
>
>> Hi Qang,
>>
>> this is right. you just need to replace oss.cache with oss.space
>> (actually both work, but ongoing, we will only use oss.space). and
>> please move to 3.3.1 if possible. there is a few issues with 3.3.0
>> but i forgot what are they.
>>
>> regards,
>> Wei Yang | [log in to unmask] | 650-926-3338(O)
>>
>>
>> On May 23, 2013, at 6:56 AM, Gang Qin wrote:
>>
>>> Dear Expert:
>>>
>>> We are migrating xrootd from version 3.0.5 to 3.3.0, in our
>>> previous configuration file we have:
>>>
>>> all.export /localdisk/xrootd
>>> set osscachepath = /localdisk/xrootd
>>> oss.cache public $(osscachepath)/* xa
>>> oss.cache public /localdisk2/xrootd xa
>>> oss.cache public /localdisk3/xrootd xa
>>> oss.cache public /localdisk4/xrootd xa
>>> oss.cache public /localdisk5/xrootd xa
>>> oss.cache public /localdisk6/xrootd xa
>>>
>>> However, seems oss.cache is phased out in the new version. so can
>>> we simply change it to something like:
>>>
>>> all.export /localdisk/xrootd
>>> set osscachepath = /localdisk/xrootd
>>> oss.space public $(osscachepath)/* xa
>>> oss.space public /localdisk2/xrootd xa
>>> oss.space public /localdisk3/xrootd xa
>>> oss.space public /localdisk4/xrootd xa
>>> oss.space public /localdisk5/xrootd xa
>>> oss.space public /localdisk6/xrootd xa
>>>
>>> Cheers,Gang
>>>
>>>
>>>
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
