Dear Andrew and Yang-Wei:
The problem is fixed by remounting the /localdisk* with 'user_xattr'
option.
Cheers,Gang
On 11/09/2013 05:46 PM, qing wrote:
> Dear Andrew and Yangwei:
>
> Thanks for the support. Now there is a new problem, xrdcp reports
> that the data server valtical00 is full:
>
> xrdcp: Copy to valtical00.cern.ch failed on close!
> xrdcp: Unable to write
> /localdisk/xrootd/users/qing/data12_8TeV/SMDILEP_p1328_p1329/user.qing.data12_8TeV.periodA.physics_Muons.PhysCont.NTUP_SMWZ.grp14_v01_p1328_p1329_2LepSkim_v2/user.qing.001619._00845.skimmed.root;
> no space left on device
>
> But looking at valtical00, we can see there are a lot of free space on
> /localdisk*, seems these disks are not used as extended space for
> /localdisk.
>
> [root@valtical00 localdisk]# df -h | grep localdisk
> /dev/sda1 1.8T 196M 1.7T 1% /localdisk7
> /dev/sdb1 1.8T 1.2T 569G 68% /localdisk5
> /dev/sdd1 1.8T 1.7T 0 100% /localdisk
> /dev/sde1 1.8T 1.2T 568G 68% /localdisk2
> /dev/sdf1 1.8T 1.2T 568G 68% /localdisk3
> /dev/sdg1 1.8T 1.2T 570G 68% /localdisk4
> /dev/sdh1 1.8T 1.2T 564G 68% /localdisk6
>
> The following is the local xrootd configuration file, I used oss.space
> and removed the xa option:
>
> [root@valtical00 localdisk]# cat /etc/xrootd/xrootd-clustered.cfg
> all.export /localdisk/xrootd
> oss.space public /localdisk/xrootd/*
> oss.space public /localdisk2/xrootd
> oss.space public /localdisk3/xrootd
> oss.space public /localdisk4/xrootd
> oss.space public /localdisk5/xrootd
> oss.space public /localdisk6/xrootd
> oss.space public /localdisk7/xrootd
> all.export /localdisk/proofbox
>
> set xrdr=valtical.cern.ch
> all.manager $(xrdr):1213
> if $(xrdr) && named cns
> all.export /data/inventory
> xrd.port 1095
> else if $(xrdr)
> all.role manager
> xrd.port 1094
> else
> all.role server
> oss.localroot /
> #oss.localroot /local/xrootd
> ofs.notify closew create mkdir mv rm rmdir trunc |
> /usr/bin/XrdCnsd -d -D 2 -i 90 -b root://$(xrdr):2094
> add cms.space if you have less the 11GB
> # cms.space options
> http://xrootd.slac.stanford.edu/doc/dev/cms_config.htm
> cms.space min 2g 5g
> fi
>
> From the cmsd.log I see that all the space are correctly imported into
> xrootd:
>
>
> [root@valtical00 xrootd]# cat cmsd.log
> 131109 17:18:09 4725 Scalla is starting. . .
> Copr. 2004-2012 Stanford University, xrd version v3.3.1
> ++++++ cmsd [log in to unmask] initialization started.
> Config using configuration file /etc/xrootd/xrootd-clustered.cfg
> Config maximum number of connections restricted to 65536
> Copr. 2007 Stanford University/SLAC cmsd.
> ++++++ [log in to unmask] phase 1 initialization started.
> =====> all.export /localdisk/xrootd
> =====> all.export /localdisk/proofbox
> =====> all.manager valtical.cern.ch:1213
> =====> all.role server
> =====> oss.localroot /
> =====> cms.space min 2g 5g
> The following paths are available to the redirector:
> w /localdisk/proofbox
> w /localdisk/xrootd
>
> ------ [log in to unmask] phase 1 server initialization completed.
> ++++++ [log in to unmask] phase 2 server initialization started.
> Config warning: adminpath resides in /tmp and may be unstable!
> ++++++ Storage system initialization started.
> =====> all.export /localdisk/xrootd
> =====> oss.space public /localdisk/xrootd/*
> =====> oss.space public /localdisk2/xrootd
> =====> oss.space public /localdisk3/xrootd
> =====> oss.space public /localdisk4/xrootd
> =====> oss.space public /localdisk5/xrootd
> =====> oss.space public /localdisk6/xrootd
> =====> oss.space public /localdisk7/xrootd
> =====> all.export /localdisk/proofbox
> =====> oss.localroot /
> ++++++ Configuring standalone mode . . .
> Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration:
> oss.alloc 0 0 0
> oss.cachescan 600
> oss.fdlimit 32768 65536
> oss.maxsize 0
> oss.localroot /
> oss.trace 0
> oss.xfr 1 deny 10800 keep 1200
> oss.memfile off max 12593575936
> oss.space public /localdisk/xrootd/test
> oss.space public /localdisk/xrootd/users
> oss.space public /localdisk/xrootd/test4
> oss.space public /localdisk/xrootd/sam
> oss.space public /localdisk2/xrootd
> oss.space public /localdisk3/xrootd
> oss.space public /localdisk4/xrootd
> oss.space public /localdisk5/xrootd
> oss.space public /localdisk6/xrootd
> oss.space public /localdisk7/xrootd
> oss.defaults r/w nocheck nodread nomig norcreate nopurge
> nostage xattr
> oss.path /localdisk/proofbox r/w nocheck nodread nomig
> norcreate nopurge nostage xattr
> oss.path /localdisk/xrootd r/w nocheck nodread nomig norcreate
> nopurge nostage xattr
> ------ Storage system initialization completed.
> 131109 17:18:09 4725 Meter: Found 7 filesystem(s); 12TB total (65%
> util); 4TB free (1TB max)
> ------ [log in to unmask] phase 2 server initialization completed.
> ------ cmsd [log in to unmask]:57515 initialization completed.
> 131109 17:18:09 4738 Start: Waiting for primary server to login.
> 131109 17:18:19 4740 do_Login:: Primary server 4659 logged in; data
> port is 1094
> 131109 17:18:19 4728 Config: server service enabled.
> 131109 17:18:19 4741 State: Status changed to active + nostaging
> 131109 17:18:19 4729 ManTree: Now connected to 1 root node(s)
> 131109 17:18:19 4729 Protocol: Logged into valtical
>
> So any idea where the problem stays?
>
> Cheers,Gang
>
>
>
>
>
> On 05/23/2013 08:27 PM, Andrew Hanushevsky wrote:
>> Hi Qang,
>>
>> When you convert oss.cache to oss.space, please drop the "xa" in the
>> statement. That option is not relevant for oss.space as it always
>> uses "xa" mode.
>>
>> Andy
>>
>> On Thu, 23 May 2013, Yang, Wei wrote:
>>
>>> Hi Qang,
>>>
>>> this is right. you just need to replace oss.cache with oss.space
>>> (actually both work, but ongoing, we will only use oss.space). and
>>> please move to 3.3.1 if possible. there is a few issues with 3.3.0
>>> but i forgot what are they.
>>>
>>> regards,
>>> Wei Yang | [log in to unmask] | 650-926-3338(O)
>>>
>>>
>>> On May 23, 2013, at 6:56 AM, Gang Qin wrote:
>>>
>>>> Dear Expert:
>>>>
>>>> We are migrating xrootd from version 3.0.5 to 3.3.0, in our
>>>> previous configuration file we have:
>>>>
>>>> all.export /localdisk/xrootd
>>>> set osscachepath = /localdisk/xrootd
>>>> oss.cache public $(osscachepath)/* xa
>>>> oss.cache public /localdisk2/xrootd xa
>>>> oss.cache public /localdisk3/xrootd xa
>>>> oss.cache public /localdisk4/xrootd xa
>>>> oss.cache public /localdisk5/xrootd xa
>>>> oss.cache public /localdisk6/xrootd xa
>>>>
>>>> However, seems oss.cache is phased out in the new version. so can
>>>> we simply change it to something like:
>>>>
>>>> all.export /localdisk/xrootd
>>>> set osscachepath = /localdisk/xrootd
>>>> oss.space public $(osscachepath)/* xa
>>>> oss.space public /localdisk2/xrootd xa
>>>> oss.space public /localdisk3/xrootd xa
>>>> oss.space public /localdisk4/xrootd xa
>>>> oss.space public /localdisk5/xrootd xa
>>>> oss.space public /localdisk6/xrootd xa
>>>>
>>>> Cheers,Gang
>>>>
>>>>
>>>>
>>>>
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>>
>>>
>>> ########################################################################
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|