 |
 |
Dear Andrew and Yang-Wei: The problem is fixed by remounting the /localdisk* with 'user_xattr' option. Cheers,Gang On 11/09/2013 05:46 PM, qing wrote: > Dear Andrew and Yangwei: > > Thanks for the support. Now there is a new problem, xrdcp reports > that the data server valtical00 is full: > > xrdcp: Copy to valtical00.cern.ch failed on close! > xrdcp: Unable to write > /localdisk/xrootd/users/qing/data12_8TeV/SMDILEP_p1328_p1329/user.qing.data12_8TeV.periodA.physics_Muons.PhysCont.NTUP_SMWZ.grp14_v01_p1328_p1329_2LepSkim_v2/user.qing.001619._00845.skimmed.root; > no space left on device > > But looking at valtical00, we can see there are a lot of free space on > /localdisk*, seems these disks are not used as extended space for > /localdisk. > > [root@valtical00 localdisk]# df -h | grep localdisk > /dev/sda1 1.8T 196M 1.7T 1% /localdisk7 > /dev/sdb1 1.8T 1.2T 569G 68% /localdisk5 > /dev/sdd1 1.8T 1.7T 0 100% /localdisk > /dev/sde1 1.8T 1.2T 568G 68% /localdisk2 > /dev/sdf1 1.8T 1.2T 568G 68% /localdisk3 > /dev/sdg1 1.8T 1.2T 570G 68% /localdisk4 > /dev/sdh1 1.8T 1.2T 564G 68% /localdisk6 > > The following is the local xrootd configuration file, I used oss.space > and removed the xa option: > > [root@valtical00 localdisk]# cat /etc/xrootd/xrootd-clustered.cfg > all.export /localdisk/xrootd > oss.space public /localdisk/xrootd/* > oss.space public /localdisk2/xrootd > oss.space public /localdisk3/xrootd > oss.space public /localdisk4/xrootd > oss.space public /localdisk5/xrootd > oss.space public /localdisk6/xrootd > oss.space public /localdisk7/xrootd > all.export /localdisk/proofbox > > set xrdr=valtical.cern.ch > all.manager $(xrdr):1213 > if $(xrdr) && named cns > all.export /data/inventory > xrd.port 1095 > else if $(xrdr) > all.role manager > xrd.port 1094 > else > all.role server > oss.localroot / > #oss.localroot /local/xrootd > ofs.notify closew create mkdir mv rm rmdir trunc | > /usr/bin/XrdCnsd -d -D 2 -i 90 -b root://$(xrdr):2094 > add cms.space if you have less the 11GB > # cms.space options > http://xrootd.slac.stanford.edu/doc/dev/cms_config.htm > cms.space min 2g 5g > fi > > From the cmsd.log I see that all the space are correctly imported into > xrootd: > > > [root@valtical00 xrootd]# cat cmsd.log > 131109 17:18:09 4725 Scalla is starting. . . > Copr. 2004-2012 Stanford University, xrd version v3.3.1 > ++++++ cmsd [log in to unmask] initialization started. > Config using configuration file /etc/xrootd/xrootd-clustered.cfg > Config maximum number of connections restricted to 65536 > Copr. 2007 Stanford University/SLAC cmsd. > ++++++ [log in to unmask] phase 1 initialization started. > =====> all.export /localdisk/xrootd > =====> all.export /localdisk/proofbox > =====> all.manager valtical.cern.ch:1213 > =====> all.role server > =====> oss.localroot / > =====> cms.space min 2g 5g > The following paths are available to the redirector: > w /localdisk/proofbox > w /localdisk/xrootd > > ------ [log in to unmask] phase 1 server initialization completed. > ++++++ [log in to unmask] phase 2 server initialization started. > Config warning: adminpath resides in /tmp and may be unstable! > ++++++ Storage system initialization started. > =====> all.export /localdisk/xrootd > =====> oss.space public /localdisk/xrootd/* > =====> oss.space public /localdisk2/xrootd > =====> oss.space public /localdisk3/xrootd > =====> oss.space public /localdisk4/xrootd > =====> oss.space public /localdisk5/xrootd > =====> oss.space public /localdisk6/xrootd > =====> oss.space public /localdisk7/xrootd > =====> all.export /localdisk/proofbox > =====> oss.localroot / > ++++++ Configuring standalone mode . . . > Config effective /etc/xrootd/xrootd-clustered.cfg oss configuration: > oss.alloc 0 0 0 > oss.cachescan 600 > oss.fdlimit 32768 65536 > oss.maxsize 0 > oss.localroot / > oss.trace 0 > oss.xfr 1 deny 10800 keep 1200 > oss.memfile off max 12593575936 > oss.space public /localdisk/xrootd/test > oss.space public /localdisk/xrootd/users > oss.space public /localdisk/xrootd/test4 > oss.space public /localdisk/xrootd/sam > oss.space public /localdisk2/xrootd > oss.space public /localdisk3/xrootd > oss.space public /localdisk4/xrootd > oss.space public /localdisk5/xrootd > oss.space public /localdisk6/xrootd > oss.space public /localdisk7/xrootd > oss.defaults r/w nocheck nodread nomig norcreate nopurge > nostage xattr > oss.path /localdisk/proofbox r/w nocheck nodread nomig > norcreate nopurge nostage xattr > oss.path /localdisk/xrootd r/w nocheck nodread nomig norcreate > nopurge nostage xattr > ------ Storage system initialization completed. > 131109 17:18:09 4725 Meter: Found 7 filesystem(s); 12TB total (65% > util); 4TB free (1TB max) > ------ [log in to unmask] phase 2 server initialization completed. > ------ cmsd [log in to unmask]:57515 initialization completed. > 131109 17:18:09 4738 Start: Waiting for primary server to login. > 131109 17:18:19 4740 do_Login:: Primary server 4659 logged in; data > port is 1094 > 131109 17:18:19 4728 Config: server service enabled. > 131109 17:18:19 4741 State: Status changed to active + nostaging > 131109 17:18:19 4729 ManTree: Now connected to 1 root node(s) > 131109 17:18:19 4729 Protocol: Logged into valtical > > So any idea where the problem stays? > > Cheers,Gang > > > > > > On 05/23/2013 08:27 PM, Andrew Hanushevsky wrote: >> Hi Qang, >> >> When you convert oss.cache to oss.space, please drop the "xa" in the >> statement. That option is not relevant for oss.space as it always >> uses "xa" mode. >> >> Andy >> >> On Thu, 23 May 2013, Yang, Wei wrote: >> >>> Hi Qang, >>> >>> this is right. you just need to replace oss.cache with oss.space >>> (actually both work, but ongoing, we will only use oss.space). and >>> please move to 3.3.1 if possible. there is a few issues with 3.3.0 >>> but i forgot what are they. >>> >>> regards, >>> Wei Yang | [log in to unmask] | 650-926-3338(O) >>> >>> >>> On May 23, 2013, at 6:56 AM, Gang Qin wrote: >>> >>>> Dear Expert: >>>> >>>> We are migrating xrootd from version 3.0.5 to 3.3.0, in our >>>> previous configuration file we have: >>>> >>>> all.export /localdisk/xrootd >>>> set osscachepath = /localdisk/xrootd >>>> oss.cache public $(osscachepath)/* xa >>>> oss.cache public /localdisk2/xrootd xa >>>> oss.cache public /localdisk3/xrootd xa >>>> oss.cache public /localdisk4/xrootd xa >>>> oss.cache public /localdisk5/xrootd xa >>>> oss.cache public /localdisk6/xrootd xa >>>> >>>> However, seems oss.cache is phased out in the new version. so can >>>> we simply change it to something like: >>>> >>>> all.export /localdisk/xrootd >>>> set osscachepath = /localdisk/xrootd >>>> oss.space public $(osscachepath)/* xa >>>> oss.space public /localdisk2/xrootd xa >>>> oss.space public /localdisk3/xrootd xa >>>> oss.space public /localdisk4/xrootd xa >>>> oss.space public /localdisk5/xrootd xa >>>> oss.space public /localdisk6/xrootd xa >>>> >>>> Cheers,Gang >>>> >>>> >>>> >>>> >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-L list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>>> >>> >>> ######################################################################## >>> >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1