 |
 |
Hi Wei, Thanks for the tip. It appears that it is a problem with one of the disks /data/2. It is a xfs file system on top of lvm and xfs_repair is not locating any problems. The xfs database is stating that ifree=0. Not sure how I can fix the issue with this file system. Is there a way to have one of the two drives in the oss.space public /data/* be readonly and the other r/w ? Thanks, Doug On 03/26/2015 03:20 PM, Yang, Wei wrote: > This doesn’t seems like a CMS issue because the client is talking the xrootd instance of the data server. > > Can you become the user that run xrootd and try to create a file on /data/1 or 2? > > > Wei Yang | [log in to unmask] | 650-926-3338(O) > > > > On Mar 25, 2015, at 2:59 PM, Doug Benjamin <[log in to unmask]> wrote: > >> Hi Patrick, >> >> oss.space public /data/* directive is used to point where to hold the data files. >> >> /atlas/local/* contains symlinks to the data files. >> >> I removed the cms.space directive from the configuration file without success. >> >> Also SELinux is running in permissive mode so I do not think that that is the issue. >> >> >> Doug >> >> On 03/25/2015 04:41 PM, Patrick McGuigan wrote: >>> Hi Doug, >>> >>> My suspicion: >>> >>> 1) I don't see where /data/* is supporting anything under the /atlas namespace, so I won't look at that. >>> >>> 2) You are specifying that xrootd keep a minimum free space of 10GB and you only have 7.7GB free under /atlas. Hence it won't write there. >>> >>> Patrick >>> >>> On 03/25/2015 04:32 PM, Doug Benjamin wrote: >>>> Hi, >>>> >>>> I having a heck of a time trying to debug this problem. >>>> >>>> I have a stand alone file server with two partitions /data/1 and /data/2 that >>>> are glued together with xrootd >>>> >>>> Here is the xrootd config file important bits - >>>> >>>> all.adminpath /var/spool/xrootd/admin >>>> all.export /atlas r/w >>>> oss.space public /data/* >>>> xrootd.chksum max 3 adler32 /usr/bin/xrdadler32 >>>> xrootd.seclib /usr/lib64/libXrdSec.so >>>> sec.protocol /usr/lib64 unix >>>> acc.authdb /etc/xrootd/auth_file >>>> ofs.authorize >>>> cms.space min 10g 15g >>>> xrootd.trace all >>>> >>>> >>>> >>>> Both of the data partitions have sufficient space and inodes free. >>>> >>>> [root@atlasfs ~]# df -h /data/* >>>> Filesystem Size Used Avail Use% Mounted on >>>> /dev/mapper/vgdata1-lvdata1 >>>> 19T 17T 2.2T 89% /data/1 >>>> /dev/mapper/vgdata2-lvdata2 >>>> 19T 16T 3.0T 84% /data/2 >>>> [root@atlasfs ~]# df -ih /data/* >>>> Filesystem Inodes IUsed IFree IUse% Mounted on >>>> /dev/mapper/vgdata1-lvdata1 >>>> 3.7G 211K 3.7G 1% /data/1 >>>> /dev/mapper/vgdata2-lvdata2 >>>> 3.7G 165K 3.7G 1% /data/2 >>>> >>>> The exported area /atlas is on a disk with sufficient space and inodes : >>>> >>>> >>>> [root@atlasfs ~]# df -ih /atlas >>>> Filesystem Inodes IUsed IFree IUse% Mounted on >>>> /dev/mapper/sysvg-LVsys >>>> 1.6M 639K 926K 41% / >>>> [root@atlasfs ~]# df -h /atlas >>>> Filesystem Size Used Avail Use% Mounted on >>>> /dev/mapper/sysvg-LVsys >>>> 24G 16G 7.7G 67% / >>>> >>>> >>>> Yet the xrootd data server is reporting no space left on the device. >>>> >>>> >>>> 150325 16:04:16 10920 ?:7@atlas28 XrootdProtocol: 0000 req=3007 dlen=71 >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0000 sending 23 >>>> data bytes; status=0 >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0000 req=3000 dlen=21 >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0000 sending OK >>>> 150325 16:04:16 10920 XrootdXeq: dbenjami.9782:7@atlas28 pub IPv4 login as >>>> dbenjamin >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 req=3017 dlen=39 >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 rc=0 stat >>>> /atlas/local/dbenjamin/xroot_copy_test/ >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0100 sending 36 >>>> data bytes >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 req=3010 dlen=94 >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 open unmat >>>> /atlas/local/dbenjamin/xroot_copy_test//NTUP_SMWZ.01122074._000001.root.1?oss.asize=3303014400 >>>> >>>> 150325 16:04:16 10920 ofs_open: dbenjami.9782:7@atlas28 Unable to create >>>> /atlas/local/dbenjamin/xroot_copy_test/NTUP_SMWZ.01122074._000001.root.1; no >>>> space left on device >>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0100 sending err >>>> 3009: Unable to create >>>> /atlas/local/dbenjamin/xroot_copy_test/NTUP_SMWZ.01122074._000001.root.1; no >>>> space left on device >>>> 150325 16:04:16 10920 XrootdXeq: dbenjami.9782:7@atlas28 disc 0:00:00 >>>> 150325 16:09:19 10950 ?:18@atlas17 XrootdProtocol: 0000 req=3007 dlen=84 >>>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdResponse: 0000 sending 23 >>>> data bytes; status=0 >>>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdProtocol: 0000 req=3000 dlen=18 >>>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdResponse: 0000 sending OK >>>> >>>> Also using xrd - >>>> >>>> [dbenjamin@atlas28 bnl_openstack]$ xrd atlasfs.hep.anl.gov statvfs /atlas/local >>>> r/w nodes: 1 r/w free space: 3064292 r/w utilization: 0 staging nodes: 1 staging >>>> free space: 3064292 staging utilization: 0 >>>> >>>> [dbenjamin@atlas28 bnl_openstack]$ xrd atlasfs.hep.anl.gov queryspace /atlas/local >>>> Disk space approximations (MB): >>>> Total : 38139916 >>>> Free : 5324910 >>>> Used : 0 >>>> Largest chunk : 3064292 >>>> >>>> >>>> >>>> What am I missing? >>>> >>>> Thanks, >>>> >>>> Doug Benjamin >>>> >>>> ######################################################################## >>>> Use REPLY-ALL to reply to list >>>> >>>> To unsubscribe from the XROOTD-L list, click the following link: >>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1