 |
 |
This doesn’t seems like a CMS issue because the client is talking the xrootd instance of the data server. Can you become the user that run xrootd and try to create a file on /data/1 or 2? Wei Yang | [log in to unmask] | 650-926-3338(O) On Mar 25, 2015, at 2:59 PM, Doug Benjamin <[log in to unmask]> wrote: > Hi Patrick, > > oss.space public /data/* directive is used to point where to hold the data files. > > /atlas/local/* contains symlinks to the data files. > > I removed the cms.space directive from the configuration file without success. > > Also SELinux is running in permissive mode so I do not think that that is the issue. > > > Doug > > On 03/25/2015 04:41 PM, Patrick McGuigan wrote: >> Hi Doug, >> >> My suspicion: >> >> 1) I don't see where /data/* is supporting anything under the /atlas namespace, so I won't look at that. >> >> 2) You are specifying that xrootd keep a minimum free space of 10GB and you only have 7.7GB free under /atlas. Hence it won't write there. >> >> Patrick >> >> On 03/25/2015 04:32 PM, Doug Benjamin wrote: >>> Hi, >>> >>> I having a heck of a time trying to debug this problem. >>> >>> I have a stand alone file server with two partitions /data/1 and /data/2 that >>> are glued together with xrootd >>> >>> Here is the xrootd config file important bits - >>> >>> all.adminpath /var/spool/xrootd/admin >>> all.export /atlas r/w >>> oss.space public /data/* >>> xrootd.chksum max 3 adler32 /usr/bin/xrdadler32 >>> xrootd.seclib /usr/lib64/libXrdSec.so >>> sec.protocol /usr/lib64 unix >>> acc.authdb /etc/xrootd/auth_file >>> ofs.authorize >>> cms.space min 10g 15g >>> xrootd.trace all >>> >>> >>> >>> Both of the data partitions have sufficient space and inodes free. >>> >>> [root@atlasfs ~]# df -h /data/* >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/mapper/vgdata1-lvdata1 >>> 19T 17T 2.2T 89% /data/1 >>> /dev/mapper/vgdata2-lvdata2 >>> 19T 16T 3.0T 84% /data/2 >>> [root@atlasfs ~]# df -ih /data/* >>> Filesystem Inodes IUsed IFree IUse% Mounted on >>> /dev/mapper/vgdata1-lvdata1 >>> 3.7G 211K 3.7G 1% /data/1 >>> /dev/mapper/vgdata2-lvdata2 >>> 3.7G 165K 3.7G 1% /data/2 >>> >>> The exported area /atlas is on a disk with sufficient space and inodes : >>> >>> >>> [root@atlasfs ~]# df -ih /atlas >>> Filesystem Inodes IUsed IFree IUse% Mounted on >>> /dev/mapper/sysvg-LVsys >>> 1.6M 639K 926K 41% / >>> [root@atlasfs ~]# df -h /atlas >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/mapper/sysvg-LVsys >>> 24G 16G 7.7G 67% / >>> >>> >>> Yet the xrootd data server is reporting no space left on the device. >>> >>> >>> 150325 16:04:16 10920 ?:7@atlas28 XrootdProtocol: 0000 req=3007 dlen=71 >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0000 sending 23 >>> data bytes; status=0 >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0000 req=3000 dlen=21 >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0000 sending OK >>> 150325 16:04:16 10920 XrootdXeq: dbenjami.9782:7@atlas28 pub IPv4 login as >>> dbenjamin >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 req=3017 dlen=39 >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 rc=0 stat >>> /atlas/local/dbenjamin/xroot_copy_test/ >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0100 sending 36 >>> data bytes >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 req=3010 dlen=94 >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 open unmat >>> /atlas/local/dbenjamin/xroot_copy_test//NTUP_SMWZ.01122074._000001.root.1?oss.asize=3303014400 >>> >>> 150325 16:04:16 10920 ofs_open: dbenjami.9782:7@atlas28 Unable to create >>> /atlas/local/dbenjamin/xroot_copy_test/NTUP_SMWZ.01122074._000001.root.1; no >>> space left on device >>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0100 sending err >>> 3009: Unable to create >>> /atlas/local/dbenjamin/xroot_copy_test/NTUP_SMWZ.01122074._000001.root.1; no >>> space left on device >>> 150325 16:04:16 10920 XrootdXeq: dbenjami.9782:7@atlas28 disc 0:00:00 >>> 150325 16:09:19 10950 ?:18@atlas17 XrootdProtocol: 0000 req=3007 dlen=84 >>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdResponse: 0000 sending 23 >>> data bytes; status=0 >>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdProtocol: 0000 req=3000 dlen=18 >>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdResponse: 0000 sending OK >>> >>> Also using xrd - >>> >>> [dbenjamin@atlas28 bnl_openstack]$ xrd atlasfs.hep.anl.gov statvfs /atlas/local >>> r/w nodes: 1 r/w free space: 3064292 r/w utilization: 0 staging nodes: 1 staging >>> free space: 3064292 staging utilization: 0 >>> >>> [dbenjamin@atlas28 bnl_openstack]$ xrd atlasfs.hep.anl.gov queryspace /atlas/local >>> Disk space approximations (MB): >>> Total : 38139916 >>> Free : 5324910 >>> Used : 0 >>> Largest chunk : 3064292 >>> >>> >>> >>> What am I missing? >>> >>> Thanks, >>> >>> Doug Benjamin >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the XROOTD-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1