This doesn’t seems like a CMS issue because the client is talking the xrootd instance of the data server.
Can you become the user that run xrootd and try to create a file on /data/1 or 2?
Wei Yang | [log in to unmask] | 650-926-3338(O)
On Mar 25, 2015, at 2:59 PM, Doug Benjamin <[log in to unmask]> wrote:
> Hi Patrick,
>
> oss.space public /data/* directive is used to point where to hold the data files.
>
> /atlas/local/* contains symlinks to the data files.
>
> I removed the cms.space directive from the configuration file without success.
>
> Also SELinux is running in permissive mode so I do not think that that is the issue.
>
>
> Doug
>
> On 03/25/2015 04:41 PM, Patrick McGuigan wrote:
>> Hi Doug,
>>
>> My suspicion:
>>
>> 1) I don't see where /data/* is supporting anything under the /atlas namespace, so I won't look at that.
>>
>> 2) You are specifying that xrootd keep a minimum free space of 10GB and you only have 7.7GB free under /atlas. Hence it won't write there.
>>
>> Patrick
>>
>> On 03/25/2015 04:32 PM, Doug Benjamin wrote:
>>> Hi,
>>>
>>> I having a heck of a time trying to debug this problem.
>>>
>>> I have a stand alone file server with two partitions /data/1 and /data/2 that
>>> are glued together with xrootd
>>>
>>> Here is the xrootd config file important bits -
>>>
>>> all.adminpath /var/spool/xrootd/admin
>>> all.export /atlas r/w
>>> oss.space public /data/*
>>> xrootd.chksum max 3 adler32 /usr/bin/xrdadler32
>>> xrootd.seclib /usr/lib64/libXrdSec.so
>>> sec.protocol /usr/lib64 unix
>>> acc.authdb /etc/xrootd/auth_file
>>> ofs.authorize
>>> cms.space min 10g 15g
>>> xrootd.trace all
>>>
>>>
>>>
>>> Both of the data partitions have sufficient space and inodes free.
>>>
>>> [root@atlasfs ~]# df -h /data/*
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/mapper/vgdata1-lvdata1
>>> 19T 17T 2.2T 89% /data/1
>>> /dev/mapper/vgdata2-lvdata2
>>> 19T 16T 3.0T 84% /data/2
>>> [root@atlasfs ~]# df -ih /data/*
>>> Filesystem Inodes IUsed IFree IUse% Mounted on
>>> /dev/mapper/vgdata1-lvdata1
>>> 3.7G 211K 3.7G 1% /data/1
>>> /dev/mapper/vgdata2-lvdata2
>>> 3.7G 165K 3.7G 1% /data/2
>>>
>>> The exported area /atlas is on a disk with sufficient space and inodes :
>>>
>>>
>>> [root@atlasfs ~]# df -ih /atlas
>>> Filesystem Inodes IUsed IFree IUse% Mounted on
>>> /dev/mapper/sysvg-LVsys
>>> 1.6M 639K 926K 41% /
>>> [root@atlasfs ~]# df -h /atlas
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/mapper/sysvg-LVsys
>>> 24G 16G 7.7G 67% /
>>>
>>>
>>> Yet the xrootd data server is reporting no space left on the device.
>>>
>>>
>>> 150325 16:04:16 10920 ?:7@atlas28 XrootdProtocol: 0000 req=3007 dlen=71
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0000 sending 23
>>> data bytes; status=0
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0000 req=3000 dlen=21
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0000 sending OK
>>> 150325 16:04:16 10920 XrootdXeq: dbenjami.9782:7@atlas28 pub IPv4 login as
>>> dbenjamin
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 req=3017 dlen=39
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 rc=0 stat
>>> /atlas/local/dbenjamin/xroot_copy_test/
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0100 sending 36
>>> data bytes
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 req=3010 dlen=94
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdProtocol: 0100 open unmat
>>> /atlas/local/dbenjamin/xroot_copy_test//NTUP_SMWZ.01122074._000001.root.1?oss.asize=3303014400
>>>
>>> 150325 16:04:16 10920 ofs_open: dbenjami.9782:7@atlas28 Unable to create
>>> /atlas/local/dbenjamin/xroot_copy_test/NTUP_SMWZ.01122074._000001.root.1; no
>>> space left on device
>>> 150325 16:04:16 10920 dbenjami.9782:7@atlas28 XrootdResponse: 0100 sending err
>>> 3009: Unable to create
>>> /atlas/local/dbenjamin/xroot_copy_test/NTUP_SMWZ.01122074._000001.root.1; no
>>> space left on device
>>> 150325 16:04:16 10920 XrootdXeq: dbenjami.9782:7@atlas28 disc 0:00:00
>>> 150325 16:09:19 10950 ?:18@atlas17 XrootdProtocol: 0000 req=3007 dlen=84
>>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdResponse: 0000 sending 23
>>> data bytes; status=0
>>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdProtocol: 0000 req=3000 dlen=18
>>> 150325 16:09:19 10950 xrootd.7213:18@atlas17 XrootdResponse: 0000 sending OK
>>>
>>> Also using xrd -
>>>
>>> [dbenjamin@atlas28 bnl_openstack]$ xrd atlasfs.hep.anl.gov statvfs /atlas/local
>>> r/w nodes: 1 r/w free space: 3064292 r/w utilization: 0 staging nodes: 1 staging
>>> free space: 3064292 staging utilization: 0
>>>
>>> [dbenjamin@atlas28 bnl_openstack]$ xrd atlasfs.hep.anl.gov queryspace /atlas/local
>>> Disk space approximations (MB):
>>> Total : 38139916
>>> Free : 5324910
>>> Used : 0
>>> Largest chunk : 3064292
>>>
>>>
>>>
>>> What am I missing?
>>>
>>> Thanks,
>>>
>>> Doug Benjamin
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
|