Hi Adrian,
OK, so the partition being used here has only 176 GB left on it and the
alloc parameter says each request needs to reserve 20 GB of that. So, if you
get 8 requests in short order the 9th one will get not enough space error.
That won't be readjusted until 10 minutes go by (see the oss.cachescan
directive). Thus far, I don't see anything unusual given the circumstances.
Andy
-----Original Message-----
From: Adrian Sevcenco
Sent: Monday, January 30, 2017 1:23 AM
To: Andrew Hanushevsky
Cc: [log in to unmask]
Subject: Re: strange "no space left" messages
On 01/30/2017 09:41 AM, Andrew Hanushevsky wrote:
> Hi Adrian,
Hi!
> How about checking if you jave enough inodes? What does "df -h" and "df
> -hi" say?
yeap, i already checked :
aliprod@storage02: ~ $ df -h /storage*
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.0T 8.8T 192G 98% /storage01
/dev/sdb1 9.0T 8.8T 176G 99% /storage02
aliprod@storage02: ~ $ df -hi /storage*
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 1.2G 599K 1.2G 1% /storage01
/dev/sdb1 1.2G 185K 1.2G 1% /storage02
oss.localroot is on /storage01
Thank you!
Adrian
>
> Andy
>
> On Sun, 29 Jan 2017, Adrian Sevcenco wrote:
>
>> On 01/27/2017 08:05 PM, Andrew Hanushevsky wrote:
>>> Hi Adrian,
>> Hi!
>>
>>> The problem likey is that you are getting a lot of requests at once and
>>> each time you get a request to create a file he oss.alloc directive says
>>> that 20% of the requested amount (or 4 GB if the amount was not
>>> specified) is to be reerved. If you get enough allocation requests the
>>> sum of the reserve amounts will exceed the available space. At that
>>> point you get the error. I assume the error comes from the server not
>>> the redirector.
>> so i set up
>> oss.alloc 20G 0 0
>>
>> but today i got a lot of messages of "no space left on device" and
>> this is happening only on this server ..
>> ssh_run_cmd storage02 grep "no space left"
>> /home/aliprod/alicexrdrun/logs/server/xrdlog | wc -l
>> 452
>>
>> space is enough :
>> [localhost:1094] / > query space /
>> oss.cgroup=public&oss.space=19685898543104&oss.free=352717732414&oss.maxf=184908956313&oss.used=49994824333&oss.quota=-1
>>
>>
>> Any idea how can i fix this?
>> my only idea now is to declare the space and files lost to the
>> experiment (with a full listing of the current files) and start the
>> server from scratch ...
>>
>> Thank you!!
>> Adrian
>>
>>>
>>> Andy
>>>
>>> On Fri, 27 Jan 2017, Adrian Sevcenco wrote:
>>>
>>>> Hi! I have a strange situation where i receive from time to time
>>>> errors like this :
>>>> 170127 13:21:18 18229 aliprod.14481:34@[::ffff:172.18.3.237]
>>>> XrootdResponse: 0100 sending err 3009: Unable to create
>>>> /home/aliprod/data/13/00751/b616b1c8-e482-11e6-98ad-779a6f1c6846; no
>>>> space left on device
>>>>
>>>> the thing is that i have plenty of space left :
>>>> aliprod@storage02: server $ df -h | grep storage
>>>> /dev/sda1 9.0T 8.8T 216G 98% /storage01
>>>> /dev/sdb1 9.0T 8.8T 203G 98% /storage02
>>>>
>>>> on the server i have this setting :
>>>> aliprod@storage02: server $ grep alloc xrootd.xrootd.cf
>>>> oss.alloc 20G 20 0
>>>>
>>>> and on redirector i have :
>>>> aliprod@rd: manager $ grep cms.space xrootd.xrootd.cf
>>>> cms.space min 48g 16g
>>>>
>>>> how can i debug why i get this errors?
>>>>
>>>> Thank you!
>>>> Adrian
>>>>
>>>>
>>>> ########################################################################
>>>>
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>>
>>>
>>
>>
>> --
>> ----------------------------------------------
>> Adrian Sevcenco, Ph.D. |
>> Institute of Space Science - ISS, Romania |
>> adrian.sevcenco at {cern.ch,spacescience.ro} |
>> ----------------------------------------------
>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>
>
--
----------------------------------------------
Adrian Sevcenco, Ph.D. |
Institute of Space Science - ISS, Romania |
adrian.sevcenco at {cern.ch,spacescience.ro} |
----------------------------------------------
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
