On 01/31/2017 02:19 AM, Andrew Hanushevsky wrote:
> Hi Adrian,
Hi!
> OK, so the partition being used here has only 176 GB left on it and the
> alloc parameter says each request needs to reserve 20 GB of that. So, if
> you get 8 requests in short order the 9th one will get not enough space
> error. That won't be readjusted until 10 minutes go by (see the
> oss.cachescan directive). Thus far, I don't see anything unusual given
> the circumstances.
great, thank for explanation! beside lowering the oss.alloc is there a
way to limit the number of write requests? maybe some combination of
cms.sched on redirector?
Thank you!
Adrian
>
> Andy
>
> -----Original Message----- From: Adrian Sevcenco
> Sent: Monday, January 30, 2017 1:23 AM
> To: Andrew Hanushevsky
> Cc: [log in to unmask]
> Subject: Re: strange "no space left" messages
>
> On 01/30/2017 09:41 AM, Andrew Hanushevsky wrote:
>> Hi Adrian,
> Hi!
>
>> How about checking if you jave enough inodes? What does "df -h" and "df
>> -hi" say?
> yeap, i already checked :
>
> aliprod@storage02: ~ $ df -h /storage*
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 9.0T 8.8T 192G 98% /storage01
> /dev/sdb1 9.0T 8.8T 176G 99% /storage02
>
> aliprod@storage02: ~ $ df -hi /storage*
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/sda1 1.2G 599K 1.2G 1% /storage01
> /dev/sdb1 1.2G 185K 1.2G 1% /storage02
>
> oss.localroot is on /storage01
>
> Thank you!
> Adrian
>
>>
>> Andy
>>
>> On Sun, 29 Jan 2017, Adrian Sevcenco wrote:
>>
>>> On 01/27/2017 08:05 PM, Andrew Hanushevsky wrote:
>>>> Hi Adrian,
>>> Hi!
>>>
>>>> The problem likey is that you are getting a lot of requests at once and
>>>> each time you get a request to create a file he oss.alloc directive
>>>> says
>>>> that 20% of the requested amount (or 4 GB if the amount was not
>>>> specified) is to be reerved. If you get enough allocation requests the
>>>> sum of the reserve amounts will exceed the available space. At that
>>>> point you get the error. I assume the error comes from the server not
>>>> the redirector.
>>> so i set up
>>> oss.alloc 20G 0 0
>>>
>>> but today i got a lot of messages of "no space left on device" and
>>> this is happening only on this server ..
>>> ssh_run_cmd storage02 grep "no space left"
>>> /home/aliprod/alicexrdrun/logs/server/xrdlog | wc -l
>>> 452
>>>
>>> space is enough :
>>> [localhost:1094] / > query space /
>>> oss.cgroup=public&oss.space=19685898543104&oss.free=352717732414&oss.maxf=184908956313&oss.used=49994824333&oss.quota=-1
>>>
>>>
>>>
>>> Any idea how can i fix this?
>>> my only idea now is to declare the space and files lost to the
>>> experiment (with a full listing of the current files) and start the
>>> server from scratch ...
>>>
>>> Thank you!!
>>> Adrian
>>>
>>>>
>>>> Andy
>>>>
>>>> On Fri, 27 Jan 2017, Adrian Sevcenco wrote:
>>>>
>>>>> Hi! I have a strange situation where i receive from time to time
>>>>> errors like this :
>>>>> 170127 13:21:18 18229 aliprod.14481:34@[::ffff:172.18.3.237]
>>>>> XrootdResponse: 0100 sending err 3009: Unable to create
>>>>> /home/aliprod/data/13/00751/b616b1c8-e482-11e6-98ad-779a6f1c6846; no
>>>>> space left on device
>>>>>
>>>>> the thing is that i have plenty of space left :
>>>>> aliprod@storage02: server $ df -h | grep storage
>>>>> /dev/sda1 9.0T 8.8T 216G 98% /storage01
>>>>> /dev/sdb1 9.0T 8.8T 203G 98% /storage02
>>>>>
>>>>> on the server i have this setting :
>>>>> aliprod@storage02: server $ grep alloc xrootd.xrootd.cf
>>>>> oss.alloc 20G 20 0
>>>>>
>>>>> and on redirector i have :
>>>>> aliprod@rd: manager $ grep cms.space xrootd.xrootd.cf
>>>>> cms.space min 48g 16g
>>>>>
>>>>> how can i debug why i get this errors?
>>>>>
>>>>> Thank you!
>>>>> Adrian
>>>>>
>>>>>
>>>>> ########################################################################
>>>>>
>>>>>
>>>>> Use REPLY-ALL to reply to list
>>>>>
>>>>> To unsubscribe from the XROOTD-L list, click the following link:
>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>>>
>>>>
>>>
>>>
>>> --
>>> ----------------------------------------------
>>> Adrian Sevcenco, Ph.D. |
>>> Institute of Space Science - ISS, Romania |
>>> adrian.sevcenco at {cern.ch,spacescience.ro} |
>>> ----------------------------------------------
>>>
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the XROOTD-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
>>>
>>
>
>
--
----------------------------------------------
Adrian Sevcenco, Ph.D. |
Institute of Space Science - ISS, Romania |
adrian.sevcenco at {cern.ch,spacescience.ro} |
----------------------------------------------
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
