Doug,
There is one redirector -- it doesn't seem to have a problem with any of
this -- and one data server. The "problem" namespace is built on five
underlying filesystems:
/dev/sde1 9.0T 4.4T 4.2T 51% /disk/data3
/dev/sdi1 9.0T 4.4T 4.2T 51% /disk/data4
/dev/sdd1 9.0T 4.2T 4.4T 49% /disk/data5
/dev/sdj1 19T 6.3T 12T 35% /disk/data6
/dev/sdf1 19T 6.3T 12T 35% /disk/data7
The other namespace is built on a single filesystem:
/dev/sdg1 5.4T 282G 4.9T 6% /disk/data0
There is no fancy configuration:
all.manager <redacted>:1213
if named srm
all.export /srm
oss.localroot /disk/data3/ns-srm
oss.space public /disk/data3/xrddata
oss.space public /disk/data4/xrddata
oss.space public /disk/data5/xrddata
oss.space public /disk/data6/xrddata
oss.space public /disk/data7/xrddata
xrd.port 1094
fi
if named atlasc
all.export /atlasC
oss.localroot /disk/data0/ns-atlasC
oss.space public /disk/data0/xrddata
xrd.port 1095
fi
xrd.allow host <redacted>
xrd.allow host *.at3f
cms.allow host <redacted>
cms.allow host *.at3f
all.role server
Plus some security configuration...
-paul
> Hi Paul,
>
> What is the configuration of your xrootd system? How many data
> servers, redirector etc.
>
> thanks,
>
> Doug
>
> On 10/07/2011 02:37 PM, Paul T. Keener wrote:
>> Lukasz,
>>
>>> 2011/10/7 Paul T. Keener <[log in to unmask]>:
>>>> I see load averages of 140 to 200 or more and I see a high percent
>>>> of CPU
>>>> time in wait states (70-80%) with a large number of interrupts
>>>> (10-15K),
>>>> both of these reported by dstat.
>>>
>>> Hi Paul,
>>>
>>> if a significant fraction of the CPU is used by ksoftirqd it's
>>> usually an indication that the kernel has troubles managing the
>>> interrupts coming from the NIC (too much data coming from or going to
>>> the wire). If possible, try to put the network driver for your device
>>> in polling mode (the kernel then devotes one core to handling the
>>> stuff coming from NIC and disables the NIC's interrupts), you should
>>> see an improvement.
>>
>> I will look at this, but if it really is a network problem, wouldn't I
>> see it regardless of which xrootd namespace I was talking to?
>>
>> Thanks.
>>
>> -paul
|