 |
 |
Hello all, Just wanted to let the list know that the solution to this was to add the user_xattr option in the fstab entry for all the filesystems on my data server nodes. We switched to ext4, from ext3, but they still needed the user_xattr option. The docs describe this here: http://xrootd.org/doc/prod/frm_migr.htm#_Toc284350516 Though I found that the command to test is: frm_admin -c /path/to/cfile convert -t old2new Cheers, ksb On 02/17/12 16:53, Keith Beattie wrote: > Here it is: > > ----- > all.role server > all.manager santa 3121 > > all.adminpath /export/data/xrd/admin > all.pidpath /export/data/xrd/admin > > all.export / nostage > oss.localroot /export/data/xrd/ns > > oss.space public /export/data/xrd/data > oss.space public /export/data/xrd/data > oss.space public /export/data1/xrd/data > oss.space public /export/data1/xrd/data > oss.space public /export/data2/xrd/data > oss.space public /export/data2/xrd/data > oss.space public /export/data3/xrd/data > oss.space public /export/data3/xrd/data > ------ > > On 02/17/12 16:46, Andrew Hanushevsky wrote: >> Hi Keith, >> >> Please send me your config file. >> >> Andy >> >> On Fri, 17 Feb 2012, Keith Beattie wrote: >> >>> ok, I spoke too soon, this problem seems to be back. >>> >>> I've added data server nodes, now 13 of them, each with 3 drives and >>> only the first drive is getting populated with any data - except for 1 >>> of them. After loading the cluster with lots of files, the first drive >>> on all of them are at 100% usage with the rest of the drives at 3-5% >>> usage. The one outlier's 2nd drive is at 20% usage. All these drives >>> are 6-700 GB total size. >>> >>> Digging deeper on each data server node, on all of them (except for >>> that outlier) oss.localroot (/export/data/xrd/ns/) is not getting >>> populated with symlinks into the 'oss.space public' locations >>> (/export/data*/xrd/data/) but the data files are directly there. >>> Further, there *are* lots of files under the oss.space public >>> locations (e.g. /export/data2/xrd/data/public/1B/271B3B4FDB000000176%) >>> but they are all empty! >>> >>> On the one outlier there are symlinks to the 2nd drive, but that only >>> accounts for ~200 of the ~1000 entries under oss.localroot there. >>> >>> The config files for each of the data server nodes are identical. >>> >>> I tried duplicating the oss.space public lines in the configs, as was >>> previously suggested but that didn't seem to change anything. >>> >>> I'll try some other test, like using xrootd 3.0.5, different configs, >>> but these test take a while to complete, so I'm hoping this rings some >>> bells here as to what I might be missing. >>> >>> Thanks, >>> ksb >>> >>> On 01/30/12 12:47, Keith Beattie wrote: >>>> to follow up here... >>>> >>>> I've tried it again, with the same configs, and now it works. To the >>>> best of my knowledge nothing is different, except the absence of this >>>> problem - both drives on all the clients are receiving data. >>>> >>>> I'd rather know what fixed it, but I'll take it and move on. >>>> >>>> Thanks for your help, >>>> ksb >>>> >>>> On 12/20/11 5:08 PM, Keith Beattie wrote: >>>>> Here you go, attached. Both from xrootd and cmsd. >>>>> >>>>> Thanks, >>>>> ksb >>>>> >>>>> On 12/16/11 11:55 PM, Wilko Kroeger wrote: >>>>>> >>>>>> Hello Keith >>>>>> >>>>>> I am using the release 3.1.0 at slac with cache systems (using >>>>>> oss.space >>>>>> public ...) and files are placed in all cache systems. >>>>>> Could you maybe post the startup message from the xrdlog? >>>>>> >>>>>> Cheers, >>>>>> Wilko >>>>>> >>>>>> >>>>>> >>>>>> On Fri, 16 Dec 2011, Keith Beattie wrote: >>>>>> >>>>>>> Hi Andy, >>>>>>> >>>>>>> I'm using 3.1.0, built from source. I tried doubling the oss.cache >>>>>>> lines and the servers failed to start. When using oss.space rather >>>>>>> than oss.cache, doubling those lines isn't fatal, but regardless I >>>>>>> get >>>>>>> the same behavior - data only to the first entry. >>>>>>> >>>>>>> Thanks, >>>>>>> ksb >>>>>>> >>>>>>> On 12/16/11 6:06 PM, Andrew Hanushevsky wrote: >>>>>>>> Hi Keith, >>>>>>>> >>>>>>>> You may not be missing anything, depending on what version you are >>>>>>>> using. There was a bug that exhibited exactly this behaviour. The >>>>>>>> bypass >>>>>>>> (other than upgrading to atleast 3.0.2) is to list the oss.cache >>>>>>>> entries >>>>>>>> twice for each path. >>>>>>>> >>>>>>>> Andy >>>>>>>> >>>>>>>> On Fri, 16 Dec 2011, Keith Beattie wrote: >>>>>>>> >>>>>>>>> Hello all, >>>>>>>>> >>>>>>>>> This seems like I'm missing something simple but can't seem to >>>>>>>>> find >>>>>>>>> it. >>>>>>>>> >>>>>>>>> On my data servers nodes, only the first 'oss.cache' entry is >>>>>>>>> getting >>>>>>>>> data, the following disks don't seem to ever get written to. >>>>>>>>> Below is >>>>>>>>> an example of what the config looks like on the data nodes (not >>>>>>>>> santa >>>>>>>>> where the manager is running). What am I missing? >>>>>>>>> >>>>>>>>> ---- >>>>>>>>> all.role server >>>>>>>>> all.manager santa 3121 >>>>>>>>> >>>>>>>>> all.export / >>>>>>>>> oss.localroot /data/ns >>>>>>>>> oss.cache public /data/xrddata >>>>>>>>> oss.cache public /data1/xrddata >>>>>>>>> ---- >>>>>>>>> >>>>>>>>> i.e. /data/xrddata gets filled, /data1/xrddata does not. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> ksb >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the XROOTD-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the XROOTD-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1