Hi Andy,
I took your sample and replaced necessary entries with ucsd cluster
specific data.
When I compared your configuration with Matevz's configuration, I saw he
had one additional oss.space directive [1]. When I include it, the
caching plugin library failed to instantiate [2].
Shoud oss.space be removed from configuration?
I also added statlib directive. Where I implemented XrdStat info
function in XrdFileCache library [3]. With this setup and the stat info
implementation the cluster behaves same as before -- the redirector goes
to a random proxy server. I looked at the logs and it looks that custom
stat function was never called on the proxy server.
Thanks,
Alja
[1] https://github.com/alja/vaya/blob/master/proxy-cluster.cfg#L47
[2] http://uaf-2.t2.ucsd.edu/~alja/xrootd.log
[3] https://github.com/alja/xrootd/compare/stat
On 12/06/14 08:38, Andrew Hanushevsky wrote:
> Hi Matevz,
>
> Below is the sample config fle that will appear in he manual. Once we
> resolve the "statlib" issue, I will add that as well.
>
> Andy
>
> # Tell everyone who the manager is
> #
> all.manager redirector:1213
>
> # Everyone will export /data red-only with the stage option.
> # The stage option requests that if the file isn't found in
> # the cluster the redirector should send the client to a PFC
> # server with enough space to cache the file.
> #
> all.export /data stage r/o
>
> # First configure the redirector using if-else-fi
> #
> if redirector
> all.role manager
> else
> # All the PFC's act as virtual data servers
> #
> all.role server
>
> # However, load the proxy plugin and the disk caching plugin.
> # In reality it will function as a disk caching proxy server.
> #
> ofs.osslib libXrdPss.so
> pss.cachelib libFileCache.so
>
> # Tell the proxy where the data is coming from (arbitrary).
> #
> pss.origin someserver.domain.org
>
> # Tell the PFC's where the disk cache resides (arbitrary).
> #
> pfc.cachedir /pfc-cache
>
>
>
> On Fri, 5 Dec 2014, Matevz Tadel wrote:
>
>> Hi,
>>
>> Alja and I have been testing a caching-proxy cluster, the idea is that
>> there are several independent caching-proxies all reporting to a
>> common redirector so that one gets more disk space, better performance
>> and redundancy. Here are the scripts we've been using:
>>
>> http://uaf-2.t2.ucsd.edu/~matevz/xrd/proxy-cluster/
>>
>> pooxy-klus.cfg is the config and start-pooxy.sh the startup script.
>> cabinet-10-10-10 is the redirector.
>>
>> All servers export /store with the stage option and there is a trivial
>> oss.stagecmd /opt/stage-fake.pl that basically just touches the file,
>> assuming that proxy will then pull in the parts of the file that are
>> actually needed.
>>
>> On proxy servers, we configure xrootd to use
>> ofs.osslib /opt/xrootd/lib64/libXrdPss-4.so
>> pss.origin xrootd.t2.ucsd.edu:1094
>> pss.cachelib libXrdFileCache-4.so
>> while cmsd just gets
>> xrootd.fslib /opt/xrootd/lib64/libXrdOfs-4.so
>> so that it is able to look what files already exist on the disk.
>>
>> The whole thing works ... but apparently stage option is too strong.
>> When opening a file that exists on proxy machine A it also happens
>> that the client gets redirected to machine X that does not have it --
>> so it has to be pulled in from remote location one more time.
>>
>> In particular, this happens after the cluster restart. xrdcp is used
>> to access the files. xrdcp does stat before the open. both are
>> redirected randomly the first time after restart ... after that the
>> server that was selected for open keeps getting selected consistently.
>>
>> Any ideas what we could do? Or should we be looking for a completely
>> different solution?
>>
>> Could it be proxy cmsds are not seeing the files right? I tried
>> bumping trace to debug but there is no info there about found/not
>> found cmsd queries.
>>
>> Cheers,
>> Matevz
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the XROOTD-DEV list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the XROOTD-DEV list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-DEV list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-DEV&A=1
|