Hi Andy:
So, this redirector is a load-distributing redirector for a set of
internal caches at Glasgow; it has, by design, no route to the outside
world.
Point 1) setting: cms.delay servers 1
makes no difference to the behaviour of the redirector after a restart
of the services.
Point 2)
The result of xrdfs cephs03.beowulf.cluster:1094 locate * is
[cephs03.beowulf.cluster:1094] / > locate *
[::10.1.50.11]:1094 Server Read
10.1.50.11 is the internal IP address of cephc01.beowulf.cluster,
which is that one server which we agree is connected to the
redirector.
Is there supposed to be something else shown?
Looking at the logs (after stopping all services and then bringing
them up cleanly, to be sure I get a clean reconnection), I see:
++++++ Configuring manager role. . .
=====> all.manager cephs03.beowulf.cluster:1213
Config 1 manager(s) started.
Config effective /etc/xrootd/xrootd-cache.cfg ofs configuration:
all.role manager
ofs.maxdelay 60
ofs.persist manual hold 600 logdir /tmp/cache/.ofs/posc.log
ofs.trace 0
------ File system manager initialization completed.
Config warning: 'xrootd.prepare logdir' not specified; prepare
tracking disabled.
------ xrootd protocol initialization completed.
------ xrootd [log in to unmask]:1094 initialization completed.
191205 10:06:25 43562 cms_ClientMan: Connected to suspended
cephs03.beowulf.cluster v 3
191205 10:07:48 43562 cms_setStatus: Manager cephs03.beowulf.cluster resumed
- this is the end of the log, there are no further entries after
startup completes.
It's not clear to me that this shows me *why* the cmsd is suspended,
and then resumes?
(The redirector cmsd's own log from startup shows:
Config round robin scheduling in effect.
------ [log in to unmask] phase 2 manager initialization completed.
------ cmsd [log in to unmask]:1213 initialization completed.
191205 10:06:25 43527 Protocol: redirector.43550:20@cephs03 logged in.
191205 10:06:39 43542 Protocol: Primary server.273818:21@cephc01:1094 logged in.
191205 10:06:39 43542 Protocol: server.273818:21@cephc01:1094 system
ID: [log in to unmask] 1213cephs03.beowulf.cluster
=====> Routing for 10.1.50.11: local pub4 prv4
=====> Route all4: 10.1.50.11 Dest=[::10.1.50.11]:1094
191205 10:07:48 43526 Config: manager service enabled.
191205 10:07:48 43541 State: Status changed to active + staging
)
Similarly, the xrootd server on the *server node*, cephc01, shows:
------ File system server initialization completed.
Config warning: 'xrootd.prepare logdir' not specified; prepare
tracking disabled.
191205 10:06:39 273858 cms_Finder: Connected to cmsd via
/tmp/cache/.olb/olbd.admin
------ xrootd protocol initialization completed.
------ xrootd [log in to unmask]:1094 initialization completed.
191205 10:06:40 273857 XrdFileCache_Manager: info Cache::Purge() Started.
191205 10:06:40 273857 XrdFileCache_Manager: info Cache::Purge()
Finished, removed 0 data files, total size 0, bytes to remove at end:
0
191205 10:11:40 273857 XrdFileCache_Manager: info Cache::Purge() Started.
191205 10:11:40 273857 XrdFileCache_Manager: info Cache::Purge()
Finished, removed 0 data files, total size 0, bytes to remove at end:
0
there's no mention of any suspension in the log.
What should I try next? Is there a "troubleshooting guide" anywhere
for this kind of thing?
Sam
Sam
On Wed, 4 Dec 2019 at 23:08, Andrew Hanushevsky <[log in to unmask]> wrote:
>
> Hi Sam,
>
> Can I connect to it or is i behind a firewall? If I can connect do I need
> any credentials at least for he redirector? If it's not redirecting then
> it means it's in suspend status. Here are the reasons a redirector can
> suspend itself:
>
> 1) The number of nodes connected to the redirector has fallen below the
> mnimum required (configurable). The default is good for lage clusters but
> not for small ones. See, "delay servers" in:
> https://xrootd.slac.stanford.edu/doc/dev410/cms_config.htm#_Toc8247276
>
> 2) All nodes connected to the redirector have suspended themselves. This
> is usually caused by the xrootd being unable to connect to it's twin cmsd.
> The xrootd log will show why. You can figure this out by greping the
> redirector log for "suspend" using '-i' (case insenstive).
>
> You can also do something as follows to ascertain the status of your
> cluster:
>
> xrdfs <redirector> 'locate *'
>
> which will tell you the state of things at the redirector. Might
> interesting to see what it says.
>
> Andy
>
> On Wed, 4 Dec 2019, Sam Skipsey wrote:
>
> > Hi Andy,
> >
> > The config is literally copied from the boilerplate example in the
> > Disk Caching Proxy Cluster section in the documentation :) - I did the
> > replication because it's how the example does it.
> >
> > It is entirely correct that there is just one server for now - I was
> > building up the system from a single server once it actually worked.
> > (If you read up, you will see that in this case, it's not even
> > redirecting when there's just 1 server to consider.)
> >
> > Sam
> >
> > On Wed, 4 Dec 2019 at 22:38, Andrew Hanushevsky <[log in to unmask]> wrote:
> >>
> >> Hi Sam,
> >>
> >> Well, xrdmapc shows a redirector (cephs03) with one server (cephc01). I
> >> sthat not correct? I assume you expected more nodes, yes? The config file
> >> looks reasonable, though i doesn't have to repeat so much information.
> >> Also, for consistency oss.localroot should be the sameverywhere (though
> >> as specified it's essentially the same).
> >>
> >> Andy
> >>
> >> On Wed, 4 Dec 2019, Sam Skipsey wrote:
> >>
> >>> Hi Matevz,
> >>>
> >>> So:
> >>>
> >>> [root@cephs03 ~]# xrdmapc cephs03:1094
> >>> 0**** cephs03.beowulf.cluster:1094
> >>> Srv cephc01.beowulf.cluster:1094
> >>>
> >>>
> >>> I agree, though, that it looks like the cmsd and xrootd aren't
> >>> properly connecting - I see none of the expected stuff in the xrootd
> >>> log on the redirector.
> >>>
> >>> The config file (shared between the redirector and the server) is:
> >>>
> >>> #Clustered cache config (redirector also does authentication with lcmaps)
> >>>
> >>> all.manager cephs03.beowulf.cluster:1213
> >>> all.export /xrootd:/ stage r/o
> >>> all.export /root:/ stage r/o
> >>> all.export * stage r/o
> >>>
> >>> if cephs03+
> >>>
> >>> all.role manager
> >>> all.export /xrootd:/ stage r/o
> >>> all.export /root:/ stage r/o
> >>> all.export * stage r/o
> >>>
> >>> xrootd.trace emsg login stall redirect
> >>> xrd.trace conn
> >>> cms.trace defer files redirect stage
> >>>
> >>>
> >>>
> >>>
> >>> #the cmsd on the cache node
> >>> else if exec cmsd
> >>>
> >>> all.role server
> >>> all.export /xrootd:/ stage
> >>> all.export /root:/ stage
> >>> all.export * stage
> >>> oss.localroot /cache
> >>>
> >>> #the xrootd on the cache node
> >>> else
> >>>
> >>> all.role server
> >>>
> >>> ofs.osslib libXrdPss.so
> >>> pss.cachelib libXrdFileCache.so
> >>> pfc.ram 16g
> >>> pfc.trace info
> >>> pfc.diskusage 0.90 0.95
> >>> oss.localroot /cache/
> >>> cms.trace defer redirect stage
> >>>
> >>> all.export /xroot:/ r/w
> >>> all.export /root:/ r/w
> >>> all.export * r/w
> >>>
> >>> pss.origin localhost:1095
> >>>
> >>> xrd.allow host *.beowulf.cluster
> >>>
> >>> #inbound security protocol, for authenticating to the xrootd-ceph
> >>> #sec.protocol
> >>> xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
> >>> sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c
> >>> /etc/gateway/xrootd/sss.keytab.grp
> >>> sec.protbind localhost:1095 only sss
> >>>
> >>> xrd.report 127.0.0.1:9527 every 5s all
> >>>
> >>>
> >>> #try changing some pss options to tune this connection
> >>>
> >>> pss.setopt DebugLevel 3
> >>> pss.setopt ParStreamsPerPhyConn 15
> >>> pfc.blocksize 16M
> >>>
> >>>
> >>> fi
> >>>
> >>>
> >>> [Note that the shared secret is the same for all parts of the cluster,
> >>> at the moment]
> >>> At various points, I've messed with bits of the file (removing some of
> >>> the stages, or even entire exports, and a bunch of other things), but
> >>> this is the current config - using the default setup derived from the
> >>> Proxy config documentation pg 38, and adding your debugging lines.
> >>>
> >>> Sam
> >>>
> >>>
> >>>
> >>> On Tue, 3 Dec 2019 at 20:37, Matevz Tadel <[log in to unmask]> wrote:
> >>>>
> >>>> On 2019-12-03 06:38, Sam Skipsey wrote:
> >>>>> Okay, so what I have, including logs from all the relevant bits is below.
> >>>>>
> >>>>> I restarted all of the services (after adding the traces as requested)
> >>>>> at around 14:30, so all of the servers come up.
> >>>>> About 2 minutes later, I tried (from cephc01, using xrdcp) to copy a
> >>>>> file via the redirector xrootd (on cephs03). This only seems to show
> >>>>> up on the redirector xrootd itself - there's no record of anything in
> >>>>> the other logs as far as I can see.
> >>>>
> >>>> Hmmh, this is stange, cmsd on redirector never seems to ask the servers.
> >>>>
> >>>> redirector cmsd:
> >>>> 191203 11:57:35 51827 Select seeking /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> >>>> 191203 11:57:35 51827 redirector.51097:17@xrootd XrdLink: Setting ref to 2+-1 post=0
> >>>> 191203 11:57:35 51475 Dispatch server.6875:24@xcache-05:1094 for have dlen=107
> >>>> 191203 11:57:35 51475 server.6875:24@xcache-05 XrdLink: Setting ref to 1+1 post=0
> >>>> 191203 11:57:35 51341 XrdSched: running server inq=0
> >>>> 191203 11:57:35 51341 server.6875:24@xcache-05:1094 do_Have: /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> >>>>
> >>>> redirector xrootd:
> >>>> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 req=open dlen=106
> >>>> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 open r /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> >>>> 191203 11:57:35 51514 nobody.351:[log in to unmask] ofs_open: 0-660 fn=/store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> >>>> 191203 11:57:35 51109 Receive xrootd 26 bytes on 60327927
> >>>> 191203 11:57:35 51109 Decode xrootd redirects nobody.351:[log in to unmask] to xcache-05.t2.ucsd.edu:1094 /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> >>>> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 redirecting to xcache-05.t2.ucsd.edu:1094
> >>>> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdResponse: 0100 sending 25 data bytes; status=4004
> >>>> 191203 11:57:35 51514 nobody.351:[log in to unmask] ofs_close: use=0 fn=dummy
> >>>> 191203 11:57:35 51668 XrdSched: running cuser3.270:1838@cabinet-5-5-5 inq=0
> >>>>
> >>>> So, I'm suspecting redirector cmsd and xrootd never get connected, do you see something like this in the redirector xrootd log:
> >>>> #### Initialization
> >>>> ------ File system manager initialization completed.
> >>>> 191203 11:14:07 51097 XrootdAioReq: Max aio/req=8; aio/srv=4096; Quantum=131072
> >>>> 191203 11:14:07 51097 XrootdAioReq: Adding 18 aioreq objects.
> >>>> 191203 11:14:07 51097 XrootdAio: Adding 18 aio objects; 4096 pending.
> >>>> 191203 11:14:07 51097 XrdSched: scheduling xrootd protocol anchor in 3600 seconds
> >>>> 191203 11:14:07 51097 XrdSched: scheduling transit protocol anchor in 3600 seconds
> >>>> Config warning: 'xrootd.prepare logdir' not specified; prepare tracking disabled.
> >>>> 191203 11:14:07 51111 XrdXeq: Admin traffic thread started
> >>>> 191203 11:14:07 51109 XrdInet: Connected to xrootd.t2.ucsd.edu:2041 <-------
> >>>> 191203 11:14:07 51109 cms_ClientMan: Connected to xrootd.t2.ucsd.edu v 3 <-------
> >>>> 191203 11:14:07 51109 Hookup xrootd.t2.ucsd.edu qt=178ms rw=2
> >>>> ------ xrootd protocol initialization completed.
> >>>>
> >>>> What does 'xrdmapc redirector:1094' give you?
> >>>>
> >>>> Would you mind sharing your redirector config?
> >>>>
> >>>> Andy, do you have a better idea what to look for / try?
> >>>>
> >>>> Matevz
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> Sam
> >>>>>
> >>>>>
> >>>>>
> >>>>> redirector cmsd
> >>>>>
> >>>>> 191203 14:30:14 18957 Protocol: Primary server.202124:22@cephc01:1094 logged in.
> >>>>> 191203 14:30:14 18957 Protocol: server.202124:22@cephc01:1094 system
> >>>>> ID: [log in to unmask] 1213cephs03.beowulf.cluster
> >>>>> =====> Routing for 10.1.50.11: local pub4 prv4
> >>>>> =====> Route all4: 10.1.50.11 Dest=[::10.1.50.11]:1094
> >>>>> 191203 14:31:16 18939 Config: manager service enabled.
> >>>>> 191203 14:31:16 18953 State: Status changed to active + staging
> >>>>>
> >>>>>
> >>>>> ----
> >>>>> redirector xrootd
> >>>>>
> >>>>> ------ xrootd [log in to unmask]:1094 initialization completed.
> >>>>> 191203 14:32:26 18970 XrootdXeq: root.218779:20@cephc01 pvt IPv4 login
> >>>>> 191203 14:32:26 18970 root.218779:20@cephc01 XrootdResponse: sending
> >>>>> err 3011: No servers have read access to the file
> >>>>> 191203 14:32:26 18970 XrootdXeq: root.218779:20@cephc01 disc 0:00:00
> >>>>>
> >>>>>
> >>>>> -----
> >>>>> server cmsd
> >>>>>
> >>>>> ------ cmsd [log in to unmask]:46427 initialization completed.
> >>>>> 191203 14:30:14 218768 do_Login:: Primary server 218748 logged in;
> >>>>> data port is 1094
> >>>>> Config Connecting to 1 manager and 1 site.
> >>>>> 191203 14:30:14 218729 Config: server service enabled.
> >>>>> 191203 14:30:14 218770 State: Status changed to active + staging
> >>>>> 191203 14:30:14 218740 ManTree: Now connected to 1 root node(s)
> >>>>> 191203 14:30:14 218740 Protocol: Logged into cephs03
> >>>>>
> >>>>>
> >>>>> -----
> >>>>> server xrootd
> >>>>>
> >>>>> 191203 14:30:14 218765 cms_Finder: Connected to cmsd via
> >>>>> /tmp/cache/.olb/olbd.admin
> >>>>> ------ xrootd protocol initialization completed.
> >>>>> ------ xrootd [log in to unmask]:1094 initialization completed.
> >>>>> 191203 14:30:15 218764 XrdFileCache_Manager: info Cache::Purge() Started.
> >>>>> 191203 14:30:15 218764 XrdFileCache_Manager: info Cache::Purge()
> >>>>> Finished, removed 0 data files, total size 0, bytes to remove at end:
> >>>>> 0
> >>>>> 191203 14:35:15 218764 XrdFileCache_Manager: info Cache::Purge() Started.
> >>>>> 191203 14:35:15 218764 XrdFileCache_Manager: info Cache::Purge()
> >>>>> Finished, removed 0 data files, total size 0, bytes to remove at end:
> >>>>> 0
> >>>>>
> >>>>> On Mon, 2 Dec 2019 at 19:33, Matevz Tadel <[log in to unmask]> wrote:
> >>>>>>
> >>>>>> I'd try this:
> >>>>>>
> >>>>>> redirector:
> >>>>>> xrootd.trace emsg login stall redirect
> >>>>>> xrd.trace conn
> >>>>>> cms.trace defer files redirect stage
> >>>>>>
> >>>>>> server:
> >>>>>> # For debug, to see files being searched
> >>>>>> # cms.trace defer files redirect stage
> >>>>>> cms.trace defer redirect stage
> >>>>>>
> >>>>>>
> >>>>>> You say xrdmapc shows the configured servers, right?
> >>>>>>
> >>>>>> We had some trouble with ipv4/6 at ucsd lately, clients will be redirected to
> >>>>>> ipv6 servers only if they come in via ipv6 to the redirector.
> >>>>>>
> >>>>>> Can you restart redirector cmsd and then (after 30sec) look at:
> >>>>>>
> >>>>>> [1131] root@xrootd /var/log/xrootd/xcacheucsd# grep Routing cmsd.log | sort
> >>>>>>
> >>>>>> =====> Routing for bcache-1.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for bcache-1.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-00.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-01.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-02.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-03.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-04.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-05.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-06.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-07.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-08.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-09.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xcache-10.t2.ucsd.edu: local pub4 prv4
> >>>>>> =====> Routing for xcache-11.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xrd-cache-1.ultralight.org: local pub4 prv4 pub6 prv6
> >>>>>> =====> Routing for xrd-cache-2.ultralight.org: local pub4 prv4 pub6 prv6
> >>>>>>
> >>>>>> Matevz
> >>>>>>
> >>>>>> On 2019-12-02 11:24, Sam Skipsey wrote:
> >>>>>>> No, I explicitly did that. (As I noted, there's a typo for that in the
> >>>>>>> example, as it uses "rw" not "r/w" , which doesn't work).
> >>>>>>>
> >>>>>>> I've tried basically every variation of stage/nostage/ r/w / r/o at
> >>>>>>> different parts of the network, but the manager cmsd never seems to
> >>>>>>> actually consider the servers (even when I've already pre-staged the
> >>>>>>> file it's looking for by directly talking to the server xrootd service
> >>>>>>> and getting it to cache).
> >>>>>>>
> >>>>>>> Sam
> >>>>>>>
> >>>>>>> On Mon, 2 Dec 2019 at 19:09, Matevz Tadel <[log in to unmask]> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Before I go looking at what's wrong on the web page, have a look at this, page 21:
> >>>>>>>>
> >>>>>>>> https://indico.cern.ch/event/727208/contributions/3444604/
> >>>>>>>>
> >>>>>>>> Maybe you're missing the r/w for xrootd, stage r/o for cmsd trick?
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Matevz
> >>>>>>>>
> >>>>>>>> On 2019-11-28 08:22, Sam Skipsey wrote:
> >>>>>>>>> Hello everyone,
> >>>>>>>>>
> >>>>>>>>> So, I have another question, working entirely from the documentation
> >>>>>>>>> on xrootd.org
> >>>>>>>>>
> >>>>>>>>> In the documentation for cache configuration, there's an example of
> >>>>>>>>> how to set up a cluster of disk caching proxies:
> >>>>>>>>>
> >>>>>>>>> https://xrootd.slac.stanford.edu/doc/dev410/pss_config.pdf [page 38,
> >>>>>>>>> you can't copy it because weirdly its an image]
> >>>>>>>>>
> >>>>>>>>> I'm following that exactly (except for fixing the typo where the
> >>>>>>>>> example has an export using "rw" and not "r/w" as an option), and,
> >>>>>>>>> well, it just doesn't seem to work.
> >>>>>>>>>
> >>>>>>>>> If I talk directly to the server that the proxies talk to: I can get a file.
> >>>>>>>>> If I talk to an individual proxy: I can also get a file (and it is cached)
> >>>>>>>>> If I talk to the *redirector*, I get, with debugging on "Open has
> >>>>>>>>> returned with status [ERROR] Server responded with an error: [3011] No
> >>>>>>>>> servers have read access to the file"
> >>>>>>>>>
> >>>>>>>>> The redirector logs show that the cmsd on the proxy logs in (and is
> >>>>>>>>> listed as a "server" in its list of servers), and the proxy cmsd logs
> >>>>>>>>> also show that it happily registers to the redirector.
> >>>>>>>>>
> >>>>>>>>> How do I debug this?
> >>>>>>>>>
> >>>>>>>>> I've already tried adding and removing options to the various exports,
> >>>>>>>>> making sure that all the relevant ports are open, etc.
> >>>>>>>>>
> >>>>>>>>> Sam
> >>>>>>>>>
> >>>>>>>>> ########################################################################
> >>>>>>>>> Use REPLY-ALL to reply to list
> >>>>>>>>>
> >>>>>>>>> To unsubscribe from the XROOTD-L list, click the following link:
> >>>>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>>
> >>> ########################################################################
> >>> Use REPLY-ALL to reply to list
> >>>
> >>> To unsubscribe from the XROOTD-L list, click the following link:
> >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
> >>>
> >
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
