Hi Matevz,
So:
[root@cephs03 ~]# xrdmapc cephs03:1094
0**** cephs03.beowulf.cluster:1094
Srv cephc01.beowulf.cluster:1094
I agree, though, that it looks like the cmsd and xrootd aren't
properly connecting - I see none of the expected stuff in the xrootd
log on the redirector.
The config file (shared between the redirector and the server) is:
#Clustered cache config (redirector also does authentication with lcmaps)
all.manager cephs03.beowulf.cluster:1213
all.export /xrootd:/ stage r/o
all.export /root:/ stage r/o
all.export * stage r/o
if cephs03+
all.role manager
all.export /xrootd:/ stage r/o
all.export /root:/ stage r/o
all.export * stage r/o
xrootd.trace emsg login stall redirect
xrd.trace conn
cms.trace defer files redirect stage
#the cmsd on the cache node
else if exec cmsd
all.role server
all.export /xrootd:/ stage
all.export /root:/ stage
all.export * stage
oss.localroot /cache
#the xrootd on the cache node
else
all.role server
ofs.osslib libXrdPss.so
pss.cachelib libXrdFileCache.so
pfc.ram 16g
pfc.trace info
pfc.diskusage 0.90 0.95
oss.localroot /cache/
cms.trace defer redirect stage
all.export /xroot:/ r/w
all.export /root:/ r/w
all.export * r/w
pss.origin localhost:1095
xrd.allow host *.beowulf.cluster
#inbound security protocol, for authenticating to the xrootd-ceph
#sec.protocol
xrootd.seclib /opt/xrootd/lib64/libXrdSec.so
sec.protocol sss -s /etc/gateway/xrootd/sss.keytab.grp -c
/etc/gateway/xrootd/sss.keytab.grp
sec.protbind localhost:1095 only sss
xrd.report 127.0.0.1:9527 every 5s all
#try changing some pss options to tune this connection
pss.setopt DebugLevel 3
pss.setopt ParStreamsPerPhyConn 15
pfc.blocksize 16M
fi
[Note that the shared secret is the same for all parts of the cluster,
at the moment]
At various points, I've messed with bits of the file (removing some of
the stages, or even entire exports, and a bunch of other things), but
this is the current config - using the default setup derived from the
Proxy config documentation pg 38, and adding your debugging lines.
Sam
On Tue, 3 Dec 2019 at 20:37, Matevz Tadel <[log in to unmask]> wrote:
>
> On 2019-12-03 06:38, Sam Skipsey wrote:
> > Okay, so what I have, including logs from all the relevant bits is below.
> >
> > I restarted all of the services (after adding the traces as requested)
> > at around 14:30, so all of the servers come up.
> > About 2 minutes later, I tried (from cephc01, using xrdcp) to copy a
> > file via the redirector xrootd (on cephs03). This only seems to show
> > up on the redirector xrootd itself - there's no record of anything in
> > the other logs as far as I can see.
>
> Hmmh, this is stange, cmsd on redirector never seems to ask the servers.
>
> redirector cmsd:
> 191203 11:57:35 51827 Select seeking /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> 191203 11:57:35 51827 redirector.51097:17@xrootd XrdLink: Setting ref to 2+-1 post=0
> 191203 11:57:35 51475 Dispatch server.6875:24@xcache-05:1094 for have dlen=107
> 191203 11:57:35 51475 server.6875:24@xcache-05 XrdLink: Setting ref to 1+1 post=0
> 191203 11:57:35 51341 XrdSched: running server inq=0
> 191203 11:57:35 51341 server.6875:24@xcache-05:1094 do_Have: /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
>
> redirector xrootd:
> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 req=open dlen=106
> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 open r /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> 191203 11:57:35 51514 nobody.351:[log in to unmask] ofs_open: 0-660 fn=/store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> 191203 11:57:35 51109 Receive xrootd 26 bytes on 60327927
> 191203 11:57:35 51109 Decode xrootd redirects nobody.351:[log in to unmask] to xcache-05.t2.ucsd.edu:1094 /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root
> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 redirecting to xcache-05.t2.ucsd.edu:1094
> 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdResponse: 0100 sending 25 data bytes; status=4004
> 191203 11:57:35 51514 nobody.351:[log in to unmask] ofs_close: use=0 fn=dummy
> 191203 11:57:35 51668 XrdSched: running cuser3.270:1838@cabinet-5-5-5 inq=0
>
> So, I'm suspecting redirector cmsd and xrootd never get connected, do you see something like this in the redirector xrootd log:
> #### Initialization
> ------ File system manager initialization completed.
> 191203 11:14:07 51097 XrootdAioReq: Max aio/req=8; aio/srv=4096; Quantum=131072
> 191203 11:14:07 51097 XrootdAioReq: Adding 18 aioreq objects.
> 191203 11:14:07 51097 XrootdAio: Adding 18 aio objects; 4096 pending.
> 191203 11:14:07 51097 XrdSched: scheduling xrootd protocol anchor in 3600 seconds
> 191203 11:14:07 51097 XrdSched: scheduling transit protocol anchor in 3600 seconds
> Config warning: 'xrootd.prepare logdir' not specified; prepare tracking disabled.
> 191203 11:14:07 51111 XrdXeq: Admin traffic thread started
> 191203 11:14:07 51109 XrdInet: Connected to xrootd.t2.ucsd.edu:2041 <-------
> 191203 11:14:07 51109 cms_ClientMan: Connected to xrootd.t2.ucsd.edu v 3 <-------
> 191203 11:14:07 51109 Hookup xrootd.t2.ucsd.edu qt=178ms rw=2
> ------ xrootd protocol initialization completed.
>
> What does 'xrdmapc redirector:1094' give you?
>
> Would you mind sharing your redirector config?
>
> Andy, do you have a better idea what to look for / try?
>
> Matevz
>
>
>
>
> > Sam
> >
> >
> >
> > redirector cmsd
> >
> > 191203 14:30:14 18957 Protocol: Primary server.202124:22@cephc01:1094 logged in.
> > 191203 14:30:14 18957 Protocol: server.202124:22@cephc01:1094 system
> > ID: [log in to unmask] 1213cephs03.beowulf.cluster
> > =====> Routing for 10.1.50.11: local pub4 prv4
> > =====> Route all4: 10.1.50.11 Dest=[::10.1.50.11]:1094
> > 191203 14:31:16 18939 Config: manager service enabled.
> > 191203 14:31:16 18953 State: Status changed to active + staging
> >
> >
> > ----
> > redirector xrootd
> >
> > ------ xrootd [log in to unmask]:1094 initialization completed.
> > 191203 14:32:26 18970 XrootdXeq: root.218779:20@cephc01 pvt IPv4 login
> > 191203 14:32:26 18970 root.218779:20@cephc01 XrootdResponse: sending
> > err 3011: No servers have read access to the file
> > 191203 14:32:26 18970 XrootdXeq: root.218779:20@cephc01 disc 0:00:00
> >
> >
> > -----
> > server cmsd
> >
> > ------ cmsd [log in to unmask]:46427 initialization completed.
> > 191203 14:30:14 218768 do_Login:: Primary server 218748 logged in;
> > data port is 1094
> > Config Connecting to 1 manager and 1 site.
> > 191203 14:30:14 218729 Config: server service enabled.
> > 191203 14:30:14 218770 State: Status changed to active + staging
> > 191203 14:30:14 218740 ManTree: Now connected to 1 root node(s)
> > 191203 14:30:14 218740 Protocol: Logged into cephs03
> >
> >
> > -----
> > server xrootd
> >
> > 191203 14:30:14 218765 cms_Finder: Connected to cmsd via
> > /tmp/cache/.olb/olbd.admin
> > ------ xrootd protocol initialization completed.
> > ------ xrootd [log in to unmask]:1094 initialization completed.
> > 191203 14:30:15 218764 XrdFileCache_Manager: info Cache::Purge() Started.
> > 191203 14:30:15 218764 XrdFileCache_Manager: info Cache::Purge()
> > Finished, removed 0 data files, total size 0, bytes to remove at end:
> > 0
> > 191203 14:35:15 218764 XrdFileCache_Manager: info Cache::Purge() Started.
> > 191203 14:35:15 218764 XrdFileCache_Manager: info Cache::Purge()
> > Finished, removed 0 data files, total size 0, bytes to remove at end:
> > 0
> >
> > On Mon, 2 Dec 2019 at 19:33, Matevz Tadel <[log in to unmask]> wrote:
> >>
> >> I'd try this:
> >>
> >> redirector:
> >> xrootd.trace emsg login stall redirect
> >> xrd.trace conn
> >> cms.trace defer files redirect stage
> >>
> >> server:
> >> # For debug, to see files being searched
> >> # cms.trace defer files redirect stage
> >> cms.trace defer redirect stage
> >>
> >>
> >> You say xrdmapc shows the configured servers, right?
> >>
> >> We had some trouble with ipv4/6 at ucsd lately, clients will be redirected to
> >> ipv6 servers only if they come in via ipv6 to the redirector.
> >>
> >> Can you restart redirector cmsd and then (after 30sec) look at:
> >>
> >> [1131] root@xrootd /var/log/xrootd/xcacheucsd# grep Routing cmsd.log | sort
> >>
> >> =====> Routing for bcache-1.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for bcache-1.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-00.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-01.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-02.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-03.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-04.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-05.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-06.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-07.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-08.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-09.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xcache-10.t2.ucsd.edu: local pub4 prv4
> >> =====> Routing for xcache-11.t2.ucsd.edu: local pub4 prv4 pub6 prv6
> >> =====> Routing for xrd-cache-1.ultralight.org: local pub4 prv4 pub6 prv6
> >> =====> Routing for xrd-cache-2.ultralight.org: local pub4 prv4 pub6 prv6
> >>
> >> Matevz
> >>
> >> On 2019-12-02 11:24, Sam Skipsey wrote:
> >>> No, I explicitly did that. (As I noted, there's a typo for that in the
> >>> example, as it uses "rw" not "r/w" , which doesn't work).
> >>>
> >>> I've tried basically every variation of stage/nostage/ r/w / r/o at
> >>> different parts of the network, but the manager cmsd never seems to
> >>> actually consider the servers (even when I've already pre-staged the
> >>> file it's looking for by directly talking to the server xrootd service
> >>> and getting it to cache).
> >>>
> >>> Sam
> >>>
> >>> On Mon, 2 Dec 2019 at 19:09, Matevz Tadel <[log in to unmask]> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> Before I go looking at what's wrong on the web page, have a look at this, page 21:
> >>>>
> >>>> https://indico.cern.ch/event/727208/contributions/3444604/
> >>>>
> >>>> Maybe you're missing the r/w for xrootd, stage r/o for cmsd trick?
> >>>>
> >>>> Cheers,
> >>>> Matevz
> >>>>
> >>>> On 2019-11-28 08:22, Sam Skipsey wrote:
> >>>>> Hello everyone,
> >>>>>
> >>>>> So, I have another question, working entirely from the documentation
> >>>>> on xrootd.org
> >>>>>
> >>>>> In the documentation for cache configuration, there's an example of
> >>>>> how to set up a cluster of disk caching proxies:
> >>>>>
> >>>>> https://xrootd.slac.stanford.edu/doc/dev410/pss_config.pdf [page 38,
> >>>>> you can't copy it because weirdly its an image]
> >>>>>
> >>>>> I'm following that exactly (except for fixing the typo where the
> >>>>> example has an export using "rw" and not "r/w" as an option), and,
> >>>>> well, it just doesn't seem to work.
> >>>>>
> >>>>> If I talk directly to the server that the proxies talk to: I can get a file.
> >>>>> If I talk to an individual proxy: I can also get a file (and it is cached)
> >>>>> If I talk to the *redirector*, I get, with debugging on "Open has
> >>>>> returned with status [ERROR] Server responded with an error: [3011] No
> >>>>> servers have read access to the file"
> >>>>>
> >>>>> The redirector logs show that the cmsd on the proxy logs in (and is
> >>>>> listed as a "server" in its list of servers), and the proxy cmsd logs
> >>>>> also show that it happily registers to the redirector.
> >>>>>
> >>>>> How do I debug this?
> >>>>>
> >>>>> I've already tried adding and removing options to the various exports,
> >>>>> making sure that all the relevant ports are open, etc.
> >>>>>
> >>>>> Sam
> >>>>>
> >>>>> ########################################################################
> >>>>> Use REPLY-ALL to reply to list
> >>>>>
> >>>>> To unsubscribe from the XROOTD-L list, click the following link:
> >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
> >>>>>
> >>>>
> >>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the XROOTD-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1
