 |
 |
On 2019-12-03 06:38, Sam Skipsey wrote: > Okay, so what I have, including logs from all the relevant bits is below. > > I restarted all of the services (after adding the traces as requested) > at around 14:30, so all of the servers come up. > About 2 minutes later, I tried (from cephc01, using xrdcp) to copy a > file via the redirector xrootd (on cephs03). This only seems to show > up on the redirector xrootd itself - there's no record of anything in > the other logs as far as I can see. Hmmh, this is stange, cmsd on redirector never seems to ask the servers. redirector cmsd: 191203 11:57:35 51827 Select seeking /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root 191203 11:57:35 51827 redirector.51097:17@xrootd XrdLink: Setting ref to 2+-1 post=0 191203 11:57:35 51475 Dispatch server.6875:24@xcache-05:1094 for have dlen=107 191203 11:57:35 51475 server.6875:24@xcache-05 XrdLink: Setting ref to 1+1 post=0 191203 11:57:35 51341 XrdSched: running server inq=0 191203 11:57:35 51341 server.6875:24@xcache-05:1094 do_Have: /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root redirector xrootd: 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 req=open dlen=106 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 open r /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root 191203 11:57:35 51514 nobody.351:[log in to unmask] ofs_open: 0-660 fn=/store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root 191203 11:57:35 51109 Receive xrootd 26 bytes on 60327927 191203 11:57:35 51109 Decode xrootd redirects nobody.351:[log in to unmask] to xcache-05.t2.ucsd.edu:1094 /store/data/Run2018D/MET/MINIAOD/PromptReco-v2/000/322/431/00000/4AD1B758-3BB8-E811-92B8-FA163EA98227.root 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdProtocol: 0100 redirecting to xcache-05.t2.ucsd.edu:1094 191203 11:57:35 51514 nobody.351:[log in to unmask] XrootdResponse: 0100 sending 25 data bytes; status=4004 191203 11:57:35 51514 nobody.351:[log in to unmask] ofs_close: use=0 fn=dummy 191203 11:57:35 51668 XrdSched: running cuser3.270:1838@cabinet-5-5-5 inq=0 So, I'm suspecting redirector cmsd and xrootd never get connected, do you see something like this in the redirector xrootd log: #### Initialization ------ File system manager initialization completed. 191203 11:14:07 51097 XrootdAioReq: Max aio/req=8; aio/srv=4096; Quantum=131072 191203 11:14:07 51097 XrootdAioReq: Adding 18 aioreq objects. 191203 11:14:07 51097 XrootdAio: Adding 18 aio objects; 4096 pending. 191203 11:14:07 51097 XrdSched: scheduling xrootd protocol anchor in 3600 seconds 191203 11:14:07 51097 XrdSched: scheduling transit protocol anchor in 3600 seconds Config warning: 'xrootd.prepare logdir' not specified; prepare tracking disabled. 191203 11:14:07 51111 XrdXeq: Admin traffic thread started 191203 11:14:07 51109 XrdInet: Connected to xrootd.t2.ucsd.edu:2041 <------- 191203 11:14:07 51109 cms_ClientMan: Connected to xrootd.t2.ucsd.edu v 3 <------- 191203 11:14:07 51109 Hookup xrootd.t2.ucsd.edu qt=178ms rw=2 ------ xrootd protocol initialization completed. What does 'xrdmapc redirector:1094' give you? Would you mind sharing your redirector config? Andy, do you have a better idea what to look for / try? Matevz > Sam > > > > redirector cmsd > > 191203 14:30:14 18957 Protocol: Primary server.202124:22@cephc01:1094 logged in. > 191203 14:30:14 18957 Protocol: server.202124:22@cephc01:1094 system > ID: [log in to unmask] 1213cephs03.beowulf.cluster > =====> Routing for 10.1.50.11: local pub4 prv4 > =====> Route all4: 10.1.50.11 Dest=[::10.1.50.11]:1094 > 191203 14:31:16 18939 Config: manager service enabled. > 191203 14:31:16 18953 State: Status changed to active + staging > > > ---- > redirector xrootd > > ------ xrootd [log in to unmask]:1094 initialization completed. > 191203 14:32:26 18970 XrootdXeq: root.218779:20@cephc01 pvt IPv4 login > 191203 14:32:26 18970 root.218779:20@cephc01 XrootdResponse: sending > err 3011: No servers have read access to the file > 191203 14:32:26 18970 XrootdXeq: root.218779:20@cephc01 disc 0:00:00 > > > ----- > server cmsd > > ------ cmsd [log in to unmask]:46427 initialization completed. > 191203 14:30:14 218768 do_Login:: Primary server 218748 logged in; > data port is 1094 > Config Connecting to 1 manager and 1 site. > 191203 14:30:14 218729 Config: server service enabled. > 191203 14:30:14 218770 State: Status changed to active + staging > 191203 14:30:14 218740 ManTree: Now connected to 1 root node(s) > 191203 14:30:14 218740 Protocol: Logged into cephs03 > > > ----- > server xrootd > > 191203 14:30:14 218765 cms_Finder: Connected to cmsd via > /tmp/cache/.olb/olbd.admin > ------ xrootd protocol initialization completed. > ------ xrootd [log in to unmask]:1094 initialization completed. > 191203 14:30:15 218764 XrdFileCache_Manager: info Cache::Purge() Started. > 191203 14:30:15 218764 XrdFileCache_Manager: info Cache::Purge() > Finished, removed 0 data files, total size 0, bytes to remove at end: > 0 > 191203 14:35:15 218764 XrdFileCache_Manager: info Cache::Purge() Started. > 191203 14:35:15 218764 XrdFileCache_Manager: info Cache::Purge() > Finished, removed 0 data files, total size 0, bytes to remove at end: > 0 > > On Mon, 2 Dec 2019 at 19:33, Matevz Tadel <[log in to unmask]> wrote: >> >> I'd try this: >> >> redirector: >> xrootd.trace emsg login stall redirect >> xrd.trace conn >> cms.trace defer files redirect stage >> >> server: >> # For debug, to see files being searched >> # cms.trace defer files redirect stage >> cms.trace defer redirect stage >> >> >> You say xrdmapc shows the configured servers, right? >> >> We had some trouble with ipv4/6 at ucsd lately, clients will be redirected to >> ipv6 servers only if they come in via ipv6 to the redirector. >> >> Can you restart redirector cmsd and then (after 30sec) look at: >> >> [1131] root@xrootd /var/log/xrootd/xcacheucsd# grep Routing cmsd.log | sort >> >> =====> Routing for bcache-1.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for bcache-1.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-00.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-01.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-02.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-03.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-04.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-05.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-06.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-07.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-08.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-09.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xcache-10.t2.ucsd.edu: local pub4 prv4 >> =====> Routing for xcache-11.t2.ucsd.edu: local pub4 prv4 pub6 prv6 >> =====> Routing for xrd-cache-1.ultralight.org: local pub4 prv4 pub6 prv6 >> =====> Routing for xrd-cache-2.ultralight.org: local pub4 prv4 pub6 prv6 >> >> Matevz >> >> On 2019-12-02 11:24, Sam Skipsey wrote: >>> No, I explicitly did that. (As I noted, there's a typo for that in the >>> example, as it uses "rw" not "r/w" , which doesn't work). >>> >>> I've tried basically every variation of stage/nostage/ r/w / r/o at >>> different parts of the network, but the manager cmsd never seems to >>> actually consider the servers (even when I've already pre-staged the >>> file it's looking for by directly talking to the server xrootd service >>> and getting it to cache). >>> >>> Sam >>> >>> On Mon, 2 Dec 2019 at 19:09, Matevz Tadel <[log in to unmask]> wrote: >>>> >>>> Hi, >>>> >>>> Before I go looking at what's wrong on the web page, have a look at this, page 21: >>>> >>>> https://indico.cern.ch/event/727208/contributions/3444604/ >>>> >>>> Maybe you're missing the r/w for xrootd, stage r/o for cmsd trick? >>>> >>>> Cheers, >>>> Matevz >>>> >>>> On 2019-11-28 08:22, Sam Skipsey wrote: >>>>> Hello everyone, >>>>> >>>>> So, I have another question, working entirely from the documentation >>>>> on xrootd.org >>>>> >>>>> In the documentation for cache configuration, there's an example of >>>>> how to set up a cluster of disk caching proxies: >>>>> >>>>> https://xrootd.slac.stanford.edu/doc/dev410/pss_config.pdf [page 38, >>>>> you can't copy it because weirdly its an image] >>>>> >>>>> I'm following that exactly (except for fixing the typo where the >>>>> example has an export using "rw" and not "r/w" as an option), and, >>>>> well, it just doesn't seem to work. >>>>> >>>>> If I talk directly to the server that the proxies talk to: I can get a file. >>>>> If I talk to an individual proxy: I can also get a file (and it is cached) >>>>> If I talk to the *redirector*, I get, with debugging on "Open has >>>>> returned with status [ERROR] Server responded with an error: [3011] No >>>>> servers have read access to the file" >>>>> >>>>> The redirector logs show that the cmsd on the proxy logs in (and is >>>>> listed as a "server" in its list of servers), and the proxy cmsd logs >>>>> also show that it happily registers to the redirector. >>>>> >>>>> How do I debug this? >>>>> >>>>> I've already tried adding and removing options to the various exports, >>>>> making sure that all the relevant ports are open, etc. >>>>> >>>>> Sam >>>>> >>>>> ######################################################################## >>>>> Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the XROOTD-L list, click the following link: >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1 >>>>> >>>> >> ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the XROOTD-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=XROOTD-L&A=1