 |
 |
This setup was sketchy and didn't have any MySQL configuration, so it's normal we have this message I'm currently testing on openstack/docker and I let you know in a few minutes. Cheers, On 11/12/2015 02:17 PM, Andrew Hanushevsky wrote: > Hi Fabrice, > > Well, we have substituted one problem for another. While now it tries > to initialize, the problem is that we can't because mySQL hasn't been > setup. I don't understand the subcomponent sequencing but this need to > be taked into account. Or maybe I am misunderstanding what you were > testing. I'll talk with John & Fritz about this. > > Andy > > On Thu, 12 Nov 2015, Fabrice Jammes wrote: > >> Thanks Andy, >> >> Simple test case is working now. Thanks so much for this quick fix. >> >> qserv@ccqserv126:~$ cat cmsd.conf >> all.role server >> all.manager ccqserv125.in2p3.fr:2131 >> ssi.svclib libxrdsvc.so >> oss.statlib -2 libXrdSsi.so >> qserv@ccqserv126:~$ cmsd -c cmsd.conf >> 151112 20:27:48 4622 Starting on Linux 3.10.0-229.20.1.el7.x86_64 >> Copr. 2004-2012 Stanford University, xrd version 4.3.0.rc4.lsst1 >> ++++++ cmsd [log in to unmask] initialization started. >> Config using configuration file cmsd.conf >> Config maximum number of connections restricted to 1048576 >> Copr. 2007 Stanford University/SLAC cmsd. >> ++++++ [log in to unmask] phase 1 initialization started. >> =====> all.role server >> =====> all.manager ccqserv125.in2p3.fr:2131 >> The following paths are available to the redirector: >> r / >> >> ------ [log in to unmask] phase 1 server initialization completed. >> ++++++ [log in to unmask] phase 2 server initialization started. >> Config warning: adminpath resides in /tmp and may be unstable! >> ++++++ Storage system initialization started. >> =====> oss.statlib -2 libXrdSsi.so >> Plugin No such file or directory loading statlib libXrdSsi-4.so >> Config Falling back to using libXrdSsi.so >> ++++++ ssi phase 1 initialization started. >> =====> all.role server >> =====> ssi.svclib libxrdsvc.so >> ------ ssi phase 1 initialization completed. >> ++++++ ssi phase 2 initialization started. >> 0 [0x7f413e597740] INFO root null - SsiProvider initializing... >> 151112 20:27:48 4622 sysQserv: Provider Initializing >> 1 [0x7f413e597740] DEBUG root null - Launching query : SELECT db FROM >> qservw_anon.Dbs >> 1 [0x7f413e597740] ERROR root null - ChunkInventory can't get list of >> publishable dbs. >> 1 [0x7f413e597740] ERROR root null - Error -999: Error connecting to >> mysql with config:[host=, port=0, usr=qsmaster, pass=, dbName=mysql, >> socket=/var/lib/mysql/mysql.sock] >> >> 1 [0x7f413e597740] INFO root null - Provider valid paths(ci): >> ChunkInventory() >> 151112 20:27:48 4622 sysQserv: Provider valid paths(ci): >> ChunkInventory() >> ------ ssi phase 2 initialization completed. >> ++++++ Configuring standalone mode . . . >> Config effective cmsd.conf oss configuration: >> oss.alloc 0 0 0 >> oss.cachescan 600 >> oss.fdlimit 524288 1048576 >> oss.maxsize 0 >> oss.trace 0 >> oss.xfr 1 deny 10800 keep 1200 >> oss.memfile off max 8355569664 >> oss.defaults r/w nocheck nodread nomig norcreate nopurge >> nostage xattr >> ------ Storage system initialization completed. >> 151112 20:27:48 4622 Meter: Warning! No writable filesystems found. >> 151112 20:27:48 4622 Meter: Write access and staging prohibited. >> ------ [log in to unmask] phase 2 server initialization completed. >> ------ cmsd [log in to unmask]:33737 initialization completed. >> 151112 20:27:48 4635 Start: Waiting for primary server to login. >> >> >> On 11/11/2015 04:04 PM, Fabrice Jammes wrote: >>> Hi Andy, >>> >>> Thanks for this quick fix, Fritz explained me the deployment stuff >>> so I will try to do it and let you know tomorrow. >>> >>> Thanks :-) >>> >>> On 11/11/2015 03:29 PM, Andrew Hanushevsky wrote: >>>> Hi Fabrice, >>>> >>>> OK, the fix has been pushed to the xrootd git head xrdssi branch. >>>> Now, all that needs to be done is to include this in the LSST >>>> version of the branch so you can recompile. Unfortuantely, Fritz >>>> would do all of this but I think he is away, sigh. >>>> >>>> Andy >>>> >>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote: >>>> >>>>> Hi Andy, >>>>> >>>>> I'm afraid our distributed setup is broken for a few time... >>>>> Thanks for your help and for the future fix ;-) >>>>> >>>>> Regards, >>>>> >>>>> On 11/11/2015 03:13 PM, Andrew Hanushevsky wrote: >>>>>> Hi Fabrice, >>>>>> >>>>>> Ah, OK, I see. This is a problem. There is no easy solution here. >>>>>> I need to rework a bit of code to get the cmsd running. It has to >>>>>> with the way the initialization is ordered, sigh. I won't have >>>>>> something immediately an it will require code changes in the SSI. >>>>>> >>>>>> Andy >>>>>> >>>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote: >>>>>> >>>>>>> Hi Andy, >>>>>>> >>>>>>> Here's the requested traces: >>>>>>> >>>>>>> *cmsd starts successfully with the first config:* >>>>>>> >>>>>>> qserv@ccqserv126:~$ cat cmsd.conf >>>>>>> all.role server >>>>>>> all.manager ccqserv125.in2p3.fr:2131 >>>>>>> ssi.svclib libxrdsvc.so >>>>>>> #oss.statlib -2 libXrdSsi.so >>>>>>> qserv@ccqserv126:~$ cmsd -d -c cmsd.conf >>>>>>> 151111 22:45:19 103 Starting on Linux 3.10.0-229.20.1.el7.x86_64 >>>>>>> Copr. 2004-2012 Stanford University, xrd version unknown >>>>>>> ++++++ cmsd [log in to unmask] initialization started. >>>>>>> Config using configuration file cmsd.conf >>>>>>> Config maximum number of connections restricted to 1048576 >>>>>>> Config maximum number of threads restricted to 1048576 >>>>>>> 151111 22:45:19 103 XrdConfig: sendfile enabled. >>>>>>> 151111 22:45:19 103 XrdSched: scheduling underused thread >>>>>>> monitor in 780 seconds >>>>>>> 151111 22:45:19 104 XrdXeq: Buffer Manager reshaper thread started >>>>>>> 151111 22:45:19 105 XrdXeq: Time scheduler thread started >>>>>>> 151111 22:45:19 103 XrdSched: Starting with 2 workers >>>>>>> 151111 22:45:19 103 XrdLink: Allocating 8 link objects at a time >>>>>>> 151111 22:45:19 107 XrdXeq: Worker thread started >>>>>>> 151111 22:45:19 106 XrdXeq: Worker thread started >>>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 0 >>>>>>> 151111 22:45:19 108 XrdXeq: Poller thread started >>>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 1 >>>>>>> 151111 22:45:19 109 XrdXeq: Poller thread started >>>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 2 >>>>>>> 151111 22:45:19 110 XrdXeq: Poller thread started >>>>>>> 151111 22:45:19 103 XrdProtocol: getting port from protocol cmsd >>>>>>> Copr. 2007 Stanford University/SLAC cmsd. >>>>>>> ++++++ [log in to unmask] phase 1 initialization started. >>>>>>> =====> all.role server >>>>>>> =====> all.manager ccqserv125.in2p3.fr:2131 >>>>>>> The following paths are available to the redirector: >>>>>>> r / >>>>>>> >>>>>>> ------ [log in to unmask] phase 1 server initialization >>>>>>> completed. >>>>>>> 151111 22:45:19 103 XrdConfig: LCL port 37568 wsz=87380 (87380) >>>>>>> 151111 22:45:19 103 XrdProtocol: getting protocol object cmsd >>>>>>> ++++++ [log in to unmask] phase 2 server initialization >>>>>>> started. >>>>>>> Config warning: adminpath resides in /tmp and may be unstable! >>>>>>> 151111 22:45:19 103 Configure2 Global System Identification: >>>>>>> anon-s 2131ccqserv125.in2p3.fr >>>>>>> ++++++ Storage system initialization started. >>>>>>> ++++++ Configuring standalone mode . . . >>>>>>> 151111 22:45:19 103 oss_AioInit: started AIO read signal thread; >>>>>>> tid=1278469888 >>>>>>> 151111 22:45:19 103 oss_AioInit: started AIO write signal >>>>>>> thread; tid=1277417216 >>>>>>> Config effective cmsd.conf oss configuration: >>>>>>> oss.alloc 0 0 0 >>>>>>> oss.cachescan 600 >>>>>>> oss.fdlimit 524288 1048576 >>>>>>> oss.maxsize 0 >>>>>>> oss.trace fff >>>>>>> oss.xfr 1 deny 10800 keep 1200 >>>>>>> oss.memfile off max 8355569664 >>>>>>> oss.defaults r/w nocheck nodread nomig norcreate nopurge >>>>>>> nostage xattr >>>>>>> ------ Storage system initialization completed. >>>>>>> 151111 22:45:19 103 Start Srv=0 dfs=0 lcl=0 Pre=1 dmLife=0 0 >>>>>>> 151111 22:45:19 103 Start Lim=0 0 fix=0 Qmax=1 >>>>>>> 151111 22:45:19 103 Meter: Warning! No writable filesystems found. >>>>>>> 151111 22:45:19 103 Update Space Parm1=0 Parm2=0 >>>>>>> 151111 22:45:19 103 Meter: Write access and staging prohibited. >>>>>>> ------ [log in to unmask] phase 2 server initialization >>>>>>> completed. >>>>>>> 151111 22:45:19 107 XrdSched: running cmsd startup inq=0 >>>>>>> 151111 22:45:19 113 XrdXeq: Notification handler thread started >>>>>>> 151111 22:45:19 115 XrdXeq: Admin traffic thread started >>>>>>> 151111 22:45:19 114 XrdXeq: Prep handler thread started >>>>>>> 151111 22:45:19 115 Start: Waiting for primary server to login. >>>>>>> ------ cmsd [log in to unmask]:37568 initialization >>>>>>> completed. >>>>>>> 151111 22:45:19 106 XrdSched: Now have 3 workers >>>>>>> 151111 22:45:19 106 XrdSched: running main accept inq=0 >>>>>>> 151111 22:45:19 117 XrdXeq: Worker thread started >>>>>>> >>>>>>> *cmsd crashes with the second config:* >>>>>>> >>>>>>> qserv@ccqserv126:~$ cat cmsd.conf >>>>>>> all.role server >>>>>>> all.manager ccqserv125.in2p3.fr:2131 >>>>>>> ssi.svclib libxrdsvc.so >>>>>>> oss.statlib -2 libXrdSsi.so >>>>>>> qserv@ccqserv126:~$ >>>>>>> qserv@ccqserv126:~$ cmsd -d -c cmsd.conf >>>>>>> 151111 22:58:54 137 Starting on Linux 3.10.0-229.20.1.el7.x86_64 >>>>>>> Copr. 2004-2012 Stanford University, xrd version unknown >>>>>>> ++++++ cmsd [log in to unmask] initialization started. >>>>>>> Config using configuration file cmsd.conf >>>>>>> Config maximum number of connections restricted to 1048576 >>>>>>> Config maximum number of threads restricted to 1048576 >>>>>>> 151111 22:58:54 137 XrdConfig: sendfile enabled. >>>>>>> 151111 22:58:54 137 XrdSched: scheduling underused thread >>>>>>> monitor in 780 seconds >>>>>>> 151111 22:58:54 138 XrdXeq: Buffer Manager reshaper thread started >>>>>>> 151111 22:58:54 141 XrdXeq: Worker thread started >>>>>>> 151111 22:58:54 137 XrdSched: Starting with 2 workers >>>>>>> 151111 22:58:54 137 XrdLink: Allocating 8 link objects at a time >>>>>>> 151111 22:58:54 139 XrdXeq: Time scheduler thread started >>>>>>> 151111 22:58:54 140 XrdXeq: Worker thread started >>>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 0 >>>>>>> 151111 22:58:54 142 XrdXeq: Poller thread started >>>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 1 >>>>>>> 151111 22:58:54 143 XrdXeq: Poller thread started >>>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 2 >>>>>>> 151111 22:58:54 144 XrdXeq: Poller thread started >>>>>>> 151111 22:58:54 137 XrdProtocol: getting port from protocol cmsd >>>>>>> Copr. 2007 Stanford University/SLAC cmsd. >>>>>>> ++++++ [log in to unmask] phase 1 initialization started. >>>>>>> =====> all.role server >>>>>>> =====> all.manager ccqserv125.in2p3.fr:2131 >>>>>>> The following paths are available to the redirector: >>>>>>> r / >>>>>>> >>>>>>> ------ [log in to unmask] phase 1 server initialization >>>>>>> completed. >>>>>>> 151111 22:58:54 137 XrdConfig: LCL port 52851 wsz=87380 (87380) >>>>>>> 151111 22:58:54 137 XrdProtocol: getting protocol object cmsd >>>>>>> ++++++ [log in to unmask] phase 2 server initialization >>>>>>> started. >>>>>>> Config warning: adminpath resides in /tmp and may be unstable! >>>>>>> 151111 22:58:54 137 Configure2 Global System Identification: >>>>>>> anon-s 2131ccqserv125.in2p3.fr >>>>>>> ++++++ Storage system initialization started. >>>>>>> =====> oss.statlib -2 libXrdSsi.so >>>>>>> Plugin No such file or directory loading statlib libXrdSsi-4.so >>>>>>> Config Falling back to using libXrdSsi.so >>>>>>> ++++++ ssi phase 1 initialization started. >>>>>>> =====> all.role server >>>>>>> =====> ssi.svclib libxrdsvc.so >>>>>>> ------ ssi phase 1 initialization completed. >>>>>>> ++++++ ssi phase 2 initialization started. >>>>>>> 151111 22:58:54 137 sysFinder: Network i/f undefined; unable to >>>>>>> self-locate. >>>>>>> ------ ssi phase 2 initialization failed. >>>>>>> ++++++ Configuring standalone mode . . . >>>>>>> ------ Storage system initialization failed. >>>>>>> ------ [log in to unmask] phase 2 server initialization >>>>>>> failed. >>>>>>> 151111 22:58:54 137 XrdProtocol: Protocol cmsd could not be loaded >>>>>>> ------ cmsd [log in to unmask]:-1 initialization failed. >>>>>>> >>>>>>> Hope it'll help. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> On 11/11/2015 02:10 PM, Andrew Hanushevsky wrote: >>>>>>>> Hi Fabrice, >>>>>>>> >>>>>>>> Odd. OK, my answers.... >>>>>>>> >>>>>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote: >>>>>>>> >>>>>>>>>> 1) Who is producing the following messages? >>>>>>>>> This messages are in cmsd logs and are produced by xrootd: >>>>>>>> Got it. OK, this is because of static initialization of >>>>>>>> something we will not use but cannot easily avoid initializing. >>>>>>>> It should be OK. >>>>>>>> >>>>>>>>>> 2) The "statlib" uses the libXrdSsi.so because we packaged it >>>>>>>>>> there as a convenience since we need to use the file >>>>>>>>>> registry. Do you have a static initialization section that >>>>>>>>>> expects it will fire up all of qserv? We don't want that. >>>>>>>>> I don't really understand this question, sorry. Here's our >>>>>>>>> configuration file, it may help? >>>>>>>> I just answered in in (1). This is the xrootd client doing >>>>>>>> static initialization and this is because the SSI library uses >>>>>>>> the client so it is forced to be initialized when the client >>>>>>>> library is loaded. >>>>>>>> >>>>>>>>>> 3) This is a container, right? >>>>>>>>> Yes. FYI, our previous cmsd version was running fine under the >>>>>>>>> same sort of container with same network setting. >>>>>>>> Then is should run here. >>>>>>>> >>>>>>>>>> 5) I assume things are registered in DNS or at least appear >>>>>>>>>> correctly in /etc/hosts otherwise we will have a problem. The >>>>>>>>>> container has to look like an actual machine. >>>>>>>>> # runned inside he container >>>>>>>>> root@ccqserv126:/qserv# ping ccqserv126 >>>>>>>>> PING ccqserv126.in2p3.fr (172.17.0.7): 56 data bytes >>>>>>>>> 64 bytes from 172.17.0.7: icmp_seq=0 ttl=64 time=0.061 ms >>>>>>>>> 64 bytes from 172.17.0.7: icmp_seq=1 ttl=64 time=0.049 ms >>>>>>>> OK, it's properly registered. So, type up a small config file, >>>>>>>> as follows: >>>>>>>> >>>>>>>> all.role server >>>>>>>> all.manager ccqserv125.in2p3.fr:2131 >>>>>>>> ssi.svclib libxrdsvc.so >>>>>>>> #oss.statlib -2 libXrdSsi.so >>>>>>>> >>>>>>>> Setup the environment as you normally would but don't start >>>>>>>> anything. By hand do: >>>>>>>> >>>>>>>> <path>/cmsd -d -c <path to config file above> >>>>>>>> >>>>>>>> Send the output to me. The uncomment the "statlib" directive >>>>>>>> and so the same thing again. Send that output to me as well. >>>>>>>> >>>>>>>> Andy >>>>>>> >>>>>>> >>>>> >>>>> ######################################################################## >>>>> Use REPLY-ALL to reply to list >>>>> >>>>> To unsubscribe from the QSERV-L list, click the following link: >>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >>>>> >>> >>> ######################################################################## >>> >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the QSERV-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1