Hi Andy,
cmsd is now starting correctly, thanks for the fix :-)
Regards
Le 13/11/2015 01:33, Fabrice Jammes a écrit :
> This setup was sketchy and didn't have any MySQL configuration, so
> it's normal we have this message
> I'm currently testing on openstack/docker and I let you know in a few
> minutes.
>
> Cheers,
>
> On 11/12/2015 02:17 PM, Andrew Hanushevsky wrote:
>> Hi Fabrice,
>>
>> Well, we have substituted one problem for another. While now it tries
>> to initialize, the problem is that we can't because mySQL hasn't been
>> setup. I don't understand the subcomponent sequencing but this need
>> to be taked into account. Or maybe I am misunderstanding what you
>> were testing. I'll talk with John & Fritz about this.
>>
>> Andy
>>
>> On Thu, 12 Nov 2015, Fabrice Jammes wrote:
>>
>>> Thanks Andy,
>>>
>>> Simple test case is working now. Thanks so much for this quick fix.
>>>
>>> qserv@ccqserv126:~$ cat cmsd.conf
>>> all.role server
>>> all.manager ccqserv125.in2p3.fr:2131
>>> ssi.svclib libxrdsvc.so
>>> oss.statlib -2 libXrdSsi.so
>>> qserv@ccqserv126:~$ cmsd -c cmsd.conf
>>> 151112 20:27:48 4622 Starting on Linux 3.10.0-229.20.1.el7.x86_64
>>> Copr. 2004-2012 Stanford University, xrd version 4.3.0.rc4.lsst1
>>> ++++++ cmsd [log in to unmask] initialization started.
>>> Config using configuration file cmsd.conf
>>> Config maximum number of connections restricted to 1048576
>>> Copr. 2007 Stanford University/SLAC cmsd.
>>> ++++++ [log in to unmask] phase 1 initialization started.
>>> =====> all.role server
>>> =====> all.manager ccqserv125.in2p3.fr:2131
>>> The following paths are available to the redirector:
>>> r /
>>>
>>> ------ [log in to unmask] phase 1 server initialization
>>> completed.
>>> ++++++ [log in to unmask] phase 2 server initialization started.
>>> Config warning: adminpath resides in /tmp and may be unstable!
>>> ++++++ Storage system initialization started.
>>> =====> oss.statlib -2 libXrdSsi.so
>>> Plugin No such file or directory loading statlib libXrdSsi-4.so
>>> Config Falling back to using libXrdSsi.so
>>> ++++++ ssi phase 1 initialization started.
>>> =====> all.role server
>>> =====> ssi.svclib libxrdsvc.so
>>> ------ ssi phase 1 initialization completed.
>>> ++++++ ssi phase 2 initialization started.
>>> 0 [0x7f413e597740] INFO root null - SsiProvider initializing...
>>> 151112 20:27:48 4622 sysQserv: Provider Initializing
>>> 1 [0x7f413e597740] DEBUG root null - Launching query : SELECT db
>>> FROM qservw_anon.Dbs
>>> 1 [0x7f413e597740] ERROR root null - ChunkInventory can't get list
>>> of publishable dbs.
>>> 1 [0x7f413e597740] ERROR root null - Error -999: Error connecting to
>>> mysql with config:[host=, port=0, usr=qsmaster, pass=, dbName=mysql,
>>> socket=/var/lib/mysql/mysql.sock]
>>>
>>> 1 [0x7f413e597740] INFO root null - Provider valid paths(ci):
>>> ChunkInventory()
>>> 151112 20:27:48 4622 sysQserv: Provider valid paths(ci):
>>> ChunkInventory()
>>> ------ ssi phase 2 initialization completed.
>>> ++++++ Configuring standalone mode . . .
>>> Config effective cmsd.conf oss configuration:
>>> oss.alloc 0 0 0
>>> oss.cachescan 600
>>> oss.fdlimit 524288 1048576
>>> oss.maxsize 0
>>> oss.trace 0
>>> oss.xfr 1 deny 10800 keep 1200
>>> oss.memfile off max 8355569664
>>> oss.defaults r/w nocheck nodread nomig norcreate nopurge
>>> nostage xattr
>>> ------ Storage system initialization completed.
>>> 151112 20:27:48 4622 Meter: Warning! No writable filesystems found.
>>> 151112 20:27:48 4622 Meter: Write access and staging prohibited.
>>> ------ [log in to unmask] phase 2 server initialization
>>> completed.
>>> ------ cmsd [log in to unmask]:33737 initialization completed.
>>> 151112 20:27:48 4635 Start: Waiting for primary server to login.
>>>
>>>
>>> On 11/11/2015 04:04 PM, Fabrice Jammes wrote:
>>>> Hi Andy,
>>>>
>>>> Thanks for this quick fix, Fritz explained me the deployment stuff
>>>> so I will try to do it and let you know tomorrow.
>>>>
>>>> Thanks :-)
>>>>
>>>> On 11/11/2015 03:29 PM, Andrew Hanushevsky wrote:
>>>>> Hi Fabrice,
>>>>>
>>>>> OK, the fix has been pushed to the xrootd git head xrdssi branch.
>>>>> Now, all that needs to be done is to include this in the LSST
>>>>> version of the branch so you can recompile. Unfortuantely, Fritz
>>>>> would do all of this but I think he is away, sigh.
>>>>>
>>>>> Andy
>>>>>
>>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote:
>>>>>
>>>>>> Hi Andy,
>>>>>>
>>>>>> I'm afraid our distributed setup is broken for a few time...
>>>>>> Thanks for your help and for the future fix ;-)
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> On 11/11/2015 03:13 PM, Andrew Hanushevsky wrote:
>>>>>>> Hi Fabrice,
>>>>>>>
>>>>>>> Ah, OK, I see. This is a problem. There is no easy solution
>>>>>>> here. I need to rework a bit of code to get the cmsd running. It
>>>>>>> has to with the way the initialization is ordered, sigh. I won't
>>>>>>> have something immediately an it will require code changes in
>>>>>>> the SSI.
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote:
>>>>>>>
>>>>>>>> Hi Andy,
>>>>>>>>
>>>>>>>> Here's the requested traces:
>>>>>>>>
>>>>>>>> *cmsd starts successfully with the first config:*
>>>>>>>>
>>>>>>>> qserv@ccqserv126:~$ cat cmsd.conf
>>>>>>>> all.role server
>>>>>>>> all.manager ccqserv125.in2p3.fr:2131
>>>>>>>> ssi.svclib libxrdsvc.so
>>>>>>>> #oss.statlib -2 libXrdSsi.so
>>>>>>>> qserv@ccqserv126:~$ cmsd -d -c cmsd.conf
>>>>>>>> 151111 22:45:19 103 Starting on Linux 3.10.0-229.20.1.el7.x86_64
>>>>>>>> Copr. 2004-2012 Stanford University, xrd version unknown
>>>>>>>> ++++++ cmsd [log in to unmask] initialization started.
>>>>>>>> Config using configuration file cmsd.conf
>>>>>>>> Config maximum number of connections restricted to 1048576
>>>>>>>> Config maximum number of threads restricted to 1048576
>>>>>>>> 151111 22:45:19 103 XrdConfig: sendfile enabled.
>>>>>>>> 151111 22:45:19 103 XrdSched: scheduling underused thread
>>>>>>>> monitor in 780 seconds
>>>>>>>> 151111 22:45:19 104 XrdXeq: Buffer Manager reshaper thread started
>>>>>>>> 151111 22:45:19 105 XrdXeq: Time scheduler thread started
>>>>>>>> 151111 22:45:19 103 XrdSched: Starting with 2 workers
>>>>>>>> 151111 22:45:19 103 XrdLink: Allocating 8 link objects at a time
>>>>>>>> 151111 22:45:19 107 XrdXeq: Worker thread started
>>>>>>>> 151111 22:45:19 106 XrdXeq: Worker thread started
>>>>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 0
>>>>>>>> 151111 22:45:19 108 XrdXeq: Poller thread started
>>>>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 1
>>>>>>>> 151111 22:45:19 109 XrdXeq: Poller thread started
>>>>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 2
>>>>>>>> 151111 22:45:19 110 XrdXeq: Poller thread started
>>>>>>>> 151111 22:45:19 103 XrdProtocol: getting port from protocol cmsd
>>>>>>>> Copr. 2007 Stanford University/SLAC cmsd.
>>>>>>>> ++++++ [log in to unmask] phase 1 initialization started.
>>>>>>>> =====> all.role server
>>>>>>>> =====> all.manager ccqserv125.in2p3.fr:2131
>>>>>>>> The following paths are available to the redirector:
>>>>>>>> r /
>>>>>>>>
>>>>>>>> ------ [log in to unmask] phase 1 server initialization
>>>>>>>> completed.
>>>>>>>> 151111 22:45:19 103 XrdConfig: LCL port 37568 wsz=87380 (87380)
>>>>>>>> 151111 22:45:19 103 XrdProtocol: getting protocol object cmsd
>>>>>>>> ++++++ [log in to unmask] phase 2 server initialization
>>>>>>>> started.
>>>>>>>> Config warning: adminpath resides in /tmp and may be unstable!
>>>>>>>> 151111 22:45:19 103 Configure2 Global System Identification:
>>>>>>>> anon-s 2131ccqserv125.in2p3.fr
>>>>>>>> ++++++ Storage system initialization started.
>>>>>>>> ++++++ Configuring standalone mode . . .
>>>>>>>> 151111 22:45:19 103 oss_AioInit: started AIO read signal
>>>>>>>> thread; tid=1278469888
>>>>>>>> 151111 22:45:19 103 oss_AioInit: started AIO write signal
>>>>>>>> thread; tid=1277417216
>>>>>>>> Config effective cmsd.conf oss configuration:
>>>>>>>> oss.alloc 0 0 0
>>>>>>>> oss.cachescan 600
>>>>>>>> oss.fdlimit 524288 1048576
>>>>>>>> oss.maxsize 0
>>>>>>>> oss.trace fff
>>>>>>>> oss.xfr 1 deny 10800 keep 1200
>>>>>>>> oss.memfile off max 8355569664
>>>>>>>> oss.defaults r/w nocheck nodread nomig norcreate
>>>>>>>> nopurge nostage xattr
>>>>>>>> ------ Storage system initialization completed.
>>>>>>>> 151111 22:45:19 103 Start Srv=0 dfs=0 lcl=0 Pre=1 dmLife=0 0
>>>>>>>> 151111 22:45:19 103 Start Lim=0 0 fix=0 Qmax=1
>>>>>>>> 151111 22:45:19 103 Meter: Warning! No writable filesystems found.
>>>>>>>> 151111 22:45:19 103 Update Space Parm1=0 Parm2=0
>>>>>>>> 151111 22:45:19 103 Meter: Write access and staging prohibited.
>>>>>>>> ------ [log in to unmask] phase 2 server initialization
>>>>>>>> completed.
>>>>>>>> 151111 22:45:19 107 XrdSched: running cmsd startup inq=0
>>>>>>>> 151111 22:45:19 113 XrdXeq: Notification handler thread started
>>>>>>>> 151111 22:45:19 115 XrdXeq: Admin traffic thread started
>>>>>>>> 151111 22:45:19 114 XrdXeq: Prep handler thread started
>>>>>>>> 151111 22:45:19 115 Start: Waiting for primary server to login.
>>>>>>>> ------ cmsd [log in to unmask]:37568 initialization
>>>>>>>> completed.
>>>>>>>> 151111 22:45:19 106 XrdSched: Now have 3 workers
>>>>>>>> 151111 22:45:19 106 XrdSched: running main accept inq=0
>>>>>>>> 151111 22:45:19 117 XrdXeq: Worker thread started
>>>>>>>>
>>>>>>>> *cmsd crashes with the second config:*
>>>>>>>>
>>>>>>>> qserv@ccqserv126:~$ cat cmsd.conf
>>>>>>>> all.role server
>>>>>>>> all.manager ccqserv125.in2p3.fr:2131
>>>>>>>> ssi.svclib libxrdsvc.so
>>>>>>>> oss.statlib -2 libXrdSsi.so
>>>>>>>> qserv@ccqserv126:~$
>>>>>>>> qserv@ccqserv126:~$ cmsd -d -c cmsd.conf
>>>>>>>> 151111 22:58:54 137 Starting on Linux 3.10.0-229.20.1.el7.x86_64
>>>>>>>> Copr. 2004-2012 Stanford University, xrd version unknown
>>>>>>>> ++++++ cmsd [log in to unmask] initialization started.
>>>>>>>> Config using configuration file cmsd.conf
>>>>>>>> Config maximum number of connections restricted to 1048576
>>>>>>>> Config maximum number of threads restricted to 1048576
>>>>>>>> 151111 22:58:54 137 XrdConfig: sendfile enabled.
>>>>>>>> 151111 22:58:54 137 XrdSched: scheduling underused thread
>>>>>>>> monitor in 780 seconds
>>>>>>>> 151111 22:58:54 138 XrdXeq: Buffer Manager reshaper thread started
>>>>>>>> 151111 22:58:54 141 XrdXeq: Worker thread started
>>>>>>>> 151111 22:58:54 137 XrdSched: Starting with 2 workers
>>>>>>>> 151111 22:58:54 137 XrdLink: Allocating 8 link objects at a time
>>>>>>>> 151111 22:58:54 139 XrdXeq: Time scheduler thread started
>>>>>>>> 151111 22:58:54 140 XrdXeq: Worker thread started
>>>>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 0
>>>>>>>> 151111 22:58:54 142 XrdXeq: Poller thread started
>>>>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 1
>>>>>>>> 151111 22:58:54 143 XrdXeq: Poller thread started
>>>>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 2
>>>>>>>> 151111 22:58:54 144 XrdXeq: Poller thread started
>>>>>>>> 151111 22:58:54 137 XrdProtocol: getting port from protocol cmsd
>>>>>>>> Copr. 2007 Stanford University/SLAC cmsd.
>>>>>>>> ++++++ [log in to unmask] phase 1 initialization started.
>>>>>>>> =====> all.role server
>>>>>>>> =====> all.manager ccqserv125.in2p3.fr:2131
>>>>>>>> The following paths are available to the redirector:
>>>>>>>> r /
>>>>>>>>
>>>>>>>> ------ [log in to unmask] phase 1 server initialization
>>>>>>>> completed.
>>>>>>>> 151111 22:58:54 137 XrdConfig: LCL port 52851 wsz=87380 (87380)
>>>>>>>> 151111 22:58:54 137 XrdProtocol: getting protocol object cmsd
>>>>>>>> ++++++ [log in to unmask] phase 2 server initialization
>>>>>>>> started.
>>>>>>>> Config warning: adminpath resides in /tmp and may be unstable!
>>>>>>>> 151111 22:58:54 137 Configure2 Global System Identification:
>>>>>>>> anon-s 2131ccqserv125.in2p3.fr
>>>>>>>> ++++++ Storage system initialization started.
>>>>>>>> =====> oss.statlib -2 libXrdSsi.so
>>>>>>>> Plugin No such file or directory loading statlib libXrdSsi-4.so
>>>>>>>> Config Falling back to using libXrdSsi.so
>>>>>>>> ++++++ ssi phase 1 initialization started.
>>>>>>>> =====> all.role server
>>>>>>>> =====> ssi.svclib libxrdsvc.so
>>>>>>>> ------ ssi phase 1 initialization completed.
>>>>>>>> ++++++ ssi phase 2 initialization started.
>>>>>>>> 151111 22:58:54 137 sysFinder: Network i/f undefined; unable to
>>>>>>>> self-locate.
>>>>>>>> ------ ssi phase 2 initialization failed.
>>>>>>>> ++++++ Configuring standalone mode . . .
>>>>>>>> ------ Storage system initialization failed.
>>>>>>>> ------ [log in to unmask] phase 2 server initialization
>>>>>>>> failed.
>>>>>>>> 151111 22:58:54 137 XrdProtocol: Protocol cmsd could not be loaded
>>>>>>>> ------ cmsd [log in to unmask]:-1 initialization failed.
>>>>>>>>
>>>>>>>> Hope it'll help.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/11/2015 02:10 PM, Andrew Hanushevsky wrote:
>>>>>>>>> Hi Fabrice,
>>>>>>>>>
>>>>>>>>> Odd. OK, my answers....
>>>>>>>>>
>>>>>>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote:
>>>>>>>>>
>>>>>>>>>>> 1) Who is producing the following messages?
>>>>>>>>>> This messages are in cmsd logs and are produced by xrootd:
>>>>>>>>> Got it. OK, this is because of static initialization of
>>>>>>>>> something we will not use but cannot easily avoid
>>>>>>>>> initializing. It should be OK.
>>>>>>>>>
>>>>>>>>>>> 2) The "statlib" uses the libXrdSsi.so because we packaged
>>>>>>>>>>> it there as a convenience since we need to use the file
>>>>>>>>>>> registry. Do you have a static initialization section that
>>>>>>>>>>> expects it will fire up all of qserv? We don't want that.
>>>>>>>>>> I don't really understand this question, sorry. Here's our
>>>>>>>>>> configuration file, it may help?
>>>>>>>>> I just answered in in (1). This is the xrootd client doing
>>>>>>>>> static initialization and this is because the SSI library uses
>>>>>>>>> the client so it is forced to be initialized when the client
>>>>>>>>> library is loaded.
>>>>>>>>>
>>>>>>>>>>> 3) This is a container, right?
>>>>>>>>>> Yes. FYI, our previous cmsd version was running fine under
>>>>>>>>>> the same sort of container with same network setting.
>>>>>>>>> Then is should run here.
>>>>>>>>>
>>>>>>>>>>> 5) I assume things are registered in DNS or at least appear
>>>>>>>>>>> correctly in /etc/hosts otherwise we will have a problem.
>>>>>>>>>>> The container has to look like an actual machine.
>>>>>>>>>> # runned inside he container
>>>>>>>>>> root@ccqserv126:/qserv# ping ccqserv126
>>>>>>>>>> PING ccqserv126.in2p3.fr (172.17.0.7): 56 data bytes
>>>>>>>>>> 64 bytes from 172.17.0.7: icmp_seq=0 ttl=64 time=0.061 ms
>>>>>>>>>> 64 bytes from 172.17.0.7: icmp_seq=1 ttl=64 time=0.049 ms
>>>>>>>>> OK, it's properly registered. So, type up a small config file,
>>>>>>>>> as follows:
>>>>>>>>>
>>>>>>>>> all.role server
>>>>>>>>> all.manager ccqserv125.in2p3.fr:2131
>>>>>>>>> ssi.svclib libxrdsvc.so
>>>>>>>>> #oss.statlib -2 libXrdSsi.so
>>>>>>>>>
>>>>>>>>> Setup the environment as you normally would but don't start
>>>>>>>>> anything. By hand do:
>>>>>>>>>
>>>>>>>>> <path>/cmsd -d -c <path to config file above>
>>>>>>>>>
>>>>>>>>> Send the output to me. The uncomment the "statlib" directive
>>>>>>>>> and so the same thing again. Send that output to me as well.
>>>>>>>>>
>>>>>>>>> Andy
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> ########################################################################
>>>>>> Use REPLY-ALL to reply to list
>>>>>>
>>>>>> To unsubscribe from the QSERV-L list, click the following link:
>>>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>>>>>
>>>>
>>>> ########################################################################
>>>>
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the QSERV-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|
