Thanks Andy,
Simple test case is working now. Thanks so much for this quick fix.
qserv@ccqserv126:~$ cat cmsd.conf
all.role server
all.manager ccqserv125.in2p3.fr:2131
ssi.svclib libxrdsvc.so
oss.statlib -2 libXrdSsi.so
qserv@ccqserv126:~$ cmsd -c cmsd.conf
151112 20:27:48 4622 Starting on Linux 3.10.0-229.20.1.el7.x86_64
Copr. 2004-2012 Stanford University, xrd version 4.3.0.rc4.lsst1
++++++ cmsd [log in to unmask] initialization started.
Config using configuration file cmsd.conf
Config maximum number of connections restricted to 1048576
Copr. 2007 Stanford University/SLAC cmsd.
++++++ [log in to unmask] phase 1 initialization started.
=====> all.role server
=====> all.manager ccqserv125.in2p3.fr:2131
The following paths are available to the redirector:
r /
------ [log in to unmask] phase 1 server initialization completed.
++++++ [log in to unmask] phase 2 server initialization started.
Config warning: adminpath resides in /tmp and may be unstable!
++++++ Storage system initialization started.
=====> oss.statlib -2 libXrdSsi.so
Plugin No such file or directory loading statlib libXrdSsi-4.so
Config Falling back to using libXrdSsi.so
++++++ ssi phase 1 initialization started.
=====> all.role server
=====> ssi.svclib libxrdsvc.so
------ ssi phase 1 initialization completed.
++++++ ssi phase 2 initialization started.
0 [0x7f413e597740] INFO root null - SsiProvider initializing...
151112 20:27:48 4622 sysQserv: Provider Initializing
1 [0x7f413e597740] DEBUG root null - Launching query : SELECT db FROM
qservw_anon.Dbs
1 [0x7f413e597740] ERROR root null - ChunkInventory can't get list of
publishable dbs.
1 [0x7f413e597740] ERROR root null - Error -999: Error connecting to
mysql with config:[host=, port=0, usr=qsmaster, pass=, dbName=mysql,
socket=/var/lib/mysql/mysql.sock]
1 [0x7f413e597740] INFO root null - Provider valid paths(ci):
ChunkInventory()
151112 20:27:48 4622 sysQserv: Provider valid paths(ci): ChunkInventory()
------ ssi phase 2 initialization completed.
++++++ Configuring standalone mode . . .
Config effective cmsd.conf oss configuration:
oss.alloc 0 0 0
oss.cachescan 600
oss.fdlimit 524288 1048576
oss.maxsize 0
oss.trace 0
oss.xfr 1 deny 10800 keep 1200
oss.memfile off max 8355569664
oss.defaults r/w nocheck nodread nomig norcreate nopurge
nostage xattr
------ Storage system initialization completed.
151112 20:27:48 4622 Meter: Warning! No writable filesystems found.
151112 20:27:48 4622 Meter: Write access and staging prohibited.
------ [log in to unmask] phase 2 server initialization completed.
------ cmsd [log in to unmask]:33737 initialization completed.
151112 20:27:48 4635 Start: Waiting for primary server to login.
On 11/11/2015 04:04 PM, Fabrice Jammes wrote:
> Hi Andy,
>
> Thanks for this quick fix, Fritz explained me the deployment stuff so
> I will try to do it and let you know tomorrow.
>
> Thanks :-)
>
> On 11/11/2015 03:29 PM, Andrew Hanushevsky wrote:
>> Hi Fabrice,
>>
>> OK, the fix has been pushed to the xrootd git head xrdssi branch.
>> Now, all that needs to be done is to include this in the LSST version
>> of the branch so you can recompile. Unfortuantely, Fritz would do all
>> of this but I think he is away, sigh.
>>
>> Andy
>>
>> On Wed, 11 Nov 2015, Fabrice Jammes wrote:
>>
>>> Hi Andy,
>>>
>>> I'm afraid our distributed setup is broken for a few time... Thanks
>>> for your help and for the future fix ;-)
>>>
>>> Regards,
>>>
>>> On 11/11/2015 03:13 PM, Andrew Hanushevsky wrote:
>>>> Hi Fabrice,
>>>>
>>>> Ah, OK, I see. This is a problem. There is no easy solution here. I
>>>> need to rework a bit of code to get the cmsd running. It has to
>>>> with the way the initialization is ordered, sigh. I won't have
>>>> something immediately an it will require code changes in the SSI.
>>>>
>>>> Andy
>>>>
>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote:
>>>>
>>>>> Hi Andy,
>>>>>
>>>>> Here's the requested traces:
>>>>>
>>>>> *cmsd starts successfully with the first config:*
>>>>>
>>>>> qserv@ccqserv126:~$ cat cmsd.conf
>>>>> all.role server
>>>>> all.manager ccqserv125.in2p3.fr:2131
>>>>> ssi.svclib libxrdsvc.so
>>>>> #oss.statlib -2 libXrdSsi.so
>>>>> qserv@ccqserv126:~$ cmsd -d -c cmsd.conf
>>>>> 151111 22:45:19 103 Starting on Linux 3.10.0-229.20.1.el7.x86_64
>>>>> Copr. 2004-2012 Stanford University, xrd version unknown
>>>>> ++++++ cmsd [log in to unmask] initialization started.
>>>>> Config using configuration file cmsd.conf
>>>>> Config maximum number of connections restricted to 1048576
>>>>> Config maximum number of threads restricted to 1048576
>>>>> 151111 22:45:19 103 XrdConfig: sendfile enabled.
>>>>> 151111 22:45:19 103 XrdSched: scheduling underused thread monitor
>>>>> in 780 seconds
>>>>> 151111 22:45:19 104 XrdXeq: Buffer Manager reshaper thread started
>>>>> 151111 22:45:19 105 XrdXeq: Time scheduler thread started
>>>>> 151111 22:45:19 103 XrdSched: Starting with 2 workers
>>>>> 151111 22:45:19 103 XrdLink: Allocating 8 link objects at a time
>>>>> 151111 22:45:19 107 XrdXeq: Worker thread started
>>>>> 151111 22:45:19 106 XrdXeq: Worker thread started
>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 0
>>>>> 151111 22:45:19 108 XrdXeq: Poller thread started
>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 1
>>>>> 151111 22:45:19 109 XrdXeq: Poller thread started
>>>>> 151111 22:45:19 103 XrdPoll: Starting poller 2
>>>>> 151111 22:45:19 110 XrdXeq: Poller thread started
>>>>> 151111 22:45:19 103 XrdProtocol: getting port from protocol cmsd
>>>>> Copr. 2007 Stanford University/SLAC cmsd.
>>>>> ++++++ [log in to unmask] phase 1 initialization started.
>>>>> =====> all.role server
>>>>> =====> all.manager ccqserv125.in2p3.fr:2131
>>>>> The following paths are available to the redirector:
>>>>> r /
>>>>>
>>>>> ------ [log in to unmask] phase 1 server initialization
>>>>> completed.
>>>>> 151111 22:45:19 103 XrdConfig: LCL port 37568 wsz=87380 (87380)
>>>>> 151111 22:45:19 103 XrdProtocol: getting protocol object cmsd
>>>>> ++++++ [log in to unmask] phase 2 server initialization
>>>>> started.
>>>>> Config warning: adminpath resides in /tmp and may be unstable!
>>>>> 151111 22:45:19 103 Configure2 Global System Identification:
>>>>> anon-s 2131ccqserv125.in2p3.fr
>>>>> ++++++ Storage system initialization started.
>>>>> ++++++ Configuring standalone mode . . .
>>>>> 151111 22:45:19 103 oss_AioInit: started AIO read signal thread;
>>>>> tid=1278469888
>>>>> 151111 22:45:19 103 oss_AioInit: started AIO write signal thread;
>>>>> tid=1277417216
>>>>> Config effective cmsd.conf oss configuration:
>>>>> oss.alloc 0 0 0
>>>>> oss.cachescan 600
>>>>> oss.fdlimit 524288 1048576
>>>>> oss.maxsize 0
>>>>> oss.trace fff
>>>>> oss.xfr 1 deny 10800 keep 1200
>>>>> oss.memfile off max 8355569664
>>>>> oss.defaults r/w nocheck nodread nomig norcreate nopurge
>>>>> nostage xattr
>>>>> ------ Storage system initialization completed.
>>>>> 151111 22:45:19 103 Start Srv=0 dfs=0 lcl=0 Pre=1 dmLife=0 0
>>>>> 151111 22:45:19 103 Start Lim=0 0 fix=0 Qmax=1
>>>>> 151111 22:45:19 103 Meter: Warning! No writable filesystems found.
>>>>> 151111 22:45:19 103 Update Space Parm1=0 Parm2=0
>>>>> 151111 22:45:19 103 Meter: Write access and staging prohibited.
>>>>> ------ [log in to unmask] phase 2 server initialization
>>>>> completed.
>>>>> 151111 22:45:19 107 XrdSched: running cmsd startup inq=0
>>>>> 151111 22:45:19 113 XrdXeq: Notification handler thread started
>>>>> 151111 22:45:19 115 XrdXeq: Admin traffic thread started
>>>>> 151111 22:45:19 114 XrdXeq: Prep handler thread started
>>>>> 151111 22:45:19 115 Start: Waiting for primary server to login.
>>>>> ------ cmsd [log in to unmask]:37568 initialization completed.
>>>>> 151111 22:45:19 106 XrdSched: Now have 3 workers
>>>>> 151111 22:45:19 106 XrdSched: running main accept inq=0
>>>>> 151111 22:45:19 117 XrdXeq: Worker thread started
>>>>>
>>>>> *cmsd crashes with the second config:*
>>>>>
>>>>> qserv@ccqserv126:~$ cat cmsd.conf
>>>>> all.role server
>>>>> all.manager ccqserv125.in2p3.fr:2131
>>>>> ssi.svclib libxrdsvc.so
>>>>> oss.statlib -2 libXrdSsi.so
>>>>> qserv@ccqserv126:~$
>>>>> qserv@ccqserv126:~$ cmsd -d -c cmsd.conf
>>>>> 151111 22:58:54 137 Starting on Linux 3.10.0-229.20.1.el7.x86_64
>>>>> Copr. 2004-2012 Stanford University, xrd version unknown
>>>>> ++++++ cmsd [log in to unmask] initialization started.
>>>>> Config using configuration file cmsd.conf
>>>>> Config maximum number of connections restricted to 1048576
>>>>> Config maximum number of threads restricted to 1048576
>>>>> 151111 22:58:54 137 XrdConfig: sendfile enabled.
>>>>> 151111 22:58:54 137 XrdSched: scheduling underused thread monitor
>>>>> in 780 seconds
>>>>> 151111 22:58:54 138 XrdXeq: Buffer Manager reshaper thread started
>>>>> 151111 22:58:54 141 XrdXeq: Worker thread started
>>>>> 151111 22:58:54 137 XrdSched: Starting with 2 workers
>>>>> 151111 22:58:54 137 XrdLink: Allocating 8 link objects at a time
>>>>> 151111 22:58:54 139 XrdXeq: Time scheduler thread started
>>>>> 151111 22:58:54 140 XrdXeq: Worker thread started
>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 0
>>>>> 151111 22:58:54 142 XrdXeq: Poller thread started
>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 1
>>>>> 151111 22:58:54 143 XrdXeq: Poller thread started
>>>>> 151111 22:58:54 137 XrdPoll: Starting poller 2
>>>>> 151111 22:58:54 144 XrdXeq: Poller thread started
>>>>> 151111 22:58:54 137 XrdProtocol: getting port from protocol cmsd
>>>>> Copr. 2007 Stanford University/SLAC cmsd.
>>>>> ++++++ [log in to unmask] phase 1 initialization started.
>>>>> =====> all.role server
>>>>> =====> all.manager ccqserv125.in2p3.fr:2131
>>>>> The following paths are available to the redirector:
>>>>> r /
>>>>>
>>>>> ------ [log in to unmask] phase 1 server initialization
>>>>> completed.
>>>>> 151111 22:58:54 137 XrdConfig: LCL port 52851 wsz=87380 (87380)
>>>>> 151111 22:58:54 137 XrdProtocol: getting protocol object cmsd
>>>>> ++++++ [log in to unmask] phase 2 server initialization
>>>>> started.
>>>>> Config warning: adminpath resides in /tmp and may be unstable!
>>>>> 151111 22:58:54 137 Configure2 Global System Identification:
>>>>> anon-s 2131ccqserv125.in2p3.fr
>>>>> ++++++ Storage system initialization started.
>>>>> =====> oss.statlib -2 libXrdSsi.so
>>>>> Plugin No such file or directory loading statlib libXrdSsi-4.so
>>>>> Config Falling back to using libXrdSsi.so
>>>>> ++++++ ssi phase 1 initialization started.
>>>>> =====> all.role server
>>>>> =====> ssi.svclib libxrdsvc.so
>>>>> ------ ssi phase 1 initialization completed.
>>>>> ++++++ ssi phase 2 initialization started.
>>>>> 151111 22:58:54 137 sysFinder: Network i/f undefined; unable to
>>>>> self-locate.
>>>>> ------ ssi phase 2 initialization failed.
>>>>> ++++++ Configuring standalone mode . . .
>>>>> ------ Storage system initialization failed.
>>>>> ------ [log in to unmask] phase 2 server initialization failed.
>>>>> 151111 22:58:54 137 XrdProtocol: Protocol cmsd could not be loaded
>>>>> ------ cmsd [log in to unmask]:-1 initialization failed.
>>>>>
>>>>> Hope it'll help.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On 11/11/2015 02:10 PM, Andrew Hanushevsky wrote:
>>>>>> Hi Fabrice,
>>>>>>
>>>>>> Odd. OK, my answers....
>>>>>>
>>>>>> On Wed, 11 Nov 2015, Fabrice Jammes wrote:
>>>>>>
>>>>>>>> 1) Who is producing the following messages?
>>>>>>> This messages are in cmsd logs and are produced by xrootd:
>>>>>> Got it. OK, this is because of static initialization of something
>>>>>> we will not use but cannot easily avoid initializing. It should
>>>>>> be OK.
>>>>>>
>>>>>>>> 2) The "statlib" uses the libXrdSsi.so because we packaged it
>>>>>>>> there as a convenience since we need to use the file registry.
>>>>>>>> Do you have a static initialization section that expects it
>>>>>>>> will fire up all of qserv? We don't want that.
>>>>>>> I don't really understand this question, sorry. Here's our
>>>>>>> configuration file, it may help?
>>>>>> I just answered in in (1). This is the xrootd client doing static
>>>>>> initialization and this is because the SSI library uses the
>>>>>> client so it is forced to be initialized when the client library
>>>>>> is loaded.
>>>>>>
>>>>>>>> 3) This is a container, right?
>>>>>>> Yes. FYI, our previous cmsd version was running fine under the
>>>>>>> same sort of container with same network setting.
>>>>>> Then is should run here.
>>>>>>
>>>>>>>> 5) I assume things are registered in DNS or at least appear
>>>>>>>> correctly in /etc/hosts otherwise we will have a problem. The
>>>>>>>> container has to look like an actual machine.
>>>>>>> # runned inside he container
>>>>>>> root@ccqserv126:/qserv# ping ccqserv126
>>>>>>> PING ccqserv126.in2p3.fr (172.17.0.7): 56 data bytes
>>>>>>> 64 bytes from 172.17.0.7: icmp_seq=0 ttl=64 time=0.061 ms
>>>>>>> 64 bytes from 172.17.0.7: icmp_seq=1 ttl=64 time=0.049 ms
>>>>>> OK, it's properly registered. So, type up a small config file, as
>>>>>> follows:
>>>>>>
>>>>>> all.role server
>>>>>> all.manager ccqserv125.in2p3.fr:2131
>>>>>> ssi.svclib libxrdsvc.so
>>>>>> #oss.statlib -2 libXrdSsi.so
>>>>>>
>>>>>> Setup the environment as you normally would but don't start
>>>>>> anything. By hand do:
>>>>>>
>>>>>> <path>/cmsd -d -c <path to config file above>
>>>>>>
>>>>>> Send the output to me. The uncomment the "statlib" directive and
>>>>>> so the same thing again. Send that output to me as well.
>>>>>>
>>>>>> Andy
>>>>>
>>>>>
>>>
>>> ########################################################################
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the QSERV-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>>
>
> ########################################################################
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|