Vaikunth,

1. I'd be good if you'd use the fixes we just pushed to the tip
    of the master in the last 2-3 days, so rebase your branch and
    rebuild things and try it.

2. If "1" does not help, run
mysql --host=127.0.0.1 --port=13306 -u root -p -A

and then:

show status like 'Threads_connected';

and

show variables like 'max_connections';

If Threads_connected approaches max_connections, that is your problem.

3. If 2 does not help, try running your failing query by hand,
    using the same credentials as what Qserv is using when it runs
    the failing query.

Jacek




On 05/07/2015 11:10 AM, Vaikunth Thukral wrote:
> Hi all,
>
> So I've been able to resolve the remaining issues with the data loader
> (thanks AndyS for your help) for integration tests in multi-node and
> cases 1, 2, 3 and 5 are now working. For case 4 however the last query
> fails (query 0030_largeResult.sql). The czar log gives the following
> (full logs available on dbdev1 at
> /usr/local/home/vaikunth/qserv-run/var/log/):
>
> 0507 12:45:37.872 [0x7f7cf2137700] INFO  root
> (build/rproc/InfileMerger.cc:351) - InfileMerger 0x7f7cdc0105b0
> connected to db.
> 0507 12:45:37.872 [0x7f7cf2137700] ERROR root
> (build/rproc/InfileMerger.cc:359) - InfileMerger sql error: Error
> applying sql. Error 1146: Tabl\
> e 'qservResult.result_11900632096_m' doesn't exist Unable to execute
> query: CREATE TABLE qservResult.result_11900632096 SELECT deepForcedSourc\
> eId,scienceCcdExposureId,filterId,deepSourceId,timeMid,expTime,ra,decl,raVar,declVar,radeclCov,htmId20,x,y,xVar,yVar,xyCov,psfFlux,psfFluxSigm\
> a,apFlux,apFluxSigma,modelFlux,modelFluxSigma,instFlux,instFluxSigma,apCorrection,apCorrectionSigma,shapeIx,shapeIy,shapeIxVar,shapeIyVar,shap\
> eIxIyCov,shapeIxx,shapeIyy,shapeIxy,shapeIxxVar,shapeIyyVar,shapeIxyVar,shapeIxxIyyCov,shapeIxxIxyCov,shapeIyyIxyCov,extendedness,flagNegative\
> ,flagBadMeasCentroid,flagPixEdge,flagPixInterpAny,flagPixInterpCen,flagPixSaturAny,flagPixSaturCen,flagBadPsfFlux,flagBadApFlux,flagBadModelFl\
> ux,flagBadInstFlux,flagBadCentroid,flagBadShape,raDeepSource,declDeepSource
> FROM qservResult.result_11900632096_m ORDER BY deepForcedSourceId
>
> 0507 12:45:37.872 [0x7f7cf2137700] INFO  root
> (build/rproc/InfileMerger.cc:313) - Cleaning up
> qservResult.result_11900632096_m
> 0507 12:45:37.872 [0x7f7cf2137700] INFO  root
> (build/rproc/InfileMerger.cc:325) - Merged
> qservResult.result_11900632096_m into qservResult.res\
> ult_11900632096
> 0507 12:45:37.872 [0x7f7cf2137700] ERROR root
> (build/ccontrol/UserQuery.cc:206) - Joined everything (failure!)
> 0507 12:45:37.874 [0x7f7cf2137700] INFO  root (app.py:408) - Query exec
> (5) took 1.873482 seconds
> 0507 12:45:37.874 [0x7f7cf2137700] ERROR root
> (build/qdisp/Executive.cc:360) - Ref=1
> Resource(/chk/qservTest_case04_qserv/6970): 20150507-12:4\
> 5:37, Error in result data., 1,
> Ref=2 Resource(/chk/qservTest_case04_qserv/7138): 20150507-12:45:37,
> Error in result data., 1,
> Ref=3 Resource(/chk/qservTest_case04_qserv/7140): 20150507-12:45:37,
> Error in result data., 1,
> Ref=4 Resource(/chk/qservTest_case04_qserv/7308): 20150507-12:45:37,
> Error in result data., 1,
> Ref=5 Resource(/chk/qservTest_case04_qserv/7310): 20150507-12:45:37,
> Error in result data., 1,
> 0507 12:45:37.875 [0x7f7cf2137700] DEBUG root (app.py:343) - reporting
> -1 -1 Ref=1 Resource(/chk/qservTest_case04_qserv/6970): 20150507-12:45:\
> 37, Error in result data., 1,
> Ref=2 Resource(/chk/qservTest_case04_qserv/7138): 20150507-12:45:37,
> Error in result data., 1,
> Ref=3 Resource(/chk/qservTest_case04_qserv/7140): 20150507-12:45:37,
> Error in result data., 1,
> Ref=4 Resource(/chk/qservTest_case04_qserv/7308): 20150507-12:45:37,
> Error in result data., 1,
> Ref=5 Resource(/chk/qservTest_case04_qserv/7310): 20150507-12:45:37,
> Error in result data., 1,
> 0507 12:45:37.875 [0x7f7cf2137700] ERROR root
> (build/qdisp/MessageStore.cc:47) - Msg: -1 -1 Ref=1
> Resource(/chk/qservTest_case04_qserv/6970): \
> 20150507-12:45:37, Error in result data., 1,
> Ref=2 Resource(/chk/qservTest_case04_qserv/7138): 20150507-12:45:37,
> Error in result data., 1,
> Ref=3 Resource(/chk/qservTest_case04_qserv/7140): 20150507-12:45:37,
> Error in result data., 1,
> Ref=4 Resource(/chk/qservTest_case04_qserv/7308): 20150507-12:45:37,
> Error in result data., 1,
> Ref=5 Resource(/chk/qservTest_case04_qserv/7310): 20150507-12:45:37,
> Error in result data., 1,
> 0507 12:45:37.876 [0x7f7cf2137700] INFO  root (app.py:413) - Final state
> of all queries error
> /usr/local/home/vaikunth/src/qserv/lib/python/lsst/qserv/czar/db.py:100:
> Warning: Data truncated for column 'message' at row 1
>    c.execute(sql, params)
> 0507 12:45:37.878 [0x7f7cf1736700] INFO  root
> (build/ccontrol/UserQuery.cc:236) - Discarded UserQuery(5)
>
> I am not sure if this is related to what Fabrice is seeing at in2p3, but
> when I run a simpler version of the 0030_ query myself I get (with error
> log also in the same location as mentioned above):
>
> [vaikunth@lsst-dbdev1 qserv_testdata]$ mysql
> --host=lsst-dbdev1.ncsa.illinois.edu
> <http://lsst-dbdev1.ncsa.illinois.edu> --port=4040 --user=qsmaster
> --batch qservTest_case04_qserv -e "SELECT * FROM DeepForcedSource ORDER
> BY deepForcedSourceId"
> ERROR 4120 (Proxy) at line 1: Error during execution:
> -1 Ref=1 Resource(/chk/qservTest_case04_qserv/6970): 20150507-13:07:32,
> Error in result data., 1,
> Ref=2 Resource(/chk/qservTest_case04_qserv/7138): 20150507-13:07:32,
> Error in result data., 1,
> Ref=3 Resource(/chk/qservTest_case04_qserv/7140): 20150507-13:0 (-1)
>
> -Vaikunth
>
> ------------------------------------------------------------------------
>
> Use REPLY-ALL to reply to list
>
> To unsubscribe from the QSERV-L list, click the following link:
> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1