To IN2P3 team: I wanted to draw you attention to two things we discussed today here in the US: 1) qserv master is using lots of threads. We may have to tinker with threads-max to increase it for the large scale test. 2) please do not commit from this account: "Compte Installe Qserv <[log in to unmask]>" (or any other generic account :), always use user-name account full notes below in case you didn't see that through lsst-data. Thanks, Jacek -------- Original Message -------- Subject: notes from qserv mtg (May 2) Date: Thu, 02 May 2013 22:24:42 -0700 From: Jacek Becla <[log in to unmask]> Organization: SLAC National Accelerator Laboratory To: LSST Data Management <[log in to unmask]> Attendees: Serge, Douglas, Daniel, Jacek W13 (Douglas) - all good, row counts correct, etc - had issues: one table was missing in the dump - csv dump periodically had a blank line, which crushed partitioner --> try to find out how these lines got inserted [Jacek] - next steps: a) transfer to lsst10 [Douglas] b) install qserv on lsst10 [Douglas] buildbot - was waiting for fixes from in2p3 - have it now (in 0.5rc1) --> Douglas will follow up w/Robyn qserv releases - when cutting a release, keep release notes: - version - date - short list of major things that went in - list of tickets parser (Daniel) - common queries, eg selects, scans, near neighbor, count, limit etc all supported - working on "average" - can of worms, need to support math, affects most of the parser. - Implemented, stabilizing these changes now - Daniel leaving tomorrow for over a week (out of country) - will have it ready for review after next week - move parser code to separate directory? - 20 new files, 10-15 go away - not yet, maybe in the future metadata (Jacek) - migrated lots more qserv code, all table parameters are now configured via metadata - also fixed mysql connection timeout issue in qms (#2851) - need to fix in qserv, Douglas already saw issues with qserv after day+ inactivity (#2854) "next" branch - just created - in the next few weeks will be used to merge new parser code+queryRewriting with new metadata partitioner (Serge) - started crushing when turned on compiler optimizations (-o2), in thread segfaults - because gcc string not thread safe - warning: we are using strings in master, too and it is multi-threaded - probably didn't see the issues because we don't use compiler optimizations (yet...) --> try with the optimizations [Serge] - do we need self-overlap? - probably not threading in the master (Serge) - started looking into it managing info about schema - qms keeps full info about schema - new partitioner needs info about partitioned tables - currently managed through config - see source.cfg in #2458 - suggestion: new partitioner should query qms to get info it needs --> create ticket: [Serge] --> document better the metadata [Jacek] large scale tests prep - nodes arriving in May - would be good if Douglas could go to in2p3 - eg, june 29-july 6 --> re-ask Jeff [Jacek] --> talk to in2p3 about need to increase max_threads count on the test machines There are commits in qserv done by "Compte Installe Qserv <[log in to unmask]>" - we want all commits to be signed by real people --> mention to the IN2P3 team do we need the new duplicator for the large scale test? - yes! --> Douglas will test it --> Daniel will review unclassified - scheduling mtg with Monty in July - trying to find dates that work for db architecture potential reviewers - Douglas gone July 30-Aug 8 writing documentation (today!) - duplicator [Serge] - partitioning (Serge added last night - all: read) - parser [Daniel] - metadata [Jacek] - partitioning w13 using batch nodes [douglas] next meetings - next week (without Daniel) - 2 weeks from now (without Jacek) - 2 weeks from now is the larger mtg, Daniel can coordinate Jacek ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1