To IN2P3 team:
I wanted to draw you attention to two things we
discussed today here in the US:
1) qserv master is using lots of threads.
We may have to tinker with threads-max
to increase it for the large scale test.
2) please do not commit from this account:
"Compte Installe Qserv <[log in to unmask]>"
(or any other generic account :), always use user-name
account
full notes below in case you didn't see that
through lsst-data.
Thanks,
Jacek
-------- Original Message --------
Subject: notes from qserv mtg (May 2)
Date: Thu, 02 May 2013 22:24:42 -0700
From: Jacek Becla <[log in to unmask]>
Organization: SLAC National Accelerator Laboratory
To: LSST Data Management <[log in to unmask]>
Attendees: Serge, Douglas, Daniel, Jacek
W13 (Douglas)
- all good, row counts correct, etc
- had issues: one table was missing in the dump
- csv dump periodically had a blank line, which crushed
partitioner
--> try to find out how these lines got inserted [Jacek]
- next steps:
a) transfer to lsst10 [Douglas]
b) install qserv on lsst10 [Douglas]
buildbot
- was waiting for fixes from in2p3
- have it now (in 0.5rc1)
--> Douglas will follow up w/Robyn
qserv releases
- when cutting a release, keep release notes:
- version
- date
- short list of major things that went in
- list of tickets
parser (Daniel)
- common queries, eg selects, scans, near neighbor, count,
limit etc all supported
- working on "average" - can of worms, need to support math,
affects most of the parser.
- Implemented, stabilizing these changes now
- Daniel leaving tomorrow for over a week (out of country)
- will have it ready for review after next week
- move parser code to separate directory?
- 20 new files, 10-15 go away
- not yet, maybe in the future
metadata (Jacek)
- migrated lots more qserv code, all table parameters are now
configured via metadata
- also fixed mysql connection timeout issue in qms (#2851)
- need to fix in qserv, Douglas already saw issues
with qserv after day+ inactivity (#2854)
"next" branch
- just created
- in the next few weeks will be used to merge new
parser code+queryRewriting with new metadata
partitioner (Serge)
- started crushing when turned on compiler optimizations (-o2),
in thread segfaults
- because gcc string not thread safe
- warning: we are using strings in master, too and it is
multi-threaded
- probably didn't see the issues because we don't use
compiler optimizations (yet...)
--> try with the optimizations [Serge]
- do we need self-overlap?
- probably not
threading in the master (Serge)
- started looking into it
managing info about schema
- qms keeps full info about schema
- new partitioner needs info about partitioned tables
- currently managed through config
- see source.cfg in #2458
- suggestion: new partitioner should query qms
to get info it needs
--> create ticket: [Serge]
--> document better the metadata [Jacek]
large scale tests prep
- nodes arriving in May
- would be good if Douglas could go to in2p3
- eg, june 29-july 6
--> re-ask Jeff [Jacek]
--> talk to in2p3 about need to increase max_threads
count on the test machines
There are commits in qserv done by
"Compte Installe Qserv <[log in to unmask]>"
- we want all commits to be signed by real people
--> mention to the IN2P3 team
do we need the new duplicator for the large scale test?
- yes!
--> Douglas will test it
--> Daniel will review
unclassified
- scheduling mtg with Monty in July
- trying to find dates that work for db architecture
potential reviewers
- Douglas gone July 30-Aug 8
writing documentation (today!)
- duplicator [Serge]
- partitioning (Serge added last night - all: read)
- parser [Daniel]
- metadata [Jacek]
- partitioning w13 using batch nodes [douglas]
next meetings
- next week (without Daniel)
- 2 weeks from now (without Jacek)
- 2 weeks from now is the larger mtg, Daniel can coordinate
Jacek
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|
