LISTSERV mailing list manager LISTSERV 16.5

Help for QSERV-L Archives


QSERV-L Archives

QSERV-L Archives


QSERV-L@LISTSERV.SLAC.STANFORD.EDU


View:

Message:

[

First

|

Previous

|

Next

|

Last

]

By Topic:

[

First

|

Previous

|

Next

|

Last

]

By Author:

[

First

|

Previous

|

Next

|

Last

]

Font:

Proportional Font

LISTSERV Archives

LISTSERV Archives

QSERV-L Home

QSERV-L Home

QSERV-L  November 2013

QSERV-L November 2013

Subject:

notes from Qserv mtg Nov 7

From:

Jacek Becla <[log in to unmask]>

Reply-To:

General discussion for qserv (LSST prototype baseline catalog)

Date:

Fri, 8 Nov 2013 14:38:04 -0800

Content-Type:

text/plain

Parts/Attachments:

Parts/Attachments

text/plain (178 lines)

Fabrice, Serge, Daniel, Douglas, Jacek,



Agenda:
  * quick round table status report
  * outstanding action items
  * coding standards
  * layout for Qserv hackathon day 1
  * Qserv modularization



quick round table status report
===============================

Daniel
  - join syntax, including logic for substituting names
    for overlap, previous version was "terrible", current
    version is more general

Serge
  - concurrency-related tickets
  - progress with SciDB tests, but run into hardware problems

Fabrice
  - working on auto testing, have clean system

Jacek
  - final touches to W14 Qserv plan
  - prep for hackathon:
    - proposal for day 1 layout on trac
    - added "tentative list of outcomes", see
 
https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/201311_hackathon
    - coordinated build/package discussion time
      w/Mario and Robert (Mon 9:30am)
    - arranged meeting rooms
  - points from wishlists documented in appropriate places
    (where obvious)
  - documented python usage, see:
    https://dev.lsstcorp.org/trac/wiki/db/Qserv/RedesignFY2014/pythonCPP
  - researched and documented mysql wrappers in python, see:
    https://dev.lsstcorp.org/trac/wiki/db/MySQLWrapper4Python
  - started work on interfaces for Data Loading, see:
 
https://dev.lsstcorp.org/trac/wiki/db/Qserv/DataLoading/Interfaces/InputData
  - working on letter of collaboration for Florin's DOE grant

Douglas
  - evaluated Zookeeper
  - now looking into Doozerd, etcd, Concoord and Openreplica
  - all documented at:
    https://dev.lsstcorp.org/trac/wiki/db/Qserv/ClusterControl


Side discussions:
  - wrapper
    - qserv might need more raw access that other parts that
      talk to mysql
    - want to avoid accessing mysql through python in Qserv core,
      only from admin
    - qms likely goes away, replaced by something like Zookeeper
    - btw, we have another wrapper in admin used for data loader


Cluster mgmt tools
  - Qserv will likely run on variety of clusters, some of
    them not under our control, and they might require
    specialized cluster mgmt tools (and won't accept our).
    We can't support all of them, we will pick one and will
    support it, if others want to use different cluster mgmt
    tools, they will have to do the porting

One large ticket vs several smaller tickets?
  - from review point, smaller tickets ar better
  - but there is overhead per ticket (branch, merging etc)

Qserv code reviews
  - Issue K-T raises from time to time: want non-qserv
    people to review Qserv tickets
  - Often push back "why was I assigned to this ticket?"
  - We still want to review our own tickets to keep up
    to date with changes

Keeping documentation of tests run by Fabrice on in2p3 servers
  - not big issue for now, but better to keep all
    in one place, eg. on trac

Monitor trac changes daily
  - e.g., use timeline, or "watch" pages


outstanding action items
========================
  - #3007 CLOSED
  - #3013 reviewed, change owners to FJ and close
  - #3004 depends on #3013, close when #3013 closed
  - then close #3006 when above done [FJ]
  - #3019 (concurrency) almost done
  - #3020 not done yet, unlikely to happen in the
    next few days
  - cut release
    - include xrootd patch 5
  - will cut another release with 3019, 3020
  - document what is now done in python DONE
  - hackathon planning DONE
  - document ideas about user friendly communication
    and syntax parsing ("--unsafe flag") STILL TODO [DW]
  - modularization DONE
  - lay out list of things that can be done
    on either worker or master STILL TODO [DW]
  - bring Mario/Robert discussion about
    building/packaging, during hackathon week DONE
  - add to trac hackathon planning: "tentative list
     of outcomes" DONE
  - take apart wishlists and document each point
    raised in appropriate place DONE
  - work on objectId index STILL TODO, CAN WAIT [DW]
  - join syntax improvements DONE
  - small tweaks to shared scan ticket DONE
  - follow up with Florin Rusu, check his code DONE

  - talk to Robyn about integrating with buildbot
    sometime later next week DONE
  - send Fabrice info about policies/restrictions
    regarding in2p3 contributions *during construction*
    DONE (Dominique plugged into the loop)
  - work on collaboration with Florin Rusu ON TRACK



coding standards
================
  - most files in admin don't have any headers, comments
    see https://dev.lsstcorp.org/trac/wiki/CodeStandards
  - raising/handling errors, incl throwing exceptions
    - would be useful to have one style and stick with it



Qserv modularization
====================

See:

https://listserv.slac.stanford.edu/cgi-bin/wa?A2=ind1311&L=QSERV-L&F=&S=&P=5294

  - this covers C++ only
  - note, that replaces common, worker and master directories
  - looks fine at first glance
  - need to cover python to see complete flow
  - think about modules for admin [FJ, DS]
  - would be useful to prefix some files,
    otherwise once we add (many) files from admin,
    it'll be too messy
    - e.g., prefix all files that belong to master (with 'm'?)
      and all files that belong to admin (with 'a'?)



proposed layout for Qserv hackathon day 1
=========================================

https://listserv.slac.stanford.edu/cgi-bin/wa?A2=ind1311&L=QSERV-L&F=&S=&P=574

sounds like a good plan for 1st day, then we will revisit
  - possibly, might need few larger mtgs


Jacek

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1

Top of Message | Previous Page | Permalink

Advanced Options


Options

Log In

Log In

Get Password

Get Password


Search Archives

Search Archives


Subscribe or Unsubscribe

Subscribe or Unsubscribe


Archives

March 2018
February 2018
January 2018
December 2017
August 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012

ATOM RSS1 RSS2



LISTSERV.SLAC.STANFORD.EDU

Secured by F-Secure Anti-Virus CataList Email List Search Powered by the LISTSERV Email List Manager

Privacy Notice, Security Notice and Terms of Use