Jacek,

Thanks for your answer. I will let Fabrice and Osman to deal with this. 
I guess that Fabrice knows how to run the partitioner but he may have 
some specific questions.

Dominique
Le 13/01/2014 11:54, Jacek Becla a écrit :
> Dominique
>
> I think the best would be to break the largest tables into
> a reasonably small number of smaller tables (say 16, that
> should put the table sizes back into the regime where things
> are working reasonably fast.). I think writing a trivial
> python script to do that would be easier than going through
> partitioner. But indeed, if it is experimental, we could
> run it by the partitioner. Would you like the instructions
> how to use the partitioner, or will you be trying to divide
> the largest tables through a simple python script first?
>
> Jacek
>
>
>
>
>
> On 01/13/2014 10:36 AM, Dominique Boutigny wrote:
>> Hi Jacek,
>>
>> As far as I understand there is no unsolved issues with the standard
>> (non qserv) procedure, it is just awfully (and at the limit of the
>> usability) long. The idea here was to try qserv partitioning to see if
>> it is improving the performances for the ingestion step as well as for
>> the user access.
>> There are no issues with the Data Challenge, as IN2P3 already fulfilled
>> its commitment with LSST, the goal is to explore this partitioning
>> functionality on a single MySQL node.
>>
>> Dominique
>>
>> Le 13/01/2014 10:22, Jacek Becla a écrit :
>>> Fabrice
>>>
>>> It'd be better to not pull Qserv into any Data Challenges yet.
>>>
>>> Is there an issue with using the merge engine? If we use
>>> partitioner to just break data into separate tables, we can
>>> do that easily without partitioner, and if we start using the
>>> advanced features like overlap, it will quickly get out of hand.
>>>
>>> Can Osman send more specific information what is not working?
>>> As I said earlier, I have not heard about any issues since I
>>> exchanged emails with him on Dec 11 (which I just forwarded
>>> to you).
>>>
>>> Jacek
>>>
>>>
>>>
>>> On 01/13/2014 09:53 AM, Fabrice Jammes wrote:
>>>> Hello,
>>>>
>>>> Osman Aidel, a CC-IN2P3 expert in databases administration, try
>>>> currently to load in MySQL the 3TB dataset produced during last data
>>>> challenge.
>>>> Osman and Dominique Boutigny succeeded in loading the whole dataset in
>>>> MySQL, but some post-processing steps of this dataset (like removal of
>>>> duplicates) take a infinite time.
>>>>
>>>> Please remark that these issues are good news for Qserv as it 
>>>> validates
>>>> its distributed data model ;-).
>>>>
>>>> Christian Arnault, french manager for LSST computing, thinks that some
>>>> of the tools developed by Qserv team could help Osman, and CC-IN2P3.
>>>> Indeed, the partitioning algorithm developed by Serge could be used to
>>>> partition the DC dataset in a collection of chunks.
>>>> Osman could then load a part of contiguous chunks of this collection
>>>> into a single-node MySQL server.
>>>>
>>>> Do you think this proposal could be attainable soon ? Indeed, French
>>>> physicists are interested in studying a representative sample of 
>>>> the DC
>>>> dataset and this solution would help them a lot.
>>>>
>>>> Thanks,
>>>>
>>>> Fabrice
>>>>
>>>
>>> ######################################################################## 
>>>
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the QSERV-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>
>

-- 
Dominique Boutigny  -  CNRS / CC-IN2P3
Now at SLAC National Accelerator Laboratory

Mail     : [log in to unmask] -  [log in to unmask]
Office   : +1 650-926-5759   -  Cellular : +1 774-232-0912



########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1