Hi Fabrice,
yes, for now we need to provide both schema for table and
a separate column list for partitioner. It would be better
to avoid this duplication, for that we need to either parse
schema definition or create table in the database and then
extract column list from database. I guess we need a
separate ticket for this and maybe a discussion about overall
design, I'm not sure I understand all issues involved here.
Cheers,
Andy
Fabrice Jammes wrote on 2014-12-18:
> Hi Andy,
>
> Right now, I have to provide twice time to the data-loader the column
> list for a partitionted table, once in the cfg file and once
> in the schema file, isn't it?
>
> Could we deduce the cfg from the schema in the data-loader? It would
> produce more maintainable configuration.
>
> Thanks
>
> Fabrice
>
>
> On 12/18/2014 01:14 PM, Daniel L. Wang wrote:
>> Hi Andy,
>>
>> I think what I will do is have qserv-admin just read it in as-is and
>> pass it to qservAdmin. It will differ from what the loader is doing,
>> but that wont matter until there is code that actually using it. If we
>> need the schema to look a certain way, we will put that processing
>> code somewhere so that the loader and css can both use it.
>>
>> -Daniel
>>
>>
>> On 12/18/2014 01:09 PM, Salnikov, Andrei A. wrote:
>>> Hi Daniel,
>>>
>>> right, in dataLoader I prepare schema myself before passing it to
>>> qservAdmin. I'm not sure that you can read schema reliably from a
>>> file, the files that we have contain lots of comments which you
>>> probably want to strip away. What I do in dataLoader is first to
>>> create table in mysql and then run "SHOW CREATE TABLE ..." and
>>> dump its output to CSS (stripping "CREATE TABLE TableName" part).
>>> I do not really know if something like it can be done inside
>>> qserv-admin.py.
>>>
>>> Cheers,
>>> Andy
>>>
>>>
>>> Daniel L. Wang wrote on 2014-12-17:
>>>> Hello,
>>>>
>>>> I'm taking a look at validating the arguments for qserv-admin.py, and
>>>> in particular, I've noticed that it ignores the "schemaFile"
>>>> parameter, which means that it doesn't bother loading the schema
>>>> file, which means that all the example *.params files that we have
>>>> that say "schemaFile: ....../something.sql" are including that line
>>>> as a placeholder.
>>>>
>>>> I think this is the reason why dataLoader.py stuffs the "schema"
>>>> option manually when invoking qservAdmin, rather than passing a
>>>> schema file.
>>>>
>>>> Is this correct? I think this means that I need to add logic for
>>>> loading
>>>> the schema file in qserv-admin.py .
>>>>
>>>> DM-370 was supposed to improve handling of default values for
>>>> parameters, but in order to do that, it seems to need a solid base of
>>>> validating parameters in order to do the right thing for defaulting.
>>>> So,
>>>> the scope is bigger than I thought. I will try to complete it in Dec,
>>>> but not positive anymore.
>>>>
>>>> -Daniel
>>>>
>>>>
>>>> ######################################################################
>>>> ##
>>>>
>>>> Use REPLY-ALL to reply to list
>>>>
>>>> To unsubscribe from the QSERV-L list, click the following link:
>>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>>
>>>
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|