Jacek,

I'm happy to discuss it. FYI, the system already accepts multiple 
emptyChunk files (one per db), but it defaults to emptyChunks.txt if 
none is found.
Have a look at populateEmptyChunkInfo() in metadata.py.
Also, note that in the czar log you should see a message saying what it 
was looking for. Sorry, it's not configurable. It will always look in 
$CWD/empty_<dbname>.txt . At the time, I was hoping we could eliminate 
the empty chunks file with a lightweight secondary index, but that got 
shelved because I didn't have a good solution to handle updates.

-Daniel

On 05/16/2014 01:03 PM, Becla, Jacek wrote:
> Daniel
>
> I feel like it'd be good to get to the bottom of it this FY,
> at minimum we should allow multiple emptyChunk lists to
> coexist (one per database). I'll add to the list of to-discuss
> topics for next Wed.
>
> Thanks,
> Jacek
>
>
>
>
>
>
> On 05/16/2014 12:29 PM, Wang, Daniel Liwei wrote:
>> Ideally, we would rebuild it upon any changes to the dirTable. There was
>> code in indexing.py(I think?) that was a placeholder attempt of code
>> that could generate it.
>>
>> There are a couple ways of generating it:
>> 1) from the range of numbers defined by the min and max chunk number,
>> filter out chunks determined to be non-empty by the existence of
>> dirtable_NNN tables. This is what my scripts did, and it was a hassle to
>> get it to work.
>>
>> 2) Do a special all-chunks query (count(*) or similar), but don't squash
>> on errors-- add them to the empty chunks list.
>>
>> 3) Populate the empty chunks file when you create a database, or when a
>> czar becomes aware that a database exists. Every time you load data, you
>> know what chunks you are creating, so remove those chunks from the empty
>> chunks file/list. The czar always checks the db entry in css to see if
>> its empty chunks file is out of date. It is impossible to delete
>> partitioned rows, so chunks never become empty after being non-empty.
>>
>> 4) Compute a non-empty chunk list from the sec-index list: select
>> distinct chunkId from blah (will go obsolete soon) and create a
>> hash-table or std::map, and use it. The czar can compute and cache this
>> the first time the db is accessed.
>>
>> I know, it's annoying.
>> -Daniel
>>
>>
>> -Daniel
>>
>>
>> On 05/16/2014 11:59 AM, Jacek Becla wrote:
>>> Daniel
>>>
>>> Can you remind me what the plan are with regards to the
>>> emptyChunk.txt? Will it be going away any time soon?
>>> It can still lead to a lot of confusion: I had a fully
>>> working environment based on PT1.1 data set, but after
>>> I run qserv-testdata it silently overwrote the version
>>> I had (in build/dist/etc), and as a result I started
>>> getting an error:
>>>
>>> Table 'LSST.Object_1234567890' doesn't exist
>>>
>>> which I did not connect with bad emptyChunk.txt for
>>> quite some time.
>>>
>>> Thanks,
>>> Jacek
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the QSERV-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1