Hi Fabrice
Looks great! If you have access to this machine for longer,
can you try the last test with 2 GB, 16 GB and 48 GB sort
buffer size?
I see one issue: why are you torturing our poor qserv
repo so much? It looks like you have a branch
u/fjammes/mysqlLargeTables
for that, but I can't get to your code by checking out
your branch, and I had to do:
git checkout 0b84d416e95a
which is in "detached from HEAD" state.
Please create directory "mysqlLargeTable" directory in
dbutils and move your scripts there
(git clone [log in to unmask]:/contrib/dbutils.git dbutils
to get there)
thanks
Jacek
On 05/28/2013 07:56 AM, Fabrice Jammes wrote:
> Hello Jacek,
>
> I've done and documented the tests run with mariadb, myisamchk, and
> RunDeepForcedSource of Winter2013.
> It's here :
> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3
>
> Feel free to ask for any additional information.
>
> Fabrice
>
> On 05/07/2013 09:51 PM, Jacek Becla wrote:
>> Fabrice
>>
>>> Of course we can have a look a it.
>>
>> That is great!
>>
>>> Is there additional instructions for b) point ? If yes, could you send
>>> it to us ?
>>
>> 1) Have a RunDeepForcedSource table ready. It should not be
>> a production database, because we will be doing bad things
>> to it.
>>
>> 2) Disable indexes (alter table RunDeepForcedSource disable index,
>> than )
>>
>> 3) do not use that table through mysql while doing everything
>> below. Either shut down mysqld, or run "flush cache" and
>> don't touch that table
>>
>> 4) then test myisamchk, note that this is command line tool
>>
>> time /usr/local/mysql/bin/myisamchk
>> -v -t <tmp_dir>
>> --force
>> -o
>> --update-state
>> -p
>> --key_buffer_size=8M
>> --sort_buffer_size= <~1/2 of your RAM available>
>> --read_buffer_size=16M
>> --write_buffer_size=16M
>> <mysql data dir>/<db>/RunDeepForcedSource.MYI
>>
>>
>> that will most likely fail, see
>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTables
>>
>> Then apply the patch and retry, Monty mentioned some
>> extra switches (--force --force, etc), see his email,
>> and try them
>>
>> Please capture output (and the timing!) for each
>> thing you run.
>>
>> it is probably better to use a machine that has
>> reasonable amount of memory (say 32 or 64G), not
>> insanely large amount of memory (on a machine with
>> 256G ram I couldn't reproduce this problem).
>>
>> It is important that the <tmp dir> has enough space,
>> at least the size of one copy of RunDeepForcedSource.*
>> files.
>>
>> thanks
>> Jacek
>>
>
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
|