Fabrice

It is more interesting than I thought!

I am going to forward to Monty (mysql founder/best guru)
to see what he thinks about your results. I certainly
didn't expect some of these tests to fail.

I will cc you, he will probably have questions
about details or suggestions how to try it to
pin down the problems.

Jacek

p.s. I have new, VERY interesting and important
set of tests to run. More soon.




On 06/06/2013 01:47 AM, Fabrice Jammes wrote:
> Hello Jacek,
>
> The last test has been launched with sort_buffer_size parameter values
> you asked for.
> The results are here :
> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3
>
> Best regards,
>
> Fabrice
>
> On 05/28/2013 06:36 PM, Fabrice Jammes wrote:
>> Hi Jacek,
>>
>>
>> On 05/28/2013 05:26 PM, Jacek Becla wrote:
>>> Hi Fabrice
>>>
>>> Looks great! If you have access to this machine for longer,
>>> can you try the last test with 2 GB, 16 GB and 48 GB sort
>>> buffer size?
>> It's possible, nevertheless i've a lot of tasks to do for in2p3 until
>> the 24.
>>>
>>> I see one issue: why are you torturing our poor qserv
>>> repo so much? It looks like you have a branch
>>> u/fjammes/mysqlLargeTables
>>> for that, but I can't get to your code by checking out
>>> your branch, and I had to do:
>>>
>>> git checkout 0b84d416e95a
>>>
>>> which is in "detached from HEAD" state.
>> cd /tmp/
>> git clone ssh:[log in to unmask] -b
>> u/fjammes/mysqlLargeTables
>> works fine, doesn't it ?
>>>
>>> Please create directory "mysqlLargeTable" directory in
>>> dbutils and move your scripts there
>>>
>>> (git clone [log in to unmask]:/contrib/dbutils.git dbutils
>>> to get there)
>> Ok, i've created a directory "mysqlLargeTable" on master branch and
>> next job will be done here.
>>>
>>> thanks
>> thanks for this interesting job.
>>> Jacek
>>>
>>>
>>> On 05/28/2013 07:56 AM, Fabrice Jammes wrote:
>>>> Hello Jacek,
>>>>
>>>> I've done and documented the tests run with mariadb, myisamchk, and
>>>> RunDeepForcedSource of Winter2013.
>>>> It's here :
>>>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3
>>>>
>>>> Feel free to ask for any additional information.
>>>>
>>>> Fabrice
>>>>
>>>> On 05/07/2013 09:51 PM, Jacek Becla wrote:
>>>>> Fabrice
>>>>>
>>>>>> Of course we can have a look a it.
>>>>>
>>>>> That is great!
>>>>>
>>>>>> Is there additional instructions for b) point ? If yes, could you
>>>>>> send
>>>>>> it to us ?
>>>>>
>>>>> 1) Have a RunDeepForcedSource table ready. It should not be
>>>>>      a production database, because we will be doing bad things
>>>>>      to it.
>>>>>
>>>>> 2) Disable indexes (alter table RunDeepForcedSource disable index,
>>>>>      than )
>>>>>
>>>>> 3) do not use that table through mysql while doing everything
>>>>>      below. Either shut down mysqld, or run "flush cache" and
>>>>>      don't touch that table
>>>>>
>>>>> 4) then test myisamchk, note that this is command line tool
>>>>>
>>>>> time /usr/local/mysql/bin/myisamchk
>>>>>      -v -t <tmp_dir>
>>>>>      --force
>>>>>      -o
>>>>>      --update-state
>>>>>      -p
>>>>>      --key_buffer_size=8M
>>>>>      --sort_buffer_size= <~1/2 of your RAM available>
>>>>>      --read_buffer_size=16M
>>>>>      --write_buffer_size=16M
>>>>> <mysql data dir>/<db>/RunDeepForcedSource.MYI
>>>>>
>>>>>
>>>>> that will most likely fail, see
>>>>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTables
>>>>>
>>>>> Then apply the patch and retry, Monty mentioned some
>>>>> extra switches (--force --force, etc), see his email,
>>>>> and try them
>>>>>
>>>>> Please capture output (and the timing!) for each
>>>>> thing you run.
>>>>>
>>>>> it is probably better to use a machine that has
>>>>> reasonable amount of memory (say 32 or 64G), not
>>>>> insanely large amount of memory (on a machine with
>>>>> 256G ram I couldn't reproduce this problem).
>>>>>
>>>>> It is important that the <tmp dir> has enough space,
>>>>> at least the size of one copy of RunDeepForcedSource.*
>>>>> files.
>>>>>
>>>>> thanks
>>>>> Jacek
>>>>>
>>>>
>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the QSERV-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
>>
>> ########################################################################
>> Use REPLY-ALL to reply to list
>>
>> To unsubscribe from the QSERV-L list, click the following link:
>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1