 |
 |
Fabrice It is more interesting than I thought! I am going to forward to Monty (mysql founder/best guru) to see what he thinks about your results. I certainly didn't expect some of these tests to fail. I will cc you, he will probably have questions about details or suggestions how to try it to pin down the problems. Jacek p.s. I have new, VERY interesting and important set of tests to run. More soon. On 06/06/2013 01:47 AM, Fabrice Jammes wrote: > Hello Jacek, > > The last test has been launched with sort_buffer_size parameter values > you asked for. > The results are here : > https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3 > > Best regards, > > Fabrice > > On 05/28/2013 06:36 PM, Fabrice Jammes wrote: >> Hi Jacek, >> >> >> On 05/28/2013 05:26 PM, Jacek Becla wrote: >>> Hi Fabrice >>> >>> Looks great! If you have access to this machine for longer, >>> can you try the last test with 2 GB, 16 GB and 48 GB sort >>> buffer size? >> It's possible, nevertheless i've a lot of tasks to do for in2p3 until >> the 24. >>> >>> I see one issue: why are you torturing our poor qserv >>> repo so much? It looks like you have a branch >>> u/fjammes/mysqlLargeTables >>> for that, but I can't get to your code by checking out >>> your branch, and I had to do: >>> >>> git checkout 0b84d416e95a >>> >>> which is in "detached from HEAD" state. >> cd /tmp/ >> git clone ssh:[log in to unmask] -b >> u/fjammes/mysqlLargeTables >> works fine, doesn't it ? >>> >>> Please create directory "mysqlLargeTable" directory in >>> dbutils and move your scripts there >>> >>> (git clone [log in to unmask]:/contrib/dbutils.git dbutils >>> to get there) >> Ok, i've created a directory "mysqlLargeTable" on master branch and >> next job will be done here. >>> >>> thanks >> thanks for this interesting job. >>> Jacek >>> >>> >>> On 05/28/2013 07:56 AM, Fabrice Jammes wrote: >>>> Hello Jacek, >>>> >>>> I've done and documented the tests run with mariadb, myisamchk, and >>>> RunDeepForcedSource of Winter2013. >>>> It's here : >>>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3 >>>> >>>> Feel free to ask for any additional information. >>>> >>>> Fabrice >>>> >>>> On 05/07/2013 09:51 PM, Jacek Becla wrote: >>>>> Fabrice >>>>> >>>>>> Of course we can have a look a it. >>>>> >>>>> That is great! >>>>> >>>>>> Is there additional instructions for b) point ? If yes, could you >>>>>> send >>>>>> it to us ? >>>>> >>>>> 1) Have a RunDeepForcedSource table ready. It should not be >>>>> a production database, because we will be doing bad things >>>>> to it. >>>>> >>>>> 2) Disable indexes (alter table RunDeepForcedSource disable index, >>>>> than ) >>>>> >>>>> 3) do not use that table through mysql while doing everything >>>>> below. Either shut down mysqld, or run "flush cache" and >>>>> don't touch that table >>>>> >>>>> 4) then test myisamchk, note that this is command line tool >>>>> >>>>> time /usr/local/mysql/bin/myisamchk >>>>> -v -t <tmp_dir> >>>>> --force >>>>> -o >>>>> --update-state >>>>> -p >>>>> --key_buffer_size=8M >>>>> --sort_buffer_size= <~1/2 of your RAM available> >>>>> --read_buffer_size=16M >>>>> --write_buffer_size=16M >>>>> <mysql data dir>/<db>/RunDeepForcedSource.MYI >>>>> >>>>> >>>>> that will most likely fail, see >>>>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTables >>>>> >>>>> Then apply the patch and retry, Monty mentioned some >>>>> extra switches (--force --force, etc), see his email, >>>>> and try them >>>>> >>>>> Please capture output (and the timing!) for each >>>>> thing you run. >>>>> >>>>> it is probably better to use a machine that has >>>>> reasonable amount of memory (say 32 or 64G), not >>>>> insanely large amount of memory (on a machine with >>>>> 256G ram I couldn't reproduce this problem). >>>>> >>>>> It is important that the <tmp dir> has enough space, >>>>> at least the size of one copy of RunDeepForcedSource.* >>>>> files. >>>>> >>>>> thanks >>>>> Jacek >>>>> >>>> >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the QSERV-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1