 |
 |
Hello Jacek, The last test has been launched with sort_buffer_size parameter values you asked for. The results are here : https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3 Best regards, Fabrice On 05/28/2013 06:36 PM, Fabrice Jammes wrote: > Hi Jacek, > > > On 05/28/2013 05:26 PM, Jacek Becla wrote: >> Hi Fabrice >> >> Looks great! If you have access to this machine for longer, >> can you try the last test with 2 GB, 16 GB and 48 GB sort >> buffer size? > It's possible, nevertheless i've a lot of tasks to do for in2p3 until > the 24. >> >> I see one issue: why are you torturing our poor qserv >> repo so much? It looks like you have a branch >> u/fjammes/mysqlLargeTables >> for that, but I can't get to your code by checking out >> your branch, and I had to do: >> >> git checkout 0b84d416e95a >> >> which is in "detached from HEAD" state. > cd /tmp/ > git clone ssh:[log in to unmask] -b > u/fjammes/mysqlLargeTables > works fine, doesn't it ? >> >> Please create directory "mysqlLargeTable" directory in >> dbutils and move your scripts there >> >> (git clone [log in to unmask]:/contrib/dbutils.git dbutils >> to get there) > Ok, i've created a directory "mysqlLargeTable" on master branch and > next job will be done here. >> >> thanks > thanks for this interesting job. >> Jacek >> >> >> On 05/28/2013 07:56 AM, Fabrice Jammes wrote: >>> Hello Jacek, >>> >>> I've done and documented the tests run with mariadb, myisamchk, and >>> RunDeepForcedSource of Winter2013. >>> It's here : >>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3 >>> >>> Feel free to ask for any additional information. >>> >>> Fabrice >>> >>> On 05/07/2013 09:51 PM, Jacek Becla wrote: >>>> Fabrice >>>> >>>>> Of course we can have a look a it. >>>> >>>> That is great! >>>> >>>>> Is there additional instructions for b) point ? If yes, could you >>>>> send >>>>> it to us ? >>>> >>>> 1) Have a RunDeepForcedSource table ready. It should not be >>>> a production database, because we will be doing bad things >>>> to it. >>>> >>>> 2) Disable indexes (alter table RunDeepForcedSource disable index, >>>> than ) >>>> >>>> 3) do not use that table through mysql while doing everything >>>> below. Either shut down mysqld, or run "flush cache" and >>>> don't touch that table >>>> >>>> 4) then test myisamchk, note that this is command line tool >>>> >>>> time /usr/local/mysql/bin/myisamchk >>>> -v -t <tmp_dir> >>>> --force >>>> -o >>>> --update-state >>>> -p >>>> --key_buffer_size=8M >>>> --sort_buffer_size= <~1/2 of your RAM available> >>>> --read_buffer_size=16M >>>> --write_buffer_size=16M >>>> <mysql data dir>/<db>/RunDeepForcedSource.MYI >>>> >>>> >>>> that will most likely fail, see >>>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTables >>>> >>>> Then apply the patch and retry, Monty mentioned some >>>> extra switches (--force --force, etc), see his email, >>>> and try them >>>> >>>> Please capture output (and the timing!) for each >>>> thing you run. >>>> >>>> it is probably better to use a machine that has >>>> reasonable amount of memory (say 32 or 64G), not >>>> insanely large amount of memory (on a machine with >>>> 256G ram I couldn't reproduce this problem). >>>> >>>> It is important that the <tmp dir> has enough space, >>>> at least the size of one copy of RunDeepForcedSource.* >>>> files. >>>> >>>> thanks >>>> Jacek >>>> >>> >> >> ######################################################################## >> Use REPLY-ALL to reply to list >> >> To unsubscribe from the QSERV-L list, click the following link: >> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1