 |
 |
Hi Jacek, On 05/28/2013 05:26 PM, Jacek Becla wrote: > Hi Fabrice > > Looks great! If you have access to this machine for longer, > can you try the last test with 2 GB, 16 GB and 48 GB sort > buffer size? It's possible, nevertheless i've a lot of tasks to do for in2p3 until the 24. > > I see one issue: why are you torturing our poor qserv > repo so much? It looks like you have a branch > u/fjammes/mysqlLargeTables > for that, but I can't get to your code by checking out > your branch, and I had to do: > > git checkout 0b84d416e95a > > which is in "detached from HEAD" state. cd /tmp/ git clone ssh:[log in to unmask] -b u/fjammes/mysqlLargeTables works fine, doesn't it ? > > Please create directory "mysqlLargeTable" directory in > dbutils and move your scripts there > > (git clone [log in to unmask]:/contrib/dbutils.git dbutils > to get there) Ok, i've created a directory "mysqlLargeTable" on master branch and next job will be done here. > > thanks thanks for this interesting job. > Jacek > > > On 05/28/2013 07:56 AM, Fabrice Jammes wrote: >> Hello Jacek, >> >> I've done and documented the tests run with mariadb, myisamchk, and >> RunDeepForcedSource of Winter2013. >> It's here : >> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTablesAtIn2p3 >> >> Feel free to ask for any additional information. >> >> Fabrice >> >> On 05/07/2013 09:51 PM, Jacek Becla wrote: >>> Fabrice >>> >>>> Of course we can have a look a it. >>> >>> That is great! >>> >>>> Is there additional instructions for b) point ? If yes, could you send >>>> it to us ? >>> >>> 1) Have a RunDeepForcedSource table ready. It should not be >>> a production database, because we will be doing bad things >>> to it. >>> >>> 2) Disable indexes (alter table RunDeepForcedSource disable index, >>> than ) >>> >>> 3) do not use that table through mysql while doing everything >>> below. Either shut down mysqld, or run "flush cache" and >>> don't touch that table >>> >>> 4) then test myisamchk, note that this is command line tool >>> >>> time /usr/local/mysql/bin/myisamchk >>> -v -t <tmp_dir> >>> --force >>> -o >>> --update-state >>> -p >>> --key_buffer_size=8M >>> --sort_buffer_size= <~1/2 of your RAM available> >>> --read_buffer_size=16M >>> --write_buffer_size=16M >>> <mysql data dir>/<db>/RunDeepForcedSource.MYI >>> >>> >>> that will most likely fail, see >>> https://dev.lsstcorp.org/trac/wiki/mysqlLargeTables >>> >>> Then apply the patch and retry, Monty mentioned some >>> extra switches (--force --force, etc), see his email, >>> and try them >>> >>> Please capture output (and the timing!) for each >>> thing you run. >>> >>> it is probably better to use a machine that has >>> reasonable amount of memory (say 32 or 64G), not >>> insanely large amount of memory (on a machine with >>> 256G ram I couldn't reproduce this problem). >>> >>> It is important that the <tmp dir> has enough space, >>> at least the size of one copy of RunDeepForcedSource.* >>> files. >>> >>> thanks >>> Jacek >>> >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1