 |
 |
Hmm, 4+8=12. Let's hope that InnoDB does not try to align things on a double word boundary if both are stored together (which I understand they will be). If it does, you will be surprised :-( And if you are surprised then perhaps you can come up with something useful to put into another 4 bytes to use up the wasted storage (e.g. sub-chunk ID). Andy -----Original Message----- From: Fabrice Jammes Sent: Thursday, February 18, 2016 12:13 PM To: Jacek Becla ; Serge Monkewitz Cc: qserv-l ; Fabio Hernandez Subject: Re: [QSERV-L] About secondary index size Hi Serge, Fabio might get some SSD or SSD+NVM-express storage to test secondary index performance. That's why I'm trying to better understand how secondary index size is computed. So, in my current understanding, having chunkID on 4 bytes and objectId on 8 bytes seems reasonable and would lead to a InnoDB table file of 456GB*3 = 1.5TB. If this is correct, a 2TB SSD machine would be fine for our tests. Serge, would you agree with this proposal? Regards, On 02/18/2016 11:50 AM, Jacek Becla wrote: > also, smallint/mediumint types are mysql specific and we were > trying to avoid using them. > > > On 02/18/2016 11:00 AM, Serge Monkewitz wrote: >> Chunk ids probably fit into 2 bytes with the current baseline 20k chunks, >> but chunk IDs are not contiguous integers so I’m not 100% sure. It’s >> obviously not my call, but personally I would like to have the >> flexibility to try > 65536 chunks, or to encode spatial binning at >> finer-than-chunk granularity in the secondary index. Is the storage >> overhead of an extra 2 bytes per row a deal breaker somehow? >> >>> On Feb 18, 2016, at 10:50 AM, Jacek Becla <[log in to unmask]> >>> wrote: >>> >>> Fabrice, I think we planned for regular 4 byte int for chunkid. >>> >>> But check with others from the group, I am way too busy with the >>> overall dm type things right now... >>> >>> >>> >>> On 02/18/2016 09:45 AM, Fabrice Jammes wrote: >>>> Sorry for nitpicking, but it seems that 8 bytes (a SMALLINT for chunkID >>>> and a BIGINT for objectId) would be enough to store one raw: >>>> http://dev.mysql.com/doc/refman/5.5/en/integer-types.html >>>> >>>> This would lead to an index of: >>>> 38 * 10^9 objects * (2+8 bytes) = 380 GB >>>> >>>> And, with overhead, 1.2TB. >>>> >>>> Can I send these number to Fabio please? >>>> >>>> Cheers, >>> >>> ######################################################################## >>> Use REPLY-ALL to reply to list >>> >>> To unsubscribe from the QSERV-L list, click the following link: >>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 >> > > ######################################################################## > Use REPLY-ALL to reply to list > > To unsubscribe from the QSERV-L list, click the following link: > https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1 ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1