I'm not sure this still matters, but I'm confused. I thought we were 
talking about the transfer time for a single "chunk". I took "chunk" to 
mean qserv-chunk-of-a-table. This means that we need to transfer a 
number of bytes equal to the on-disk representation of that table. For 
MyISAM, this means the raw row data + myisam overhead + index files. 
rowdata+overhead+index all need to accessible for chunk X of table A to 
be usable. It is not clear to me that transferring rowdata and index 
concurrently is faster unless you are getting around TCP 
window/congestion control (or you have multiple source nodes, multiple 
pipes, and multiple destinations).

Still not sure this is worth arguing about. I feel that the larger 
question of whether it's a good idea to have an external qserv cluster 
that has no local (data) storage and uses the LSST cluster as a backing 
store is a bit orthogonal. For the larger question, I am a bit concerned 
that I haven't heard of implementations auto-caching HDFS clusters or 
distributed dbms clusters that use other clusters as backing stores, 
whether research, proprietary, or open-source. This concerns me. (On the 
other hand, it's probably worth a master's or phd thesis.)

-Daniel

On 09/24/2013 05:19 PM, Becla, Jacek wrote:
> We are talking here about sizes of *individual*
> chunks that are transferred, my point is that
> data+index are not a single file.
>
> All 20,000 chunks are going through the same
> pipe too, right? So if we consider db and index,
> it 40,000 chunks.
>
> Jacek
>
>
>
>
> On 9/24/2013 5:15 PM, Wang, Daniel Liwei wrote:
>> Wait, why is it faster in parallel? Same pipe, right? Unless you are
>> thinking disjoint sets of source-pipe-dest.
>>
>> -Daniel
>>
>> On 09/24/2013 04:44 PM, Jacek Becla wrote:
>>> As we just talked, my numbers are for data chunks,
>>> index is up to 2x larger, so we can use 2x larger
>>> numbers. Data+index come in separate files, so
>>> they can be transferred in parallel, so I think
>>> it'd be unfair to assume 3x my numbers though
>>>
>>> Jacek
>>>
>>>
>>>
>>> On 9/24/2013 3:07 PM, Jacek Becla wrote:
>>>>> 	Chunks are expected to be multiple terabytes in size, which
>>>>> means that downloads are hours long.
>>>> K-T,
>>>>
>>>> Based on the baseline, which assumes flat 20K chunks per tables,
>>>> the largest chunk will be 255 GB. The numbers are (in GB,
>>>> DR1 --> DR11)
>>>>       - Object:    2 -->   4
>>>>       - ObjExtra: 25 -->  69
>>>>       - Source:    9 --> 255
>>>>       - ForcedSrc: 2 -->  98
>>>>
>>>> This is in LDM-141, dbL2, L141 (and nearby)
>>>>
>>>> And, that is before compression.
>>>>
>>>> We talked about keeping chunk size const rather than #chunks
>>>> constants, which will probably make us go with DR1-size chunk
>>>> sizes, thus keeping chunk size closer to 25 GB than 1/4 TB)
>>>>
>>>> Jacek
>>>>
>>> ########################################################################
>>> Use REPLY-ALL to reply to list
>>>
>>> To unsubscribe from the QSERV-L list, click the following link:
>>> https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1