There's relationship between memory, cores/threads, and chunk size. If 
we're running t threads and the tables are about s GB/chunk for Source, 
f GB/chunk for ForcedSource, and e GB/chunk for ObjectExtra) are each 
about g GB/chunk, we would have at most 3*tsfe gigabytes of memory 
locked, with each thread running a join between the three tables on a 
different chunk (which shouldn't happen), but does give us a point at 
which more memory for locking tables is pointless.

At a minimum, we will want to be able to lock at least 2 copies of each 
table in memory, and we'd be much happier with a couple more. The more 
threads we run, probably the more copies we will want, but I'm not sure 
what the exact relation is going to be, but less than 2 copies will 
likely slow things down with memory contention issues. The number of 
threads we run is going to be a function of the number of cores. The 
amount of memory on a worker that should be reserved for locking is not 
really know, but Andy H thinks half or a bit more than half is a good 
estimate and I've got no reason to disagree.

I put together a quick spreadsheet (which is attached) and at 20k 
chunks, we will need 810 GB of memory for locking, which means the 
machines would need something like 1400GB or RAM. Using 200k chunks, we 
only need 81GB for locking, so we would do ok with 160GB of RAM and 
probably be pretty happy with 250GB of RAM. The downside is the czars 
get to do 10 times as much book keeping (which is only maybe about 
10-20% of what they spend their time doing) so adding another czar or 
two might be needed.

  -John






########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1