 |
 |
There's relationship between memory, cores/threads, and chunk size. If we're running t threads and the tables are about s GB/chunk for Source, f GB/chunk for ForcedSource, and e GB/chunk for ObjectExtra) are each about g GB/chunk, we would have at most 3*tsfe gigabytes of memory locked, with each thread running a join between the three tables on a different chunk (which shouldn't happen), but does give us a point at which more memory for locking tables is pointless. At a minimum, we will want to be able to lock at least 2 copies of each table in memory, and we'd be much happier with a couple more. The more threads we run, probably the more copies we will want, but I'm not sure what the exact relation is going to be, but less than 2 copies will likely slow things down with memory contention issues. The number of threads we run is going to be a function of the number of cores. The amount of memory on a worker that should be reserved for locking is not really know, but Andy H thinks half or a bit more than half is a good estimate and I've got no reason to disagree. I put together a quick spreadsheet (which is attached) and at 20k chunks, we will need 810 GB of memory for locking, which means the machines would need something like 1400GB or RAM. Using 200k chunks, we only need 81GB for locking, so we would do ok with 160GB of RAM and probably be pretty happy with 250GB of RAM. The downside is the czars get to do 10 times as much book keeping (which is only maybe about 10-20% of what they spend their time doing) so adding another czar or two might be needed. -John ######################################################################## Use REPLY-ALL to reply to list To unsubscribe from the QSERV-L list, click the following link: https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1