There's some existing code for shared scan scheduling. I'm writing blurb 
about what I know about the worker scheduling and task handling. In the 
last couple of months I re-wrote a very large fraction of this code so 
query cancellation would work. I tried to make it flexible and 
understandable as I expect scheduling will be one of those places where 
there will be a lot of experimentation and it could get complicated.

The worker gets TaskMsg's from the czar and uses these to create 
wbase::Task objects (the czar does all the analysis). The important 
thing about a Task is that it can be given to a wcontrol::Scheduler (to 
queue it), the worker knows how to run the query it contains, and it can 
be canceled.  Whatever else happens, for the cancellation code to work 
any scan scheduler needs to work with Task's.

The scheduling is now done by wsched::BlendScheduler, which in turn 
passes Task's flagged as part of a query needing multiple chunks to the 
wsched::ScanScheduler, and other Tasks go to the GroupScheduler.

The GroupScheduler is much like a fifo with the exception that it tries 
to group queries by chunk id in an attempt to reduce disk I/O.

The ScanScheduler groups all Tasks by chunk id and then works through 
sequentially through all the chunk ids on the worker, and finally wraps 
back around to the lowest chunk id. It makes no attempt to lock chunks 
in memory, it is only trying to limit disk I/O to reading one chunk into 
memory at a time.

To change the behavior of the schedulers, ::queCmd() and ::_ready() are 
the primary functions that need to be changed.

The Schedulers all get their threads from the same util::ThreadPool 
found in EventThread.h. I have some concern about a lots of context 
switching and complicated _ready() functions taking up too much CPU 
time, and adding a util::PseudoThreadPool that creates/destorys threads 
up to a maximum as needed and shares the same interface as ThreadPool. I 
believe this would be simple with the biggest change being that 
schedulers would need to know about the ThreadPool.

The code for scheduling is based on the code found in util::Command.h 
and util::EventThread.h. Tasks are based off of util::Command and 
Command's are easy to pass around and run.


It is easy to switch between Schedulers and change the number of threads 
each can use as well as the total available from the pool, which is set 
in the code below.

SsiService::SsiService(XrdSsiLogger* log) {

  ...

     // TODO: set poolSize and all maxThreads values from config file.
     uint poolSize = std::max(static_cast<uint>(24), 
std::thread::hardware_concurrency());
     // TODO: set GroupScheduler group size from configuration file
     // TODO: Consider limiting the number of chunks being accessed at a 
time
     //       by GroupScheduler and ScanScheduler
     //_foreman = 
wcontrol::Foreman::newForeman(std::make_shared<wsched::FifoScheduler>(), 
poolSize);
     //_foreman = 
wcontrol::Foreman::newForeman(std::make_shared<wsched::GroupScheduler>(12), 
poolSize);
     // poolSize should be greater than either 
GroupScheduler::maxThreads or ScanScheduler::maxThreads
     _foreman = wcontrol::Foreman::newForeman(
             std::make_shared<wsched::BlendScheduler>(
std::make_shared<wsched::GroupScheduler>(20, 10),
std::make_shared<wsched::ScanScheduler>(20)),
             poolSize);
}

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1