There's some existing code for shared scan scheduling. I'm writing blurb
about what I know about the worker scheduling and task handling. In the
last couple of months I re-wrote a very large fraction of this code so
query cancellation would work. I tried to make it flexible and
understandable as I expect scheduling will be one of those places where
there will be a lot of experimentation and it could get complicated.
The worker gets TaskMsg's from the czar and uses these to create
wbase::Task objects (the czar does all the analysis). The important
thing about a Task is that it can be given to a wcontrol::Scheduler (to
queue it), the worker knows how to run the query it contains, and it can
be canceled. Whatever else happens, for the cancellation code to work
any scan scheduler needs to work with Task's.
The scheduling is now done by wsched::BlendScheduler, which in turn
passes Task's flagged as part of a query needing multiple chunks to the
wsched::ScanScheduler, and other Tasks go to the GroupScheduler.
The GroupScheduler is much like a fifo with the exception that it tries
to group queries by chunk id in an attempt to reduce disk I/O.
The ScanScheduler groups all Tasks by chunk id and then works through
sequentially through all the chunk ids on the worker, and finally wraps
back around to the lowest chunk id. It makes no attempt to lock chunks
in memory, it is only trying to limit disk I/O to reading one chunk into
memory at a time.
To change the behavior of the schedulers, ::queCmd() and ::_ready() are
the primary functions that need to be changed.
The Schedulers all get their threads from the same util::ThreadPool
found in EventThread.h. I have some concern about a lots of context
switching and complicated _ready() functions taking up too much CPU
time, and adding a util::PseudoThreadPool that creates/destorys threads
up to a maximum as needed and shares the same interface as ThreadPool. I
believe this would be simple with the biggest change being that
schedulers would need to know about the ThreadPool.
The code for scheduling is based on the code found in util::Command.h
and util::EventThread.h. Tasks are based off of util::Command and
Command's are easy to pass around and run.
It is easy to switch between Schedulers and change the number of threads
each can use as well as the total available from the pool, which is set
in the code below.
SsiService::SsiService(XrdSsiLogger* log) {
...
// TODO: set poolSize and all maxThreads values from config file.
uint poolSize = std::max(static_cast<uint>(24),
std::thread::hardware_concurrency());
// TODO: set GroupScheduler group size from configuration file
// TODO: Consider limiting the number of chunks being accessed at a
time
// by GroupScheduler and ScanScheduler
//_foreman =
wcontrol::Foreman::newForeman(std::make_shared<wsched::FifoScheduler>(),
poolSize);
//_foreman =
wcontrol::Foreman::newForeman(std::make_shared<wsched::GroupScheduler>(12),
poolSize);
// poolSize should be greater than either
GroupScheduler::maxThreads or ScanScheduler::maxThreads
_foreman = wcontrol::Foreman::newForeman(
std::make_shared<wsched::BlendScheduler>(
std::make_shared<wsched::GroupScheduler>(20, 10),
std::make_shared<wsched::ScanScheduler>(20)),
poolSize);
}
########################################################################
Use REPLY-ALL to reply to list
To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1
