Bill

Sure! Please go ahead

Jacek


On 09/22/2013 12:32 PM, Bill Chickering wrote:
> Hi Jacek -
>
> I'd like to create two new tickets within TRAC. These are related to my fault tolerance work and build on top of ticket 2531, which implements status/error messages and basic fault tolerance (but not recovery) (note that 2531 has been submitted for review). The two additional tickets are for (development is complete for both):
>
> 1) Logging: This has been very helpful when debugging. Includes both the logging mechanism and integration throughout qserv master. For example, references to std::cout within the C++ layer of qserv are replaced with the logging stream. Implemented using boost filtering streams; very lightweight. Features includes:
>      -- Thread safe.
>      -- Severity levels allow the verbosity of logging output to be easily throttled (helpful when debugging).
>      -- Automatically includes timestamp, thread id, and severity level.
>      -- Integrated into error messaging (ticket 2531) so any/all error messages are automatically logged.
>      -- Buffers each line to minimize output from different threads "stepping on each other".
>      -- Placed in common directory so can be used by both master and worker.
>      -- Implemented Swig enabled interface so it's accessible from python layer.
>
> 2) Error recovery: Builds on both ticket 2531 and logging features. This is the basic chunkQuery-level error recovery described in the review:
>> Consider the event of a disk failure. Qserv's worker logic is not equipped to manage such a failure on localized regions of disk and would behave as if a software fault had occurred. The worker process would therefore crash and all chunk queries belonging to that worker would be lost. The in-flight queries on its local mysqld would be cleaned up and have resources freed. The Qserv master's requests to retrieve these chunk queries via XRootD would then return an error code. The master responds by re-initializing the chunk queries and re-submits them to XRootD. Ideally, duplicate data associated with the chunk queries exists on other nodes. In this case, XRootD silently re-routes the request(s) to the surviving node(s) and all associated queries are completed as usual. In the event that duplicate data does not exist for one or more chunk queries, XRootD would again return an error code. The master will re-initialize and re-submit a chunk query a fixed number of times (determined by a
 parameter within Qserv) before giving up, logging information about the failure, and returning an error message to the user in response to the associated query.
>
> -- Bill
>

########################################################################
Use REPLY-ALL to reply to list

To unsubscribe from the QSERV-L list, click the following link:
https://listserv.slac.stanford.edu/cgi-bin/wa?SUBED1=QSERV-L&A=1