Job controller / queue system

This document covers the details of how the job controller / queuing system works.

  1. The queue
  2. The queue manager / job controller

Last updated: $Date: 2009-04-06 14:52:39 +0200 (må, 06 apr 2009) $

1. The queue

  1. The queuing system is used for operations that the core or a client application estimates to have a relativeley long execution time, for example importing a raw data set.
  2. The queue can also be used to hold operations that need to be executed at regular intervals, for example cleaning and house-keeping operations.
  3. The queue holds all information that is needed to complete the operation, including:

2. The queue manager / job controller

  1. An external application (queue manager) running in the background regularly checks the queue for new entries.
  2. The queue manager doesn't have to run on the same server as the web server or database.
  3. It should be possible to have more than one queue manager running on different servers. Each one may be configured to only execute certain type of operations or all types of operations. For example
  4. The queue manager executes the operations in the order of priority and the time they were added to the queue.
  5. The queue manager must execute an operation as if it were logged in as the user who added the operation to the queue.
  6. The queue manager should avoid overloading the server. For example by:
  7. While an operation is executing it should, if possible, report the progress (ie. percent done) to the core.
  8. The queue manager should be a flexible application that easily can be extended by plugins to allow more complex server configurations. For example executing operations on a cluster.
  9. If an operation fails because of an unexpected failure it should be possible to re-execute the operation at a later time.