Job controller / queue system
1. The queue
- The queuing system is used for operations that the core or
a client application estimates to have a relativeley long execution time,
for example importing a raw data set.
- The queue can also be used to hold operations that need to
be executed at regular intervals, for example cleaning and
- The queue holds all information that is needed to complete
the operation, including:
- the command to run
- the input parameters needed
- a priority value
- the user that added the operaion to the queue
- a repeat interval if desired
2. The queue manager / job controller
- An external application (queue manager) running in the background
regularly checks the queue for new entries.
- The queue manager doesn't have to run on the same server as the web server
- It should be possible to have more than one queue manager running
on different servers. Each one may be configured to only execute
certain type of operations or all types of operations. For example
- The queue manager executes the operations in the order of priority
and the time they were added to the queue.
- The queue manager must execute an operation as if it were logged in
as the user who added the operation to the queue.
- The queue manager should avoid overloading the server. For
- Only allow a maximum number of threads to be executed at the same time.
- Executing the operations with a lower thread priority
- While an operation is executing it should, if possible, report
the progress (ie. percent done) to the core.
- The queue manager should be a flexible application that easily can
be extended by plugins to allow more complex server configurations.
For example executing operations on a cluster.
- If an operation fails because of an unexpected failure it should
be possible to re-execute the operation at a later time.