The queuing system is used for operations that the core or
a client application estimates to have a relativeley long execution time,
for example importing a raw data set.
The queue can also be used to hold operations that need to
be executed at regular intervals, for example cleaning and
house-keeping operations.
The queue holds all information that is needed to complete
the operation, including:
An external application (queue manager) running in the background
regularly checks the queue for new entries.
The queue manager doesn't have to run on the same server as the web server
or database.
It should be possible to have more than one queue manager running
on different servers. Each one may be configured to only execute
certain type of operations or all types of operations. For example
The queue manager executes the operations in the order of priority
and the time they were added to the queue.
The queue manager must execute an operation as if it were logged in
as the user who added the operation to the queue.
The queue manager should avoid overloading the server. For
example by:
Only allow a maximum number of threads to be executed at the same time.
Executing the operations with a lower thread priority
While an operation is executing it should, if possible, report
the progress (ie. percent done) to the core.
The queue manager should be a flexible application that easily can
be extended by plugins to allow more complex server configurations.
For example executing operations on a cluster.
If an operation fails because of an unexpected failure it should
be possible to re-execute the operation at a later time.
Core specification