Use multiple threads in Lowess

[Nicklas Nordborg]

The Lowess plug-in is spending most if it's time doing calculations, and not so much reading and writing data from/to the database. Since the bioassays can be normalized independently of each other it would probably not be very problematic to do parallell calculations in two or more threads and have the main thread responsible for the reading/writing from/to the database. The cost is a higher memory usage since we need to keep data from more bioassays in memory (one per worker thread and one for the database thread).

[Nicklas Nordborg]

Here are the original timings for the Lowess plug-in on 'grey'. For more info see attached file on ticket #1238. All parameters use the default values. Times are in minutes and seconds.

Assays	T1	T2
--------------------
10	2.42	1.49
20	3.16	2.57
40	5.15	4.55
80	13.41	14.17
160	32.16	25.01
320	59.55	52.20

[Nicklas Nordborg]

(In [4811]) References #1257: Use multiple threads in Lowess

Checking in a changed plug-in that uses two worker threads so we can test this on 'grey'. Initial tests on 'auster' indicate that execution times decrease by 50%.

[Nicklas Nordborg]

Timings for Lowess on 'grey' when using two worker threads:

Assays  T1     T2
---------------------
10      1.39   1.26
20      3.01   3.13
40      7.24   -
80      11.59  -
160     24.14  -

Not so good improvement as we had hoped for. But it seems like there may be a flaw in the implementation were the worker threads may have to wait for input.

[Nicklas Nordborg]

(In [4812]) References #1257: Use multiple threads in Lowess

Trying a different algorithm for reading/writing data.

[Nicklas Nordborg]

(In [4813]) References #1257: Use multiple threads in Lowess

Now uses functionality in java.util.concurrent to queue and schedule worker threads. It makes the code easier and it seems like threads doens't have to wait for each other as much as before.

[Nicklas Nordborg]

(In [4814]) Fixes #1257: Use multiple threads in Lowess

Removed debug output.

[Nicklas Nordborg]

The debugging on 'grey' showed that the initial statement that the Lowess plug-in is spending most if it's time doing calculations was not correct. 'grey' is considerably faster than my development machine and spent maybe 80% of the time reading/writing from/to the database and only 20% with calculations (on 'auster' it was more like 40/60). Thus, we should not expect too much using multiple threads on a production server.

Here are the final test results from 'grey':

Assays  T1     T2
---------------------
10      0.44   1.20
20      2.42   2.21
40      5.21   5.11
80      11.23  -
160     23.42  -
320     52.04  -