Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#1257 closed enhancement (fixed)

Use multiple threads in Lowess

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: major Milestone: BASE 2.11
Component: coreplugins Version:
Keywords: Cc:

Description

The Lowess plug-in is spending most if it's time doing calculations, and not so much reading and writing data from/to the database. Since the bioassays can be normalized independently of each other it would probably not be very problematic to do parallell calculations in two or more threads and have the main thread responsible for the reading/writing from/to the database. The cost is a higher memory usage since we need to keep data from more bioassays in memory (one per worker thread and one for the database thread).

Change History (8)

comment:1 Changed 12 years ago by Nicklas Nordborg

Owner: changed from everyone to Nicklas Nordborg
Status: newassigned

comment:2 Changed 12 years ago by Nicklas Nordborg

Here are the original timings for the Lowess plug-in on 'grey'. For more info see attached file on ticket #1238. All parameters use the default values. Times are in minutes and seconds.

Assays	T1	T2
--------------------
10	2.42	1.49
20	3.16	2.57
40	5.15	4.55
80	13.41	14.17
160	32.16	25.01
320	59.55	52.20

comment:3 Changed 12 years ago by Nicklas Nordborg

(In [4811]) References #1257: Use multiple threads in Lowess

Checking in a changed plug-in that uses two worker threads so we can test this on 'grey'. Initial tests on 'auster' indicate that execution times decrease by 50%.

comment:4 Changed 12 years ago by Nicklas Nordborg

Timings for Lowess on 'grey' when using two worker threads:

Assays  T1     T2
---------------------
10      1.39   1.26
20      3.01   3.13
40      7.24   -
80      11.59  -
160     24.14  -

Not so good improvement as we had hoped for. But it seems like there may be a flaw in the implementation were the worker threads may have to wait for input.

comment:5 Changed 12 years ago by Nicklas Nordborg

(In [4812]) References #1257: Use multiple threads in Lowess

Trying a different algorithm for reading/writing data.

comment:6 Changed 12 years ago by Nicklas Nordborg

(In [4813]) References #1257: Use multiple threads in Lowess

Now uses functionality in java.util.concurrent to queue and schedule worker threads. It makes the code easier and it seems like threads doens't have to wait for each other as much as before.

comment:7 Changed 12 years ago by Nicklas Nordborg

Resolution: fixed
Status: assignedclosed

(In [4814]) Fixes #1257: Use multiple threads in Lowess

Removed debug output.

comment:8 Changed 12 years ago by Nicklas Nordborg

The debugging on 'grey' showed that the initial statement that the Lowess plug-in is spending most if it's time doing calculations was not correct. 'grey' is considerably faster than my development machine and spent maybe 80% of the time reading/writing from/to the database and only 20% with calculations (on 'auster' it was more like 40/60). Thus, we should not expect too much using multiple threads on a production server.

Here are the final test results from 'grey':

Assays  T1     T2
---------------------
10      0.44   1.20
20      2.42   2.21
40      5.21   5.11
80      11.23  -
160     23.42  -
320     52.04  -
Note: See TracTickets for help on using tickets.