Opened 18 years ago

Closed 18 years ago

#242 closed task (fixed)

Investigate if it would be better to preload features when importing raw data

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: minor Milestone: BASE 2.0
Component: core Version:
Keywords: Cc:

Description

We need the features for validating the raw data and connecting with reporters. The current implementation issues one SQL statement for each position. Maybe it is better to preload all features and store in a hashmap. I think it can be a lot faster, and hope that not too much memory is used. The preloading would be done in the RawDataBatcher class.

Change History (3)

comment:1 by Nicklas Nordborg, 18 years ago

Status: newassigned

I made a simple test and it is certain that preloading has a big potential of beeing faster. A normal import took almost for minutes. A preloading import took two and a half, where the preloading phase took two minutes...

Maybe it is possible to speed up the preloading phase with a PredefinedQuery that loads exactly what we want. It would be nice if the speed could be doubled!

comment:2 by Nicklas Nordborg, 18 years ago

YES!!! A PredefinedQuery probably does the trick. It now imports the same data in less than a minute!

comment:3 by Nicklas Nordborg, 18 years ago

Resolution: fixed
Status: assignedclosed

(In [2459]) Fixes #242: Investigate if it would be better to preload features when importing raw data

Also fixed two old bugs in the test programs:

  • Load features to an affy design
  • New parameter in NullPlugin was required
Note: See TracTickets for help on using tickets.