Opened 16 years ago

Closed 15 years ago

#242 closed task (fixed)

Investigate if it would be better to preload features when importing raw data

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: minor Milestone: BASE 2.0
Component: core Version:
Keywords: Cc:

Description

We need the features for validating the raw data and connecting with reporters. The current implementation issues one SQL statement for each position. Maybe it is better to preload all features and store in a hashmap. I think it can be a lot faster, and hope that not too much memory is used. The preloading would be done in the RawDataBatcher? class.

Change History (3)

comment:1 Changed 15 years ago by Nicklas Nordborg

Status: newassigned

I made a simple test and it is certain that preloading has a big potential of beeing faster. A normal import took almost for minutes. A preloading import took two and a half, where the preloading phase took two minutes...

Maybe it is possible to speed up the preloading phase with a PredefinedQuery? that loads exactly what we want. It would be nice if the speed could be doubled!

comment:2 Changed 15 years ago by Nicklas Nordborg

YES!!! A PredefinedQuery? probably does the trick. It now imports the same data in less than a minute!

comment:3 Changed 15 years ago by Nicklas Nordborg

Resolution: fixed
Status: assignedclosed

(In [2459]) Fixes #242: Investigate if it would be better to preload features when importing raw data

Also fixed two old bugs in the test programs:

  • Load features to an affy design
  • New parameter in NullPlugin? was required
Note: See TracTickets for help on using tickets.