Opened 17 years ago
Closed 17 years ago
#242 closed task (fixed)
Investigate if it would be better to preload features when importing raw data
| Reported by: | Nicklas Nordborg | Owned by: | Nicklas Nordborg |
|---|---|---|---|
| Priority: | minor | Milestone: | BASE 2.0 |
| Component: | core | Version: | |
| Keywords: | Cc: |
Description
We need the features for validating the raw data and connecting with reporters. The current implementation issues one SQL statement for each position. Maybe it is better to preload all features and store in a hashmap. I think it can be a lot faster, and hope that not too much memory is used. The preloading would be done in the RawDataBatcher? class.
Change History (3)
comment:1 Changed 17 years ago by
| Status: | new → assigned |
|---|
comment:2 Changed 17 years ago by
YES!!! A PredefinedQuery? probably does the trick. It now imports the same data in less than a minute!
comment:3 Changed 17 years ago by
| Resolution: | → fixed |
|---|---|
| Status: | assigned → closed |
(In [2459]) Fixes #242: Investigate if it would be better to preload features when importing raw data
Also fixed two old bugs in the test programs:
- Load features to an affy design
- New parameter in NullPlugin? was required
Note: See
TracTickets for help on using
tickets.
I made a simple test and it is certain that preloading has a big potential of beeing faster. A normal import took almost for minutes. A preloading import took two and a half, where the preloading phase took two minutes...
Maybe it is possible to speed up the preloading phase with a PredefinedQuery? that loads exactly what we want. It would be nice if the speed could be doubled!