Opened 18 years ago
Closed 18 years ago
#242 closed task (fixed)
Investigate if it would be better to preload features when importing raw data
| Reported by: | Nicklas Nordborg | Owned by: | Nicklas Nordborg |
|---|---|---|---|
| Priority: | minor | Milestone: | BASE 2.0 |
| Component: | core | Version: | |
| Keywords: | Cc: |
Description
We need the features for validating the raw data and connecting with reporters. The current implementation issues one SQL statement for each position. Maybe it is better to preload all features and store in a hashmap. I think it can be a lot faster, and hope that not too much memory is used. The preloading would be done in the RawDataBatcher class.
Change History (3)
comment:1 by , 18 years ago
| Status: | new → assigned |
|---|
comment:2 by , 18 years ago
YES!!! A PredefinedQuery probably does the trick. It now imports the same data in less than a minute!
comment:3 by , 18 years ago
| Resolution: | → fixed |
|---|---|
| Status: | assigned → closed |
(In [2459]) Fixes #242: Investigate if it would be better to preload features when importing raw data
Also fixed two old bugs in the test programs:
- Load features to an affy design
- New parameter in NullPlugin was required
Note:
See TracTickets
for help on using tickets.
I made a simple test and it is certain that preloading has a big potential of beeing faster. A normal import took almost for minutes. A preloading import took two and a half, where the preloading phase took two minutes...
Maybe it is possible to speed up the preloading phase with a PredefinedQuery that loads exactly what we want. It would be nice if the speed could be doubled!