Create plug-in for importing raw data to multiple raw bioassays in one go

[Jari Häkkinen]

The batch importers now support creation of items down to RawBioAssay level if raw data files are uploaded by other means like ftp or zip-files. Experiments are created through the web interface and raw bioassays are associated to the experiment. However, no data is automatically imported into raw data tables and currently this has to be done manually for each raw bioassay. We should create an import functionality for an experiment that imports data into the raw data tables (if appropriate for the platform and that data was not already imported).

This is needed for Illumina IBS and GenePix and other data. The problem is that there is a need to select plug-in configurations for imports since several configurations may match the file to import (think dye swap). To resolve the issue of different file formats and multiple matching configurations the import should start with a selection of the raw bioassays to import and a configuration should be selected. In cases where no configuration or a single configuration exists maybe this choice can be skipped. Of the selected set of raw bioassays of course data is only imported for those raw bioassays with no data in the raw tables.

The plug-in should check whether it can import each file with the selected import and only perform an import of the formats matching the configuration selected. The plug-in should report the number of successful, skipped and failed imports.

[Nicklas Nordborg]

Implementation specifications v.1

[Nicklas Nordborg]

It doesn't work! The Plugin API has not been design for a plug-in (master) to run other plug-ins (worker). There are several problems:

1. The parameters needed by the worker plug-in, for example the file to import from, doesn't make sense when used in a multi-import environment. It is hard to design something that can automatically detect the parameters that doesn't make sense and at the same time provide some useful values for those parameters so that the configuration of the worker plug-in doesn't generate an error.
2. If job agents has been set up it is not sure that the worker plug-in is available on the same job agent as the master plug-in. This is a configuration issue, but it means that it is not possible to be flexible in the setup.

The multi-import feature has to be solved by each import plug-in individually. I'll replace this ticket with new tickets for the generic raw data importer and (on the BASE plug-ins site) the Illumina importer.