Store data in files instead of in the database

[Nicklas Nordborg]

As I see it there are four basic problems:

1. Make it possible to select files in the GUI in a flexible way. For example, if we are doing Affymetrix experiments, we should be able to select CEL and CDF files. If we are doing other experiments there are other types of files. We need the concept of a "platform" and some information about what types of files are used. Platforms can be managed by server adminitrators.

2. We must of course be able to store the selected files and link them to the items the relate to. For example: the CEL file to a raw bioassay and the CDF file to an array design. Are there other item types that may need files?

3. Read metadata from the files. We want file storage to be equivalent to database storage, so we need a way to extract metadata from the files. For example the number of features for an array design and the number of spots and file headers for a raw bioassay. Should the extraction of metadata be required or optional? It will be easier for users to start with a new platform if it is optional. Downstream features must be aware of this and handle three cases: no files at all, files are present but no metadata, both files and metadata is present.

4. Import data to the database. Actually, I don't know if this is a problem or not. The existing import plug-ins could be used in this case. A slight modification is probably needed since they should use the file the user selected in step 1. It should be possible to have the data both in the database and in the file(s).

See ​http://base.thep.lu.se/chrome/site/doc/development/overview/core/data_in_files.html for more info about specification/implementation details

[Nicklas Nordborg]

(In [3681]) References #721: Store data in files instead of in the database

First draft of a specification/implementation design