Store data in files instead of in the database

[Nicklas Nordborg]

As I see it there are four basic problems:

1. Make it possible to select files in the GUI in a flexible way. For example, if we are doing Affymetrix experiments, we should be able to select CEL and CDF files. If we are doing other experiments there are other types of files. We need the concept of a "platform" and some information about what types of files are used. Platforms can be managed by server adminitrators.

2. We must of course be able to store the selected files and link them to the items the relate to. For example: the CEL file to a raw bioassay and the CDF file to an array design. Are there other item types that may need files?

3. Read metadata from the files. We want file storage to be equivalent to database storage, so we need a way to extract metadata from the files. For example the number of features for an array design and the number of spots and file headers for a raw bioassay. Should the extraction of metadata be required or optional? It will be easier for users to start with a new platform if it is optional. Downstream features must be aware of this and handle three cases: no files at all, files are present but no metadata, both files and metadata is present.

4. Import data to the database. Actually, I don't know if this is a problem or not. The existing import plug-ins could be used in this case. A slight modification is probably needed since they should use the file the user selected in step 1. It should be possible to have the data both in the database and in the file(s).

For more information:

• ​Database/UML schema
• ​Core API : Using files to store data
• ​Default platforms installed with BASE
• ​The older draft may still have some relevant information.
• ​The discussion on the mailing list

[Nicklas Nordborg]

(In [3681]) References #721: Store data in files instead of in the database

First draft of a specification/implementation design

[Nicklas Nordborg]

(In [3757]) References #721: Store data in files instead of in the database

Updated specification and documentation.

[Nicklas Nordborg]

(In [3764]) References #721. Added data layer classes and some core classes.

[Nicklas Nordborg]

(In [3793]) References #721: Many core classes in place. Work has started on the web interface for registering data file types and platforms.

[Nicklas Nordborg]

(In [3795]) References #721: Updated documentation to reflect changes in the code [3793]

[Nicklas Nordborg]

(In [3797]) References #721: Changes systemId to externalId

[Nicklas Nordborg]

(In [3798]) References #721: Changed systemId to externalId in JSP as well

[Nicklas Nordborg]

(In [3799]) References #721: Added platform variants to web gui.

[Nicklas Nordborg]

(In [3800]) References #721: It is now possible to edit platforms and data file types throguh the web interface. Added proper permissions to all items.

[Nicklas Nordborg]

(In [3801]) References #721: It is now possible to select files for an array design

[Nicklas Nordborg]

(In [3802]) References #721: File validation is now in place. More robust code for users with not enough permissions.

[Nicklas Nordborg]

(In [3804]) References #721: Most things for attaching files to array designs are now in place.

• Added update script for updating existing installations
• Re-coded parts of the code that depended on depracated methods
• Changed depracated methods to be backwards compatible
• Plugins that import to or use array designs can now use the attached files

[Nicklas Nordborg]

(In [3807]) References #721: Most things for attaching files to rw bioassays are now in place.

[Nicklas Nordborg]

(In [3812]) References #721: Store data in files instead of in the database

• Re-coded most of the code that depended on depracated methods
• Updated import plug-ins to use attached file and to save the file which file has been used
• Minor fixes to raw bioassay platform/file selection

[Nicklas Nordborg]

(In [3814]) References #721: Store data in files instead of in the database

• Experiments are now aware of file-only raw data types
• Fixed test programs
• A few remaining Affymetrix issues

[Nicklas Nordborg]

(In [3817]) References #721: Store data in files instead of in the database

• Fixes remaining backwards compatibility and update issues

[Nicklas Nordborg]

(In [3818]) References #721: Store data in files instead of in the database

• Synchronized Javadoc with code
• Added missing methods to some classes

[Nicklas Nordborg]

(In [3835]) References #721. Updated documentation.

[Nicklas Nordborg]

(In [3848]) References #721: Can now edit raw bioassays, array designs and projects where the platform/variant has been marked for removal

[Nicklas Nordborg]

(In [3851]) References #721: Filter on file extension when selecting platform data files

[Nicklas Nordborg]

(In [3852]) References #721: Auto-detect function is now aware of FileStoreEnabled items and automatically selects a default file for import.

[Nicklas Nordborg]

(In [3860]) References #721: Started to update the docbook documentation with respect to platforms

[Nicklas Nordborg]

(In [3865]) References #721: Create doc/html/userdoc/platforms directory to make docbook work on windows.

[Nicklas Nordborg]

(In [3867]) References #721: Store data in files instead of in the database

• Minor GUI fixes
• Some fixes related to Internet Explorer problems

[Nicklas Nordborg]

(In [3873]) References #721, fixes #789: Extended test program to include more Affymetrix tests and to attach file to array designs and raw bioassays.

[Nicklas Nordborg]

(In [3899]) References #721: Store data in files instead of in the database

Generated incorrect join on query.

[Nicklas Nordborg]

(In [3909]) References #721: Store data in files instead of in the database

Remember most recently used platform/variant when creating raw bioassays/array designs

[Nicklas Nordborg]

(In [3912]) References #721. Fixes NPE in autodetect file format function when used with BasicData classes

[Nicklas Nordborg]

(In [4006]) Fixes #844: File.isUsed() doesn't check if a file has been used in a FileSet. References #721