Ticket #1028: batchimport_userperspective.txt

There is a need to facilitate batch upload, creation, and modification
of items in BASE. Some batch tools already exists such as

 - batch upload of files using zip files
 - batch creation of array slides
 - batch addition/deletion of reporters
 - import of annotations
 - list views offer an import button but which view do actually offer
   a plug-in that does anything?

For a single or few experiment setting there is not so urgent need for
batch tools but for a microarray facility where many experiments are
prepared by facility staff the need is eminent. At a facility site
many experiments are conducted by few people and all data upload is
done by these staff members. To ease the upload of data to BASE we
suggest to create one or several plug-ins that can create or modify
several items in a batch by reading information from tab separated
files. The idea here is not to create one single monolithic plug-in
that imports a complete experiment and creates all necessary items,
but rather imports, creates, or modifies items for a given context and
makes the proper associations to parents. The word 'import' is used in
this document but it could just as well be create or modify depending
on user requirements.

There is ongoing work on a full experiment import from tab2mage
formatted files, see
http://baseplugins.thep.lu.se/wiki/uk.ac.ebi.Tab2MageImporter.

The plug-in requirements outlined here is to be used in a context
where the user ideally works interactively with BASE in a step-by-step
procedure. The idea is that the interaction with BASE starts on some
level and data is added from this level down. Here a sample work
session is outlined where RNA is extracted and labeled starting from
some source of biomaterial. In BASE this follows the path of biosource
- sample - extract - labeled extract, and the continuing with
hybridization - scan - raw bioassay - bioassay - bioassay sets -
analysis.

Starting at the bio source level, the user must make an initial import
of biosource information or use the BASE web interface for adding
biosource items. Samples are created from these biosources, in BASE
context this means that sample information needs to be added. In this
example we want to associate the samples to their parents, changing
sample properties follows a similar path but the import files do not
require parent information. The import of sample data is started with
selecting the biosources associated with the samples in BASE, and then
exporting this information to a file. This file is used as a template
for entering sample data to be stored in BASE. The reason for using
this template is to ensure that the correct biosource identifiers are
used for the samples. (A user can of course create the file without the
export from BASE but has to make sure that items are properly
referenced.) The biosource identifiers are required for making
parent-child association within BASE. When the samples are added to
this file, the file is imported into BASE. After this import, the
sample information is exported to a file again, and this file is used
as a template for the extracts information. Again, the reason for this
is to make sure that proper BASE identifiers are used. Extract
information is added to the template and imported back to BASE. This
procedure is performed for each level of data entry.

The information optionally exported to be used as templates above are
simple tab separated files with a few columns of information about the
items. The columns exported have a two-fold purpose; i) make sure that
BASE can make the proper associations when importing data, ii) guide
the users when adding information to the template file, i.e.,
descriptive names for human interpretation.

Items that should be supported by the item importer(s) are:

 - Biosources: top level, currently no parent items to associate

 - Samples: biosource (or pooled samples) to associate. Create sample
   events for pools to decrease pooled samples.

 - Extracts: samples (or pooled extracts) and protocols to
   associate. Create sample events to decrease sample amounts and
   extract events to decrease extract amounts for pooled extracts.

 - Labeled extracts: extracts (or pooled labeled extracts) and
   protocols to associate. Create extract events to decrease extract
   amounts and labeled extract events to decrease pooled amounts.

 - Hybridizations: multiple lableled extracts, ... more to come
 - scan
 - Raw bioassays: ...
 - Experiments: ...


A detailed discussion on the different export/import steps, sample
files for the different item types are available as attachements to
ticket 1028 (http://base.thep.lu.se/ticket/1028) at the BASE web site.

Sample files based on trunk revision 4301 were exported and modified
for items from biosource level down to the labeled extracts level. An
OpenOffice.org spreadsheet (batchimport_sample.ods) that contain
format information with explanations in one document is also
available. A tentatiove aim is that the spreadsheet may be used by
laborative staff to fill information to be used in import to BASE.

Dry-run that explain what will be done during import should be
supported.  Potential dangers and errors should be reported. This will
allow the user to check that the import will behave as expected.


Biosource

This is currently the top level of associations. No association are
needed except for the optional reference to an external item (a
property of the biosource). The import is a straightforward tab
separated import to fill the item properties.

The available fields to import are: 'Name', 'Description', 'External
id'

Mandatory columns for imports: 'Name'

Sample export file: biosource_out.txt
Sample import file: biosource_in.txt


Sample

The import of item properties is a straightforward tab separated
import. Compared to biosource items there are additional columns for
associations to other items (the parent biosource and protocol). There
is one parent only if the parent is a biosource, pooled samples may
have multiple parents (other samples) defined using multiple lines.

Pooled samples create 'Event's that decrease the parent amount.

The available fields to import are: 'Name', 'Original quantity (µg)',
'Description', 'External id', 'Protocol', 'Created', 'Pooled'

Mandatory columns for imports: 'Name'

The important difference compared with biosource items is the possible
associations to bioassays and protcols.

Sample export file: sample_out.txt
Sample import file: sample_in.txt


Extract

The import of item properties is a straightforward tab separated
import. There are additional columns for associations to the parent
item and other items. There is one parent only if the parent is a
sample, pooled extracts may have multiple parents (other extracts)
defined using multiple lines.

Extracts and pooled extracts create 'Event's that decrease the parent
amount.

The available fields to import are: 'Name', 'Original quantity (µg)',
'Description', 'External id', 'Protocol', 'Created', 'Pooled'

Mandatory columns for imports: 'Name'

Extract export file: extract_out.txt
Extract import file: extract_in.txt


Labeled Extract

The import of item properties is a straightforward tab separated
import. There are additional columns for associations to the parent
item and other items. There is one parent only if the
parent is an extract, pooled labeled extracts may have multiple
parents (other labeled extracts) defined using multiple lines.

There is an additional column as compared to the extract items, Label.

The available fields to import are: 'Name', 'Label', 'Original
quantity (µg)', 'Description', 'External id', 'Protocol', 'Created',
'Pooled'

Mandatory columns for imports: 'Name'

Labeledextract export file: labeledextract_out.txt
Labeledextract import file: labeledextract_in.txt