2.17.2: 2011-06-17

net.sf.basedb.plugins
Class RawDataFlatFileImporter

java.lang.Object
  extended by net.sf.basedb.core.plugin.AbstractPlugin
      extended by net.sf.basedb.plugins.AbstractFlatFileImporter
          extended by net.sf.basedb.plugins.RawDataFlatFileImporter
All Implemented Interfaces:
AnnotationSetterPlugin, AutoDetectingImporter, InteractivePlugin, Plugin, SignalTarget, ConfigureByExample

public class RawDataFlatFileImporter
extends AbstractFlatFileImporter
implements InteractivePlugin, ConfigureByExample, AnnotationSetterPlugin

A plugin that imports rawdata from a flatfile.

Version:
2.0
Author:
Enell
Last modified
$Date: 2011-02-18 13:19:38 +0100 (Fri, 18 Feb 2011) $

Nested Class Summary
(package private)  class RawDataFlatFileImporter.FileIterator
           
 
Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin
Plugin.MainType
 
Field Summary
private static About about
           
private  List<PluginParameter<String>> allColumnMappings
           
private  RawDataBatcher batcher
           
private static PluginParameter<String> blockColumnMapping
           
private  Mapper blockMapper
           
private static PluginParameter<String> columnColumnMapping
           
private  Mapper columnMapper
           
static String COMMAND_CONFIGURE_PARSER
           
private  RequestInformation configureJob
           
private  RequestInformation configureParser
           
private  RequestInformation configureRawDataType
           
private  boolean continueAfterError
           
private  DbControl dc
           
protected static PluginParameter<String> duplicateFeatureErrorParameter
           
private  Map<RawDataProperty,Mapper> extendedMappers
           
private static PluginParameter<String> featureIdColumnMapping
           
private  Mapper featureIdMapper
           
protected static PluginParameter<String> featureMismatchErrorParameter
           
private  FlatFileParser ffp
           
private  RawDataFlatFileImporter.FileIterator fileIterator
           
private static Set<GuiContext> guiContexts
           
private static PluginParameter<String> metaGridXColumnMapping
           
private  Mapper metaGridXMapper
           
private static PluginParameter<String> metaGridYColumnMapping
           
private  Mapper metaGridYMapper
           
protected static PluginParameter<String> missingReporterErrorParameter
           
private  boolean nullIfException
           
private  NumberFormat numberFormat
           
private static StringParameterType optionalColumnMapping
           
private static Set<Permissions> permissions
           
private static PluginParameter<String> positionColumnMapping
           
private  Mapper positionMapper
           
private  RawBioAssay rawBioAssay
           
private  FileSetMember rawDataMember
           
private  PluginParameter<String> rawDataTypeParameter
           
private  StringParameterType rawDataTypeType
           
private static PluginParameter<String> reporterIdColumnMapping
           
private  Mapper reporterMapper
           
private static StringParameterType requiredColumnMapping
           
private static PluginParameter<String> rowColumnMapping
           
private  Mapper rowMapper
           
private  long totalFileSize
           
private  boolean transactionPerFile
           
private static PluginParameter<String> xColumnMapping
           
private  Mapper xMapper
           
private static PluginParameter<String> yColumnMapping
           
private  Mapper yMapper
           
 
Fields inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter
CHARSET, charsetType, complexMappings, dataFooterRegexpParameter, dataHeaderRegexpParameter, dataSplitterRegexpParameter, DECIMAL_SEPARATOR, decimalSeparatorType, defaultErrorParameter, errorSection, fileParameter, fileType, headerRegexpParameter, ignoreRegexpParameter, invalidUseOfNullErrorParameter, mappingSection, maxDataColumnsParameter, minDataColumnsParameter, numberFormatErrorParameter, numberOutOfRangeErrorParameter, numDataColumnsType, optionalRegexpType, parserSection, requiredRegexpType, sectionRegexpParameter, stringTooLongErrorParameter, trimQuotesParameter
 
Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin
annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc
 
Constructor Summary
RawDataFlatFileImporter()
           
 
Method Summary
private  void addExperimentParameters(DbControl dc, Experiment currentExp, List<PluginParameter<?>> parameters)
           
private  void addRawBioAssayParameters(DbControl dc, RawBioAssay currentRaw, List<PluginParameter<?>> parameters)
           
protected  void begin(FlatFileParser ffp)
          Prepare for importing the next file.
protected  void beginData()
          Initialise column Mapper:s.
 void configure(GuiContext context, Request request, Response response)
          Store configuration settings for Request.COMMAND_CONFIGURE_PLUGIN, COMMAND_CONFIGURE_PARSER and Request.COMMAND_CONFIGURE_JOB.
protected  boolean continueWithNextFileAfterError(Throwable t)
          We will continue if: The file is not of the correct format, eg. t = UnsupportedFileFormatException.
protected  void end(boolean success)
          Close and commit/rollback the RawDataBatcher and DbControl.
 String finish(Throwable t)
          Commit/rollback transaction if needed.
 About getAbout()
          Get information about the plugin, such as name, version, authors, etc.
private  List<PluginParameter<String>> getAllColumnMappings(RawDataType rdt)
          Loads the column mappings if they hasn't been loaded already.
 String getAnnotationMessage()
          Get a message that explains what the plug-in does with the annotations it has.
 ItemQuery<AnnotationType> getAnnotationTypes()
          Return a query that finds annotation types for raw bioassays.
private  RequestInformation getConfigureJobParameters(GuiContext context)
           
private  RequestInformation getConfigureParserParameters(RawDataType rdt)
           
private  RequestInformation getConfigureRawDataTypeParameters()
           
protected  Iterator<File> getFileIterator()
          Get an iterator that returns the files to be imported.
 Set<GuiContext> getGuiContexts()
          Get a set containing all items that the plugin handles.
 Collection<Permissions> getPermissions()
          Request write access to RawBioAssay:s and read access to ArrayDesign:s, File:s, Reporter:s, Platform:s and DataFileType:s
private  ItemQuery<RawBioAssay> getRawBioAssayQuery(Experiment exp)
          Get a query that returns all raw bioassays which it may be possible to import raw data to for the given experiment.
 RequestInformation getRequestInformation(GuiContext context, String command)
          The Request.COMMAND_CONFIGURE_PLUGIN command will ask for what type of rawdata to import.
protected  String getSuccessMessage(int skippedLines)
          Return x spots inserted.
protected  long getTotalFileSize()
          Get the total file size of all files that are going to be imported.
protected  void handleData(FlatFileParser.Data data)
          Called by the parser for every line in the file that is a data line.
protected  void handleHeader(FlatFileParser.Line line)
          Called by the parser for every line in the file that is a header line.
 String isInContext(GuiContext context, Object item)
          Returns null if the item is a RawBioAssay of the correct RawDataType and doesn't already have spots.
 boolean requiresConfiguration()
          Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser.
protected  void start()
          Prepare for the import.
 boolean supportsConfigurations()
          Returns TRUE, since that is how the plugins used to work before this method was introduced.
 
Methods inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter
addErrorHandler, checkColumnMapping, checkColumnMapping, doImport, getCharset, getCharset, getCharsetParameter, getDecimalSeparator, getDecimalSeparatorParameter, getErrorHandler, getErrorOption, getInitializedFlatFileParser, getMainType, getMapper, getNumberFormat, getNumBytes, getPrimaryLocationFilter, getProgress, getSignalHandler, handleSection, isImportable, isImportable, log, log, log, log, run, setUpErrorHandling, wrapInputStream
 
Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin
checkInterrupted, cloneParameterWithDefaultValue, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getJobOrConfigurationValue, getOverwriteAnnotationsParameters, init, log, log, storeValue, storeValue, storeValues, validateRequestParameters
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface net.sf.basedb.core.plugin.Plugin
done, getMainType, init, run
 

Field Detail

about

private static final About about

guiContexts

private static final Set<GuiContext> guiContexts

permissions

private static final Set<Permissions> permissions

optionalColumnMapping

private static final StringParameterType optionalColumnMapping

requiredColumnMapping

private static final StringParameterType requiredColumnMapping

positionColumnMapping

private static final PluginParameter<String> positionColumnMapping

blockColumnMapping

private static final PluginParameter<String> blockColumnMapping

columnColumnMapping

private static final PluginParameter<String> columnColumnMapping

rowColumnMapping

private static final PluginParameter<String> rowColumnMapping

metaGridXColumnMapping

private static final PluginParameter<String> metaGridXColumnMapping

metaGridYColumnMapping

private static final PluginParameter<String> metaGridYColumnMapping

xColumnMapping

private static final PluginParameter<String> xColumnMapping

yColumnMapping

private static final PluginParameter<String> yColumnMapping

reporterIdColumnMapping

private static final PluginParameter<String> reporterIdColumnMapping

featureIdColumnMapping

private static final PluginParameter<String> featureIdColumnMapping

missingReporterErrorParameter

protected static final PluginParameter<String> missingReporterErrorParameter

featureMismatchErrorParameter

protected static final PluginParameter<String> featureMismatchErrorParameter

duplicateFeatureErrorParameter

protected static final PluginParameter<String> duplicateFeatureErrorParameter

COMMAND_CONFIGURE_PARSER

public static final String COMMAND_CONFIGURE_PARSER
See Also:
Constant Field Values

allColumnMappings

private List<PluginParameter<String>> allColumnMappings

configureParser

private RequestInformation configureParser

configureJob

private RequestInformation configureJob

configureRawDataType

private RequestInformation configureRawDataType

rawDataTypeType

private StringParameterType rawDataTypeType

rawDataTypeParameter

private PluginParameter<String> rawDataTypeParameter

dc

private DbControl dc

ffp

private FlatFileParser ffp

numberFormat

private NumberFormat numberFormat

nullIfException

private boolean nullIfException

continueAfterError

private boolean continueAfterError

transactionPerFile

private boolean transactionPerFile

fileIterator

private RawDataFlatFileImporter.FileIterator fileIterator

totalFileSize

private long totalFileSize

rawBioAssay

private RawBioAssay rawBioAssay

batcher

private RawDataBatcher batcher

rawDataMember

private FileSetMember rawDataMember

featureIdMapper

private Mapper featureIdMapper

reporterMapper

private Mapper reporterMapper

positionMapper

private Mapper positionMapper

blockMapper

private Mapper blockMapper

columnMapper

private Mapper columnMapper

rowMapper

private Mapper rowMapper

metaGridXMapper

private Mapper metaGridXMapper

metaGridYMapper

private Mapper metaGridYMapper

xMapper

private Mapper xMapper

yMapper

private Mapper yMapper

extendedMappers

private Map<RawDataProperty,Mapper> extendedMappers
Constructor Detail

RawDataFlatFileImporter

public RawDataFlatFileImporter()
Method Detail

getAbout

public About getAbout()
Description copied from interface: Plugin
Get information about the plugin, such as name, version, authors, etc.

Specified by:
getAbout in interface Plugin
Returns:
An About object

supportsConfigurations

public boolean supportsConfigurations()
Description copied from class: AbstractPlugin
Returns TRUE, since that is how the plugins used to work before this method was introduced.

Specified by:
supportsConfigurations in interface Plugin
Overrides:
supportsConfigurations in class AbstractPlugin
Returns:
TRUE or FALSE

requiresConfiguration

public boolean requiresConfiguration()
Description copied from class: AbstractFlatFileImporter
Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser. If this method is overridden and returns FALSE, the subclass must also override the AbstractFlatFileImporter.getInitializedFlatFileParser() method and provide a parser with all regular expressions and other options set.

Specified by:
requiresConfiguration in interface Plugin
Overrides:
requiresConfiguration in class AbstractFlatFileImporter
Returns:
TRUE or FALSE

getPermissions

public Collection<Permissions> getPermissions()
Request write access to RawBioAssay:s and read access to ArrayDesign:s, File:s, Reporter:s, Platform:s and DataFileType:s

Specified by:
getPermissions in interface Plugin
Overrides:
getPermissions in class AbstractPlugin
Returns:
A collection of permissions or null to not use permissions

getGuiContexts

public Set<GuiContext> getGuiContexts()
Description copied from interface: InteractivePlugin
Get a set containing all items that the plugin handles. Ie. if the plugin imports reporters, return a set containing Item.REPORTER. This information is used by client applications to put the plugin in the proper place in the user interface.

Specified by:
getGuiContexts in interface InteractivePlugin
Returns:
A Set containing Item:s, or null if the plugin is not concerned about items

isInContext

public String isInContext(GuiContext context,
                          Object item)
Returns null if the item is a RawBioAssay of the correct RawDataType and doesn't already have spots.

Specified by:
isInContext in interface InteractivePlugin
Parameters:
context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()
item - The currently active item, it's type should match the GuiContext.getItem() type, or null if the context is a list context
Returns:
Null if the plugin can use that item, or a warning-level message explaining why the plugin can't be used
Throws:
PermissionDeniedException - If the raw bioasssay already has raw data or if the logged in user doesn't have write permission

getRequestInformation

public RequestInformation getRequestInformation(GuiContext context,
                                                String command)
                                         throws BaseException
The Request.COMMAND_CONFIGURE_PLUGIN command will ask for what type of rawdata to import. The COMMAND_CONFIGURE_PARSER command will ask for parser regular expressions and column mappings. The Request.COMMAND_CONFIGURE_JOB command will ask for a file and the RawBioAssay where the data should be added.

Specified by:
getRequestInformation in interface InteractivePlugin
Parameters:
command -
context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()
Returns:
The RequestInformation for the command
Throws:
BaseException - if there is an error

configure

public void configure(GuiContext context,
                      Request request,
                      Response response)
Store configuration settings for Request.COMMAND_CONFIGURE_PLUGIN, COMMAND_CONFIGURE_PARSER and Request.COMMAND_CONFIGURE_JOB.

Specified by:
configure in interface InteractivePlugin
Parameters:
context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()
request - Request object with the command and parameters
response - Response object in for the plugin to response through

getAnnotationTypes

public ItemQuery<AnnotationType> getAnnotationTypes()
Return a query that finds annotation types for raw bioassays.

Specified by:
getAnnotationTypes in interface AnnotationSetterPlugin
Returns:
A query returning annotation types, or null if the current setup doesn't allow any annotations
Since:
2.5

getAnnotationMessage

public String getAnnotationMessage()
Description copied from interface: AnnotationSetterPlugin
Get a message that explains what the plug-in does with the annotations it has.

Specified by:
getAnnotationMessage in interface AnnotationSetterPlugin
Returns:
A message or null

start

protected void start()
Prepare for the import. If used from an experiment context prepare the list of files/raw bioassays to import.

Overrides:
start in class AbstractFlatFileImporter

continueWithNextFileAfterError

protected boolean continueWithNextFileAfterError(Throwable t)
We will continue if:

Overrides:
continueWithNextFileAfterError in class AbstractFlatFileImporter
Parameters:
t - The error that happened
Returns:
TRUE to contine, FALSE to abort

getFileIterator

protected Iterator<File> getFileIterator()
Description copied from class: AbstractFlatFileImporter
Get an iterator that returns the files to be imported. The default implementation returns the single file found in the job's "file" parameter. Subclasses that needs multi-file/item import should override this method to provide their own iterator. They should also override the AbstractFlatFileImporter.getTotalFileSize() method to return sum of all file sizes. Eg. File.getSize().

Overrides:
getFileIterator in class AbstractFlatFileImporter

getTotalFileSize

protected long getTotalFileSize()
Description copied from class: AbstractFlatFileImporter
Get the total file size of all files that are going to be imported. A subclass that is going to import from multiple files needs to override this method. The default implementation return the size of the file in the job's "file" parameter.

Overrides:
getTotalFileSize in class AbstractFlatFileImporter
Returns:
The sum of the file sizes, or -1 if not known

begin

protected void begin(FlatFileParser ffp)
              throws BaseException
Prepare for importing the next file. Eg. create a DbControl (if needed) and a RawDataBatcher. Load the RawBioAssay that has been specified.

Overrides:
begin in class AbstractFlatFileImporter
Throws:
BaseException
See Also:
AbstractFlatFileImporter.end(boolean)

beginData

protected void beginData()
Initialise column Mapper:s.

Overrides:
beginData in class AbstractFlatFileImporter

end

protected void end(boolean success)
            throws BaseException
Close and commit/rollback the RawDataBatcher and DbControl.

Overrides:
end in class AbstractFlatFileImporter
Parameters:
success - TRUE if the file was parsed successfully, FALSE otherwise
Throws:
BaseException
See Also:
AbstractFlatFileImporter.begin(FlatFileParser)

finish

public String finish(Throwable t)
Commit/rollback transaction if needed. If everything went ok, t=null.

Overrides:
finish in class AbstractFlatFileImporter
Parameters:
t - Null if no error has happened
Returns:
A message or null

handleData

protected void handleData(FlatFileParser.Data data)
                   throws BaseException
Description copied from class: AbstractFlatFileImporter
Called by the parser for every line in the file that is a data line.

Specified by:
handleData in class AbstractFlatFileImporter
Throws:
BaseException

handleHeader

protected void handleHeader(FlatFileParser.Line line)
                     throws BaseException
Description copied from class: AbstractFlatFileImporter
Called by the parser for every line in the file that is a header line.

Overrides:
handleHeader in class AbstractFlatFileImporter
Throws:
BaseException

getSuccessMessage

protected String getSuccessMessage(int skippedLines)
Return x spots inserted.

Overrides:
getSuccessMessage in class AbstractFlatFileImporter
Parameters:
skippedLines - The number of data lines that were skipped due to errors

getAllColumnMappings

private List<PluginParameter<String>> getAllColumnMappings(RawDataType rdt)
Loads the column mappings if they hasn't been loaded already.

Parameters:
rdt - the RawDataType that defines the extra properties.

getConfigureJobParameters

private RequestInformation getConfigureJobParameters(GuiContext context)

addRawBioAssayParameters

private void addRawBioAssayParameters(DbControl dc,
                                      RawBioAssay currentRaw,
                                      List<PluginParameter<?>> parameters)

addExperimentParameters

private void addExperimentParameters(DbControl dc,
                                     Experiment currentExp,
                                     List<PluginParameter<?>> parameters)

getConfigureRawDataTypeParameters

private RequestInformation getConfigureRawDataTypeParameters()

getConfigureParserParameters

private RequestInformation getConfigureParserParameters(RawDataType rdt)

getRawBioAssayQuery

private ItemQuery<RawBioAssay> getRawBioAssayQuery(Experiment exp)
Get a query that returns all raw bioassays which it may be possible to import raw data to for the given experiment. The raw bioassays should have no raw data already and they must have at least one file attached.

Parameters:
exp - The experiment

2.17.2: 2011-06-17