Package net.sf.basedb.plugins

Class RawDataFlatFileImporter

java.lang.Object

net.sf.basedb.core.plugin.AbstractPlugin

net.sf.basedb.plugins.AbstractFlatFileImporter

net.sf.basedb.plugins.RawDataFlatFileImporter

All Implemented Interfaces:

AnnotationSetterPlugin, AutoDetectingImporter, InteractivePlugin, Plugin, SignalTarget, ConfigureByExample

public class RawDataFlatFileImporter
extends AbstractFlatFileImporter
implements InteractivePlugin, ConfigureByExample, AnnotationSetterPlugin

A plugin that imports rawdata from a flatfile.

Version:

2.0

Author:

Enell

Last modified

$Date: 2021-06-04 12:40:45 +0200 (Fri, 04 Jun 2021) $

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) class	RawDataFlatFileImporter.FileIterator

Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin

Plugin.MainType

Field Summary

Fields

Modifier and Type	Field	Description
private List<PluginParameter<String>>	allColumnMappings
private RawDataBatcher	batcher
private static final PluginParameter<String>	blockColumnMapping
private Mapper	blockMapper
private static final PluginParameter<String>	columnColumnMapping
private Mapper	columnMapper
static final String	COMMAND_CONFIGURE_PARSER
private RequestInformation	configureJob
private RequestInformation	configureParser
private RequestInformation	configureRawDataType
private boolean	continueAfterError
private DbControl	dc
protected static final PluginParameter<String>	duplicateFeatureErrorParameter
private Map<RawDataProperty,Mapper>	extendedMappers
private static final PluginParameter<String>	featureIdColumnMapping
private Mapper	featureIdMapper
protected static final PluginParameter<String>	featureMismatchErrorParameter
private FlatFileParser	ffp
private RawDataFlatFileImporter.FileIterator	fileIterator
private static final Set<GuiContext>	guiContexts
private static final PluginParameter<String>	metaGridXColumnMapping
private Mapper	metaGridXMapper
private static final PluginParameter<String>	metaGridYColumnMapping
private Mapper	metaGridYMapper
protected static final PluginParameter<String>	missingReporterErrorParameter
private boolean	nullIfException
private static final StringParameterType	optionalColumnMapping
private static final Set<Permissions>	permissions
private static final PluginParameter<String>	positionColumnMapping
private Mapper	positionMapper
private RawBioAssay	rawBioAssay
private FileSetMember	rawDataMember
private PluginParameter<String>	rawDataTypeParameter
private StringParameterType	rawDataTypeType
private static final PluginParameter<String>	reporterIdColumnMapping
private Mapper	reporterMapper
private static final StringParameterType	requiredColumnMapping
private static final PluginParameter<String>	rowColumnMapping
private Mapper	rowMapper
private long	totalFileSize
private boolean	transactionPerFile
private static final PluginParameter<String>	xColumnMapping
private Mapper	xMapper
private static final PluginParameter<String>	yColumnMapping
private Mapper	yMapper

Fields inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter

complexMappings, dataFooterRegexpParameter, dataHeaderRegexpParameter, dataSplitterRegexpParameter, defaultErrorParameter, errorSection, excelSheetParameter, fileParameter, fileType, headerRegexpParameter, ignoreRegexpParameter, invalidUseOfNullErrorParameter, mappingSection, maxDataColumnsParameter, minDataColumnsParameter, numberFormatErrorParameter, numberOutOfRangeErrorParameter, numDataColumnsType, optionalRegexpType, parserSection, requiredRegexpType, sectionRegexpParameter, stringTooLongErrorParameter, trimQuotesParameter

Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin

annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc

Constructor Summary

Constructors

Constructor	Description
RawDataFlatFileImporter()

Method Summary

Modifier and Type	Method	Description
private void	addExperimentParameters(DbControl dc, Experiment currentExp, List<PluginParameter<?>> parameters)
private void	addRawBioAssayParameters(DbControl dc, RawBioAssay currentRaw, List<PluginParameter<?>> parameters)
protected void	begin(FlatFileParser ffp)	Prepare for importing the next file.
protected void	beginData()	Initialise column Mapper:s.
void	configure(GuiContext context, Request request, Response response)	Store configuration settings for Request.COMMAND_CONFIGURE_PLUGIN, COMMAND_CONFIGURE_PARSER and Request.COMMAND_CONFIGURE_JOB.
protected boolean	continueWithNextFileAfterError(Throwable t)	We will continue if: The file is not of the correct format, eg. t = UnsupportedFileFormatException.
protected void	end(boolean success)	Close and commit/rollback the RawDataBatcher and DbControl.
String	finish(Throwable t)	Commit/rollback transaction if needed.
private List<PluginParameter<String>>	getAllColumnMappings(RawDataType rdt)	Loads the column mappings if they hasn't been loaded already.
String	getAnnotationMessage()	Get a message that explains what the plug-in does with the annotations it has.
ItemQuery<AnnotationType>	getAnnotationTypes()	Return a query that finds annotation types for raw bioassays.
private RequestInformation	getConfigureJobParameters(GuiContext context)
private RequestInformation	getConfigureParserParameters(RawDataType rdt)
private RequestInformation	getConfigureRawDataTypeParameters()
protected Iterator<File>	getFileIterator()	Get an iterator that returns the files to be imported.
Set<GuiContext>	getGuiContexts()	Get a set containing all items that the plugin handles.
Collection<Permissions>	getPermissions()	Request write access to RawBioAssay:s and read access to ArrayDesign:s, File:s, Reporter:s, Platform:s and DataFileType:s
private ItemQuery<RawBioAssay>	getRawBioAssayQuery(Experiment exp)	Get a query that returns all raw bioassays which it may be possible to import raw data to for the given experiment.
RequestInformation	getRequestInformation(GuiContext context, String command)	The Request.COMMAND_CONFIGURE_PLUGIN command will ask for what type of rawdata to import.
protected String	getSuccessMessage(int skippedLines)	Return x spots inserted.
protected long	getTotalFileSize()	Get the total file size of all files that are going to be imported.
protected void	handleData(FlatFileParser.Data data)	Called by the parser for every line in the file that is a data line.
protected void	handleHeader(FlatFileParser.Line line)	Called by the parser for every line in the file that is a header line.
String	isInContext(GuiContext context, Object item)	Returns null if the item is a RawBioAssay of the correct RawDataType and doesn't already have spots.
boolean	requiresConfiguration()	Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser.
protected void	start()	Prepare for the import.
boolean	supportsConfigurations()	Returns TRUE, since that is how the plugins used to work before this method was introduced.

Methods inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter

addErrorHandler, checkColumnMapping, checkColumnMapping, doImport, getCharset, getCharset, getDateFormatter, getDecimalSeparator, getErrorHandler, getErrorOption, getInitializedFlatFileParser, getInitializedFlatFileParser, getMainType, getMapper, getMapper, getNumberFormat, getNumBytes, getPattern, getPattern, getProgress, getSignalHandler, getTimestampFormatter, handleSection, isImportable, isImportable, log, log, log, log, run, setUpErrorHandling, wrapInputStream

Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin

cloneParameterWithDefaultValue, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getJobOrConfigurationValue, getOverwriteAnnotationsParameters, init, isLogging, log, log, storeValue, storeValue, storeValues, validateRequestParameters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.sf.basedb.core.plugin.Plugin

done, getMainType, init, run

Field Details

guiContexts

private static final Set<GuiContext> guiContexts

permissions

private static final Set<Permissions> permissions

optionalColumnMapping

private static final StringParameterType optionalColumnMapping

requiredColumnMapping

private static final StringParameterType requiredColumnMapping

positionColumnMapping

private static final PluginParameter<String> positionColumnMapping

blockColumnMapping

private static final PluginParameter<String> blockColumnMapping

columnColumnMapping

private static final PluginParameter<String> columnColumnMapping

rowColumnMapping

private static final PluginParameter<String> rowColumnMapping

metaGridXColumnMapping

private static final PluginParameter<String> metaGridXColumnMapping

metaGridYColumnMapping

private static final PluginParameter<String> metaGridYColumnMapping

xColumnMapping

private static final PluginParameter<String> xColumnMapping

yColumnMapping

private static final PluginParameter<String> yColumnMapping

reporterIdColumnMapping

private static final PluginParameter<String> reporterIdColumnMapping

featureIdColumnMapping

private static final PluginParameter<String> featureIdColumnMapping

missingReporterErrorParameter

protected static final PluginParameter<String> missingReporterErrorParameter

featureMismatchErrorParameter

protected static final PluginParameter<String> featureMismatchErrorParameter

duplicateFeatureErrorParameter

protected static final PluginParameter<String> duplicateFeatureErrorParameter

COMMAND_CONFIGURE_PARSER

public static final String COMMAND_CONFIGURE_PARSER

See Also:

• Constant Field Values

allColumnMappings

private List<PluginParameter<String>> allColumnMappings

configureParser

private RequestInformation configureParser

configureJob

private RequestInformation configureJob

configureRawDataType

private RequestInformation configureRawDataType

rawDataTypeType

private StringParameterType rawDataTypeType

rawDataTypeParameter

private PluginParameter<String> rawDataTypeParameter

dc

private DbControl dc

ffp

private FlatFileParser ffp

nullIfException

private boolean nullIfException

continueAfterError

private boolean continueAfterError

transactionPerFile

private boolean transactionPerFile

fileIterator

private RawDataFlatFileImporter.FileIterator fileIterator

totalFileSize

private long totalFileSize

rawBioAssay

private RawBioAssay rawBioAssay

batcher

private RawDataBatcher batcher

rawDataMember

private FileSetMember rawDataMember

featureIdMapper

private Mapper featureIdMapper

reporterMapper

private Mapper reporterMapper

positionMapper

private Mapper positionMapper

blockMapper

private Mapper blockMapper

columnMapper

private Mapper columnMapper

rowMapper

private Mapper rowMapper

metaGridXMapper

private Mapper metaGridXMapper

metaGridYMapper

private Mapper metaGridYMapper

xMapper

private Mapper xMapper

yMapper

private Mapper yMapper

extendedMappers

private Map<RawDataProperty,Mapper> extendedMappers

Constructor Details

RawDataFlatFileImporter

public RawDataFlatFileImporter()

Method Details

supportsConfigurations

public boolean supportsConfigurations()

Description copied from class: AbstractPlugin

Returns TRUE, since that is how the plugins used to work before this method was introduced.

Specified by:

supportsConfigurations in interface Plugin

Overrides:

supportsConfigurations in class AbstractPlugin

Returns:

TRUE or FALSE

requiresConfiguration

public boolean requiresConfiguration()

Description copied from class: AbstractFlatFileImporter

Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser. If this method is overridden and returns FALSE, the subclass must also override the AbstractFlatFileImporter.getInitializedFlatFileParser() method and provide a parser with all regular expressions and other options set.

Specified by:

requiresConfiguration in interface Plugin

Overrides:

requiresConfiguration in class AbstractFlatFileImporter

Returns:

TRUE or FALSE

getPermissions

public Collection<Permissions> getPermissions()

Request write access to RawBioAssay:s and read access to ArrayDesign:s, File:s, Reporter:s, Platform:s and DataFileType:s

Specified by:

getPermissions in interface Plugin

Overrides:

getPermissions in class AbstractPlugin

Returns:

A collection of permissions or null to not use permissions

getGuiContexts

public Set<GuiContext> getGuiContexts()

Description copied from interface: InteractivePlugin

Get a set containing all items that the plugin handles. Ie. if the plugin imports reporters, return a set containing Item.REPORTER. This information is used by client applications to put the plugin in the proper place in the user interface.

Specified by:

getGuiContexts in interface InteractivePlugin

Returns:

A Set containing Item:s, or null if the plugin is not concerned about items

isInContext

public String isInContext(GuiContext context,
 Object item)

Returns null if the item is a RawBioAssay of the correct RawDataType and doesn't already have spots.

Specified by:

isInContext in interface InteractivePlugin

Parameters:

context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()

item - The currently active item, it's type should match the GuiContext.getItem() type, or null if the context is a list context

Returns:

Null if the plugin can use that item, or a warning-level message explaining why the plugin can't be used

Throws:

PermissionDeniedException - If the raw bioasssay already has raw data or if the logged in user doesn't have write permission

getRequestInformation

public RequestInformation getRequestInformation(GuiContext context,
 String command)
                                         throws BaseException

The Request.COMMAND_CONFIGURE_PLUGIN command will ask for what type of rawdata to import. The COMMAND_CONFIGURE_PARSER command will ask for parser regular expressions and column mappings. The Request.COMMAND_CONFIGURE_JOB command will ask for a file and the RawBioAssay where the data should be added.

Specified by:

getRequestInformation in interface InteractivePlugin

Parameters:

context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()

command - The command

Returns:

The RequestInformation for the command

Throws:

BaseException - if there is an error

configure

public void configure(GuiContext context,
 Request request,
 Response response)

Store configuration settings for Request.COMMAND_CONFIGURE_PLUGIN, COMMAND_CONFIGURE_PARSER and Request.COMMAND_CONFIGURE_JOB.

Specified by:

configure in interface InteractivePlugin

Parameters:

context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()

request - Request object with the command and parameters

response - Response object in for the plugin to response through

getAnnotationTypes

public ItemQuery<AnnotationType> getAnnotationTypes()

Return a query that finds annotation types for raw bioassays.

Specified by:

getAnnotationTypes in interface AnnotationSetterPlugin

Returns:

A query returning annotation types, or null if the current setup doesn't allow any annotations

Since:

2.5

getAnnotationMessage

public String getAnnotationMessage()

Description copied from interface: AnnotationSetterPlugin

Get a message that explains what the plug-in does with the annotations it has.

Specified by:

getAnnotationMessage in interface AnnotationSetterPlugin

Returns:

A message or null

start

protected void start()

Prepare for the import. If used from an experiment context prepare the list of files/raw bioassays to import.

Overrides:

start in class AbstractFlatFileImporter

continueWithNextFileAfterError

protected boolean continueWithNextFileAfterError(Throwable t)

We will continue if:

• The file is not of the correct format, eg. t = UnsupportedFileFormatException.
• The 'Continue after error' parameter is true

Overrides:

continueWithNextFileAfterError in class AbstractFlatFileImporter

Parameters:

t - The error that happened

Returns:

TRUE to contine, FALSE to abort

getFileIterator

protected Iterator<File> getFileIterator()

Description copied from class: AbstractFlatFileImporter

Get an iterator that returns the files to be imported. The default implementation returns the single file found in the job's "file" parameter. Subclasses that needs multi-file/item import should override this method to provide their own iterator. They should also override the AbstractFlatFileImporter.getTotalFileSize() method to return sum of all file sizes. Eg. File.getSize().

Overrides:

getFileIterator in class AbstractFlatFileImporter

getTotalFileSize

protected long getTotalFileSize()

Description copied from class: AbstractFlatFileImporter

Get the total file size of all files that are going to be imported. A subclass that is going to import from multiple files needs to override this method. The default implementation return the size of the file in the job's "file" parameter.

Overrides:

getTotalFileSize in class AbstractFlatFileImporter

Returns:

The sum of the file sizes, or -1 if not known

begin

protected void begin(FlatFileParser ffp)
              throws BaseException

Prepare for importing the next file. Eg. create a DbControl (if needed) and a RawDataBatcher. Load the RawBioAssay that has been specified.

Overrides:

begin in class AbstractFlatFileImporter

Throws:

BaseException

See Also:

• AbstractFlatFileImporter.end(boolean)

beginData

protected void beginData()

Initialise column Mapper:s.

Overrides:

beginData in class AbstractFlatFileImporter

end

protected void end(boolean success)
            throws BaseException

Close and commit/rollback the RawDataBatcher and DbControl.

Overrides:

end in class AbstractFlatFileImporter

Parameters:

success - TRUE if the file was parsed successfully, FALSE otherwise

Throws:

BaseException

See Also:

• AbstractFlatFileImporter.begin(FlatFileParser)

finish

public String finish(Throwable t)

Commit/rollback transaction if needed. If everything went ok, t=null.

Overrides:

finish in class AbstractFlatFileImporter

Parameters:

t - Null if no error has happened

Returns:

A message or null

handleData

protected void handleData(FlatFileParser.Data data)
                   throws BaseException

Description copied from class: AbstractFlatFileImporter

Called by the parser for every line in the file that is a data line.

Specified by:

handleData in class AbstractFlatFileImporter

Throws:

BaseException

handleHeader

protected void handleHeader(FlatFileParser.Line line)
                     throws BaseException

Description copied from class: AbstractFlatFileImporter

Called by the parser for every line in the file that is a header line.

Overrides:

handleHeader in class AbstractFlatFileImporter

Throws:

BaseException

getSuccessMessage

protected String getSuccessMessage(int skippedLines)

Return x spots inserted.

Overrides:

getSuccessMessage in class AbstractFlatFileImporter

Parameters:

skippedLines - The number of data lines that were skipped due to errors

getAllColumnMappings

private List<PluginParameter<String>> getAllColumnMappings(RawDataType rdt)

Loads the column mappings if they hasn't been loaded already.

Parameters:

rdt - the RawDataType that defines the extra properties.

getConfigureJobParameters

private RequestInformation getConfigureJobParameters(GuiContext context)

addRawBioAssayParameters

private void addRawBioAssayParameters(DbControl dc,
 RawBioAssay currentRaw,
 List<PluginParameter<?>> parameters)

addExperimentParameters

private void addExperimentParameters(DbControl dc,
 Experiment currentExp,
 List<PluginParameter<?>> parameters)

getConfigureRawDataTypeParameters

private RequestInformation getConfigureRawDataTypeParameters()

getConfigureParserParameters

private RequestInformation getConfigureParserParameters(RawDataType rdt)

getRawBioAssayQuery

private ItemQuery<RawBioAssay> getRawBioAssayQuery(Experiment exp)

Get a query that returns all raw bioassays which it may be possible to import raw data to for the given experiment. The raw bioassays should have no raw data already and they must have at least one file attached.

Parameters:

exp - The experiment