Package net.sf.basedb.plugins
Class RawDataFlatFileImporter
- java.lang.Object
-
- net.sf.basedb.core.plugin.AbstractPlugin
-
- net.sf.basedb.plugins.AbstractFlatFileImporter
-
- net.sf.basedb.plugins.RawDataFlatFileImporter
-
- All Implemented Interfaces:
AnnotationSetterPlugin
,AutoDetectingImporter
,InteractivePlugin
,Plugin
,SignalTarget
,ConfigureByExample
public class RawDataFlatFileImporter extends AbstractFlatFileImporter implements InteractivePlugin, ConfigureByExample, AnnotationSetterPlugin
A plugin that imports rawdata from a flatfile.- Version:
- 2.0
- Author:
- Enell
- Last modified
- $Date: 2019-03-21 12:50:52 +0100 (tors, 21 mars 2019) $
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description (package private) class
RawDataFlatFileImporter.FileIterator
-
Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin
Plugin.MainType
-
-
Field Summary
-
Fields inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter
complexMappings, dataFooterRegexpParameter, dataHeaderRegexpParameter, dataSplitterRegexpParameter, defaultErrorParameter, errorSection, excelSheetParameter, fileParameter, fileType, headerRegexpParameter, ignoreRegexpParameter, invalidUseOfNullErrorParameter, mappingSection, maxDataColumnsParameter, minDataColumnsParameter, numberFormatErrorParameter, numberOutOfRangeErrorParameter, numDataColumnsType, optionalRegexpType, parserSection, requiredRegexpType, sectionRegexpParameter, stringTooLongErrorParameter, trimQuotesParameter
-
Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin
annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc
-
-
Constructor Summary
Constructors Constructor Description RawDataFlatFileImporter()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description private void
addExperimentParameters(DbControl dc, Experiment currentExp, List<PluginParameter<?>> parameters)
private void
addRawBioAssayParameters(DbControl dc, RawBioAssay currentRaw, List<PluginParameter<?>> parameters)
protected void
begin(FlatFileParser ffp)
Prepare for importing the next file.protected void
beginData()
Initialise columnMapper
:s.void
configure(GuiContext context, Request request, Response response)
Store configuration settings forRequest.COMMAND_CONFIGURE_PLUGIN
,COMMAND_CONFIGURE_PARSER
andRequest.COMMAND_CONFIGURE_JOB
.protected boolean
continueWithNextFileAfterError(Throwable t)
We will continue if: The file is not of the correct format, eg. t =UnsupportedFileFormatException
.protected void
end(boolean success)
Close and commit/rollback the RawDataBatcher and DbControl.String
finish(Throwable t)
Commit/rollback transaction if needed.private List<PluginParameter<String>>
getAllColumnMappings(RawDataType rdt)
Loads the column mappings if they hasn't been loaded already.String
getAnnotationMessage()
Get a message that explains what the plug-in does with the annotations it has.ItemQuery<AnnotationType>
getAnnotationTypes()
Return a query that finds annotation types for raw bioassays.private RequestInformation
getConfigureJobParameters(GuiContext context)
private RequestInformation
getConfigureParserParameters(RawDataType rdt)
private RequestInformation
getConfigureRawDataTypeParameters()
protected Iterator<File>
getFileIterator()
Get an iterator that returns the files to be imported.Set<GuiContext>
getGuiContexts()
Get a set containing all items that the plugin handles.Collection<Permissions>
getPermissions()
Request write access to RawBioAssay:s and read access to ArrayDesign:s, File:s, Reporter:s, Platform:s and DataFileType:sprivate ItemQuery<RawBioAssay>
getRawBioAssayQuery(Experiment exp)
Get a query that returns all raw bioassays which it may be possible to import raw data to for the given experiment.RequestInformation
getRequestInformation(GuiContext context, String command)
TheRequest.COMMAND_CONFIGURE_PLUGIN
command will ask for what type of rawdata to import.protected String
getSuccessMessage(int skippedLines)
Returnx spots inserted
.protected long
getTotalFileSize()
Get the total file size of all files that are going to be imported.protected void
handleData(FlatFileParser.Data data)
Called by the parser for every line in the file that is a data line.protected void
handleHeader(FlatFileParser.Line line)
Called by the parser for every line in the file that is a header line.String
isInContext(GuiContext context, Object item)
Returns null if the item is aRawBioAssay
of the correctRawDataType
and doesn't already have spots.boolean
requiresConfiguration()
Return TRUE, since the implementation requires it for finding the regular expressions used by theFlatFileParser
.protected void
start()
Prepare for the import.boolean
supportsConfigurations()
Returns TRUE, since that is how the plugins used to work before this method was introduced.-
Methods inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter
addErrorHandler, checkColumnMapping, checkColumnMapping, doImport, getCharset, getCharset, getDateFormatter, getDecimalSeparator, getErrorHandler, getErrorOption, getInitializedFlatFileParser, getMainType, getMapper, getMapper, getNumberFormat, getNumBytes, getPattern, getProgress, getSignalHandler, getTimestampFormatter, handleSection, isImportable, isImportable, log, log, log, log, run, setUpErrorHandling, wrapInputStream
-
Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin
cloneParameterWithDefaultValue, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getJobOrConfigurationValue, getOverwriteAnnotationsParameters, init, log, log, storeValue, storeValue, storeValues, validateRequestParameters
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface net.sf.basedb.core.plugin.Plugin
done, getMainType, init, run
-
-
-
-
Field Detail
-
guiContexts
private static final Set<GuiContext> guiContexts
-
permissions
private static final Set<Permissions> permissions
-
optionalColumnMapping
private static final StringParameterType optionalColumnMapping
-
requiredColumnMapping
private static final StringParameterType requiredColumnMapping
-
positionColumnMapping
private static final PluginParameter<String> positionColumnMapping
-
blockColumnMapping
private static final PluginParameter<String> blockColumnMapping
-
columnColumnMapping
private static final PluginParameter<String> columnColumnMapping
-
rowColumnMapping
private static final PluginParameter<String> rowColumnMapping
-
metaGridXColumnMapping
private static final PluginParameter<String> metaGridXColumnMapping
-
metaGridYColumnMapping
private static final PluginParameter<String> metaGridYColumnMapping
-
xColumnMapping
private static final PluginParameter<String> xColumnMapping
-
yColumnMapping
private static final PluginParameter<String> yColumnMapping
-
reporterIdColumnMapping
private static final PluginParameter<String> reporterIdColumnMapping
-
featureIdColumnMapping
private static final PluginParameter<String> featureIdColumnMapping
-
missingReporterErrorParameter
protected static final PluginParameter<String> missingReporterErrorParameter
-
featureMismatchErrorParameter
protected static final PluginParameter<String> featureMismatchErrorParameter
-
duplicateFeatureErrorParameter
protected static final PluginParameter<String> duplicateFeatureErrorParameter
-
COMMAND_CONFIGURE_PARSER
public static final String COMMAND_CONFIGURE_PARSER
- See Also:
- Constant Field Values
-
allColumnMappings
private List<PluginParameter<String>> allColumnMappings
-
configureParser
private RequestInformation configureParser
-
configureJob
private RequestInformation configureJob
-
configureRawDataType
private RequestInformation configureRawDataType
-
rawDataTypeType
private StringParameterType rawDataTypeType
-
rawDataTypeParameter
private PluginParameter<String> rawDataTypeParameter
-
dc
private DbControl dc
-
ffp
private FlatFileParser ffp
-
nullIfException
private boolean nullIfException
-
continueAfterError
private boolean continueAfterError
-
transactionPerFile
private boolean transactionPerFile
-
fileIterator
private RawDataFlatFileImporter.FileIterator fileIterator
-
totalFileSize
private long totalFileSize
-
rawBioAssay
private RawBioAssay rawBioAssay
-
batcher
private RawDataBatcher batcher
-
rawDataMember
private FileSetMember rawDataMember
-
featureIdMapper
private Mapper featureIdMapper
-
reporterMapper
private Mapper reporterMapper
-
positionMapper
private Mapper positionMapper
-
blockMapper
private Mapper blockMapper
-
columnMapper
private Mapper columnMapper
-
rowMapper
private Mapper rowMapper
-
metaGridXMapper
private Mapper metaGridXMapper
-
metaGridYMapper
private Mapper metaGridYMapper
-
xMapper
private Mapper xMapper
-
yMapper
private Mapper yMapper
-
extendedMappers
private Map<RawDataProperty,Mapper> extendedMappers
-
-
Method Detail
-
supportsConfigurations
public boolean supportsConfigurations()
Description copied from class:AbstractPlugin
Returns TRUE, since that is how the plugins used to work before this method was introduced.- Specified by:
supportsConfigurations
in interfacePlugin
- Overrides:
supportsConfigurations
in classAbstractPlugin
- Returns:
- TRUE or FALSE
-
requiresConfiguration
public boolean requiresConfiguration()
Description copied from class:AbstractFlatFileImporter
Return TRUE, since the implementation requires it for finding the regular expressions used by theFlatFileParser
. If this method is overridden and returns FALSE, the subclass must also override theAbstractFlatFileImporter.getInitializedFlatFileParser()
method and provide a parser with all regular expressions and other options set.- Specified by:
requiresConfiguration
in interfacePlugin
- Overrides:
requiresConfiguration
in classAbstractFlatFileImporter
- Returns:
- TRUE or FALSE
-
getPermissions
public Collection<Permissions> getPermissions()
Request write access to RawBioAssay:s and read access to ArrayDesign:s, File:s, Reporter:s, Platform:s and DataFileType:s- Specified by:
getPermissions
in interfacePlugin
- Overrides:
getPermissions
in classAbstractPlugin
- Returns:
- A collection of permissions or null to not use permissions
-
getGuiContexts
public Set<GuiContext> getGuiContexts()
Description copied from interface:InteractivePlugin
Get a set containing all items that the plugin handles. Ie. if the plugin imports reporters, return a set containingItem.REPORTER
. This information is used by client applications to put the plugin in the proper place in the user interface.- Specified by:
getGuiContexts
in interfaceInteractivePlugin
- Returns:
- A
Set
containingItem
:s, or null if the plugin is not concerned about items
-
isInContext
public String isInContext(GuiContext context, Object item)
Returns null if the item is aRawBioAssay
of the correctRawDataType
and doesn't already have spots.- Specified by:
isInContext
in interfaceInteractivePlugin
- Parameters:
context
- The current context of the client application, it is one of the values found in set returned byInteractivePlugin.getGuiContexts()
item
- The currently active item, it's type should match theGuiContext.getItem()
type, or null if the context is a list context- Returns:
Null
if the plugin can use that item, or a warning-level message explaining why the plugin can't be used- Throws:
PermissionDeniedException
- If the raw bioasssay already has raw data or if the logged in user doesn't have write permission
-
getRequestInformation
public RequestInformation getRequestInformation(GuiContext context, String command) throws BaseException
TheRequest.COMMAND_CONFIGURE_PLUGIN
command will ask for what type of rawdata to import. TheCOMMAND_CONFIGURE_PARSER
command will ask for parser regular expressions and column mappings. TheRequest.COMMAND_CONFIGURE_JOB
command will ask for a file and theRawBioAssay
where the data should be added.- Specified by:
getRequestInformation
in interfaceInteractivePlugin
- Parameters:
context
- The current context of the client application, it is one of the values found in set returned byInteractivePlugin.getGuiContexts()
command
- The command- Returns:
- The
RequestInformation
for the command - Throws:
BaseException
- if there is an error
-
configure
public void configure(GuiContext context, Request request, Response response)
Store configuration settings forRequest.COMMAND_CONFIGURE_PLUGIN
,COMMAND_CONFIGURE_PARSER
andRequest.COMMAND_CONFIGURE_JOB
.- Specified by:
configure
in interfaceInteractivePlugin
- Parameters:
context
- The current context of the client application, it is one of the values found in set returned byInteractivePlugin.getGuiContexts()
request
- Request object with the command and parametersresponse
- Response object in for the plugin to response through
-
getAnnotationTypes
public ItemQuery<AnnotationType> getAnnotationTypes()
Return a query that finds annotation types for raw bioassays.- Specified by:
getAnnotationTypes
in interfaceAnnotationSetterPlugin
- Returns:
- A query returning annotation types, or null if the current setup doesn't allow any annotations
- Since:
- 2.5
-
getAnnotationMessage
public String getAnnotationMessage()
Description copied from interface:AnnotationSetterPlugin
Get a message that explains what the plug-in does with the annotations it has.- Specified by:
getAnnotationMessage
in interfaceAnnotationSetterPlugin
- Returns:
- A message or null
-
start
protected void start()
Prepare for the import. If used from an experiment context prepare the list of files/raw bioassays to import.- Overrides:
start
in classAbstractFlatFileImporter
-
continueWithNextFileAfterError
protected boolean continueWithNextFileAfterError(Throwable t)
We will continue if:- The file is not of the correct format, eg. t =
UnsupportedFileFormatException
. - The 'Continue after error' parameter is true
- Overrides:
continueWithNextFileAfterError
in classAbstractFlatFileImporter
- Parameters:
t
- The error that happened- Returns:
- TRUE to contine, FALSE to abort
- The file is not of the correct format, eg. t =
-
getFileIterator
protected Iterator<File> getFileIterator()
Description copied from class:AbstractFlatFileImporter
Get an iterator that returns the files to be imported. The default implementation returns the single file found in the job's "file" parameter. Subclasses that needs multi-file/item import should override this method to provide their own iterator. They should also override theAbstractFlatFileImporter.getTotalFileSize()
method to return sum of all file sizes. Eg.File.getSize()
.- Overrides:
getFileIterator
in classAbstractFlatFileImporter
-
getTotalFileSize
protected long getTotalFileSize()
Description copied from class:AbstractFlatFileImporter
Get the total file size of all files that are going to be imported. A subclass that is going to import from multiple files needs to override this method. The default implementation return the size of the file in the job's "file" parameter.- Overrides:
getTotalFileSize
in classAbstractFlatFileImporter
- Returns:
- The sum of the file sizes, or -1 if not known
-
begin
protected void begin(FlatFileParser ffp) throws BaseException
Prepare for importing the next file. Eg. create aDbControl
(if needed) and aRawDataBatcher
. Load theRawBioAssay
that has been specified.- Overrides:
begin
in classAbstractFlatFileImporter
- Throws:
BaseException
- See Also:
AbstractFlatFileImporter.end(boolean)
-
beginData
protected void beginData()
Initialise columnMapper
:s.- Overrides:
beginData
in classAbstractFlatFileImporter
-
end
protected void end(boolean success) throws BaseException
Close and commit/rollback the RawDataBatcher and DbControl.- Overrides:
end
in classAbstractFlatFileImporter
- Parameters:
success
- TRUE if the file was parsed successfully, FALSE otherwise- Throws:
BaseException
- See Also:
AbstractFlatFileImporter.begin(FlatFileParser)
-
finish
public String finish(Throwable t)
Commit/rollback transaction if needed. If everything went ok, t=null.- Overrides:
finish
in classAbstractFlatFileImporter
- Parameters:
t
- Null if no error has happened- Returns:
- A message or null
-
handleData
protected void handleData(FlatFileParser.Data data) throws BaseException
Description copied from class:AbstractFlatFileImporter
Called by the parser for every line in the file that is a data line.- Specified by:
handleData
in classAbstractFlatFileImporter
- Throws:
BaseException
-
handleHeader
protected void handleHeader(FlatFileParser.Line line) throws BaseException
Description copied from class:AbstractFlatFileImporter
Called by the parser for every line in the file that is a header line.- Overrides:
handleHeader
in classAbstractFlatFileImporter
- Throws:
BaseException
-
getSuccessMessage
protected String getSuccessMessage(int skippedLines)
Returnx spots inserted
.- Overrides:
getSuccessMessage
in classAbstractFlatFileImporter
- Parameters:
skippedLines
- The number of data lines that were skipped due to errors
-
getAllColumnMappings
private List<PluginParameter<String>> getAllColumnMappings(RawDataType rdt)
Loads the column mappings if they hasn't been loaded already.- Parameters:
rdt
- theRawDataType
that defines the extra properties.
-
getConfigureJobParameters
private RequestInformation getConfigureJobParameters(GuiContext context)
-
addRawBioAssayParameters
private void addRawBioAssayParameters(DbControl dc, RawBioAssay currentRaw, List<PluginParameter<?>> parameters)
-
addExperimentParameters
private void addExperimentParameters(DbControl dc, Experiment currentExp, List<PluginParameter<?>> parameters)
-
getConfigureRawDataTypeParameters
private RequestInformation getConfigureRawDataTypeParameters()
-
getConfigureParserParameters
private RequestInformation getConfigureParserParameters(RawDataType rdt)
-
getRawBioAssayQuery
private ItemQuery<RawBioAssay> getRawBioAssayQuery(Experiment exp)
Get a query that returns all raw bioassays which it may be possible to import raw data to for the given experiment. The raw bioassays should have no raw data already and they must have at least one file attached.- Parameters:
exp
- The experiment
-
-