Package net.sf.basedb.plugins
Class AnnotationFlatFileImporter
- java.lang.Object
-
- net.sf.basedb.core.plugin.AbstractPlugin
-
- net.sf.basedb.plugins.AbstractFlatFileImporter
-
- net.sf.basedb.plugins.AnnotationFlatFileImporter
-
- All Implemented Interfaces:
AutoDetectingImporter
,InteractivePlugin
,Plugin
,SignalTarget
,ConfigureByExample
public class AnnotationFlatFileImporter extends AbstractFlatFileImporter implements InteractivePlugin, ConfigureByExample
Plug-in for importing annotations from simple text files. The plug-in supports all files that can be parsed with theFlatFileParser
class. This plug-in works without a configuration but can use a configuration to store regular expressions and other settings for the flat file parser. In both cases, the job configuration is a three-step process:- Setup regular expressions and other options for the flat file parser. If a configuration is used all values should already be filled in. In this step a file to import from must also be selected.
- Map file columns to annotation types. Annotation types that support multiple values may be mapped to more than one column.
- Setup error handling options and other settings for the plug-in (for example if existing annotations should be replaced or not).
- Version:
- 2.4
- Author:
- nicklas
- Last modified
- $Date: 2019-04-11 09:31:26 +0200 (tors, 11 apr. 2019) $
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
AnnotationFlatFileImporter.AnnotationValue
private static class
AnnotationFlatFileImporter.NewAnnotations
Internal cache for storing annotation values from the file for a single item.-
Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin
Plugin.MainType
-
-
Field Summary
-
Fields inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter
complexMappings, dataFooterRegexpParameter, dataHeaderRegexpParameter, dataSplitterRegexpParameter, errorSection, excelSheetParameter, fileParameter, fileType, headerRegexpParameter, ignoreRegexpParameter, invalidUseOfNullErrorParameter, mappingSection, maxDataColumnsParameter, minDataColumnsParameter, numDataColumnsType, optionalRegexpType, parserSection, requiredRegexpType, sectionRegexpParameter, trimQuotesParameter
-
Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin
annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc
-
-
Constructor Summary
Constructors Constructor Description AnnotationFlatFileImporter()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
begin(FlatFileParser ffp)
Setup error handling and pre-load some of the configuration options.protected void
beginData()
Setup column mapping.void
configure(GuiContext context, Request request, Response response)
Configure the plugin.private ItemQuery<?>
createQuery(Item itemType, ItemList list)
Create a query that return items of the specified type.void
doImport(InputStream in, ProgressReporter progress)
Import the data from the givenInputStream
.protected void
end(boolean success)
Now it's time to update the items in the cache with the new annotation values.private RequestInformation
getConfigureImportParameters(GuiContext context)
Step 3 of 3.private RequestInformation
getConfigureMappingParameters(GuiContext context)
Step 2 of 3.private RequestInformation
getConfigureParserParameters(GuiContext context, boolean forJob)
Set<GuiContext>
getGuiContexts()
This plug-in works in list context of allAnnotatable
items, except bioassay sets, bioassays and wells because they are not standalone items.protected IdMethod
getIdMethod(DbControl dc, Item item, String method)
private List<IdMethod>
getIdMethods(DbControl dc, Item item)
protected FlatFileParser
getInitializedFlatFileParser(ParameterValues parameters)
private ItemList
getItemListFromContext(DbControl dc, GuiContext context)
If the current context is the "Members" tab for an item list we load the item list and will later require that all items to be annotated are members of that list.private Pattern
getPattern(ParameterValues parameters, String name)
Collection<Permissions>
getPermissions()
Request read access to File:s, read access to annotation types and write access to all annotatable items.protected int
getProgress(FlatFileParser ffp)
Overriding default implementation since we want file parsing to only use 'progressParser'% and the remaining 'progressDatabase'% for storing to the database.RequestInformation
getRequestInformation(GuiContext context, String command)
This method will return theRequestInformation
for a given command, i.e.protected String
getSuccessMessage(int skippedLines)
Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface.protected void
handleData(FlatFileParser.Data data)
Read annotations from a single data line.private boolean
hasExternalId(Item item)
Check if the current item has an 'externalId' property.String
isInContext(GuiContext context, Object item)
Check if the plugin can be used on the specified object.boolean
requiresConfiguration()
Return TRUE, since the implementation requires it for finding the regular expressions used by theFlatFileParser
.protected void
setUpErrorHandling()
Don't use AbstractFlatFileImporter to handle errors.boolean
supportsConfigurations()
Returns TRUE, since that is how the plugins used to work before this method was introduced.-
Methods inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter
addErrorHandler, checkColumnMapping, checkColumnMapping, continueWithNextFileAfterError, finish, getCharset, getCharset, getDateFormatter, getDecimalSeparator, getErrorHandler, getErrorOption, getFileIterator, getInitializedFlatFileParser, getMainType, getMapper, getMapper, getNumberFormat, getNumBytes, getPattern, getSignalHandler, getTimestampFormatter, getTotalFileSize, handleHeader, handleSection, isImportable, isImportable, log, log, log, log, run, start, wrapInputStream
-
Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin
cloneParameterWithDefaultValue, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getJobOrConfigurationValue, getOverwriteAnnotationsParameters, init, log, log, storeValue, storeValue, storeValues, validateRequestParameters
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface net.sf.basedb.core.plugin.Plugin
done, getMainType, init, run
-
-
-
-
Field Detail
-
guiContexts
private static volatile Set<GuiContext> guiContexts
-
permissions
private static final Set<Permissions> permissions
-
CONFIGURE_MAPPING
private static final String CONFIGURE_MAPPING
- See Also:
- Constant Field Values
-
CONFIGURE_IMPORT
private static final String CONFIGURE_IMPORT
- See Also:
- Constant Field Values
-
optionalColumnMapping
private static final StringParameterType optionalColumnMapping
-
optionalBooleanType
private static final BooleanParameterType optionalBooleanType
-
nameColumnMapping
private static final PluginParameter<String> nameColumnMapping
-
externalIdColumnMapping
private static final PluginParameter<String> externalIdColumnMapping
-
internalIdColumnMapping
private static final PluginParameter<String> internalIdColumnMapping
-
idAnnotationColumnMapping
private static final PluginParameter<String> idAnnotationColumnMapping
Column mapping parameter for the annotation column used for item identification.- Since:
- 3.15
-
annotationTypeSection
private static final PluginParameter<String> annotationTypeSection
Section definition for grouping all mappings of columns to annotation types. Parameters in this section will be generated on the fly based on the column headers from the file to import from.
-
identificationSection
private static final PluginParameter<String> identificationSection
-
addReplaceSection
private static final PluginParameter<String> addReplaceSection
-
addToUnlimitedParameter
private static final PluginParameter<Boolean> addToUnlimitedParameter
-
replaceExistingParameter
private static final PluginParameter<Boolean> replaceExistingParameter
-
removeExistingParameter
private static final PluginParameter<Boolean> removeExistingParameter
-
defaultErrorParameter
private static final PluginParameter<String> defaultErrorParameter
-
itemNotFoundErrorParameter
private static final PluginParameter<String> itemNotFoundErrorParameter
-
multipleItemsFoundErrorParameter
private static final PluginParameter<String> multipleItemsFoundErrorParameter
-
stringTooLongErrorParameter
private static final PluginParameter<String> stringTooLongErrorParameter
-
numberOutOfRangeErrorParameter
protected static final PluginParameter<String> numberOutOfRangeErrorParameter
-
numberFormatErrorParameter
protected static final PluginParameter<String> numberFormatErrorParameter
-
dateFormatErrorParameter
protected static final PluginParameter<String> dateFormatErrorParameter
-
invalidEnumErrorParameter
protected static final PluginParameter<String> invalidEnumErrorParameter
-
tooManyValuesErrorParameter
protected static final PluginParameter<String> tooManyValuesErrorParameter
-
dc
private DbControl dc
-
batcher
private AnnotationBatcher batcher
-
unitCache
private UnitCache unitCache
-
ffp
private FlatFileParser ffp
-
numberFormat
private NumberFormat numberFormat
-
itemType
private Item itemType
-
idMethod
private IdMethod idMethod
-
idMapping
private String idMapping
-
itemMapper
private Mapper itemMapper
-
mappers
private Map<Mapper,AnnotationType> mappers
-
itemCache
private Map<String,Set<AnnotationFlatFileImporter.NewAnnotations>> itemCache
-
itemQuery
private ItemQuery<?> itemQuery
-
itemList
private ItemList itemList
-
addToUnlimited
private boolean addToUnlimited
-
replaceExisting
private boolean replaceExisting
-
removeExisting
private boolean removeExisting
-
errorHandler
private ClassMapErrorHandler errorHandler
-
cropStrings
private boolean cropStrings
-
ignoreNotFoundItems
private boolean ignoreNotFoundItems
-
failIfMultipleFoundItems
private boolean failIfMultipleFoundItems
-
failIfTooManyValues
private boolean failIfTooManyValues
-
cropTooManyValues
private boolean cropTooManyValues
-
numItems
private int numItems
-
numItemNotFound
private int numItemNotFound
-
numAnnotations
private int numAnnotations
-
numReplaced
private int numReplaced
-
numError
private int numError
-
numRemoved
private int numRemoved
-
progress
private ProgressReporter progress
-
progressParser
private int progressParser
-
progressDatabase
private int progressDatabase
-
configureParser
private RequestInformation configureParser
-
configureMapping
private RequestInformation configureMapping
-
configureImport
private RequestInformation configureImport
-
-
Method Detail
-
requiresConfiguration
public boolean requiresConfiguration()
Description copied from class:AbstractFlatFileImporter
Return TRUE, since the implementation requires it for finding the regular expressions used by theFlatFileParser
. If this method is overridden and returns FALSE, the subclass must also override theAbstractFlatFileImporter.getInitializedFlatFileParser()
method and provide a parser with all regular expressions and other options set.- Specified by:
requiresConfiguration
in interfacePlugin
- Overrides:
requiresConfiguration
in classAbstractFlatFileImporter
- Returns:
- TRUE or FALSE
-
supportsConfigurations
public boolean supportsConfigurations()
Description copied from class:AbstractPlugin
Returns TRUE, since that is how the plugins used to work before this method was introduced.- Specified by:
supportsConfigurations
in interfacePlugin
- Overrides:
supportsConfigurations
in classAbstractPlugin
- Returns:
- TRUE or FALSE
-
getPermissions
public Collection<Permissions> getPermissions()
Request read access to File:s, read access to annotation types and write access to all annotatable items.- Specified by:
getPermissions
in interfacePlugin
- Overrides:
getPermissions
in classAbstractPlugin
- Returns:
- A collection of permissions or null to not use permissions
-
getGuiContexts
public Set<GuiContext> getGuiContexts()
This plug-in works in list context of allAnnotatable
items, except bioassay sets, bioassays and wells because they are not standalone items. We useMetadata.getAnnotatableItems()
to create the contexts.- Specified by:
getGuiContexts
in interfaceInteractivePlugin
- Returns:
- A
Set
containingItem
:s, or null if the plugin is not concerned about items
-
isInContext
public String isInContext(GuiContext context, Object item)
Description copied from interface:InteractivePlugin
Check if the plugin can be used on the specified object. The item is either aBasicItem
orBasicData
object, or null if called from a list context. A client application should only call this method for contexts matching one of the contexts in the set returned by theInteractivePlugin.getGuiContexts()
method. ForGuiContext.Type.ITEM
contexts an item of the correct type should be passed as a parameter.The plug-in can response with a null value indicating success, a warning-level message or throw an exception to indicate an error-level message. Client application should normally only show error-level messages to users, unless the user has requested to see warning messages.
NOTE! Before BASE 2.4 all messages were treated as error-level messages.
- Specified by:
isInContext
in interfaceInteractivePlugin
- Parameters:
context
- The current context of the client application, it is one of the values found in set returned byInteractivePlugin.getGuiContexts()
item
- The currently active item, it's type should match theGuiContext.getItem()
type, or null if the context is a list context- Returns:
Null
if the plugin can use that item, or a warning-level message explaining why the plugin can't be used
-
getRequestInformation
public RequestInformation getRequestInformation(GuiContext context, String command) throws BaseException
Description copied from interface:InteractivePlugin
This method will return theRequestInformation
for a given command, i.e. the list of parameters and some nice help text.- Specified by:
getRequestInformation
in interfaceInteractivePlugin
- Parameters:
context
- The current context of the client application, it is one of the values found in set returned byInteractivePlugin.getGuiContexts()
command
- The command- Returns:
- The
RequestInformation
for the command - Throws:
BaseException
- if there is an error
-
configure
public void configure(GuiContext context, Request request, Response response)
Description copied from interface:InteractivePlugin
Configure the plugin. Hopefully the client is supplying values for the parameters specified byInteractivePlugin.getRequestInformation(GuiContext, String)
.- Specified by:
configure
in interfaceInteractivePlugin
- Parameters:
context
- The current context of the client application, it is one of the values found in set returned byInteractivePlugin.getGuiContexts()
request
- Request object with the command and parametersresponse
- Response object in for the plugin to response through
-
doImport
public void doImport(InputStream in, ProgressReporter progress) throws BaseException
Description copied from interface:AutoDetectingImporter
Import the data from the givenInputStream
.- Specified by:
doImport
in interfaceAutoDetectingImporter
- Overrides:
doImport
in classAbstractFlatFileImporter
- Parameters:
in
- The input stream to read fromprogress
- Progress reporter used by the caller to keep track of the progress. Null is allowed- Throws:
BaseException
- If something goes wrong
-
setUpErrorHandling
protected void setUpErrorHandling()
Don't use AbstractFlatFileImporter to handle errors.- Overrides:
setUpErrorHandling
in classAbstractFlatFileImporter
-
begin
protected void begin(FlatFileParser ffp)
Setup error handling and pre-load some of the configuration options.- Overrides:
begin
in classAbstractFlatFileImporter
- See Also:
AbstractFlatFileImporter.end(boolean)
-
beginData
protected void beginData()
Setup column mapping. Creates DbControl and query to find items.- Overrides:
beginData
in classAbstractFlatFileImporter
-
handleData
protected void handleData(FlatFileParser.Data data) throws BaseException
Read annotations from a single data line. Errors are handled internally. Errors thrown from this method should be reported back to client (ie. error handling in AbstractFlatFileParser) must be disabled. This method will load items and annotation values and put them in an internal cache. No items are annotated until the entire file has been parsed.- Specified by:
handleData
in classAbstractFlatFileImporter
- Throws:
BaseException
-
end
protected void end(boolean success)
Now it's time to update the items in the cache with the new annotation values.- Overrides:
end
in classAbstractFlatFileImporter
- Parameters:
success
- TRUE if the file was parsed successfully, FALSE otherwise- See Also:
AbstractFlatFileImporter.begin(FlatFileParser)
-
getSuccessMessage
protected String getSuccessMessage(int skippedLines)
Description copied from class:AbstractFlatFileImporter
Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface. An example message might by:178 reporters imported successfully
. The default implementation always return null. Note that this method is called once for every file returned byAbstractFlatFileImporter.getFileIterator()
.- Overrides:
getSuccessMessage
in classAbstractFlatFileImporter
- Parameters:
skippedLines
- The number of data lines that were skipped due to errors
-
getProgress
protected int getProgress(FlatFileParser ffp)
Overriding default implementation since we want file parsing to only use 'progressParser'% and the remaining 'progressDatabase'% for storing to the database.- Overrides:
getProgress
in classAbstractFlatFileImporter
- Parameters:
ffp
- The file parser that is used to parsed the file- Returns:
- A value between 0 and 100
-
getPattern
private Pattern getPattern(ParameterValues parameters, String name) throws BaseException
- Throws:
BaseException
-
getInitializedFlatFileParser
protected FlatFileParser getInitializedFlatFileParser(ParameterValues parameters) throws BaseException
- Throws:
BaseException
-
getItemListFromContext
private ItemList getItemListFromContext(DbControl dc, GuiContext context)
If the current context is the "Members" tab for an item list we load the item list and will later require that all items to be annotated are members of that list.
-
getConfigureParserParameters
private RequestInformation getConfigureParserParameters(GuiContext context, boolean forJob)
-
getConfigureMappingParameters
private RequestInformation getConfigureMappingParameters(GuiContext context)
Step 2 of 3. Parameters for configuring mapping between columns and annotation types
-
getConfigureImportParameters
private RequestInformation getConfigureImportParameters(GuiContext context)
Step 3 of 3. Other options and error handling parameters
-
hasExternalId
private boolean hasExternalId(Item item)
Check if the current item has an 'externalId' property. We use reflection to look for the 'getExternalId' method in the item's data class.
-
createQuery
private ItemQuery<?> createQuery(Item itemType, ItemList list)
Create a query that return items of the specified type. We use reflection to call the static method 'getQuery' on the item's item class, for exampleSample.getQuery()
.- Parameters:
itemType
- The type of items to search- Returns:
- A query that searches items of the specified type.
-
-