net.sf.basedb.plugins
Class AnnotationFlatFileImporter

java.lang.Object
  net.sf.basedb.core.plugin.AbstractPlugin
      net.sf.basedb.plugins.AbstractFlatFileImporter
          net.sf.basedb.plugins.AnnotationFlatFileImporter

All Implemented Interfaces:

AutoDetectingImporter, InteractivePlugin, Plugin, SignalTarget, ConfigureByExample

public class AnnotationFlatFileImporter
extends AbstractFlatFileImporter
implements InteractivePlugin, ConfigureByExample

Plug-in for importing annotations from simple text files. The plug-in supports all files that can be parsed with the FlatFileParser class. This plug-in works without a configuration but can use a configuration to store regular expressions and other settings for the flat file parser. In both cases, the job configuration is a three-step process:

1. Setup regular expressions and other options for the flat file parser. If a configuration is used all values should already be filled in. In this step a file to import from must also be selected.
2. Map file columns to annotation types. Annotation types that support multiple values may be mapped to more than one column.
3. Setup error handling options and other settings for the plug-in (for example if existing annotations should be replaced or not).

Version:

2.4

Author:

nicklas

Last modified

$Date: 2010-11-08 14:49:35 +0100 (Mon, 08 Nov 2010) $

Nested Class Summary

private static class	AnnotationFlatFileImporter.AnnotationValue
private static class	AnnotationFlatFileImporter.NewAnnotations Internal cache for storing annotation values from the file for a single item.

Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin

Plugin.MainType

Field Summary

private static About	about
private static PluginParameter<String>	addReplaceSection
private boolean	addToUnlimited
private static PluginParameter<Boolean>	addToUnlimitedParameter
private static PluginParameter<String>	annotationTypeSection Section definition for grouping all mappings of columns to annotation types.
private List<String>	columnHeaders
private static String	CONFIGURE_IMPORT
private static String	CONFIGURE_MAPPING
private RequestInformation	configureImport
private RequestInformation	configureMapping
private RequestInformation	configureParser
private boolean	cropStrings
private boolean	cropTooManyValues
protected static PluginParameter<String>	dateFormatErrorParameter
private DateFormatter	dateFormatter
private DbControl	dc
private static PluginParameter<String>	defaultErrorParameter
private ClassMapErrorHandler	errorHandler
private static PluginParameter<String>	externalIdColumnMapping
private boolean	failIfMultipleFoundItems
private boolean	failIfTooManyValues
private FlatFileParser	ffp
private static PluginParameter<String>	findItemsSection
private static Set<GuiContext>	guiContexts
private boolean	ignoreNotFoundItems
private static PluginParameter<String>	internalIdColumnMapping
protected static PluginParameter<String>	invalidEnumErrorParameter
private Map<String,Set<AnnotationFlatFileImporter.NewAnnotations>>	itemCache
private Mapper	itemMapper
private static PluginParameter<String>	itemNotFoundErrorParameter
private ItemQuery<?>	itemQuery
private Item	itemType
private Map<Mapper,AnnotationType>	mappers
private static PluginParameter<String>	multipleItemsFoundErrorParameter
private static PluginParameter<String>	nameColumnMapping
private int	numAnnotations
private NumberFormat	numberFormat
protected static PluginParameter<String>	numberFormatErrorParameter
protected static PluginParameter<String>	numberOutOfRangeErrorParameter
private int	numError
private int	numItemNotFound
private int	numItems
private int	numRemoved
private int	numReplaced
private static BooleanParameterType	optionalBooleanType
private static StringParameterType	optionalColumnMapping
private static Set<Permissions>	permissions
private boolean	removeExisting
private static PluginParameter<Boolean>	removeExistingParameter
private boolean	replaceExisting
private static PluginParameter<Boolean>	replaceExistingParameter
private String	searchMapping
private Type	searchParameterType
private String	searchProperty
private static PluginParameter<String>	stringTooLongErrorParameter
private DateFormatter	timestampFormatter
protected static PluginParameter<String>	tooManyValuesErrorParameter
private UnitCache	unitCache

Fields inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter

CHARSET, charsetType, complexMappings, dataFooterRegexpParameter, dataHeaderRegexpParameter, dataSplitterRegexpParameter, DECIMAL_SEPARATOR, decimalSeparatorType, errorSection, fileParameter, fileType, headerRegexpParameter, ignoreRegexpParameter, invalidUseOfNullErrorParameter, mappingSection, maxDataColumnsParameter, minDataColumnsParameter, numDataColumnsType, optionalRegexpType, parserSection, requiredRegexpType, sectionRegexpParameter, trimQuotesParameter

Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin

annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc

Constructor Summary

AnnotationFlatFileImporter()

Method Summary

protected void	begin(FlatFileParser ffp) Setup error handling and pre-load some of the configuration options.
protected void	beginData() Setup column mapping.
void	configure(GuiContext context, Request request, Response response) Configure the plugin.
private ItemQuery<?>	createQuery(Item itemType, String searchProperty, Set<Include> includes) Create a query that return items of the specified type.
protected void	end(boolean success) Now it's time to update the items in the cache with the new annotation values.
About	getAbout() Get information about the plugin, such as name, version, authors, etc.
private RequestInformation	getConfigureImportParameters(GuiContext context) Step 3 of 3.
private RequestInformation	getConfigureMappingParameters(GuiContext context) Step 2 of 3.
private RequestInformation	getConfigureParserParameters(GuiContext context, boolean forJob)
Set<GuiContext>	getGuiContexts() This plug-in works in list context of all Annotatable items, except bioassay sets, bioassays and wells because they are not standalone items.
protected FlatFileParser	getInitializedFlatFileParser() Create and initialise a flat file parser by setting all regular expressions and other options.
protected FlatFileParser	getInitializedFlatFileParser(ParameterValues parameters)
private Pattern	getPattern(ParameterValues parameters, String name)
Collection<Permissions>	getPermissions() Request read access to File:s, read access to annotation types and write access to all annotatable items.
RequestInformation	getRequestInformation(GuiContext context, String command) This method will return the RequestInformation for a given command, i.e.
protected String	getSuccessMessage(int skippedLines) Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface.
protected void	handleData(FlatFileParser.Data data) Read annotations from a single data line.
private boolean	hasExternalId(Item item) Check if the current item has an 'externalId' property.
String	isInContext(GuiContext context, Object item) Check if the plugin can be used on the specified object.
boolean	requiresConfiguration() Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser.
protected void	setUpErrorHandling() Don't use AbstractFlatFileImporter to handle errors.
boolean	supportsConfigurations() Returns TRUE, since that is how the plugins used to work before this method was introduced.

Methods inherited from class net.sf.basedb.plugins.AbstractFlatFileImporter

addErrorHandler, checkColumnMapping, checkColumnMapping, continueWithNextFileAfterError, doImport, finish, getCharset, getCharset, getCharsetParameter, getDecimalSeparator, getDecimalSeparatorParameter, getErrorHandler, getErrorOption, getFileIterator, getMainType, getMapper, getNumberFormat, getNumBytes, getPrimaryLocationFilter, getProgress, getSignalHandler, getTotalFileSize, handleHeader, handleSection, isImportable, isImportable, log, log, log, log, run, start, wrapInputStream

Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin

checkInterrupted, cloneParameterWithDefaultValue, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getJobOrConfigurationValue, getOverwriteAnnotationsParameters, init, log, log, storeValue, storeValue, storeValues, validateRequestParameters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface net.sf.basedb.core.plugin.Plugin

done, getMainType, init, run

Field Detail

about

private static final About about

guiContexts

private static volatile Set<GuiContext> guiContexts

permissions

private static final Set<Permissions> permissions

CONFIGURE_MAPPING

private static final String CONFIGURE_MAPPING

See Also:

Constant Field Values

CONFIGURE_IMPORT

private static final String CONFIGURE_IMPORT

See Also:

Constant Field Values

optionalColumnMapping

private static final StringParameterType optionalColumnMapping

optionalBooleanType

private static final BooleanParameterType optionalBooleanType

nameColumnMapping

private static final PluginParameter<String> nameColumnMapping

externalIdColumnMapping

private static final PluginParameter<String> externalIdColumnMapping

internalIdColumnMapping

private static final PluginParameter<String> internalIdColumnMapping

annotationTypeSection

private static final PluginParameter<String> annotationTypeSection

Section definition for grouping all mappings of columns to annotation types. Parameters in this section will be generated on the fly based on the column headers from the file to import from.

findItemsSection

private static final PluginParameter<String> findItemsSection

addReplaceSection

private static final PluginParameter<String> addReplaceSection

addToUnlimitedParameter

private static final PluginParameter<Boolean> addToUnlimitedParameter

replaceExistingParameter

private static final PluginParameter<Boolean> replaceExistingParameter

removeExistingParameter

private static final PluginParameter<Boolean> removeExistingParameter

defaultErrorParameter

private static final PluginParameter<String> defaultErrorParameter

itemNotFoundErrorParameter

private static final PluginParameter<String> itemNotFoundErrorParameter

multipleItemsFoundErrorParameter

private static final PluginParameter<String> multipleItemsFoundErrorParameter

stringTooLongErrorParameter

private static final PluginParameter<String> stringTooLongErrorParameter

numberOutOfRangeErrorParameter

protected static final PluginParameter<String> numberOutOfRangeErrorParameter

numberFormatErrorParameter

protected static final PluginParameter<String> numberFormatErrorParameter

dateFormatErrorParameter

protected static final PluginParameter<String> dateFormatErrorParameter

invalidEnumErrorParameter

protected static final PluginParameter<String> invalidEnumErrorParameter

tooManyValuesErrorParameter

protected static final PluginParameter<String> tooManyValuesErrorParameter

columnHeaders

private List<String> columnHeaders

dc

private DbControl dc

unitCache

private UnitCache unitCache

ffp

private FlatFileParser ffp

numberFormat

private NumberFormat numberFormat

dateFormatter

private DateFormatter dateFormatter

timestampFormatter

private DateFormatter timestampFormatter

itemType

private Item itemType

searchProperty

private String searchProperty

searchMapping

private String searchMapping

searchParameterType

private Type searchParameterType

itemMapper

private Mapper itemMapper

mappers

private Map<Mapper,AnnotationType> mappers

itemCache

private Map<String,Set<AnnotationFlatFileImporter.NewAnnotations>> itemCache

itemQuery

private ItemQuery<?> itemQuery

addToUnlimited

private boolean addToUnlimited

replaceExisting

private boolean replaceExisting

removeExisting

private boolean removeExisting

errorHandler

private ClassMapErrorHandler errorHandler

cropStrings

private boolean cropStrings

ignoreNotFoundItems

private boolean ignoreNotFoundItems

failIfMultipleFoundItems

private boolean failIfMultipleFoundItems

failIfTooManyValues

private boolean failIfTooManyValues

cropTooManyValues

private boolean cropTooManyValues

numItems

private int numItems

numItemNotFound

private int numItemNotFound

numAnnotations

private int numAnnotations

numReplaced

private int numReplaced

numError

private int numError

numRemoved

private int numRemoved

configureParser

private RequestInformation configureParser

configureMapping

private RequestInformation configureMapping

configureImport

private RequestInformation configureImport

Constructor Detail

AnnotationFlatFileImporter

public AnnotationFlatFileImporter()

Method Detail

getAbout

public About getAbout()

Description copied from interface: Plugin

Get information about the plugin, such as name, version, authors, etc.

Specified by:

getAbout in interface Plugin

Returns:

An About object

requiresConfiguration

public boolean requiresConfiguration()

Description copied from class: AbstractFlatFileImporter

Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser. If this method is overridden and returns FALSE, the subclass must also override the AbstractFlatFileImporter.getInitializedFlatFileParser() method and provide a parser with all regular expressions and other options set.

Specified by:

requiresConfiguration in interface Plugin

Overrides:

requiresConfiguration in class AbstractFlatFileImporter

Returns:

TRUE or FALSE

supportsConfigurations

public boolean supportsConfigurations()

Description copied from class: AbstractPlugin

Returns TRUE, since that is how the plugins used to work before this method was introduced.

Specified by:

supportsConfigurations in interface Plugin

Overrides:

supportsConfigurations in class AbstractPlugin

Returns:

TRUE or FALSE

getPermissions

public Collection<Permissions> getPermissions()

Request read access to File:s, read access to annotation types and write access to all annotatable items.

Specified by:

getPermissions in interface Plugin

Overrides:

getPermissions in class AbstractPlugin

Returns:

A collection of permissions or null to not use permissions

getGuiContexts

public Set<GuiContext> getGuiContexts()

This plug-in works in list context of all Annotatable items, except bioassay sets, bioassays and wells because they are not standalone items. We use Metadata.getAnnotatableItems() to create the contexts.

Specified by:

getGuiContexts in interface InteractivePlugin

Returns:

A Set containing Item:s, or null if the plugin is not concerned about items

isInContext

public String isInContext(GuiContext context,
                          Object item)

Description copied from interface: InteractivePlugin

Check if the plugin can be used on the specified object. The item is either a BasicItem or BasicData object, or null if called from a list context. A client application should only call this method for contexts matching one of the contexts in the set returned by the InteractivePlugin.getGuiContexts() method. For GuiContext.Type.ITEM contexts an item of the correct type should be passed as a parameter.

The plug-in can response with a null value indicating success, a warning-level message or throw an exception to indicate an error-level message. Client application should normally only show error-level messages to users, unless the user has requested to see warning messages.

NOTE! Before BASE 2.4 all messages were treated as error-level messages.

Specified by:

isInContext in interface InteractivePlugin

Parameters:

context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()

item - The currently active item, it's type should match the GuiContext.getItem() type, or null if the context is a list context

Returns:

Null if the plugin can use that item, or a warning-level message explaining why the plugin can't be used

getRequestInformation

public RequestInformation getRequestInformation(GuiContext context,
                                                String command)
                                         throws BaseException

Description copied from interface: InteractivePlugin

This method will return the RequestInformation for a given command, i.e. the list of parameters and some nice help text.

Specified by:

getRequestInformation in interface InteractivePlugin

Parameters:

context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()

command - The command

Returns:

The RequestInformation for the command

Throws:

BaseException - if there is an error

configure

public void configure(GuiContext context,
                      Request request,
                      Response response)

Description copied from interface: InteractivePlugin

Configure the plugin. Hopefully the client is supplying values for the parameters specified by InteractivePlugin.getRequestInformation(GuiContext, String).

Specified by:

configure in interface InteractivePlugin

Parameters:

context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()

request - Request object with the command and parameters

response - Response object in for the plugin to response through

getInitializedFlatFileParser

protected FlatFileParser getInitializedFlatFileParser()
                                               throws BaseException

Description copied from class: AbstractFlatFileImporter

Create and initialise a flat file parser by setting all regular expressions and other options. This implementation gets all parameters from the AbstractPlugin.job or AbstractPlugin.configuration settings. If a subclass doesn't store the parameters there it must override this method and initialise the parser. Note that this method is called once for each file returned by the AbstractFlatFileImporter.getFileIterator() and that a new parser is needed for each file.

Overrides:

getInitializedFlatFileParser in class AbstractFlatFileImporter

Returns:

An intialised flat file parser

Throws:

BaseException

setUpErrorHandling

protected void setUpErrorHandling()

Don't use AbstractFlatFileImporter to handle errors.

Overrides:

setUpErrorHandling in class AbstractFlatFileImporter

begin

protected void begin(FlatFileParser ffp)

Setup error handling and pre-load some of the configuration options.

Overrides:

begin in class AbstractFlatFileImporter

See Also:

AbstractFlatFileImporter.end(boolean)

beginData

protected void beginData()

Setup column mapping. Creates DbControl and query to find items.

Overrides:

beginData in class AbstractFlatFileImporter

handleData

protected void handleData(FlatFileParser.Data data)
                   throws BaseException

Read annotations from a single data line. Errors are handled internally. Errors thrown from this method should be reported back to client (ie. error handling in AbstractFlatFileParser) must be disabled. This method will load items and annotation values and put them in an internal cache. No items are annotated until the entire file has been parsed.

Specified by:

handleData in class AbstractFlatFileImporter

Throws:

BaseException

end

protected void end(boolean success)

Now it's time to update the items in the cache with the new annotation values.

Overrides:

end in class AbstractFlatFileImporter

Parameters:

success - TRUE if the file was parsed successfully, FALSE otherwise

See Also:

AbstractFlatFileImporter.begin(FlatFileParser)

getSuccessMessage

protected String getSuccessMessage(int skippedLines)

Description copied from class: AbstractFlatFileImporter

Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface. An example message might by: 178 reporters imported successfully. The default implementation always return null. Note that this method is called once for every file returned by AbstractFlatFileImporter.getFileIterator().

Overrides:

getSuccessMessage in class AbstractFlatFileImporter

Parameters:

skippedLines - The number of data lines that were skipped due to errors

getPattern

private Pattern getPattern(ParameterValues parameters,
                           String name)
                    throws BaseException

Throws:

BaseException

getInitializedFlatFileParser

protected FlatFileParser getInitializedFlatFileParser(ParameterValues parameters)
                                               throws BaseException

Throws:

BaseException

getConfigureParserParameters

private RequestInformation getConfigureParserParameters(GuiContext context,
                                                        boolean forJob)

getConfigureMappingParameters

private RequestInformation getConfigureMappingParameters(GuiContext context)

Step 2 of 3. Parameters for configuring mapping between columns and annotation types

getConfigureImportParameters

private RequestInformation getConfigureImportParameters(GuiContext context)

Step 3 of 3. Other options and error handling parameters

hasExternalId

private boolean hasExternalId(Item item)

Check if the current item has an 'externalId' property. We use reflection to look for the 'getExternalId' method in the item's data class.

createQuery

private ItemQuery<?> createQuery(Item itemType,
                                 String searchProperty,
                                 Set<Include> includes)

Create a query that return items of the specified type. We use reflection to call the static method 'getQuery' on the item's item class, for example Sample.getQuery(). We add a restriction to either the 'name' or the 'externalId' property but the parameter is always called 'name' to make it easier to implement the code that is using the query.

Parameters:

itemType - The type of items to search

searchProperty - If we should search by name, external ID or internal ID

includes - Include options passed to AbstractEntityQuery.include(java.util.Collection)

Returns:

A query that searches items of the specified type.