Package net.sf.basedb.plugins

Class AnnotationFlatFileImporter

java.lang.Object
net.sf.basedb.core.plugin.AbstractPlugin
net.sf.basedb.plugins.AbstractFlatFileImporter
net.sf.basedb.plugins.AnnotationFlatFileImporter
All Implemented Interfaces:
AutoDetectingImporter, InteractivePlugin, Plugin, SignalTarget, ConfigureByExample
public class AnnotationFlatFileImporter extends AbstractFlatFileImporter implements InteractivePlugin, ConfigureByExample
Plug-in for importing annotations from simple text files. The plug-in supports all files that can be parsed with the FlatFileParser class. This plug-in works without a configuration but can use a configuration to store regular expressions and other settings for the flat file parser. In both cases, the job configuration is a three-step process:
  1. Setup regular expressions and other options for the flat file parser. If a configuration is used all values should already be filled in. In this step a file to import from must also be selected.
  2. Map file columns to annotation types. Annotation types that support multiple values may be mapped to more than one column.
  3. Setup error handling options and other settings for the plug-in (for example if existing annotations should be replaced or not).
Version:
2.4
Author:
nicklas
Last modified
$Date: 2024-01-08 15:38:53 +0100 (Mon, 08 Jan 2024) $

  • Field Details

    • guiContexts

      private static volatile Set<GuiContext> guiContexts

    • permissions

      private static final Set<Permissions> permissions

    • CONFIGURE_MAPPING

      private static final String CONFIGURE_MAPPING
      See Also:

    • CONFIGURE_IMPORT

      private static final String CONFIGURE_IMPORT
      See Also:

    • optionalColumnMapping

      private static final StringParameterType optionalColumnMapping

    • optionalBooleanType

      private static final BooleanParameterType optionalBooleanType

    • nameColumnMapping

      private static final PluginParameter<String> nameColumnMapping

    • pathColumnMapping

      private static final PluginParameter<String> pathColumnMapping

    • externalIdColumnMapping

      private static final PluginParameter<String> externalIdColumnMapping

    • internalIdColumnMapping

      private static final PluginParameter<String> internalIdColumnMapping

    • idAnnotationColumnMapping

      private static final PluginParameter<String> idAnnotationColumnMapping
      Column mapping parameter for the annotation column used for item identification.
      Since:
      3.15

    • annotationTypeSection

      private static final PluginParameter<String> annotationTypeSection
      Section definition for grouping all mappings of columns to annotation types. Parameters in this section will be generated on the fly based on the column headers from the file to import from.

    • identificationSection

      private static final PluginParameter<String> identificationSection

    • addReplaceSection

      private static final PluginParameter<String> addReplaceSection

    • addToUnlimitedParameter

      private static final PluginParameter<Boolean> addToUnlimitedParameter

    • replaceExistingParameter

      private static final PluginParameter<Boolean> replaceExistingParameter

    • removeExistingParameter

      private static final PluginParameter<Boolean> removeExistingParameter

    • defaultErrorParameter

      private static final PluginParameter<String> defaultErrorParameter

    • itemNotFoundErrorParameter

      private static final PluginParameter<String> itemNotFoundErrorParameter

    • multipleItemsFoundErrorParameter

      private static final PluginParameter<String> multipleItemsFoundErrorParameter

    • stringTooLongErrorParameter

      private static final PluginParameter<String> stringTooLongErrorParameter

    • numberOutOfRangeErrorParameter

      protected static final PluginParameter<String> numberOutOfRangeErrorParameter

    • numberFormatErrorParameter

      protected static final PluginParameter<String> numberFormatErrorParameter

    • dateFormatErrorParameter

      protected static final PluginParameter<String> dateFormatErrorParameter

    • invalidEnumErrorParameter

      protected static final PluginParameter<String> invalidEnumErrorParameter

    • tooManyValuesErrorParameter

      protected static final PluginParameter<String> tooManyValuesErrorParameter

    • columnHeaders

      private List<String> columnHeaders

    • dc

      private DbControl dc

    • batcher

      private AnnotationBatcher batcher

    • unitCache

      private UnitCache unitCache

    • ffp

      private FlatFileParser ffp

    • numberFormat

      private NumberFormat numberFormat

    • dateFormatter

      private Formatter<Date> dateFormatter

    • timestampFormatter

      private Formatter<Date> timestampFormatter

    • itemType

      private Item itemType

    • idMethod

      private IdMethod idMethod

    • idMapping

      private String idMapping

    • itemMapper

      private Mapper itemMapper

    • mappers

      private Map<Mapper,AnnotationType> mappers

    • itemCache

      private Map<String,Set<AnnotationFlatFileImporter.NewAnnotations>> itemCache

    • itemQuery

      private ItemQuery<?> itemQuery

    • itemList

      private ItemList itemList

    • addToUnlimited

      private boolean addToUnlimited

    • replaceExisting

      private boolean replaceExisting

    • removeExisting

      private boolean removeExisting

    • dryRun

      private boolean dryRun

    • errorHandler

      private ClassMapErrorHandler errorHandler

    • cropStrings

      private boolean cropStrings

    • ignoreNotFoundItems

      private boolean ignoreNotFoundItems

    • failIfMultipleFoundItems

      private boolean failIfMultipleFoundItems

    • failIfTooManyValues

      private boolean failIfTooManyValues

    • cropTooManyValues

      private boolean cropTooManyValues

    • numItems

      private int numItems

    • numItemNotFound

      private int numItemNotFound

    • numAnnotations

      private int numAnnotations

    • numReplaced

      private int numReplaced

    • numError

      private int numError

    • numRemoved

      private int numRemoved

    • progress

      private ProgressReporter progress

    • progressParser

      private int progressParser

    • progressDatabase

      private int progressDatabase

    • configureParser

      private RequestInformation configureParser

    • configureMapping

      private RequestInformation configureMapping

    • configureImport

      private RequestInformation configureImport

  • Constructor Details

    • AnnotationFlatFileImporter

      public AnnotationFlatFileImporter()

  • Method Details

    • requiresConfiguration

      public boolean requiresConfiguration()
      Description copied from class: AbstractFlatFileImporter
      Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser. If this method is overridden and returns FALSE, the subclass must also override the AbstractFlatFileImporter.getInitializedFlatFileParser() method and provide a parser with all regular expressions and other options set.
      Specified by:
      requiresConfiguration in interface Plugin
      Overrides:
      requiresConfiguration in class AbstractFlatFileImporter
      Returns:
      TRUE or FALSE

    • supportsConfigurations

      public boolean supportsConfigurations()
      Description copied from class: AbstractPlugin
      Returns TRUE, since that is how the plugins used to work before this method was introduced.
      Specified by:
      supportsConfigurations in interface Plugin
      Overrides:
      supportsConfigurations in class AbstractPlugin
      Returns:
      TRUE or FALSE

    • getPermissions

      public Collection<Permissions> getPermissions()
      Request read access to File:s, read access to annotation types and write access to all annotatable items.
      Specified by:
      getPermissions in interface Plugin
      Overrides:
      getPermissions in class AbstractPlugin
      Returns:
      A collection of permissions or null to not use permissions

    • getGuiContexts

      public Set<GuiContext> getGuiContexts()
      This plug-in works in list context of all Annotatable items, except bioassay sets, bioassays and wells because they are not standalone items. We use Metadata.getAnnotatableItems() to create the contexts.
      Specified by:
      getGuiContexts in interface InteractivePlugin
      Returns:
      A Set containing Item:s, or null if the plugin is not concerned about items

    • isInContext

      public String isInContext(GuiContext context, Object item)
      Description copied from interface: InteractivePlugin
      Check if the plugin can be used on the specified object. The item is either a BasicItem or BasicData object, or null if called from a list context. A client application should only call this method for contexts matching one of the contexts in the set returned by the InteractivePlugin.getGuiContexts() method. For GuiContext.Type.ITEM contexts an item of the correct type should be passed as a parameter.

      The plug-in can response with a null value indicating success, a warning-level message or throw an exception to indicate an error-level message. Client application should normally only show error-level messages to users, unless the user has requested to see warning messages.

      NOTE! Before BASE 2.4 all messages were treated as error-level messages.

      Specified by:
      isInContext in interface InteractivePlugin
      Parameters:
      context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()
      item - The currently active item, it's type should match the GuiContext.getItem() type, or null if the context is a list context
      Returns:
      Null if the plugin can use that item, or a warning-level message explaining why the plugin can't be used

    • getRequestInformation

      public RequestInformation getRequestInformation(GuiContext context, String command) throws BaseException
      Description copied from interface: InteractivePlugin
      This method will return the RequestInformation for a given command, i.e. the list of parameters and some nice help text.
      Specified by:
      getRequestInformation in interface InteractivePlugin
      Parameters:
      context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()
      command - The command
      Returns:
      The RequestInformation for the command
      Throws:
      BaseException - if there is an error

    • configure

      public void configure(GuiContext context, Request request, Response response)
      Description copied from interface: InteractivePlugin
      Configure the plugin. Hopefully the client is supplying values for the parameters specified by InteractivePlugin.getRequestInformation(GuiContext, String).
      Specified by:
      configure in interface InteractivePlugin
      Parameters:
      context - The current context of the client application, it is one of the values found in set returned by InteractivePlugin.getGuiContexts()
      request - Request object with the command and parameters
      response - Response object in for the plugin to response through

    • doImport

      public void doImport(InputStream in, ProgressReporter progress) throws BaseException
      Description copied from interface: AutoDetectingImporter
      Import the data from the given InputStream.
      Specified by:
      doImport in interface AutoDetectingImporter
      Overrides:
      doImport in class AbstractFlatFileImporter
      Parameters:
      in - The input stream to read from
      progress - Progress reporter used by the caller to keep track of the progress. Null is allowed
      Throws:
      BaseException - If something goes wrong

    • setUpErrorHandling

      protected void setUpErrorHandling()
      Don't use AbstractFlatFileImporter to handle errors.
      Overrides:
      setUpErrorHandling in class AbstractFlatFileImporter

    • begin

      protected void begin(FlatFileParser ffp)
      Setup error handling and pre-load some of the configuration options.
      Overrides:
      begin in class AbstractFlatFileImporter
      See Also:

    • beginData

      protected void beginData()
      Setup column mapping. Creates DbControl and query to find items.
      Overrides:
      beginData in class AbstractFlatFileImporter

    • handleData

      protected void handleData(FlatFileParser.Data data) throws BaseException
      Read annotations from a single data line. Errors are handled internally. Errors thrown from this method should be reported back to client (ie. error handling in AbstractFlatFileParser) must be disabled. This method will load items and annotation values and put them in an internal cache. No items are annotated until the entire file has been parsed.
      Specified by:
      handleData in class AbstractFlatFileImporter
      Throws:
      BaseException

    • end

      protected void end(boolean success)
      Now it's time to update the items in the cache with the new annotation values.
      Overrides:
      end in class AbstractFlatFileImporter
      Parameters:
      success - TRUE if the file was parsed successfully, FALSE otherwise
      See Also:

    • getSuccessMessage

      protected String getSuccessMessage(int skippedLines)
      Description copied from class: AbstractFlatFileImporter
      Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface. An example message might by: 178 reporters imported successfully. The default implementation always return null. Note that this method is called once for every file returned by AbstractFlatFileImporter.getFileIterator().
      Overrides:
      getSuccessMessage in class AbstractFlatFileImporter
      Parameters:
      skippedLines - The number of data lines that were skipped due to errors

    • getProgress

      protected int getProgress(FlatFileParser ffp)
      Overriding default implementation since we want file parsing to only use 'progressParser'% and the remaining 'progressDatabase'% for storing to the database.
      Overrides:
      getProgress in class AbstractFlatFileImporter
      Parameters:
      ffp - The file parser that is used to parsed the file
      Returns:
      A value between 0 and 100

    • getItemListFromContext

      private ItemList getItemListFromContext(DbControl dc, GuiContext context)
      If the current context is the "Members" tab for an item list we load the item list and will later require that all items to be annotated are members of that list.

    • getConfigureParserParameters

      private RequestInformation getConfigureParserParameters(GuiContext context, boolean forJob)

    • getConfigureMappingParameters

      private RequestInformation getConfigureMappingParameters(GuiContext context)
      Step 2 of 3. Parameters for configuring mapping between columns and annotation types

    • getConfigureImportParameters

      private RequestInformation getConfigureImportParameters(GuiContext context)
      Step 3 of 3. Other options and error handling parameters

    • hasExternalId

      private boolean hasExternalId(Item item)
      Check if the current item has an 'externalId' property. We use reflection to look for the 'getExternalId' method in the item's data class.

    • getIdMethods

      private List<IdMethod> getIdMethods(DbControl dc, Item item)

    • getIdMethod

      protected IdMethod getIdMethod(DbControl dc, Item item, String method)

    • createQuery

      private ItemQuery<?> createQuery(Item itemType, ItemList list)
      Create a query that return items of the specified type. We use reflection to call the static method 'getQuery' on the item's item class, for example Sample.getQuery().
      Parameters:
      itemType - The type of items to search
      Returns:
      A query that searches items of the specified type.