2.17.2: 2011-06-17

net.sf.basedb.plugins
Class AbstractFlatFileImporter

java.lang.Object
  extended by net.sf.basedb.core.plugin.AbstractPlugin
      extended by net.sf.basedb.plugins.AbstractFlatFileImporter
All Implemented Interfaces:
AutoDetectingImporter, Plugin, SignalTarget
Direct Known Subclasses:
AbstractItemImporter, AnnotationFlatFileImporter, IlluminaRawDataImporter, PlateFlatFileImporter, PlateMappingImporter, PrintMapFlatFileImporter, RawDataFlatFileImporter, ReporterFlatFileImporter, ReporterMapFlatFileImporter

public abstract class AbstractFlatFileImporter
extends AbstractPlugin
implements AutoDetectingImporter, SignalTarget

An abstract base class for all importers that imports data from one or more flat files. The implementation in this class uses a FlatFileParser for parsing the files and uses a callback method handleData(FlatFileParser.Data) that lets the subclass do whatever it needs to insert a single line of data into the database.

The subclass must also generate the RequestInformation object for both plugin and job configuration. However, this implementation expects to find the regular expression needed for the FlatFileParser to be found in the plugin configuration and the File to import to be found in the job configuration.

All of the needed PluginParameter objects needed to ask for these parameters are declared as protected variables in this class.

headerRegexpParameter);
parameters.add(dataHeaderRegexpParameter);
parameters.add(dataSplitterRegexpParameter);
parameters.add(ignoreRegexpParameter);
parameters.add(dataFooterRegexpParameter);

// Column mappings
parameters.addAll(allColumnMappings);

// Reporter type
parameters.add(reporterTypeParameter);

configurePlugin = new RequestInformation
(
   Request.COMMAND_CONFIGURE_PLUGIN,
   "Parser settings",
   "Please enter all settings needed by the flat file parser",
   parameters
);
This class implements the invoke method but only for the Request.COMMAND_EXECUTE command. The subclass must override the invoke method and provide implementations for the other commands (Request.COMMAND_CONFIGURE_PLUGIN and Request.COMMAND_CONFIGURE_JOB). The normal implementation of would be to store a request parameter.
// ReporterFlatFileImporter.java
if (command.equals(Request.COMMAND_CONFIGURE_JOB))
{
   List<Throwable> errors = validateRequestParameters(jobParameters, request);
   if (errors != null)
   {
      response.setError(errors.size()+" invalid parameter(s) were found in the request", errors);
      return;
   }
   storeValue(job, request, fileParameter);
   response.setDone(null);
}

For multi-file support (added in BASE 2.9) the subclass needs to override the getFileIterator() and getTotalFileSize() methods.

Version:
2.0
Author:
Enell, Nicklas
Last modified
$Date: 2010-09-10 13:09:05 +0200 (Fri, 10 Sep 2010) $

Nested Class Summary
 
Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin
Plugin.MainType
 
Field Summary
protected static String CHARSET
          Deprecated. Use Parameters.CHARSET_PARAMETER
protected static StringParameterType charsetType
          Deprecated. Use Parameters.charsetParameter(String, String, String)
protected static PluginParameter<String> complexMappings
          Parameter that asks if complex column mappings should be enabled or not.
private  File currentFile
           
protected static PluginParameter<String> dataFooterRegexpParameter
          Parameter definition that asks for the regular expression that matches the data footer.
protected static PluginParameter<String> dataHeaderRegexpParameter
          Parameter definition that asks for the regular expression that matches the data header.
protected static PluginParameter<String> dataSplitterRegexpParameter
          Parameter definition that asks for the regular expression that splits a data line into individual columns.
protected static String DECIMAL_SEPARATOR
          Deprecated. Use Parameters.DECIMAL_SEPARATOR_PARAMETER
protected static StringParameterType decimalSeparatorType
          Deprecated. Use Parameters.decimalSeparatorParameter(String, String, String)
protected static PluginParameter<String> defaultErrorParameter
           
private  ClassMapErrorHandler errorHandler
           
protected static PluginParameter<String> errorSection
          Section definition for grouping error handling options.
protected static PluginParameter<File> fileParameter
          Parameter definition that asks for the file that should be imported.
protected static ParameterType<File> fileType
           
private  long finishedFileSize
           
protected static PluginParameter<String> headerRegexpParameter
          Parameter definition that asks for the regular expression that splits a header line in key/value pair.
protected static PluginParameter<String> ignoreRegexpParameter
          Parameter definition that asks for the regular expression that matches a line that should be ignored.
protected static PluginParameter<String> invalidUseOfNullErrorParameter
           
protected static PluginParameter<String> mappingSection
          Section definition for grouping all column mapping expressions
protected static PluginParameter<Integer> maxDataColumnsParameter
          Parameter definition that asks for the maximum number of columns produced by the data splitter regexp for a line to be a data line.
protected static PluginParameter<Integer> minDataColumnsParameter
          Parameter definition that asks for the minimum number of columns produced by the data splitter regexp for a line to be a data line.
protected static PluginParameter<String> numberFormatErrorParameter
           
protected static PluginParameter<String> numberOutOfRangeErrorParameter
           
protected static ParameterType<Integer> numDataColumnsType
           
protected static ParameterType<String> optionalRegexpType
           
protected static PluginParameter<String> parserSection
          Section definition for grouping all file parser settings (ie. regular expressions)
protected static ParameterType<String> requiredRegexpType
           
protected static PluginParameter<String> sectionRegexpParameter
          Parameter definition that asks for the regular expression that matches a section line and extracts the section name.
private  ThreadSignalHandler signalHandler
           
private  int skippedLines
           
protected static PluginParameter<String> stringTooLongErrorParameter
           
private  long totalFileSize
           
protected static PluginParameter<Boolean> trimQuotesParameter
          Parameter definition that asks if quotes around quoted values should be removed or not.
 
Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin
annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc
 
Constructor Summary
AbstractFlatFileImporter()
           
 
Method Summary
protected  void addErrorHandler(Class<? extends Throwable> t, ErrorHandler handler)
          Add an error handler for the specified class of error.
protected  void begin(FlatFileParser ffp)
          Called just before parsing of the file begins.
protected  void beginData()
          Called by the parser after the headers have been parsed but before the first data line is parsed.
protected  String checkColumnMapping(FlatFileParser ffp, String mapExpression, boolean allowComplex, String name)
          Checks the syntax column mapping and verifies that the given file parser has found the columns that are used in the file.
protected  String checkColumnMapping(String mapExpression, boolean allowComplex, String name)
          Check if a column mapping is a valid mapping expression and optionally if it is complex mapping.
protected  boolean continueWithNextFileAfterError(Throwable t)
          If the importer should continue with the next file after an error.
 void doImport(InputStream in, ProgressReporter progress)
          Import the data from the given InputStream.
protected  void end(boolean success)
          Called just after the last line of the file has been parsed, or immediately after an error has ocurred.
protected  String finish(Throwable t)
          Called once when all files has been imported or when exiting due to an error.
protected  String getCharset()
          Get the character set the file uses.
protected  String getCharset(Request request)
          Get the character set the file uses.
protected  PluginParameter<String> getCharsetParameter(String label, String description, String defaultValue)
          Deprecated. USe Parameters.charsetParameter(String, String, String)
protected  String getDecimalSeparator()
          Get the decimal separator used by numbers in the file.
protected  PluginParameter<String> getDecimalSeparatorParameter(String label, String description, String defaultValue)
          Deprecated. Use Parameters.decimalSeparatorParameter(String, String, String)
protected  ErrorHandler getErrorHandler(String method, ErrorHandler defaultErrorHandler)
           
protected  String getErrorOption(String parameterName)
          Get the value for an error handling parameter.
protected  Iterator<File> getFileIterator()
          Get an iterator that returns the files to be imported.
protected  FlatFileParser getInitializedFlatFileParser()
          Create and initialise a flat file parser by setting all regular expressions and other options.
 Plugin.MainType getMainType()
          Return Plugin.MainType.IMPORT.
protected  Mapper getMapper(FlatFileParser ffp, String mapExpression, Integer maxStringLength, Mapper defaultMapper)
          Get a mapper for the specified flat file parser.
protected  NumberFormat getNumberFormat()
          Get a number formatter that is able to parse numbers with the specified decimal separator.
protected  long getNumBytes(FlatFileParser ffp)
          Get the number of bytes read from the file.
private  Pattern getPattern(String name)
           
protected  PropertyFilter getPrimaryLocationFilter()
          Deprecated. In 2.16. Since files can now be references to external URL:s this filter no longer makes sense. Plug-ins should not use the location to filter files.
protected  int getProgress(FlatFileParser ffp)
          Get the progress of import as a percentage value.
 SignalHandler getSignalHandler()
          Create a new ThreadSignalHandler that supports the Signal.ABORT signal.
protected  String getSuccessMessage(int skippedLines)
          Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface.
protected  long getTotalFileSize()
          Get the total file size of all files that are going to be imported.
protected abstract  void handleData(FlatFileParser.Data data)
          Called by the parser for every line in the file that is a data line.
protected  void handleHeader(FlatFileParser.Line line)
          Called by the parser for every line in the file that is a header line.
protected  void handleSection(FlatFileParser.Line line)
          Called by the parser for every line in the file that is a section line.
protected  boolean isImportable(FlatFileParser ffp)
          This method is called by the isImportable(InputStream) method after FlatFileParser.nextSection() and FlatFileParser.parseHeaders() has been called and if data has been found.
 boolean isImportable(InputStream in)
          Check if the given InputStream can be imported by this plugin.
protected  void log(String message, FlatFileParser.Data data)
          Log a message about a data line to the log file created by AbstractPlugin.createLogFile(String).
protected  void log(String message, FlatFileParser.Data data, Throwable t)
          Log an error message about a data line to the log file created by AbstractPlugin.createLogFile(String).
protected  void log(String message, FlatFileParser.Line line)
          Log a message about a header line to the log file created by AbstractPlugin.createLogFile(String).
protected  void log(String message, FlatFileParser.Line line, Throwable t)
          Log an error message about a header line to the log file created by AbstractPlugin.createLogFile(String).
 boolean requiresConfiguration()
          Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser.
 void run(Request request, Response response, ProgressReporter progress)
          Implements the Request.COMMAND_EXECUTE command.
protected  void setUpErrorHandling()
          Initialise the error handling system.
protected  void start()
          Called once before starting the import.
protected  InputStream wrapInputStream(InputStream in)
          This method is called before the parser starts reading from the input stream.
 
Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin
checkInterrupted, cloneParameterWithDefaultValue, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getJobOrConfigurationValue, getOverwriteAnnotationsParameters, getPermissions, init, log, log, storeValue, storeValue, storeValues, supportsConfigurations, validateRequestParameters
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface net.sf.basedb.core.plugin.Plugin
done, getAbout, getPermissions, init, supportsConfigurations
 

Field Detail

requiredRegexpType

protected static final ParameterType<String> requiredRegexpType

optionalRegexpType

protected static final ParameterType<String> optionalRegexpType

numDataColumnsType

protected static final ParameterType<Integer> numDataColumnsType

fileType

protected static final ParameterType<File> fileType

sectionRegexpParameter

protected static final PluginParameter<String> sectionRegexpParameter
Parameter definition that asks for the regular expression that matches a section line and extracts the section name. This is an optional parameter.

See Also:
FlatFileParser.setHeaderRegexp(Pattern)

headerRegexpParameter

protected static final PluginParameter<String> headerRegexpParameter
Parameter definition that asks for the regular expression that splits a header line in key/value pair. This is an optional parameter.

See Also:
FlatFileParser.setHeaderRegexp(Pattern)

dataHeaderRegexpParameter

protected static final PluginParameter<String> dataHeaderRegexpParameter
Parameter definition that asks for the regular expression that matches the data header. This is a required parameter for the plugin configuration.

See Also:
FlatFileParser.setDataHeaderRegexp(Pattern)

dataSplitterRegexpParameter

protected static final PluginParameter<String> dataSplitterRegexpParameter
Parameter definition that asks for the regular expression that splits a data line into individual columns. This is a required parameter for the plugin configuration.

See Also:
FlatFileParser.setDataSplitterRegexp(Pattern)

trimQuotesParameter

protected static final PluginParameter<Boolean> trimQuotesParameter
Parameter definition that asks if quotes around quoted values should be removed or not.

See Also:
FlatFileParser.setTrimQuotes(boolean)

ignoreRegexpParameter

protected static final PluginParameter<String> ignoreRegexpParameter
Parameter definition that asks for the regular expression that matches a line that should be ignored. This is an optional parameter for the plugin configuration.

See Also:
FlatFileParser.setIgnoreRegexp(Pattern)

dataFooterRegexpParameter

protected static final PluginParameter<String> dataFooterRegexpParameter
Parameter definition that asks for the regular expression that matches the data footer. This is an optional parameter for the plugin configuration.

See Also:
FlatFileParser.setDataFooterRegexp(Pattern)

minDataColumnsParameter

protected static final PluginParameter<Integer> minDataColumnsParameter
Parameter definition that asks for the minimum number of columns produced by the data splitter regexp for a line to be a data line. This is an optional parameter for the plugin configuration.

See Also:
FlatFileParser.setMinDataColumns(int)

maxDataColumnsParameter

protected static final PluginParameter<Integer> maxDataColumnsParameter
Parameter definition that asks for the maximum number of columns produced by the data splitter regexp for a line to be a data line. This is an optional parameter for the plugin configuration.

See Also:
FlatFileParser.setMaxDataColumns(int)

charsetType

protected static final StringParameterType charsetType
Deprecated. Use Parameters.charsetParameter(String, String, String)
Enumerates all available character sets.

See Also:
getCharsetParameter(String, String, String)

decimalSeparatorType

protected static final StringParameterType decimalSeparatorType
Deprecated. Use Parameters.decimalSeparatorParameter(String, String, String)
Type for selecting decimal separator.


fileParameter

protected static final PluginParameter<File> fileParameter
Parameter definition that asks for the file that should be imported. This is a required parameter for the job configuration.


parserSection

protected static final PluginParameter<String> parserSection
Section definition for grouping all file parser settings (ie. regular expressions)


mappingSection

protected static final PluginParameter<String> mappingSection
Section definition for grouping all column mapping expressions


complexMappings

protected static final PluginParameter<String> complexMappings
Parameter that asks if complex column mappings should be enabled or not. Use checkColumnMapping(String, boolean, String) to check a mapping.

Since:
2.4

errorSection

protected static final PluginParameter<String> errorSection
Section definition for grouping error handling options.

See Also:
Parameters.errorSection(String, String)

defaultErrorParameter

protected static final PluginParameter<String> defaultErrorParameter
See Also:
Parameters.defaultError(String, String, String, String...)

stringTooLongErrorParameter

protected static final PluginParameter<String> stringTooLongErrorParameter
See Also:
Parameters.stringTooLongError(String, String, String, String...)

invalidUseOfNullErrorParameter

protected static final PluginParameter<String> invalidUseOfNullErrorParameter
See Also:
Parameters.invalidUseOfNullError(String, String, String, String...)

numberOutOfRangeErrorParameter

protected static final PluginParameter<String> numberOutOfRangeErrorParameter
See Also:
Parameters.numberOutOfRangeError(String, String, String, String...)

numberFormatErrorParameter

protected static final PluginParameter<String> numberFormatErrorParameter
See Also:
Parameters.numberFormatError(String, String, String, String...)

currentFile

private File currentFile

totalFileSize

private long totalFileSize

finishedFileSize

private long finishedFileSize

skippedLines

private int skippedLines

errorHandler

private ClassMapErrorHandler errorHandler

signalHandler

private ThreadSignalHandler signalHandler

CHARSET

protected static final String CHARSET
Deprecated. Use Parameters.CHARSET_PARAMETER
The name of the parameter that asks for the character set.

See Also:
getCharsetParameter(String, String, String), Constant Field Values

DECIMAL_SEPARATOR

protected static final String DECIMAL_SEPARATOR
Deprecated. Use Parameters.DECIMAL_SEPARATOR_PARAMETER
The name of the parameter that asks for the decimal separator.

See Also:
getCharsetParameter(String, String, String), Constant Field Values
Constructor Detail

AbstractFlatFileImporter

public AbstractFlatFileImporter()
Method Detail

getMainType

public Plugin.MainType getMainType()
Return Plugin.MainType.IMPORT.

Specified by:
getMainType in interface Plugin
Returns:
One of the defined types

requiresConfiguration

public boolean requiresConfiguration()
Return TRUE, since the implementation requires it for finding the regular expressions used by the FlatFileParser. If this method is overridden and returns FALSE, the subclass must also override the getInitializedFlatFileParser() method and provide a parser with all regular expressions and other options set.

Specified by:
requiresConfiguration in interface Plugin
Overrides:
requiresConfiguration in class AbstractPlugin
Returns:
TRUE or FALSE

run

public void run(Request request,
                Response response,
                ProgressReporter progress)
Implements the Request.COMMAND_EXECUTE command. Subclasses must override this to implement other commands. Subclasses should not call this method for other commands than Request.COMMAND_EXECUTE since this method will set an error response status.

Specified by:
run in interface Plugin
Parameters:
request - Request object with the command and parameters
response - Response object in for the plugin to response through
progress - A ProgressReporter where the plugin can report its progess, can be null

isImportable

public final boolean isImportable(InputStream in)
                           throws BaseException
Description copied from interface: AutoDetectingImporter
Check if the given InputStream can be imported by this plugin.

Specified by:
isImportable in interface AutoDetectingImporter
Parameters:
in - The input stream to check
Returns:
TRUE if the stream can be imported, FALSE otherwise
Throws:
BaseException - If something goes wrong

doImport

public void doImport(InputStream in,
                     ProgressReporter progress)
              throws BaseException
Description copied from interface: AutoDetectingImporter
Import the data from the given InputStream.

Specified by:
doImport in interface AutoDetectingImporter
Parameters:
in - The input stream to read from
progress - Progress reporter used by the caller to keep track of the progress. Null is allowed
Throws:
BaseException - If something goes wrong

getSignalHandler

public SignalHandler getSignalHandler()
Create a new ThreadSignalHandler that supports the Signal.ABORT signal. Subclasses may override this to provide another signal handler, or return null if they don't support signals.

Specified by:
getSignalHandler in interface SignalTarget
Returns:
A SignalHandler object, or null if the current instance doesn't support signals
Since:
2.6

start

protected void start()
Called once before starting the import. The default implementation does nothing.

Since:
2.9

getFileIterator

protected Iterator<File> getFileIterator()
Get an iterator that returns the files to be imported. The default implementation returns the single file found in the job's "file" parameter. Subclasses that needs multi-file/item import should override this method to provide their own iterator. They should also override the getTotalFileSize() method to return sum of all file sizes. Eg. File.getSize().

Since:
2.9

getTotalFileSize

protected long getTotalFileSize()
Get the total file size of all files that are going to be imported. A subclass that is going to import from multiple files needs to override this method. The default implementation return the size of the file in the job's "file" parameter.

Returns:
The sum of the file sizes, or -1 if not known
Since:
2.9

getProgress

protected int getProgress(FlatFileParser ffp)
Get the progress of import as a percentage value. The default implementation calls sums the file size of the completed files and getNumBytes(FlatFileParser). This values is divided by getTotalFileSize().

Parameters:
ffp - The file parser that is used to parsed the file
Returns:
A value between 0 and 100
Since:
2.6

getNumBytes

protected long getNumBytes(FlatFileParser ffp)
Get the number of bytes read from the file. The value should indicate the how far into the file parsing has proceeded. The default implementation calls FlatFileParser.getParsedBytes(). If a subclass has wrapped the input stream the number of parsed byts may not correspond to the number of bytes read from the file. For example, if the file is a compressed file the number of parsed bytes will be higher than the number of bytes read from the file.

Parameters:
ffp - The file parser that is used to parsed the file
Returns:
The number of bytes read from the original file
Since:
2.6
See Also:
wrapInputStream(InputStream), getProgress(FlatFileParser)

isImportable

protected boolean isImportable(FlatFileParser ffp)
                        throws IOException
This method is called by the isImportable(InputStream) method after FlatFileParser.nextSection() and FlatFileParser.parseHeaders() has been called and if data has been found. Thus, the default implementation of this method always returns TRUE. Subclasses may override this method to do more checks, for example to make sure certain headers are present or parse more data from the file.

Parameters:
ffp - The FlatFileParser object used to parse the file
Returns:
Always TRUE
Throws:
IOException
Since:
2.4

wrapInputStream

protected InputStream wrapInputStream(InputStream in)
                               throws IOException
This method is called before the parser starts reading from the input stream. A subclass may override this method to wrap the inputstream with a filtering stream, for example, a gzip input stream. The default implementation of this method returns the original stream unmodified. If a subclass overrides this methods it may also need to override one of the getNumBytes(FlatFileParser) or getProgress(FlatFileParser) methods.

Parameters:
in - The input stream to wrap
Returns:
The same or a different input stream
Throws:
IOException
Since:
2.6
See Also:
getNumBytes(FlatFileParser)

begin

protected void begin(FlatFileParser ffp)
              throws BaseException
Called just before parsing of the file begins. A subclass may override this method if it needs to initialise some resources before the parsing starts. Note that this method is called once for each file returned by getFileIterator().

Throws:
BaseException
See Also:
end(boolean)

handleHeader

protected void handleHeader(FlatFileParser.Line line)
                     throws BaseException
Called by the parser for every line in the file that is a header line.

Throws:
BaseException

handleSection

protected void handleSection(FlatFileParser.Line line)
                      throws BaseException
Called by the parser for every line in the file that is a section line.

Throws:
BaseException

beginData

protected void beginData()
                  throws BaseException
Called by the parser after the headers have been parsed but before the first data line is parsed.

Throws:
BaseException

handleData

protected abstract void handleData(FlatFileParser.Data data)
                            throws BaseException
Called by the parser for every line in the file that is a data line.

Throws:
BaseException

end

protected void end(boolean success)
            throws BaseException
Called just after the last line of the file has been parsed, or immediately after an error has ocurred. A subclass should clean up any resources aquired in the begin(FlatFileParser) method here. Note that this metod is called once for every file returned by the getFileIterator() iterator.

Parameters:
success - TRUE if the file was parsed successfully, FALSE otherwise
Throws:
BaseException
See Also:
begin(FlatFileParser)

getSuccessMessage

protected String getSuccessMessage(int skippedLines)
Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface. An example message might by: 178 reporters imported successfully. The default implementation always return null. Note that this method is called once for every file returned by getFileIterator().

Parameters:
skippedLines - The number of data lines that were skipped due to errors

finish

protected String finish(Throwable t)
Called once when all files has been imported or when exiting due to an error. This method may return a message that is sent back to the core and user interface. If this method returns null the last message returned by getSuccessMessage(int) is used instead.

Parameters:
t - Null if no error has happened
Returns:
A message or null
Since:
2.9

getInitializedFlatFileParser

protected FlatFileParser getInitializedFlatFileParser()
                                               throws BaseException
Create and initialise a flat file parser by setting all regular expressions and other options. This implementation gets all parameters from the AbstractPlugin.job or AbstractPlugin.configuration settings. If a subclass doesn't store the parameters there it must override this method and initialise the parser. Note that this method is called once for each file returned by the getFileIterator() and that a new parser is needed for each file.

Returns:
An intialised flat file parser
Throws:
BaseException

getCharset

protected String getCharset()
Get the character set the file uses. This method looks for a character set in this order:
  1. Job parameter with name Parameters.CHARSET_PARAMETER.
  2. Configuration parameter with name Parameters.CHARSET_PARAMETER.
  3. Character set specified by file to import from File.getCharacterSet().
  4. System default characeter set Config.getCharset(). first checks the job parameters for a value, then the configuration parameters. If not found the Config.getCharset() is returned.

    Returns:
    The name of the charset to use
    See Also:
    getCharset(Request)

getCharset

protected String getCharset(Request request)
Get the character set the file uses. This method first checks the request object, and then calls getCharset().

Returns:
The name of the charset to use
Since:
2.8

getDecimalSeparator

protected String getDecimalSeparator()
Get the decimal separator used by numbers in the file. This method first checks the job parameters for a value, then the configuration parameters. If not found null is returned.

Returns:
The decimal separator or null to use Float.valueOf() or Double.valueOf()
Since:
2.2

getNumberFormat

protected NumberFormat getNumberFormat()
Get a number formatter that is able to parse numbers with the specified decimal separator. Returns null if no decimal separator has been specified which causes numbers to be parsed with Float.valueOf() or Double.valueOf().

Returns:
The number format or null to use Float.valueOf() or Double.valueOf()
Since:
2.2

getCharsetParameter

protected PluginParameter<String> getCharsetParameter(String label,
                                                      String description,
                                                      String defaultValue)
Deprecated. USe Parameters.charsetParameter(String, String, String)

Parameter definition that asks for the character encoding used in the file that is imported. This can be both a job parameter and a configuration parameter. The implementation first checks in the job parameter and if not found in the configuration parameter. If still not found the default charset is used as specified by Config.getCharset().

Parameters:
label - The label to use for the parameter or null to use the default label (Character set)
description - The description to use for the parameter or null to use the default description
defaultValue - The default value for the character set or null to use the system default

getDecimalSeparatorParameter

protected PluginParameter<String> getDecimalSeparatorParameter(String label,
                                                               String description,
                                                               String defaultValue)
Deprecated. Use Parameters.decimalSeparatorParameter(String, String, String)

Parameter definition that asks for the decimal separator used by numeric values in the file that is imported. This can be both a job parameter and a configuration parameter. The implementation first checks in the job parameter and if not found in the configuration parameter. If still not found the no decimal separator is used and number parsing is done by Float.valueOf or Double.valueOf.

Parameters:
label - The label to use for the parameter or null to use the default label (Decimal separator)
description - The description to use for the parameter or null to use the default description
defaultValue - The default value for the decimal separator
Since:
2.2

getPattern

private Pattern getPattern(String name)
                    throws BaseException
Throws:
BaseException

getPrimaryLocationFilter

@Deprecated
protected PropertyFilter getPrimaryLocationFilter()
Deprecated. In 2.16. Since files can now be references to external URL:s this filter no longer makes sense. Plug-ins should not use the location to filter files.

A property filter that can be used to filter files that are not in the primary location. Typical use is:
ItemContext fileContext = sc.getCurrentContext(Item.FILE);
fileContext.setPropertyFilter(primaryLocationFilter);
        


checkColumnMapping

protected String checkColumnMapping(String mapExpression,
                                    boolean allowComplex,
                                    String name)
                             throws InvalidDataException
Check if a column mapping is a valid mapping expression and optionally if it is complex mapping.

Parameters:
mapExpression - The mapping expression
allowComplex - If complex column mappings should be allowed
name - The name of the column (used if an error message needs to be generated)
Returns:
The mapping string
Throws:
InvalidDataException - If the mapping isn't a valid column mapping or if the allowComplex parameter is false and the mapping is complex
Since:
2.4

checkColumnMapping

protected String checkColumnMapping(FlatFileParser ffp,
                                    String mapExpression,
                                    boolean allowComplex,
                                    String name)
                             throws InvalidDataException
Checks the syntax column mapping and verifies that the given file parser has found the columns that are used in the file.

Parameters:
ffp - An optional flat file parser that should have parsed the file headers with FlatFileParser.parseHeaders()
mapExpression - The mapping expression
allowComplex - If complex column mappings should be allowed
name - The name of the column (used if an error message needs to be generated)
Returns:
The mapping string
Throws:
InvalidDataException - If the mapping isn't a valid column mapping or if the allowComplex parameter is false and the mapping is complex
Since:
2.9

getMapper

protected Mapper getMapper(FlatFileParser ffp,
                           String mapExpression,
                           Integer maxStringLength,
                           Mapper defaultMapper)
Get a mapper for the specified flat file parser. This method calls the FlatFileParser.getMapper(String) method to create a mapper, if the map expression isn't null. If a max string length has been specified the created mapper is wrapped by a CropStringMapper that crops strings returned by the Mapper.getValue(FlatFileParser.Data) method to the specified length. Use this method mainly for creating mappers for string values.

Parameters:
ffp - The flat file parser
mapExpression - The map expression, a null value is allows but no mapper is created
maxStringLength - The maximum allowed string length or null to allow any length
defaultMapper - The mapper to return if the map expression is null
Returns:
A mapper, or null if the map expression and default mapper is null

getErrorOption

protected String getErrorOption(String parameterName)
Get the value for an error handling parameter. If no value is found for the specified parameter the value for the 'defaultError' parameter is returned.

Parameters:
parameterName - The name of the error parameter
Returns:
The error option or fail if no option has been set
Since:
2.4

getErrorHandler

protected ErrorHandler getErrorHandler(String method,
                                       ErrorHandler defaultErrorHandler)

continueWithNextFileAfterError

protected boolean continueWithNextFileAfterError(Throwable t)
If the importer should continue with the next file after an error. The default implementation of this method return FALSE. A subclass may override this method to let the import continue after an error. This method isn't called until an error has happened, and it is possible to control what should happen on a file-by-file bases. The importer will of course exit if this method returns FALSE.

Parameters:
t - The error that happened
Returns:
TRUE to contine, FALSE to abort
Since:
2.9

setUpErrorHandling

protected void setUpErrorHandling()
Initialise the error handling system. This method is called just before the import of a file is starting. A subclass may override this method to add specific error handlers. If super.setUpErrorHandling() isn't called error handling in AbstractFlatFileImporter is disabled and the subclass must do all it's error handling in it's own code. The subclass may also add error handlers in the begin(FlatFileParser) method. Note that the error handling system is re-initialised for every file returned by getFileIterator().


addErrorHandler

protected void addErrorHandler(Class<? extends Throwable> t,
                               ErrorHandler handler)
Add an error handler for the specified class of error. The error handler also handles error that are subclasses of the specified class.

See Also:
ClassMapErrorHandler.addErrorHandler(Class, ErrorHandler)

log

protected void log(String message,
                   FlatFileParser.Data data)
Log a message about a data line to the log file created by AbstractPlugin.createLogFile(String). If no log file has been created, this method does nothing.

Parameters:
message - The message to log
data - The data line the log message is related to
Since:
2.8

log

protected void log(String message,
                   FlatFileParser.Data data,
                   Throwable t)
Log an error message about a data line to the log file created by AbstractPlugin.createLogFile(String). If no log file has been created, this method does nothing.

Parameters:
message - The message to log
data - The data line the log message is related to
t - The error
Since:
2.8

log

protected void log(String message,
                   FlatFileParser.Line line)
Log a message about a header line to the log file created by AbstractPlugin.createLogFile(String). If no log file has been created, this method does nothing.

Parameters:
message - The message to log
line - The header line line the log message is related to
Since:
2.8

log

protected void log(String message,
                   FlatFileParser.Line line,
                   Throwable t)
Log an error message about a header line to the log file created by AbstractPlugin.createLogFile(String). If no log file has been created, this method does nothing.

Parameters:
message - The message to log
line - The header line line the log message is related to
t - The error
Since:
2.8

2.17.2: 2011-06-17