|
2.8.2: 2008-09-18 | ||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectnet.sf.basedb.core.plugin.AbstractPlugin
net.sf.basedb.plugins.AbstractFlatFileImporter
public abstract class AbstractFlatFileImporter
An abstract base class for all importers that imports data from a single
flat file. The implementation in this class uses a FlatFileParser
for parsing the file and uses a callback method handleData(FlatFileParser.Data)
that lets the subclass do whatever it needs to insert a single line
of data into the database.
The subclass must also generate the RequestInformation
object
for both plugin and job configuration. However, this implementation expects
to find the regular expression needed for the FlatFileParser
to
be found in the plugin configuration and the File
to import to
be found in the job configuration.
All of the needed PluginParameter
objects needed to ask for these
parameters are declared as protected variables in this class.
This class implements theheaderRegexpParameter
); parameters.add(dataHeaderRegexpParameter
); parameters.add(dataSplitterRegexpParameter
); parameters.add(ignoreRegexpParameter
); parameters.add(dataFooterRegexpParameter
); // Column mappings parameters.addAll(allColumnMappings); // Reporter type parameters.add(reporterTypeParameter); configurePlugin = new RequestInformation ( Request.COMMAND_CONFIGURE_PLUGIN, "Parser settings", "Please enter all settings needed by the flat file parser", parameters );
invoke
method but only for
the Request.COMMAND_EXECUTE
command. The subclass must override
the invoke
method and provide implementations for the
other commands (Request.COMMAND_CONFIGURE_PLUGIN
and
Request.COMMAND_CONFIGURE_JOB
). The normal implementation of
would be to store a request parameter.
// ReporterFlatFileImporter.java
if (command.equals(Request.COMMAND_CONFIGURE_JOB))
{
List<Throwable> errors = validateRequestParameters(jobParameters, request);
if (errors != null)
{
response.setError(errors.size()+" invalid parameter(s) were found in the request", errors);
return;
}
storeValue(job, request, fileParameter
);
response.setDone(null);
}
Nested Class Summary |
---|
Nested classes/interfaces inherited from interface net.sf.basedb.core.plugin.Plugin |
---|
Plugin.MainType |
Field Summary | |
---|---|
protected static String |
CHARSET
Deprecated. Use Parameters.CHARSET_PARAMETER |
protected static StringParameterType |
charsetType
Deprecated. Use Parameters.charsetParameter(String, String, String) |
protected static PluginParameter<String> |
complexMappings
Parameter that asks if complex column mappings should be enabled or not. |
protected static PluginParameter<String> |
dataFooterRegexpParameter
Parameter definition that asks for the regular expression that matches the data footer. |
protected static PluginParameter<String> |
dataHeaderRegexpParameter
Parameter definition that asks for the regular expression that matches the data header. |
protected static PluginParameter<String> |
dataSplitterRegexpParameter
Parameter definition that asks for the regular expression that splits a data line into individual columns. |
protected static String |
DECIMAL_SEPARATOR
Deprecated. Use Parameters.DECIMAL_SEPARATOR_PARAMETER |
protected static StringParameterType |
decimalSeparatorType
Deprecated. Use Parameters.decimalSeparatorParameter(String, String, String) |
protected static PluginParameter<String> |
defaultErrorParameter
|
private ClassMapErrorHandler |
errorHandler
|
protected static PluginParameter<String> |
errorSection
Section definition for grouping error handling options. |
protected static PluginParameter<File> |
fileParameter
Parameter definition that asks for the file that should be imported. |
private long |
fileSize
|
protected static ParameterType<File> |
fileType
|
protected static PluginParameter<String> |
headerRegexpParameter
Parameter definition that asks for the regular expression that splits a header line in key/value pair. |
protected static PluginParameter<String> |
ignoreRegexpParameter
Parameter definition that asks for the regular expression that matches a line that should be ignored. |
protected static PluginParameter<String> |
invalidUseOfNullErrorParameter
|
protected static PluginParameter<String> |
mappingSection
Section definition for grouping all column mapping expressions |
protected static PluginParameter<Integer> |
maxDataColumnsParameter
Parameter definition that asks for the maximum number of columns produced by the data splitter regexp for a line to be a data line. |
protected static PluginParameter<Integer> |
minDataColumnsParameter
Parameter definition that asks for the minimum number of columns produced by the data splitter regexp for a line to be a data line. |
protected static PluginParameter<String> |
numberFormatErrorParameter
|
protected static PluginParameter<String> |
numberOutOfRangeErrorParameter
|
protected static ParameterType<Integer> |
numDataColumnsType
|
protected static ParameterType<String> |
optionalRegexpType
|
protected static PluginParameter<String> |
parserSection
Section definition for grouping all file parser settings (ie. regular expressions) |
protected static ParameterType<String> |
requiredRegexpType
|
protected static PluginParameter<String> |
sectionRegexpParameter
Parameter definition that asks for the regular expression that matches a section line and extracts the section name. |
private ThreadSignalHandler |
signalHandler
|
private int |
skippedLines
|
protected static PluginParameter<String> |
stringTooLongErrorParameter
|
protected static PluginParameter<Boolean> |
trimQuotesParameter
Parameter definition that asks if quotes around quoted values should be removed or not. |
Fields inherited from class net.sf.basedb.core.plugin.AbstractPlugin |
---|
annotationSection, configuration, COPY_ANNOTATIONS, job, OVERWRITE_ANNOTATIONS, sc |
Constructor Summary | |
---|---|
AbstractFlatFileImporter()
|
Method Summary | |
---|---|
protected void |
addErrorHandler(Class<? extends Throwable> t,
ErrorHandler handler)
Add an error handler for the specified class of error. |
protected void |
begin(FlatFileParser ffp)
Called just before parsing of the file begins. |
protected void |
beginData()
Called by the parser after the headers have been parsed but before the first data line is parsed. |
protected String |
checkColumnMapping(String mapExpression,
boolean allowComplex,
String name)
Check if a column mapping is a valid mapping expression and optionally if it is complex mapping. |
void |
doImport(InputStream in,
ProgressReporter progress)
Import the data from the given InputStream . |
protected void |
end(boolean success)
Called just after the last line of the file has been parsed, or immediately after an error has ocurred. |
protected String |
getCharset()
Get the character set the file uses. |
protected String |
getCharset(Request request)
Get the character set the file uses. |
protected PluginParameter<String> |
getCharsetParameter(String label,
String description,
String defaultValue)
Deprecated. USe Parameters.charsetParameter(String, String, String) |
protected String |
getDecimalSeparator()
Get the decimal separator used by numbers in the file. |
protected PluginParameter<String> |
getDecimalSeparatorParameter(String label,
String description,
String defaultValue)
Deprecated. Use Parameters.decimalSeparatorParameter(String, String, String) |
protected ErrorHandler |
getErrorHandler(String method,
ErrorHandler defaultErrorHandler)
|
protected String |
getErrorOption(String parameterName)
Get the value for an error handling parameter. |
protected FlatFileParser |
getInitializedFlatFileParser()
Create and initialise a flat file parser by setting all regular expressions and other options. |
Plugin.MainType |
getMainType()
Return Plugin.MainType.IMPORT . |
protected Mapper |
getMapper(FlatFileParser ffp,
String mapExpression,
Integer maxStringLength,
Mapper defaultMapper)
Get a mapper for the specified flat file parser. |
protected NumberFormat |
getNumberFormat()
Get a number formatter that is able to parse numbers with the specified decimal separator. |
protected long |
getNumBytes(FlatFileParser ffp)
Get the number of bytes read from the file. |
private Pattern |
getPattern(String name)
|
protected PropertyFilter |
getPrimaryLocationFilter()
A property filter that can be used to filter files that are not in the primary location. |
protected int |
getProgress(FlatFileParser ffp)
Get the progress of import as a percentage value. |
SignalHandler |
getSignalHandler()
Create a new ThreadSignalHandler that supports
the Signal.ABORT signal. |
protected String |
getSuccessMessage(int skippedLines)
Called if the parsing was successful to let the subclass generate a simple message that is sent back to the core and user interface. |
protected abstract void |
handleData(FlatFileParser.Data data)
Called by the parser for every line in the file that is a data line. |
protected void |
handleHeader(FlatFileParser.Line line)
Called by the parser for every line in the file that is a header line. |
protected void |
handleSection(FlatFileParser.Line line)
Called by the parser for every line in the file that is a section line. |
protected boolean |
isImportable(FlatFileParser ffp)
This method is called by the isImportable(InputStream) method after
FlatFileParser.nextSection() and FlatFileParser.parseHeaders()
has been called and if data has been found. |
boolean |
isImportable(InputStream in)
Check if the given InputStream can be imported by this
plugin. |
protected void |
log(String message,
FlatFileParser.Data data)
Log a message about a data line to the log file created by AbstractPlugin.createLogFile(String) . |
protected void |
log(String message,
FlatFileParser.Data data,
Throwable t)
Log an error message about a data line to the log file created by AbstractPlugin.createLogFile(String) . |
protected void |
log(String message,
FlatFileParser.Line line)
Log a message about a header line to the log file created by AbstractPlugin.createLogFile(String) . |
protected void |
log(String message,
FlatFileParser.Line line,
Throwable t)
Log an error message about a header line to the log file created by AbstractPlugin.createLogFile(String) . |
boolean |
requiresConfiguration()
Return TRUE, since the the implementation requires it for finding the regular expressions used by the FlatFileParser . |
void |
run(Request request,
Response response,
ProgressReporter progress)
Implements the Request.COMMAND_EXECUTE command. |
protected void |
setUpErrorHandling()
Initialise the error handling system. |
protected InputStream |
wrapInputStream(InputStream in)
This method is called before the parser starts reading from the input stream. |
Methods inherited from class net.sf.basedb.core.plugin.AbstractPlugin |
---|
checkInterrupted, closeLogFile, createLogFile, done, getCopyAnnotationsParmeter, getCurrentConfiguration, getCurrentJob, getOverwriteAnnotationsParameters, getPermissions, init, log, log, storeValue, storeValue, storeValues, supportsConfigurations, validateRequestParameters |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Methods inherited from interface net.sf.basedb.core.plugin.Plugin |
---|
done, getAbout, getPermissions, init, supportsConfigurations |
Field Detail |
---|
protected static final ParameterType<String> requiredRegexpType
protected static final ParameterType<String> optionalRegexpType
protected static final ParameterType<Integer> numDataColumnsType
protected static final ParameterType<File> fileType
protected static final PluginParameter<String> sectionRegexpParameter
FlatFileParser.setHeaderRegexp(Pattern)
protected static final PluginParameter<String> headerRegexpParameter
FlatFileParser.setHeaderRegexp(Pattern)
protected static final PluginParameter<String> dataHeaderRegexpParameter
FlatFileParser.setDataHeaderRegexp(Pattern)
protected static final PluginParameter<String> dataSplitterRegexpParameter
FlatFileParser.setDataSplitterRegexp(Pattern)
protected static final PluginParameter<Boolean> trimQuotesParameter
FlatFileParser.setTrimQuotes(boolean)
protected static final PluginParameter<String> ignoreRegexpParameter
FlatFileParser.setIgnoreRegexp(Pattern)
protected static final PluginParameter<String> dataFooterRegexpParameter
FlatFileParser.setDataFooterRegexp(Pattern)
protected static final PluginParameter<Integer> minDataColumnsParameter
FlatFileParser.setMinDataColumns(int)
protected static final PluginParameter<Integer> maxDataColumnsParameter
FlatFileParser.setMaxDataColumns(int)
protected static final StringParameterType charsetType
Parameters.charsetParameter(String, String, String)
getCharsetParameter(String, String, String)
protected static final StringParameterType decimalSeparatorType
Parameters.decimalSeparatorParameter(String, String, String)
protected static final PluginParameter<File> fileParameter
protected static final PluginParameter<String> parserSection
protected static final PluginParameter<String> mappingSection
protected static final PluginParameter<String> complexMappings
checkColumnMapping(String, boolean, String)
to check a mapping.
protected static final PluginParameter<String> errorSection
Parameters.errorSection(String, String)
protected static final PluginParameter<String> defaultErrorParameter
Parameters.defaultError(String, String, String, String...)
protected static final PluginParameter<String> stringTooLongErrorParameter
Parameters.stringTooLongError(String, String, String, String...)
protected static final PluginParameter<String> invalidUseOfNullErrorParameter
Parameters.invalidUseOfNullError(String, String, String, String...)
protected static final PluginParameter<String> numberOutOfRangeErrorParameter
Parameters.numberOutOfRangeError(String, String, String, String...)
protected static final PluginParameter<String> numberFormatErrorParameter
Parameters.numberFormatError(String, String, String, String...)
private long fileSize
private int skippedLines
private ClassMapErrorHandler errorHandler
private ThreadSignalHandler signalHandler
protected static final String CHARSET
Parameters.CHARSET_PARAMETER
getCharsetParameter(String, String, String)
,
Constant Field Valuesprotected static final String DECIMAL_SEPARATOR
Parameters.DECIMAL_SEPARATOR_PARAMETER
getCharsetParameter(String, String, String)
,
Constant Field ValuesConstructor Detail |
---|
public AbstractFlatFileImporter()
Method Detail |
---|
public Plugin.MainType getMainType()
Plugin.MainType.IMPORT
.
getMainType
in interface Plugin
public boolean requiresConfiguration()
FlatFileParser
. If this
method is overridden and returns FALSE, the subclass must also override
the getInitializedFlatFileParser()
method and provide a
parser with all regular expressions and other options set.
requiresConfiguration
in interface Plugin
requiresConfiguration
in class AbstractPlugin
public void run(Request request, Response response, ProgressReporter progress)
Request.COMMAND_EXECUTE
command. Subclasses
must override this to implement other commands. Subclasses should not
call this method for other commands than Request.COMMAND_EXECUTE
since this method will set an error response status.
run
in interface Plugin
request
- Request object with the command and parametersresponse
- Response object in for the plugin to response
throughprogress
- A ProgressReporter
where the plugin can report
its progess, can be nullpublic final boolean isImportable(InputStream in) throws BaseException
AutoDetectingImporter
InputStream
can be imported by this
plugin.
isImportable
in interface AutoDetectingImporter
in
- The input stream to check
BaseException
- If something goes wrongpublic void doImport(InputStream in, ProgressReporter progress) throws BaseException
AutoDetectingImporter
InputStream
.
doImport
in interface AutoDetectingImporter
in
- The input stream to read fromprogress
- Progress reporter used by the caller to keep track of the progress.
Null is allowed
BaseException
- If something goes wrongpublic SignalHandler getSignalHandler()
ThreadSignalHandler
that supports
the Signal.ABORT
signal.
Subclasses may override this to provide another signal
handler, or return null if they don't support signals.
getSignalHandler
in interface SignalTarget
protected int getProgress(FlatFileParser ffp)
getNumBytes(FlatFileParser)
and divides it by the file size.
ffp
- The file parser that is used to parsed the file
protected long getNumBytes(FlatFileParser ffp)
FlatFileParser.getParsedBytes()
. If a subclass has
wrapped the input stream the number of parsed byts may not correspond to the
number of bytes read from the file.
For example, if the file is a compressed file the number of parsed bytes will
be higher than the number of bytes read from the file.
ffp
- The file parser that is used to parsed the file
wrapInputStream(InputStream)
,
getProgress(FlatFileParser)
protected boolean isImportable(FlatFileParser ffp) throws IOException
isImportable(InputStream)
method after
FlatFileParser.nextSection()
and FlatFileParser.parseHeaders()
has been called and if data has been found. Thus, the default implementation of this
method always returns TRUE. Subclasses may override this method to do more checks, for
example to make sure certain headers are present or parse more data from the file.
ffp
- The FlatFileParser object used to parse the file
IOException
protected InputStream wrapInputStream(InputStream in) throws IOException
getNumBytes(FlatFileParser)
or getProgress(FlatFileParser)
methods.
in
- The input stream to wrap
IOException
getNumBytes(FlatFileParser)
protected void begin(FlatFileParser ffp) throws BaseException
BaseException
end(boolean)
protected void handleHeader(FlatFileParser.Line line) throws BaseException
BaseException
protected void handleSection(FlatFileParser.Line line) throws BaseException
BaseException
protected void beginData() throws BaseException
BaseException
protected abstract void handleData(FlatFileParser.Data data) throws BaseException
BaseException
protected void end(boolean success) throws BaseException
begin(FlatFileParser)
method here.
success
- TRUE if the file was parsed successfully, FALSE otherwise
BaseException
begin(FlatFileParser)
protected String getSuccessMessage(int skippedLines)
178 reporters imported successfully
.
The default implementation always return null.
skippedLines
- The number of data lines that were skipped due to errorsprotected FlatFileParser getInitializedFlatFileParser() throws BaseException
AbstractPlugin.configuration
settings. If a subclass doesn't store the parameters
there it must override this method and initialise the parser.
BaseException
protected String getCharset()
Config.getCharset()
is returned.
getCharset(Request)
protected String getCharset(Request request)
getCharset()
.
protected String getDecimalSeparator()
protected NumberFormat getNumberFormat()
protected PluginParameter<String> getCharsetParameter(String label, String description, String defaultValue)
Parameters.charsetParameter(String, String, String)
Config.getCharset()
.
label
- The label to use for the parameter or null to use the default label
(Character set)description
- The description to use for the parameter or null to use the
default descriptiondefaultValue
- The default value for the character set or null to use
the system defaultprotected PluginParameter<String> getDecimalSeparatorParameter(String label, String description, String defaultValue)
Parameters.decimalSeparatorParameter(String, String, String)
label
- The label to use for the parameter or null to use the default label
(Decimal separator)description
- The description to use for the parameter or null to use the
default descriptiondefaultValue
- The default value for the decimal separatorprivate Pattern getPattern(String name) throws BaseException
BaseException
protected PropertyFilter getPrimaryLocationFilter()
ItemContext fileContext = sc.getCurrentContext(Item.FILE); fileContext.setPropertyFilter(primaryLocationFilter);
protected String checkColumnMapping(String mapExpression, boolean allowComplex, String name) throws InvalidDataException
mapExpression
- The mapping expressionallowComplex
- If complex column mappings should be allowedname
- The name of the column (used if an error message needs to be generated)
InvalidDataException
- If the mapping isn't a valid column
mapping or if the allowComplex parameter is false and the mapping is complexprotected Mapper getMapper(FlatFileParser ffp, String mapExpression, Integer maxStringLength, Mapper defaultMapper)
FlatFileParser.getMapper(String)
method to create
a mapper, if the map expression isn't null. If a max string length has been
specified the created mapper is wrapped by a CropStringMapper
that crops strings returned by the Mapper.getValue(FlatFileParser.Data)
method
to the specified length. Use this method mainly for creating mappers for string values.
ffp
- The flat file parsermapExpression
- The map expression, a null value is allows but no mapper is createdmaxStringLength
- The maximum allowed string length or null to allow any lengthdefaultMapper
- The mapper to return if the map expression is null
protected String getErrorOption(String parameterName)
parameterName
- The name of the error parameter
fail
if no option has been setprotected ErrorHandler getErrorHandler(String method, ErrorHandler defaultErrorHandler)
protected void setUpErrorHandling()
super.setUpErrorHandling()
isn't called
error handling in AbstractFlatFileImporter is disabled and the subclass
must do all it's error handling in it's own code. The subclass may also add
error handlers in the begin(FlatFileParser)
method.
protected void addErrorHandler(Class<? extends Throwable> t, ErrorHandler handler)
ClassMapErrorHandler.addErrorHandler(Class, ErrorHandler)
protected void log(String message, FlatFileParser.Data data)
AbstractPlugin.createLogFile(String)
. If no log file has been created,
this method does nothing.
message
- The message to logdata
- The data line the log message is related toprotected void log(String message, FlatFileParser.Data data, Throwable t)
AbstractPlugin.createLogFile(String)
. If no log file has been created,
this method does nothing.
message
- The message to logdata
- The data line the log message is related tot
- The errorprotected void log(String message, FlatFileParser.Line line)
AbstractPlugin.createLogFile(String)
. If no log file has been created,
this method does nothing.
message
- The message to logline
- The header line line the log message is related toprotected void log(String message, FlatFileParser.Line line, Throwable t)
AbstractPlugin.createLogFile(String)
. If no log file has been created,
this method does nothing.
message
- The message to logline
- The header line line the log message is related tot
- The error
|
2.8.2: 2008-09-18 | ||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |