Class FlatFileParser


  • public class FlatFileParser
    extends Object
    This class can be used to parse data from flat text files and from Excel workbooks in xlsx format. If the file is a text file it must follow a few simple rules:
    • Data must be organised into columns, with one record per line
    • Each data column must be separated by some special character or character sequence not occuring in the data, for example a tab or a comma. Data in fixed-size columns cannot be parsed.
    • Data may optionally be preceeded by a data header, ie. the names of the columns
    • The data header may optionally be preceeded by file headers. A file header is something that can be split in a name-value pair.
    • The file may contain comments, which are ignored by the parser
    • The file contain section where each section can contain a header and/or a data part

    Example

    # Example of a parsable file, not actual format of a GenePix file
    section info
    Type=GenePix Results 1.3
    DateTime=2002/09/04 13:59:48
    Scanner=GenePix 4000B [83306]
    
    section data
    Block   Column  Row     Name    ID
    1       1       1       "Ly68_Lymphocyte antigen 68"    "M000205_01"
    1       2       1       "Bag1_Bcl2-associated athanogene 1"     "M000209_01"
    1       3       1       "Rps16_Ribosomal protein S16"   "M000213_01"
    1       4       1       "Col4a1_Procollagen, type IV, alpha 1"  "M000229_01"
    1       5       1       "Ace_Angiotensin converting enzyme"     "M000233_01"
    1       6       1       "Cd5_CD5 antigen"       "M000237_01"
    1       7       1       "Psme1_Protease (prosome, macropain) 28 s"
    

    If the file is an Excel file the first sheet is automatically converted to a tab-separated text file by default. To use a different sheet call the setExcelSheet(String) method with the name or index of the sheet. Initial parsing and regular expression matching is always done against the text representation of the selected sheet. Note that empty cells on the top and left are usually cut away. When retrieving values via the FlatFileParser.Data class it will typically go directly to the mapped cell from the Excel sheet and get the value, which means that numeric and date values doesn't have to be converted to and from strings if not needed. For example, if a date value is requested and the mapped cell is date that will be used as it is, but it the mapped cell is a string, the same parsers that are used for CSV files are used to convert the string to a date.

    How to use
    The parsing is controlled by regular expressions. Start by creating a new FlatFileParser object. Use the various set methods to provide regular expression used to match the data/headers.

    Use the setInputStream(InputStream, String) method to specify a file to parse, and parseHeaders() to start the parsing. Note! Even if you know that the file doesn't contain any headers, you should always call this method since the parser must initialize itself. If there are sections in the file use nextSection() first to control which section you are parsing from.

    When the headers have been found use the hasMoreData() and nextData() methods in a loop to read all data from the section.

    Example

    FlatFileParser ffp = new FlatFileParser();
    ffp.setHeaderRegexp(Pattern.compile("(.*)=(.*)"));
    ffp.setDataHeaderRegexp(Pattern.compile("Block\\tColumn\\tRow\\tName\\tID"));
    ffp.setDataSplitterRegexp(Pattern.compile("\\t"));
    ffp.setIgnoreRegexp(Pattern.compile("#.*"));
    ffp.setMinDataColumns(5);
    ffp.setMaxDataColumns(5);
    ffp.setInputStream(FileUtil.getInputStream(path_to_file), Config.getCharset());
    ffp.parseHeaders();
    for (int i = 0; i < ffp.getLineCount(); i++)
    {
       FlatFileParser.Line line = ffp.getLine(i);
       System.out.println(i+":"+line.type()+":"+line.line());
    }
    int i = 0;
    while (ffp.hasMoreData())
    {
       FlatFileParser.Data data = ffp.nextData();
       System.out.println(i+":"+data.columns()+":"+data.line());
    }
    

    Mapping column values
    With the FlatFileParser.Data object you can only access the data by column index (0-based). Another approach is to use Mapper:s. A mapper takes a string template and inserts the values of the data columns where you specify. Here are some example:

    \1\
    \row\
    Row: \row\, Col:\col\
    =2 * col('Radius')
    
    The result can be retrieved either as a string, as a numeric value or as a date. It is even possible to create expressions that does a calculation on the value before it is returned. See the getMapper(String) method for more information.
    Version:
    2.0
    Author:
    Nicklas, Enell
    Last modified
    $Date$
    • Field Detail

      • DEFAULT_MAX_UNKNOWN_LINES

        public static final int DEFAULT_MAX_UNKNOWN_LINES
        The default value for the number of unknown lines in a row that may be encountered by the parseHeaders method before it gives up.
        See Also:
        setMaxUnknownLines, Constant Field Values
      • findColumn

        private static final Pattern findColumn
        Pattern used to find column mappings in a string, ie. abc \col\ def
      • reader

        private BufferedReader reader
        Reads from the given input stream
      • tracker

        private InputStreamTracker tracker
        For keeping track of the number of bytes parsed.
      • excelSheetName

        private String excelSheetName
        Name of the Excel sheet to parse if the file is an Excel file.
      • parseAllExcelSheets

        private boolean parseAllExcelSheets
        Flag to indicate if only a single (=false) or all (=true) Excel sheets should be parsed in one go.
      • excelWorkbook

        private XlsxToCsvUtil excelWorkbook
        Excel workbook that has been loaded.
      • excelParsedLinesOffset

        private int excelParsedLinesOffset
        The value of the parsedLines when the current Excel sheet is parsed. Used to find the correct row number in the current sheet (=parsedLines - excelParsedLinesOffset)
      • bofMarker

        private Pattern bofMarker
        The regular expression for matching the beginning-of-file marker
      • header

        private Pattern header
        The regular expression for matching a header line.
      • section

        private Pattern section
        The regular expression for matching the fist line of a section. The expression must have one capturing group.
      • dataHeader

        private Pattern dataHeader
        The regular expression for matching the data header line. The expression must have two capturing groups.
      • dataSplitter

        private Pattern dataSplitter
        The regular expression for splitting a data line.
      • trimQuotes

        private boolean trimQuotes
        If quotes should be trimmed from data values or not. Default true. Quotes are double or single quotes.
      • trimWhiteSpace

        private boolean trimWhiteSpace
        If white space should be trimmed from data values or not. Default is false.
        Since:
        3.15.1
      • dataFooter

        private Pattern dataFooter
        The regular expression for matching the data footer line.
      • minDataColumns

        private int minDataColumns
        The minimun number of allowed data columns for a line to be considered a data line.
      • maxDataColumns

        private int maxDataColumns
        The maximun number of allowed data columns for a line to be considered a data line.
      • ignore

        private Pattern ignore
        The regular expression for matching a comment line.
      • maxUnknownLines

        private int maxUnknownLines
        The maximum number of unkown lines to parse before giving up.
      • emptyIsNull

        private boolean emptyIsNull
        If null should be returned for empty columns (instead of an empty string).
      • useNullIfException

        private boolean useNullIfException
        If null should be returned if a (numeric) value can't be parsed.
      • ignoreNonExistingColumns

        private boolean ignoreNonExistingColumns
        If non-existing columns should be ignored (true) or result in an exception (false)
      • nullIsNull

        private boolean nullIsNull
        If null should be returned for the string NULL (ignoring case) or not.
      • numberFormat

        private NumberFormat numberFormat
        The default number formatter to use for creating mappers.
      • dateFormat

        private Formatter<Date> dateFormat
        The default date formatter to use when creating mappers.
      • timestampFormat

        private Formatter<Date> timestampFormat
        The default timestamp format to use when creating mappers.
      • bofType

        private String bofType
        The value that was captured by the bofMarker pattern.
      • parsedLines

        private int parsedLines
        The total number of lines parsed so far.
      • parsedSections

        private int parsedSections
        The number of sections parsed so far.
      • parsedCharacters

        private long parsedCharacters
        The total number of parsed characters so far.
      • parsedDataLines

        private int parsedDataLines
        The number of data lines parsed in the current section so far. This value is reset at each new section.
      • columnHeaders

        private List<String> columnHeaders
        List of the column names found by splitting the data header using the data splitter regexp.
      • ignoredLines

        private int ignoredLines
        Number of ignored lines in the nextData() method.
      • unknownLines

        private int unknownLines
        Number of unknown lines in the nextData() method.
      • keepSkippedLines

        private boolean keepSkippedLines
        If unknown or ignored lines should be kept.
        See Also:
        getSkippedLines()
    • Constructor Detail

      • FlatFileParser

        public FlatFileParser()
        Create a new FlatFileParser object.
    • Method Detail

      • setBofMarkerRegexp

        public void setBofMarkerRegexp​(Pattern regexp)
        Set a regular expression that maches a beginning-of-file marker. This property should be set before starting to parse the file (otherwise it is ignored). The first method call that causes the parsing to be started will invoke parseToBof() (can also be invoked manually).

        The regular expression may contain a single capturing group. The matched value is returned by getBofType().

        Parameters:
        regexp - A regular expression
        Since:
        2.15
      • setHeaderRegexp

        public void setHeaderRegexp​(Pattern regexp)
        Set a regular expression that can be matched against a header. The regular expression must contain two capturing groups, the first should capture the name and the second the value of the header. For example, the file contains headers like:
        "Type=GenePix Results 1.3"
        "DateTime=2002/09/04 13:59:48"
        
        To match this we can use the following regular expression: "(.*)=(.*)".
        Parameters:
        regexp - A regular expression
      • setSectionRegexp

        public void setSectionRegexp​(Pattern regexp)
        Set a regular expression that can be matched against the section line. For example, the file contains a section like:
        [FileInformation]
        
        To match this we can use the following regular expression: section (.*). This will match to anything that starts with "section ". The section name will be in the capturing group.
        Parameters:
        regexp - A regular expression
      • setDataHeaderRegexp

        public void setDataHeaderRegexp​(Pattern regexp)
        Set a regular expression that can be matched against the data header. For example, the file contains a data header like:
        "Block"{tab}"Column"{tab}"Row"{tab}"Name"{tab}"ID" ...and so on
        
        To match this we can use the following regular expression: "(.*?)"(\t"(.*?)"). This will match to anything that has at least two columns. We could also be more specific and use: "Block"\t"Column"\t"Row"\t"Name"\t"ID"...
        Parameters:
        regexp - A regular expression
      • setDataSplitterRegexp

        public void setDataSplitterRegexp​(Pattern regexp)
        Set a regular expression that is used to split a data line into columns. To split on tabs we use: \t. This regular expression is also used to split the data header line into column names, which can then be used in the getMapper(String) method.
        Parameters:
        regexp - A regular expression
        See Also:
        setMinDataColumns, setMaxDataColumns
      • setTrimQuotes

        public void setTrimQuotes​(boolean trimQuotes)
        Set if quotes around each data value should be removed or not. A quote is either a double quote (") or a single quote ('). The default setting of this option is true.
        Parameters:
        trimQuotes - TRUE to remove quotes, FALSE to keep them
      • setTrimWhiteSpace

        public void setTrimWhiteSpace​(boolean trimWhiteSpace)
        Set a flag indicating if white-space should be trimmed from start and end of data values. The default setting is false.
        Parameters:
        trimWhiteSpace - TRUE to remove white-space, FALSE to keep them
        Since:
        3.15.1
      • setMinDataColumns

        public void setMinDataColumns​(int columns)
        Set the minimum number of columns a data line must contain in order for it to be counted as a data line.
        Parameters:
        columns - The minimum number of columns
      • setMaxDataColumns

        public void setMaxDataColumns​(int columns)
        Set the maximum number of columns a data line can contain in order for it to be counted as a data line.
        Parameters:
        columns - The maximum number of columns, or 0 for an unlimited number, or -1 to disable counting the number of columns
      • setDataFooterRegexp

        public void setDataFooterRegexp​(Pattern regexp)
        Set a regular expression that can be matched against a data footer. If a line matching this pattern is found while looking for data with the hasMoreData method it will exit and no more data will be returned.
        Parameters:
        regexp - A regular expression
      • setIgnoreRegexp

        public void setIgnoreRegexp​(Pattern regexp)
        Set a regular expression that is used to match a line that should be ignored. For example, the file may contain comments starting with a #: \#.*
        Parameters:
        regexp - A regular expression
      • setMaxUnknownLines

        public void setMaxUnknownLines​(int lines)
        The number of unknown lines in a row that can be parsed by the parseHeaders method before it gives up. The default value is specified by {#link #DEFAULT_MAX_UNKNOWN_LINES}. This value is ignored while parsing data.
        Parameters:
        lines - The number of lines
      • setUseNullIfEmpty

        public void setUseNullIfEmpty​(boolean emptyIsNull)
        Specify if null values should be returned instead of empty strings for columns that doesn't contain any value.
        Parameters:
        emptyIsNull - TRUE to return null, FALSE to return an empty string
      • setUseNullIfNull

        public void setUseNullIfNull​(boolean nullIsNull)
        Specify if null values should be returned for strings having the value "NULL" (ignoring case).
        Parameters:
        nullIsNull - TRUE to return null, FALSE to return the original string value
      • setKeepSkippedLines

        public void setKeepSkippedLines​(boolean keep)
        If the nextData() and hasMoreData() methods should keep information of lines that was skipped because they matched the ignore pattern or could be interpreted as data lines. The default is FALSE. The number of lines that was skipped is always available regardless of this setting.
        Parameters:
        keep - TRUE to keep line information, FALSE to not
        See Also:
        getSkippedLines(), getIgnoredLines(), getUnknownLines(), getNumSkippedLines()
      • setParseAllExcelSheets

        public void setParseAllExcelSheets​(boolean parseAllExcelSheets)
        If this flag is set and the source file is an Excel file, then all sheets will be parsed unless a named sheet specified. Each sheet is handled like a section with the sheet name inside brackets ([name]). The regular expression for detecting a section is automatically updated to match this pattern.
        Since:
        3.15
      • setExcelSheet

        public void setExcelSheet​(String name)
        Set the name of Excel worksheet to parse if the given file is an Excel file, otherwise this is ignored.
        Since:
        3.15
      • getExcelSheet

        public String getExcelSheet()
        Get the name of the Excel sheet that should be or is parsed.
        Since:
        3.15
      • getCurrentExcelWorkbook

        public XlsxToCsvUtil getCurrentExcelWorkbook()
        If the input stream that is being parsed is an Excel document, this method returns information about it.
        Returns:
        An XlsxToCsvUtil object or null if the stream is not an Excel document
        Since:
        3.15.1
      • getCurrentSheet

        public XlsxToCsvUtil.SheetInfo getCurrentSheet()
        If the input stream that is being parsed is an Excel document, this method returns information about the current worksheet.
        Returns:
        An XlsxToCsvUtil.SheetInfo object or null if the stream is not an Excel document
        Since:
        3.15.1
      • setInputStream

        public void setInputStream​(InputStream in,
                                   String charsetOrSheetName)
        Set the input stream that will be parsed. The stream can be either a text CSV-like stream or an Excel workbook (xlsx). If the stream is an Excel workbook the following apply: If no date format has been specified, yyyy-MM-dd is used If no timestamp format has been specified, yyyy-MM-dd HH:mm:ss is used If no number format has been specified, 'dot' is used The data splitter regular expression is changed to \\t The section regular expression is changed to [.*] (if the getParseAllExcelSheets() flag is set
        Parameters:
        in - The InputStream
        charsetOrSheetName - If CSV, the name of the character set to use when parsing the file, or null to use the default charset specified by Config.getCharset() If Excel, the name or index of the worksheet in the workbook, the default is to parse the first sheet (index=0) or the whole workbook if the getParseAllExcelSheets() flag is set
        Since:
        2.1.1
      • parseToBof

        public boolean parseToBof()
                           throws IOException
        Parse the file until the beginning-of-file marker is found. If no regular expression has been set with setBofMarkerRegexp(Pattern) or if the parsing of the file has already started, this method call is ignored.
        Returns:
        TRUE if this call resulted in parsing and the BOF marker was found, FALSE otherwise
        Throws:
        IOException
        Since:
        2.15
      • getBofType

        public String getBofType()
        Get the value captured by the BOF marker regular expression. If no capturing groups was specified in the pattern this value is the string that matched the entire pattern.
        Returns:
        The matched value, or null if BOF matching has not been done
        Since:
        2.15
      • parseHeaders

        public FlatFileParser.LineType parseHeaders()
                                             throws IOException
        Start parsing the input stream. The parser will read a single line at a time. Each line is checked in the following order:
        1. Does it match the section regular expression?
        2. Does it match the header regular expression?
        3. Does it match the data header regular expression?
        4. Does it match the comment regular expression?
        5. Can it be split by the data regular expression into the appropriate number of columns?
        The first expression that matches stops the processing of that line. If the line matched a header or comment the parser continues with the next line. If the line matched the data header or data, the method returns. If none of the above is true the line is recorded as FlatFileParser.LineType.UNKNOWN and processing is continued with the next line. If too many unkown lines in a row has been found the method also returns. This should be considered as a failure to parse the specified file.

        The method returns the type of the last line that was parsed as follows:

        Returns:
        The FlatFileParser.LineType of the last parsed line
        Throws:
        IOException - If reading the file fails.
      • convertToNull

        private String convertToNull​(String value)
      • getHeader

        public String getHeader​(String name)
        Get the value of the header with the specified name. This method should only be used after parseHeaders() has been completed.
        Parameters:
        name - The name of the header
        Returns:
        The value of the header, or null if it was not found
        See Also:
        getLine(int)
      • getLineCount

        public int getLineCount()
        Get the number of lines that the parseHeaders() method parsed.
        Returns:
        The number of lines parsed
      • getLine

        public FlatFileParser.Line getLine​(int index)
        Get the line with the specified number. This method should only be used after parseHeaders() has been completed.
        Parameters:
        index - The line number, starting at 0
        Returns:
        A Line object
        See Also:
        getHeader(String)
      • getColumnHeaderIndex

        public Integer getColumnHeaderIndex​(String name)
        Get the index of a column header with a given name. This method should only be called after parseHeaders() has been called. If more than one header with the same name exists the index of the first is returned.
        Parameters:
        name - The name of the column header
        Returns:
        The index, or null if no header with that name exists
        See Also:
        findColumnHeaderIndex(String)
      • findColumnHeaderIndex

        public Integer findColumnHeaderIndex​(String regex)
        Find the index of a column header using a regular expression for pattern matching. This method should only be called after parseHeaders() has been called. If more than one header matches the regular expression only the first one found is returned.
        Parameters:
        regex - The regular expression used to match the header names
        Returns:
        The index, or null if no header is matching the regular expression or if the string is not a valid regular expression
        Since:
        2.5
        See Also:
        getColumnHeaderIndex(String)
      • setDefaultNumberFormat

        public void setDefaultNumberFormat​(NumberFormat numberFormat)
        Set the default number format to use when creating mappers.
        Parameters:
        numberFormat - The number format to use, or null to parse numbers with Float.valueOf or Double.valueOf
        Since:
        2.2
        See Also:
        getMapper(String), getMapper(String, NumberFormat)
      • getDefaultNumberFormat

        public NumberFormat getDefaultNumberFormat()
        Get the default number format.
        Returns:
        The number format, or null if none has been specified
        Since:
        2.2
      • setDefaultDateFormat

        public void setDefaultDateFormat​(Formatter<Date> dateFormat)
        Set the default date format to use when creating mappers. If null, xxx is used.
        Since:
        3.15
      • getDefaultDateFormat

        public Formatter<Date> getDefaultDateFormat()
        Get the default date format.
        Since:
        3.15
      • setDefaultTimestampFormat

        public void setDefaultTimestampFormat​(Formatter<Date> timestampFormat)
        Set the default timestamp format to use when creating mappers. If null, xxx is used.
        Since:
        3.15
      • getDefaultTimestampFormat

        public Formatter<Date> getDefaultTimestampFormat()
        Get the default timestamp format.
        Since:
        3.15
      • setUseNullIfException

        public void setUseNullIfException​(boolean useNullIfException)
        Specify if null should be returned if a (numeric) value can't be parsed. If this setting is set to TRUE all mappers created by one of the getMapper(String) methods are wrapped in a NullIfExceptionMapper. It is not possible to log or get information about the exception.
        Parameters:
        useNullIfException - TRUE to return null, FALSE to throw an exception
        Since:
        2.4
      • getUseNullIfException

        public boolean getUseNullIfException()
        Since:
        3.15
      • setIgnoreNonExistingColumns

        public void setIgnoreNonExistingColumns​(boolean ignoreNonExistingColumns)
        Specify if trying to create a mapper with one of the getMapper(String) methods for an expression which references a non-existing column should result in an exception or be ignored.
        Parameters:
        ignoreNonExistingColumns - TRUE to ignore, or FALSE to throw an exception
        Since:
        2.6
      • getDateMapper

        public Mapper getDateMapper​(String expression)
        Get a mapper using the default date format.
        Since:
        3.15
      • getTimestampMapper

        public Mapper getTimestampMapper​(String expression)
        Get a mapper using the default timestamp format.
        Since:
        3.15
      • getMapper

        public Mapper getMapper​(String expression,
                                Formatter<Date> dateFormat,
                                boolean nullIfException)
        Get a mapper using the specified date format.
        Since:
        3.15
      • getMapper

        public Mapper getMapper​(String expression,
                                NumberFormat numberFormat,
                                Formatter<Date> dateFormat,
                                boolean nullIfException,
                                JepFunction... functions)
        Create a mapper object that maps an expression string to a value. An expression string is a regular string which contains placeholders where the data column values will be inserted. For example:
        \1\
        \row\
        Row: \row\, Col:\col\
        
        It is also possible to use expressions that are evaluated dynamically.
        =2 * col('Radius')
        
        If no column that is matching the exact name is found the placeholder is interpreted as a regular expression which is checked against each of the column headers. In all cases, the first column header found is used if there are multiple matches.

        If the expression is null, a mapper returning en empty string is returned, unless the setUseNullIfEmpty(boolean) has been activated. In that case the mapper returns null.

        Parameters:
        expression - The string containing the mapping expression
        numberFormat - The number format the mapper should use for parsing numbers, or null to use Float.valueOf or Double.valueOf
        dateFormat - The date format the mapper should use for parsing dates, or null to use Type.DATE.parseString()
        nullIfException - TRUE to return a null value instead of throwing an exception when a value can't be parsed.
        functions - Optional array with Jep functions that should be included in the parser
        Returns:
        A mapper object
        Since:
        3.15
      • hasMoreData

        public boolean hasMoreData()
                            throws IOException
        Check if the input stream contains more data. If it is unknown if there is more data or not, this method will start reading more lines from the stream. Each line is checked in the following order:
        1. Does it match the data footer regular expression?
        2. Does it match the section regular expression?
        3. Does it match the ignore regular expression?
        4. Can it be split by the data regular expression into the appropriate number of columns?
        If the first or second check is true, FALSE is returned no more data may be retrieved, but a section may be retrived with the nextSection method. If the third check is true, the line is ignored and the processing continues with the next line. If the fourth check is true, TRUE is returned and the data may be retrieved with the nextData method.
        Returns:
        TRUE if there is more data, FALSE otherwise
        Throws:
        IOException - If there is an error reading from the input stream
        See Also:
        nextData
      • trimQuotes

        public String[] trimQuotes​(String[] columns)
        Remove enclosing quotes (" or ') around all columns.
        Parameters:
        columns - The columns
        Returns:
        The trimmed columns
      • getParsedLines

        public int getParsedLines()
        Get the number of parsed lines so far.
      • getParsedSections

        public int getParsedSections()
        Get the number of found sections so far.
        Since:
        3.15
      • getParsedDataLines

        public int getParsedDataLines()
        Get the number of parsed data lines so far in the current section. This value is reset for each new section.
      • getParsedCharacters

        public long getParsedCharacters()
        Get the number of parsed characters so far. This value may or may not correspond to the number of parsed bytes depending on the character set of the file.
        See Also:
        getParsedBytes()
      • getParsedBytes

        public long getParsedBytes()
        Get the number of parsed bytes so far. This value may or may not correspond to the number of parsed characters depending on the character set of the file.
        Since:
        2.5.1
        See Also:
        getParsedCharacters()
      • getUnknownLines

        public int getUnknownLines()
        Get the number of lines that the last call to nextData() or hasMoreData() ignored because they couldn't be interpreted as data lines.
        Returns:
        The number of unknown lines
        See Also:
        setKeepSkippedLines(boolean)
      • hasMoreSections

        public boolean hasMoreSections()
                                throws IOException
        Check if the input stream contains more sections. If it is unknown if there is more sections or not, this method will start reading more lines from the stream. Each line is checked if it matches the section regular expression. The parser will continue util a section line is found or end of file is reached. If the metod return TRUE the section may be retrived with the nextSection() method. If the section regular expression isn't specified the method returns FALSE and won't parse any line.
        Returns:
        TRUE if there is more data, FALSE otherwise
        Throws:
        IOException - If there is an error reading from the input stream
        See Also:
        nextData()