Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#1444 closed task (fixed)

Implement generic BFS writers and readers/parsers

Reported by: Nicklas Nordborg Owned by: Nicklas Nordborg
Priority: critical Milestone: BASE 2.15
Component: core Version:
Keywords: Cc:

Description

Specification are found in #1440. The generic implementations should be re-usable when implementing the specific spotdata use-case.

Change History (12)

comment:1 Changed 11 years ago by Nicklas Nordborg

Owner: changed from everyone to Nicklas Nordborg
Status: newassigned

comment:2 Changed 11 years ago by Nicklas Nordborg

(In [5193]) References #1444: Implement generic BFS writers and readers/parsers

Implemented writers and parsers for metadata and annotation files. I am not 100% sure the event system for getting the data out is flexible enough.

comment:3 Changed 11 years ago by Nicklas Nordborg

(In [5194]) References #1444: Implement generic BFS writers and readers/parsers

Added writer/parser for data files. Some changes to event handling.

comment:4 Changed 11 years ago by Nicklas Nordborg

(In [5198]) References #1444: Implement generic BFS writers and readers/parsers

Improved error handling and error messages. Added DataWriterFactory? and two generic implementations.

comment:5 Changed 11 years ago by Nicklas Nordborg

(In [5212]) References #1444: Implement generic BFS writers and readers/parsers

Defined an interface for file name generation in order to separate this from the actual file creation.

comment:6 Changed 11 years ago by Nicklas Nordborg

The parser implementations needs to be re-done. The current code can only parse one file at a time, but since data is split into several files and we don't want to keep everything in memory we need to parse multiple files synchronously line-by-line. But I also think we should keep a variant of the current implementation since it may be useful in other cases.

comment:7 Changed 11 years ago by Nicklas Nordborg

(In [5221]) References #1444: Implement generic BFS writers and readers/parsers

Re-implemented parsers so that it should be possible to do line-by-line parsing of multiple files at the same time.

comment:8 Changed 11 years ago by Nicklas Nordborg

(In [5224]) References #1444: Implement generic BFS writers and readers/parsers

Added InputStreamLocator? and some implementations that are used to load files from disk and the BASE file system. Intended to be used to open data/annotation files referenced from the metadata.txt file.

Made some extensions to the EventHandler? interface so that we can get more information about the parsing process.

comment:9 Changed 11 years ago by Nicklas Nordborg

(In [5225]) References #1444: Implement generic BFS writers and readers/parsers

Added functionality that should make it easier to add progress reporting while parsing files and checking for interuption by the user.

comment:10 Changed 11 years ago by Nicklas Nordborg

Resolution: fixed
Status: assignedclosed

The parsers/writers are used in the spot data import/export.

comment:11 Changed 11 years ago by Nicklas Nordborg

(In [5263]) References #1444: Implement generic BFS writers and readers/parsers

Added some more helper methods for creating parsers for files in the local file system.

comment:12 Changed 11 years ago by Nicklas Nordborg

(In [5309]) References #1444, #1448, #1451. Added methods for closing BFS parsers and an auto-close feature on the exporter and importer. This was needed since some temporary working files could not be deleted without closing the streams.

Note: See TracTickets for help on using tickets.