Check UTF-8 for text files that are uploaded without a selected character set

[Nicklas Nordborg]

When uploading files to BASE the "Character set" option is typically set to -n/a- unless the user manually changes the encoding (which is typically very rare).

This can cause problems later on since parsing the text file have to rely on the defaultCharset setting in base.config which have a default value of ISO-8859-1.

While it is not possible to check every possible character encoding, checking if a file is UTF-8 is possible and I think we should implement a check in the file upload. If the file is detected to be UTF-8 compatible we set that, otherwise we keep the character set empty.

[Nicklas Nordborg]

In 7623:

References #2156: Check UTF-8 for text files that are uploaded without a selected character set

Implemented a utility class CharsetDetector that can be used for simple testing of encoding in text files. It works best with encodings that can be technically detected. For example, UTF-8 is very unlikely to be mixed up with other encodings while any of the ISO-8859-x encodings can typically be used for all files. The StringDetector is intended to be used for discriminating between ISO-8859-x encodings but it requires prior knowledge of text that is expected to be found in the file that is unique to an encoding.

The file upload functionality has been extended to check for UTF-8 text files. It is enabled automatically when the MIME type is set to something in the 'text/*' subset and no character set has been explicitely specified.

[Nicklas Nordborg]

In 7624:

References #2156: Check UTF-8 for text files that are uploaded without a selected character set

Added test code for testing the CharsetDetector.