Opened 11 years ago

Closed 11 years ago

#1420 closed task (invalid)

Round values imported using Manual transform

Reported by: Johan Vallon-Christersson Owned by: everyone
Priority: major Milestone:
Component: coreplugins Version:
Keywords: Cc:

Description

When using Manual transform to import spot data (Ch 1 values) rounding/mod of specified values are made during import.
Example, specified spot value in text file (Ch 1) -199999999 will be rounded to -200000000.
More examples: specified spot values (Ch 1):
-1e+08 => -1.0E8
-100000001 => -1.0E8
-1e+09 => -1.0E9
-1000000001 => -1.0E9
-1010101010 => -1.01010099E9

Is this expected behavior?

Change History (2)

comment:1 Changed 11 years ago by Nicklas Nordborg

This seems to be what one can expect. Values are stored in the database as floats which has about 6-7 significant digits. The examples given have more 8 or more signficant digits.

comment:2 Changed 11 years ago by Nicklas Nordborg

Resolution: invalid
Status: newclosed
Note: See TracTickets for help on using tickets.