Opened 15 years ago
Closed 15 years ago
#1420 closed task (invalid)
Round values imported using Manual transform
| Reported by: | Johan Vallon-Christersson | Owned by: | everyone |
|---|---|---|---|
| Priority: | major | Milestone: | |
| Component: | coreplugins | Version: | |
| Keywords: | Cc: |
Description
When using Manual transform to import spot data (Ch 1 values) rounding/mod of specified values are made during import.
Example, specified spot value in text file (Ch 1) -199999999 will be rounded to -200000000.
More examples: specified spot values (Ch 1):
-1e+08 => -1.0E8
-100000001 => -1.0E8
-1e+09 => -1.0E9
-1000000001 => -1.0E9
-1010101010 => -1.01010099E9
Is this expected behavior?
Change History (2)
comment:1 by , 15 years ago
comment:2 by , 15 years ago
| Resolution: | → invalid |
|---|---|
| Status: | new → closed |
Note:
See TracTickets
for help on using tickets.
This seems to be what one can expect. Values are stored in the database as floats which has about 6-7 significant digits. The examples given have more 8 or more signficant digits.