PROBLEMS MAPPING DECIMAL

Hi,
I'm having a problem with a decimal,

I have the value in a decimal type variable
9,820.65
But when I map it to a decimal type data that is in another cdt it saves this
9820.650000000001
Does anybody know what is it due to?
Regards, and thank you very much

  Discussion posts and replies are publicly visible

Parents
  • +1
    Certified Lead Developer

    It's because of floating point math.

    0.1 is only an approximation, because of the way Decimal is stored (as a Double precision floating point number in Java).  In binary, in that format, 1 / 10 is an infinitely repeating number, something like 0.00011001100110011. Nearly every number you see in decimal format is only an approximation of what can be represented in binary.  It's like, how many halves, plus how many fourths, plus how many eighths, plus how many sixteenths, plus how many thirty-secondths, etc.  Unless it's exactly a sum of negative powers of two, it can't be represented fully accurately.  The further away from 1 or -1 you go, the wider the gaps between the numbers you can approximate.  All software everywhere has this problem.  Try it in Java:  0.1 + 0.1 + 0.1 == 0.3 returns false.

    What can you do?  Nothing, really.  You can try multiplying it by 100, 982,065, moving it to it's own PV, then dividing that PV by 100 and putting the answer into the other CDT and see if you don't get exactly the same behavior.  You could create a custom plugin to implement C#'s Decimal type, which stores the number as a 28 or 29 digit integer and then stores the location of the decimal point.  Or you could create a custom plugin to utilize the Java arbitrary precision mathematics library.  

    Or as the other posters have said, you can use the fixed function for display purposes.  So it's off in the back-end; you just don't let the end users know about it.  For the database, just store the digits you want using number(), for instance number(12,4) is up to 8 digits before the decimal and up to 4 digits after the decimal.  It will probably still be off deep within, but should refrain from telling you about it.

  • Thank you very much David !!, very interesting information

Reply Children
No Data