ckrusen
(Calvin Krusen)
April 24, 2020, 3:24pm
1
Is precision lost when doing math with vars of type decimal?
decimal decVar1;
decimal decVar2;
decimal decVar3, decVar4, decVar5;
decVar1 = (decimal)123.45; //(a perfectly good decimal value)
decVar2 = (decimal)1000000.00; //(another perfectly good decimal value)
decVar3 = decVar1 / decVar2 * (decimal)1000;
decVar4 = decVar1 * (decimal)1000 / decVar2;
Do both decVar3 and decVar4 = 0.12 ?
If you were to do the decVar3 calc (left to right, and using decimal storage for each step), you’d get:
decVar1 / decVar2 = 123.45 / 1000000.00 = 0.00
(decVar1 / decVar2) * (decimal)1000;0.00 = 0.00 * (decimal)1000 = 0.00
ckoch
(Chris Koch)
April 24, 2020, 4:07pm
2
Have you tried using decimal literals e.g. decimal decVar1 = 123.45m;
instead of casting? The way you’re doing it now everything starts out as a float and is casted, which might be imprecise due to floating point rounding errors.
But that’s just a stab in the dark.
ckoch
(Chris Koch)
April 24, 2020, 4:14pm
3
I just tried it on repl.it: UnwieldyMediocreBinarysearchtree - Replit
There doesn’t seem to be any difference to me; I come out with 0.12345 for both decVar3 and decVar4 whether I use decimal literals or cast.
ckrusen
(Calvin Krusen)
April 24, 2020, 4:24pm
4
My post wasn’t about the casting or value initialization, but rather the intermediate values during multiple steps.
After reading further into the decimal
data type, I now realize that it’s not a fixed number of decimal places. So the order isn’t an issue.