Don't know if this has come up before, but following the example in the manual for 'digits' gives wrong values...
Code:
var x = 123456.789;
...
digits(0,0,"%f",*,1,x);
According to the manual this will display '123456.789' yet I keep getting '123456.789063'. I realise this is caused by the inaccuracies in converting the var to floating point, but it's still a mistake in the manual.
I realise this is caused by the inaccuracies in converting the var to floating point
So there is no mistake, just limit the output to %.3f. If someone does not know why the output is 123456.789063 he will probably search/ ask for it, what will increase his knowledge (about the important fact of inaccuracies of computer maths). (Sorry for being a smart-ass.)
Last edited by Superku; 01/22/1015:39.
"Falls das Resultat nicht einfach nur dermassen gut aussieht, sollten Sie nochmal von vorn anfangen..." - Manual
I realise this is caused by the inaccuracies in converting the var to floating point
So there is no mistake, just limit the output to %.3f. If someone does not know why the output is 123456.789063 he will probably search/ ask for it, what will increase his knowledge (about the important fact of inaccuracies of computer maths). (Sorry for being a smart-ass.)
Yes, but then the manual still is wrong in it's example
