LUA Rounding problem

[johnp]

Hi,

Having some trouble with math...

My device returns 8910 when it really means 89.1 MHz

So I figure why not divide by 100

8910 / 100 = 89.1 right? Wrong--

Lua Returns 89.09999999999999

So I guess I'll try 8910 * 10^-2

Lua returns the same thing

so I figure OK deal with a rounding error and round it off.

I implement this:


function rounditoff (n,shift)
shift = 10^(shift or 0)
return floor((n*shift)+0.5)/shift
end

mytunerfreq1 = 8910 /100
print (mytunerfreq1)
mytunerfreq2 = rounditoff (8910/100, 2);
print (mytunerfreq2)

--without a function
shift = 10 ^ 2
mytunerfreq3 = floor((mytunerfreq1)*(shift) + 0.5 ) / (shift)
print (mytunerfreq3)


All of these return the same thing.

What kind of screwy math am I dealing with here anyway???

I must be doing something wrong??

Oh yes, the built in round function does the same thing...

[Rob H]

That is pretty weird. Lua 5 (in Girder 4) returns 89.1 for 8910/100

Not sure what to suggest.

[johnp]

Indeed it is strange. Can anyone else reproduce this behaviour in 3.3?