johnp

August 5th, 2006, 11:37 PM

Hi,

Having some trouble with math...

My device returns 8910 when it really means 89.1 MHz

So I figure why not divide by 100

8910 / 100 = 89.1 right? Wrong--

Lua Returns 89.09999999999999

So I guess I'll try 8910 * 10^-2

Lua returns the same thing

so I figure OK deal with a rounding error and round it off.

I implement this:

function rounditoff (n,shift)

shift = 10^(shift or 0)

return floor((n*shift)+0.5)/shift

end

mytunerfreq1 = 8910 /100

print (mytunerfreq1)

mytunerfreq2 = rounditoff (8910/100, 2);

print (mytunerfreq2)

--without a function

shift = 10 ^ 2

mytunerfreq3 = floor((mytunerfreq1)*(shift) + 0.5 ) / (shift)

print (mytunerfreq3)

All of these return the same thing.

What kind of screwy math am I dealing with here anyway???

I must be doing something wrong??

Oh yes, the built in round function does the same thing...

Having some trouble with math...

My device returns 8910 when it really means 89.1 MHz

So I figure why not divide by 100

8910 / 100 = 89.1 right? Wrong--

Lua Returns 89.09999999999999

So I guess I'll try 8910 * 10^-2

Lua returns the same thing

so I figure OK deal with a rounding error and round it off.

I implement this:

function rounditoff (n,shift)

shift = 10^(shift or 0)

return floor((n*shift)+0.5)/shift

end

mytunerfreq1 = 8910 /100

print (mytunerfreq1)

mytunerfreq2 = rounditoff (8910/100, 2);

print (mytunerfreq2)

--without a function

shift = 10 ^ 2

mytunerfreq3 = floor((mytunerfreq1)*(shift) + 0.5 ) / (shift)

print (mytunerfreq3)

All of these return the same thing.

What kind of screwy math am I dealing with here anyway???

I must be doing something wrong??

Oh yes, the built in round function does the same thing...