Page 1 of 1
simple decimal addition
Posted: Tue Jan 29, 2008 1:31 am
by dinokilla
so I'm making some crappy little math tutor programs for a younger sibling, I thought doing operations with decimals would be pretty straight forward but I come up with odd answers occasionaly. The answer are always right but once in awhile there will be an extra 3 digits on the end of the answer (typically 001,002,999). This is causing me problems when verifying the answers. It will sometimes produce correct results for a long period of time and then all of a sudden I get a strange result like that. I can get around it converting it to a string and back I think but is there a better way to do this or am I doing something wrong? Thanks...
oohhhhh btw...in some of the operations I want the decimal places to line up so I use a random 5 digit integer multiplied by .001 using CSNG into a single...could that be the problem?
I want to add numbers like 21.103+13.353 but sometimes the result of this will be 34.455 why would it round it down like that? it should be 34.456
Posted: Tue Jan 29, 2008 2:05 pm
by Codemss
Yeah. This is very weird and it sucks. I have it too. Sometimes, If I add 0.2 to 0.2 I get 0.399999999999999 and that kind of things.
You should be able to get rid of this. You may round of on some decimal.
I dont know what's the best way to do that. Don't directly use CINT or INT (at least, not for this job I mean, ) if you want some decimals left
.
Well, succes. If you find something on it, I'd be interested in it.
Posted: Tue Jan 29, 2008 5:09 pm
by dinokilla
even when I try to convert it to a string and then chop off the end and convert it back to single it still does that occasionaly. Most of the time it works fine but for whatever weird reason it will do that, the strangest part to me is sometimes there will be no carry involved, like the last digit of number1 is 4 and the last digit of number2 is 3, when added together you would expect 7 yet sometimes it will come up with 6!!!! I don't know any way to do it with 100% accuracy.
Posted: Tue Jan 29, 2008 7:51 pm
by Nodtveidt
It's known as a BCD representation error, and there's no way around it. What you should do instead is used fixed point numbers.
Posted: Tue Jan 29, 2008 11:46 pm
by dinokilla
ok how do I go about doing that?
[edit] nevermind I get it, is there any reason I couldn't use long's for that? thanks I never woulda thought of that!
Posted: Wed Jan 30, 2008 1:26 am
by dinokilla
Wow what a huge pain in the butt just to do simple decimal addition!!!
Posted: Thu Jan 31, 2008 4:08 pm
by Mentat
Use large integers, then divide by a magnitude or two.
Code: Select all
'some function that gets a random integer between two others
declare function randInt(low as integer, high as integer) as integer
'integers you want to work with
dim a as integer
dim b as integer
'some multiple of ten
const scale=100
a=randInt(1,1000)
b=randInt(1,1000)
print "Here's a number: ";a/scale
print "Here's another number: ";b/scale
print "Here's the sum: ";(a+b)/scale
sleep
end
function randInt(low as integer, high as integer) as integer
return rnd*(low-high)+low
end function
Posted: Fri Feb 01, 2008 12:37 pm
by dinokilla
run your code but give it a random seed (randomize timer) and run it a few times and you will see the problem I am talking about...
Posted: Fri Feb 01, 2008 2:06 pm
by Mentat
Oh, I see what you mean. Well, I just happen to have a solution:
Code: Select all
randomize timer
'some function that gets a random integer between two others
declare function randInt(low as integer, high as integer) as integer
'integers you want to work with
dim a as integer
dim b as integer
'some multiple of ten
const scale=100
a=randInt(1,1000)
b=randInt(1,1000)
print "Here's a number: ";
print using "###.##";a/scale
print "Here's another number: ";
print using "###.##";b/scale
print "Here's the sum: ";
print using "###.##";(a+b)/scale
sleep
Posted: Fri Feb 01, 2008 5:49 pm
by dinokilla
hehehe, that wouldn't work either because it will just print it in the correct format but if you ever had to compare the answer the actual number might still be 3.230000000000000001 but I got it figured out for what I needed it for. I just make sure that I'm always using an integer for everything so it works great. Thanks for your help though!