simple decimal addition

The forum for all of your Freebasic needs!

Moderators: Pete, Mods

Post Reply
dinokilla
Coder
Posts: 12
Joined: Mon Jan 07, 2008 6:56 pm

simple decimal addition

Post by dinokilla »

so I'm making some crappy little math tutor programs for a younger sibling, I thought doing operations with decimals would be pretty straight forward but I come up with odd answers occasionaly. The answer are always right but once in awhile there will be an extra 3 digits on the end of the answer (typically 001,002,999). This is causing me problems when verifying the answers. It will sometimes produce correct results for a long period of time and then all of a sudden I get a strange result like that. I can get around it converting it to a string and back I think but is there a better way to do this or am I doing something wrong? Thanks...

oohhhhh btw...in some of the operations I want the decimal places to line up so I use a random 5 digit integer multiplied by .001 using CSNG into a single...could that be the problem?

I want to add numbers like 21.103+13.353 but sometimes the result of this will be 34.455 why would it round it down like that? it should be 34.456
User avatar
Codemss
Veteran
Posts: 124
Joined: Sun Jun 24, 2007 6:49 am
Location: Utrecht, The Netherlands
Contact:

Post by Codemss »

Yeah. This is very weird and it sucks. I have it too. Sometimes, If I add 0.2 to 0.2 I get 0.399999999999999 and that kind of things.
You should be able to get rid of this. You may round of on some decimal.
I dont know what's the best way to do that. Don't directly use CINT or INT (at least, not for this job I mean, ) if you want some decimals left :P.
Well, succes. If you find something on it, I'd be interested in it.
Check out my site: <a href="http://members.lycos.nl/rubynl">Click here</a>
Hope you like it. Send some feedback if you want: <a href="mailto:basicallybest@live.nl">Mail me</a>
Codemss, before known as RubyNL
dinokilla
Coder
Posts: 12
Joined: Mon Jan 07, 2008 6:56 pm

Post by dinokilla »

even when I try to convert it to a string and then chop off the end and convert it back to single it still does that occasionaly. Most of the time it works fine but for whatever weird reason it will do that, the strangest part to me is sometimes there will be no carry involved, like the last digit of number1 is 4 and the last digit of number2 is 3, when added together you would expect 7 yet sometimes it will come up with 6!!!! I don't know any way to do it with 100% accuracy. :cry:
Nodtveidt
Veteran
Posts: 826
Joined: Sun Jul 25, 2004 4:24 am
Location: Quebradillas, PR
Contact:

Post by Nodtveidt »

It's known as a BCD representation error, and there's no way around it. What you should do instead is used fixed point numbers.
dinokilla
Coder
Posts: 12
Joined: Mon Jan 07, 2008 6:56 pm

Post by dinokilla »

ok how do I go about doing that?

[edit] nevermind I get it, is there any reason I couldn't use long's for that? thanks I never woulda thought of that! :D
dinokilla
Coder
Posts: 12
Joined: Mon Jan 07, 2008 6:56 pm

Post by dinokilla »

Wow what a huge pain in the butt just to do simple decimal addition!!! :x
User avatar
Mentat
Veteran
Posts: 409
Joined: Tue Aug 07, 2007 3:39 pm
Location: NC, US

Post by Mentat »

Use large integers, then divide by a magnitude or two.

Code: Select all

'some function that gets a random integer between two others 
declare function randInt(low as integer, high as integer) as integer

'integers you want to work with
dim a as integer
dim b as integer

'some multiple of ten
const scale=100

a=randInt(1,1000)
b=randInt(1,1000)



print "Here's a number: ";a/scale
print "Here's another number: ";b/scale
print "Here's the sum: ";(a+b)/scale

sleep

end

function randInt(low as integer, high as integer) as integer
    return rnd*(low-high)+low
end function
For any grievances posted above, I blame whoever is in charge . . .
dinokilla
Coder
Posts: 12
Joined: Mon Jan 07, 2008 6:56 pm

Post by dinokilla »

run your code but give it a random seed (randomize timer) and run it a few times and you will see the problem I am talking about...
User avatar
Mentat
Veteran
Posts: 409
Joined: Tue Aug 07, 2007 3:39 pm
Location: NC, US

Post by Mentat »

Oh, I see what you mean. Well, I just happen to have a solution:

Code: Select all

randomize timer

'some function that gets a random integer between two others 
declare function randInt(low as integer, high as integer) as integer 

'integers you want to work with 
dim a as integer 
dim b as integer 

'some multiple of ten 
const scale=100 

a=randInt(1,1000) 
b=randInt(1,1000) 



print "Here's a number: ";
print using "###.##";a/scale 
print "Here's another number: ";
print using "###.##";b/scale 
print "Here's the sum: ";
print using "###.##";(a+b)/scale 

sleep 
For any grievances posted above, I blame whoever is in charge . . .
dinokilla
Coder
Posts: 12
Joined: Mon Jan 07, 2008 6:56 pm

Post by dinokilla »

hehehe, that wouldn't work either because it will just print it in the correct format but if you ever had to compare the answer the actual number might still be 3.230000000000000001 but I got it figured out for what I needed it for. I just make sure that I'm always using an integer for everything so it works great. Thanks for your help though!
Post Reply