Values only held to 7 significant digits?

Discuss OrbForms Designer and PocketC Architect

Postby dbprog on Wed Aug 10, 2005 7:41 pm

I am using float variables for match calculations and the results are not always accurate to 8 significant digits. For example if the final value is 122300.56 I may end up with .55, .56, or .57 for the decimal places. I've tried various rounding approaches (using integer conversion & also the format function) and am not able to get the correct value consistently. Is there any way around this or is it just a limit for the language?
Posts: 4
Joined: Mon Nov 15, 2004 3:07 pm
Location: USA

Postby dewey on Wed Aug 17, 2005 4:02 am

OrbForms using 32-bit IEEE floats, which have a little less than 8 significant bits. If you have a fixed decimal point (always 2 digits to the right of the .), you can just use integers, giving you 9 significat digits.
Site Admin
Posts: 749
Joined: Sun Oct 21, 2007 5:12 am

Postby c0d3m0nk3y on Wed Aug 24, 2005 1:09 am

Why dosen't OrbForms support 64-bit floats?

Thanks for reading my message, Have a nice day!
Thanks for reading my message, Have a nice day!
Posts: 74
Joined: Tue Jun 22, 2004 7:51 pm
Location: Canada

Return to OrbForms Designer and PocketC Architect

Who is online

Users browsing this forum: No registered users and 1 guest