manual:chapter3:numformat

This is an old revision of the document!


An approximate number can have an exponent up to 30,000 (note that the stock 50g ROM only allows an exponent up to 499). Thus, numbers can range from $10^{-30,000}$ to $10^{+30,000}$.

The precision for approximate numbers is a user selectable parameter, using SETPREC. The default is 32 digits of precision. The maximum is 2000 digits of precision (for comparison, the stock 50g ROM has a fixed precision of 12 digits displayed, 15 digits internally). One nice aspect of this precision implementation is that it can be changed mid-stream through a calculation then changed back, without affecting the rest of the chain of calculation (should certain critical parts of the calculation require higher precision).

Of course, the more precision a number has, the larger amount of memory is required to store it. Here are the storage requirements for numbers:

Number Type Number Range Bytes of storage
Exact-130,000 to +130,0004
Exact$-2^{63}$ to $2^{63}$ (but outside previous range)12
Approximateany8+4*(N/8), where N=smallest multiple of 8 > precision number of digits

At the default precision of 32 digits, an approximate number would require a maximum of 24 bytes.


  • manual/chapter3/numformat.1497228204.txt.gz
  • Last modified: 2017/06/11 17:43
  • by smartin