manual:chapter3:numformat

This is an old revision of the document!


A number can have an exponent up to 30,000 (note that the stock 50g ROM only allows an exponent up to 499). Thus, numbers can range from $10^{-30,000}$ to $10^{+30,000}$.

The precision is a user selectable parameter, using SETPREC. The default is 32 digits of precision. The maximum is 2000 digits of precision (for comparison, the stock 50g ROM has a fixed precision of 12 digits displayed, 15 digits internally). While the current precision is a system-wide setting, each number retains the precision it used when it was calculated, even if the user reduces the precision setting, a number on the stack will retain its digits. An interesting aspect of this precision implementation is that it can be changed mid-stream through a calculation if more digits are needed and then changed back, without affecting the rest of the chain of calculation (should certain critical parts of the calculation require higher precision).

Of course, the more precision a number has, the larger amount of memory is required to store it. The system will try to optimize and store the least possible number of digits. For example calculating 1 2 / will result in the number 0.5 which only needs to store one single digit regardless of the system-wide precision setting. On the other hand, calculating 1 3 / at 2000 digits precision will need to store 2000 digits.

There's one more optimization the system will attempt: when a number is an integer it will be stored as a “compact” number rather than as a variable precision real number. This is all transparent to the user, the user need not be concerned with the storage format at all except when trying to compute the required storage space.

Here are the storage requirements for numbers:

Number Type Number Range Bytes of storage
Small integer-130,000 to +130,0004
64-bit integer$-2^{63}$ to $2^{63}$ (but outside previous range)12
Larger integers or realsany8+4*(N/8), where N=smallest multiple of 8 > precision number of digits
ComplexanySum of space for real and imaginary parts plus 4 bytes

At the default precision of 32 digits, an approximate (non-complex) number would require a maximum of 24 bytes.


  • manual/chapter3/numformat.1646769574.txt.gz
  • Last modified: 2022/03/08 11:59
  • by claudio