Calculation range and precision, Function calculation input ranges and precision, E-31 – Casio FX-100MS User Manual
Page 32
E-31
The following shows the two-digit numbers for each of the metric
conversion commands.
01: in
' cm
02: cm
' in
03: ft
' m
04: m
' ft
05: yd
' m
06: m
' yd
07: mile
' km
08: km
' mile
09: n mile
' m
10: m
' n mile
11: acre
' m
2
12: m
2
' acre
13: gal (US)
'R
14:
R' gal (US)
15: gal (UK)
'R
16:
R' gal (UK)
17: pc
' km
18: km
' pc
19: km/h
' m/s
20: m/s
' km/h
21: oz
' g
22: g
' oz
23: lb
' kg
24: kg
' lb
25: atm
' Pa
26: Pa
' atm
27: mmHg
' Pa
28: Pa
' mmHg
29: hp
' kW
30: kW
' hp
31: kgf/cm
2
' Pa
32: Pa
' kgf/cm
2
33: kgf
•
m
' J
34: J
' kgf
•
m
35: lbf/in
2
' kPa
36: kPa
' lbf/in
2
37: °F
' °C
38: °C
' °F
39: J
' cal
40: cal
' J
Conversion formula data is based on the “NIST Special Publication
811 (2008)”.
Note: The J
' cal command performs conversion for values at a
temperature of 15°C.
Calculation Ranges, Number of
Digits, and Precision
Calculation Range and Precision
Calculation Range
± 1 × 10
–99
to
± 9.999999999 × 10
99
or 0
Number of Digits for
Internal Calculation
15 digits
Precision
In general,
± 1 at the 10th digit for a single
calculation. Precision for exponential
display is
± 1 at the least significant digit.
Errors are cumulative in the case of
consecutive calculations.
Function Calculation Input Ranges and
Precision
Functions
Input Range
sin
x
cos
x
DEG
0
|
x
| 9 × 10
9
RAD
0
|
x
| 157079632.7
GRA
0
|
x
| 1 Ч 10
10