Metrohm tiamo 2.3 Patch 1 (process analysis) - Manual (Page 50)

2.4 Formula editor ￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭￭
34 ￭￭￭￭￭￭￭￭ tiamo 2.3 Patch 1 (for Process Analysis)
Mean value
Absolute standard
deviation
Relative standard devi-
ation (in %)
The statistical calculations of the software have been implemented so that
they can be checked to as great a degree as possible by the user. The indi-
vidual values are therefore incorporated in the statistics with full accuracy.
It is not the number of decimal places which is decisive for the accuracy of
the calculations, but rather the number of significant digits of the decimal
numbers displayed. The binary 64 bit numerical format implemented on
the basis of the IEEE 754 standard means that the decimal numerals
resulting from this have 15 certain significant decimal places.
You can influence the number of significant digits by selecting the unit
and the number of decimal places. As the result unit to be set sometimes
contains the prefix "milli" as well as the actual physical unit, the number of
significant places changes accordingly by three places during such a con-
version.
Example
The result displayed 1234.56789158763 mg/L has 15 certain places. It
should be rounded off to three decimal places according to the above
rounding procedure:
1234.568 mg/L (7 significant places, 3 of them decimal places)
The unit "g/L" means that the same result 1.23456789158763 g/L is
also rounded off to three decimal places:
1.235 g/L (4 significant places, 3 of them decimal places)
The number of significant digits has now been reduced by three to four
digits by omitting the prefix "milli".