Rockwell Automation 2711-ND3 PanelBuilder v2.60 Software User Manual - Manual (Page 158)

10–6 Creating Data Entry Controls
Publication 2711-6.0
Precision and Rounding
Except for floating point values, the logic controller stores values as
integers, rounding off any digits to the right of the decimal point.
However, the terminal can scale decimal values to or from integer
values.
In this illustration, the entered value of 7.5 is rounded up to 8. When
this value is displayed, or used in a mathematical formula, the result
may be incorrect.
Entered Value = 7.5 Displayed Value = 8
Controller Value = 8
(rounded)
If decimal precision is needed, select appropriate values for scale and
offset.
Example
Use scaling to convert a decimal value from degrees Fahrenheit to
degrees Celsius. The precision is determined by the value the
operator enters and the decimal position of the scale.
Rounded
4964.44
4964
4964
Entered Value
121.36
50.5
51
51
Fixed Decimal
Decimal Position = 2
Field Width = 6
Keypad Controlled Decimal
Decimal Position = Operator Entered
Field Width = 6
123.8
The decimal point is positioned to provide a display appropriate to the scaled value.
Controller Value =
Entered Value – Offset
Scale
PanelView
Displayed Value = Controller Value x Scale + Offset
Equations for Scaling Example
121.36 – 32
0.018
= 4964.44 = 4964 Stored Value =
Fixed Decimal
Stored Value =
Displayed Value = (4964 x 0.018) + 32 = 121.35
122.9 – 32
1.8
= 50.5 = 51 Stored Value =
Keypad Controlled Decimal
Displayed Value
121.35
Controller
Controller
Entered Value
122.9
Displayed Value
123.8
Scale
Scale
Offset=32
Scale=.018
Offset=32
Scale=1.8
Displayed Value = (51 x 1.8) + 32 = 123.8
Rounded
The terminal stores the full precision value and the controller stores
the rounded value (if writing to an integer file).