Example of Default Precision Rules
Example of Default Precision Rules
This example shows how the default precision rules work.
Figure 1. Precision of Intermediate Results
DName+++++++++++ETDsFrom+++To/L+++IDc.Keywords+++++++++++++++++++++++++++
D FLD1 S 15P 4
D FLD2 S 15P 2
D FLD3 S 5P 2
D FLD4 S 9P 4
D FLD5 S 9P 4
CL0N01Factor1+++++++Opcode(E)+Extended-factor2+++++++++++++++++++++++++++
C EVAL FLD1 = FLD2/(((FLD3/100)*FLD4)+FLD5)
( 1 )
( 2 )
( 3 )
( 4 )When the above Calculation specification is processed, the resulting value assigned to FLD1 will have a precision of zero decimals, not the three decimals expected. The reason is that when it gets to the last evaluation ( 4 in the above example), the number to which the factor is scaled is negative. To see why, look at how the expression is evaluated.
1 : Evaluate FLD3/100
Rules:
```rpgle
Lr = 63
Dr = max(63-((L1-D1)+D2),0)
= max(63-((5-2)+0),0)
= max(63-3,0)
= 60
```2 : Evaluate (Result of 1 * FLD4)
Rules:
```rpgle
Lr = min(L1+L2,63)
= min(63+9,63)
= 63
Dr = min(D1+D2,63-min((L1-D1)+(L2-D2),63))
= min(60+4,63-min((63-60)+(9-4),63))
= min(64,63-min(4+5,63)
= min(64,55)
= 55
```3 : Evaluate (Result of 2 + FLD5)
Rules:
```rpgle
T = min(max(L1-D1,L2-D2)+1,63)
= min(max(63-55,9-4)+1,63)
= min(max(8,5)+1,63)
= min(9,63)
= 9
Dr = min(max(D1,D2),31-T)
= min(max(55,4),63-9)
= min(55,54)
= 54
Lr = T + Dr
= 9 + 54 = 63
```4 : Evaluate FLD2/Result of 3
Rules:
```rpgle
Lr = 63
Dr = max(63-((L1-D1)+D2),0)
= max(63-((15-2)+ 54),0)
= max(63-(13+54),0)
= max(-4,0)
**** NEGATIVE NUMBER TO WHICH FACTOR IS SCALED **** = 0
```To avoid this problem, you can change the above expression so that the first evaluation is a multiplication rather than a division, that is, FLD3 * 0.01 or use the %DEC built-in function to set the sub-expression FLD3/100: %DEC(FLD3/100 : 15 : 4) or use operation extender (R) to ensure that the number of decimal positions never falls below 4.